A random collection of posts

Installing OpenSolaris 2008.11 RC2 domU in a Linux (RedHat/Centos) Dom0


  • The dom0 is run by Centos 5.2.
  • The dom0 runs in 32-Bit PAE mode and so will the OpenSolaris domU.
  • The network card of the OpenSolaris domU gets attached to the green bridge (brgreen).
  • My network is managed by a DHCP server that will serve the IP address and the addresses of the gateway and the DNS-server to the new domU.
  • The OpenSolaris domU will reside on a disk: sdc.


First, download the installer CD and prepare the disc/logical volume/file or whatever you want to use as the backend device. I use a real disk, so my preparation is “plug in the disk”.
For the following instructions the CD-ISO is assumed to be at /srv/xen/iso/opensolaris/osol-0811-rc2.iso. We can directly boot the paravirtualized kernel, so make sure you copy the i86xpv kernel off the CD. I will copy the boot-files to /srv/xen/systems/opensolaris200811/be-cd-0811-rc2/

$ mkdir -p /srv/xen/systems/opensolaris200811/be-cd-0811-rc2/
$ mount -o loop,ro /srv/xen/iso/opensolaris/osol-0811-rc2.iso /mnt/
$ cp /mnt/boot/x86.microroot /srv/xen/systems/opensolaris200811/be-cd-0811-rc2/
# Copy /mnt/platform/i86xpv/kernel/amd64/unix, if you have a 64-Bit dom0.
$ cp /mnt/platform/i86xpv/kernel/amd64/unix /srv/xen/systems/opensolaris200811/be-cd-0811-rc2/
$ umount /mnt

Create a setup-file

name = "sol200811"
vcpus = 1
memory = "1024"
kernel = "/srv/xen/systems/opensolaris200811/be-cd-0811-rc2/unix"
ramdisk = "/srv/xen/systems/opensolaris200811/be-cd-0811-rc2/x86.microroot"
extra = "/platform/i86xpv/kernel/unix - nowin -B install_media=cdrom"
disk = ['file:/srv/xen/iso/opensolaris/osol-0811-rc2.iso,6:cdrom,r','phy:sdc,0,w']
vif = ['bridge=brgreen']
on_shutdown = "destroy"
on_reboot = "destroy"
on_crash = "destroy"


Boot up with

$ xm create -c

wait .. wait .. login as jack/jack and prepare VNC:

jack:~$ mkdir .vnc;cp .Xclients .vnc/xstartup
jack:~$ vncserver

If you like SSH better than the Xen console, enable the SSH-daemon:

$ svcadm enable ssh

Find out your IP with ifconfig -a and connect with your favorite VNC viewer to display :1 ( ). Start setup and let it partition the disk. Do not close the installer window, because we do not want to reboot before we have kernel and ramdisk downloaded to the dom0.

The new installer unmounts the new boot environment before we can ge a hold on it. So we mount it by ourself:

$ beadm list
BE Active Mountpoint Space Policy Created
-- ------ ---------- ----- ------ -------
opensolaris R - 2.21G static 2008-12-05 06:58
$ pfexec beadm mount opensolaris /a

Update the boot archive

$ pfexec bootadm update-archive -R /a

And now get the kernel and boot_archive for you dom0 from /a/platform/i86xpv/kernel/unix and /a/platform/i86pc/boot_archive.

Copy the Kernel to dom0

The kernel (unix) and the boot_archive should be copied into a directory in your dom0 (scp comes in handy). I choose

kernel = "/srv/xen/systems/opensolaris200811/be-0811-rc2/platform/i86xpv/kernel/unix"
ramdisk = "/srv/xen/systems/opensolaris200811/be-0811-rc2/platform/i86pc/boot_archive"

So from your dom0:

$ scp jack@ /srv/xen/systems/opensolaris200811/be-0811-rc2/platform/i86xpv/kernel/unix
$ scp jack@ /srv/xen/systems/opensolaris200811/be-0811-rc2/platform/i86pc/boot_archive

You will need the ZFS-ID of your boot-environment. Your harddisk will be known as c3d0, your boot-slice containing the ZPOOL is on c3d0s0. We need the ID of the /ROOT/opensolaris zpool. First, determine the pool_guid:

$ zpool get guid rpool
rpool guid 14025158204312238343 -

You could also have used zpool import like this — the choice is yours.

# pfexec zpool import
pool: rpool
id: 14025158204312238343
state: ONLINE
status: The pool was last accessed by another system.
action: The pool can be imported using its name or numeric identifier and
the '-f' flag.
rpool ONLINE
c3d0s0 ONLINE

The guid by iself is useless – we need the ID of the ZPOOL inside the pool.

$ pfexec zdb -d -e 14025158204312238343
Dataset mos [META], ID 0, cr_txg 4, 512K, 84 objects
Dataset 14025158204312238343/ROOT/opensolaris@install [ZPL], ID 84, cr_txg 276, 2.21G, 101427 objects
Dataset 14025158204312238343/ROOT/opensolaris [ZPL], ID 50, cr_txg 72, 2.21G, 101427 objects
Dataset 14025158204312238343/ROOT [ZPL], ID 44, cr_txg 69, 18.0K, 4 objects
Dataset 14025158204312238343/dump [ZVOL], ID 36, cr_txg 13, 512M, 3 objects
Dataset 14025158204312238343/export/home/jens [ZPL], ID 68, cr_txg 80, 20.5K, 6 objects
Dataset 14025158204312238343/export/home [ZPL], ID 62, cr_txg 78, 19.0K, 5 objects
Dataset 14025158204312238343/export [ZPL], ID 56, cr_txg 76, 19.0K, 5 objects
Dataset 14025158204312238343/swap [ZVOL], ID 30, cr_txg 8, 16K, 3 objects
Dataset 14025158204312238343 [ZPL], ID 16, cr_txg 1, 72.0K, 12 objects

Your output might look different. Look out for /ROOT/opensolaris (here: 50). Remember it!
Disable gdm (unless you want a graphical login)

$ svcadm disable svc:/application/graphical-login/gdm:default

And reboot from within the installer window.

Create the real domU

Create a config-file

name = "sol200811"
vcpus = 1
memory = "1024"
kernel = "/srv/xen/systems/opensolaris200811/be-0811-rc2/platform/i86xpv/kernel/unix"
ramdisk = "/srv/xen/systems/opensolaris200811/be-0811-rc2/platform/i86pc/boot_archive"
extra = "/platform/i86xpv/kernel/unix - nowin -B bootpath=/xpvd/xdf@0:a -B console=xen -B zfs-bootfs=rpool/50"
disk = ['phy:sdc,0,w']
vif = ['mac=aa:00:00:00:03:34, vifname=vif.solgr, bridge=brgreen']
on_shutdown = "destroy"
on_reboot = "destroy"
on_crash = "destroy"

Boot it

$ xm create -c

and enjoy.

Activating a new boot environment

After you have created a new boot environment (i.e. by pkg image-update), you need to “activate” it in your domU-configuration. Specifically, you must use the appropriate ZFS-id for your new bootfs and you need the new boot_archive/kernel combo.

Update the Xen configuration

$ beadm list
BE Active Mountpoint Space Policy Created
-- ------ ---------- ----- ------ -------
opensolaris N / 4.94M static 2008-12-05 14:58
opensolaris-1 R - 2.56G static 2008-12-06 22:57

opensolaris-1 should be the active boot environment. Find out the ZFS-ID:

$ pfexec zdb -d -e 14025158204312238343
Dataset mos [META], ID 0, cr_txg 4, 621K, 94 objects
Dataset 14025158204312238343/ROOT/opensolaris [ZPL], ID 50, cr_txg 72, 2.26G, 101574 objects
Dataset 14025158204312238343/ROOT/opensolaris-1 [ZPL], ID 93, cr_txg 4065, 2.26G, 101573 objects

And update the Xen configuration:

# ...
extra = "/platform/i86xpv/kernel/unix - nowin -B bootpath=/xpvd/xdf@0:a -B console=xen -B zfs-bootfs=rpool/93"

Copy the kernel and the boot_archive to the dom0

First, mount the new boot environment:

$ pfexec beadm mount opensolaris-1 /mnt

Update the boot_archive (this is not strictly necessary, but it happened to me that pkg image-update didn’t do it).

$ pfexec bootadm update-archive -R /mnt

Copy the kernel/boot_archive by ssh’ing from the dom0 into the domU. Switch to the dom0 and copy:

# In your dom0
[jens@dom0]$ sudo mkdir -p /srv/xen/systems/opensolaris200811/opensolaris-1/platform/i86xpv/kernel
[jens@dom0]$ sudo mkdir -p /srv/xen/systems/opensolaris200811/opensolaris-1/platform/i86pc
[jens@dom0]$ sudo scp jens@ /srv/xen/systems/opensolaris200811/opensolaris-1/platform/i86xpv/kernel/.
[jens@dom0]$ sudo scp jens@ /srv/xen/systems/opensolaris200811/opensolaris-1/platform/i86pc/

Clean up your domU and reboot:

# in your Solaris domU
$ pfexec beadm unmount opensolaris-1
$ beadm list
BE Active Mountpoint Space Policy Created
-- ------ ---------- ----- ------ -------
opensolaris N / 5.14M static 2008-12-05 14:58
opensolaris-1 R - 2.56G static 2008-12-06 22:57
$ pfexec shutdown -i0 -g0 -y

Now restart the domU into the new boot environment.

Using the installation-CD as rescue CD

Sometimes you might want to use the installation CD as a rescue CD for your domU. The following tips might be of some value.

Access the rpool from the installation CD

If you do not have a populated /a then close the installer and do the following as root (or with pfexec):

$ zpool import -f rpool
$ zfs set mountpoint=/a rpool/ROOT/opensolaris
$ zfs mount rpool/ROOT/opensolaris

Find the pool-id of the boot-environment

When you update your boot environment, you must also update your Xen domain configuration. The zfs-bootfs kernel parameter must point to your active boot environment.
Your ZFS-pools won’t be imported, when you boot from the installation CD. The following tips show you, how to find out the pool-id and using that, the ZFS-id for your Xen config.
Use zdb to scan your disc:

$ zdb -l /dev/dsk/c3d0s0 |grep pool_guid

Continue by dumping the ZFS-config, for details see above.

$ pfexec zdb -d -e 14025158204312238343


Everything, that did not fit into the text but might be helpful.

TCP connections not working

You might need to disable TCP-checksum offloading. To do so run

$ echo "set xnf:xnf_cksum_offload = 0" >>/etc/system

Do not forget to update your boot_archive afterwards (and to copy it to your dom0). The bug is filed in OpenSolaris with the id 6633784. According to the OpenSolaris issue-tracker this bug has been resolved with snv_81.

TCP stalls

When your downloads (esp. from the Internet) start good but at some point grind to a halt, it might be because of an endpoint that does not understand TCP - SAKCs. See here for a fix.

Slow network

The maximum sizes for a range of TCP buffers are not large enough for GBit-Ethernet. The following script sets the to more reasonable values.

# vim: set ft=sh :
# performance
# see
# see
ndd -set /dev/tcp tcp_xmit_hiwat 131072
ndd -set /dev/tcp tcp_recv_hiwat 131072


The following sites provided the background for this post

The following sites provided additional information