ZFS Root on FreeBSD

FreeBSD’s installation routine (sysinstall) doesn’t support to install ZFS on root, but there are some guides on the wiki which will tell you different ways of doing it. I did it my way, because all of the guides tells you have to install FreeBSD 8.0-RELEASE which does not include the latest security fixes, and because I can. Here is how:

First a few words of caution: This guide depends on that you already have FreeBSD installed on one machine, and it also depends on that you’ll need a USB stick with FreeBSD installed.

On a already existing FreeBSD system:

Create a bootable USB stick if not already done so

Compile the FreeBSD system with a ZFS aware boot loader:
# cd/usr/src
# make buildkernel buildworld SRCCONF=/dev/null __MAKE_CONF=/dev/null \
-DLOADER_ZFS_SUPPORT=YES -j3

Create a ram disk and install FreeBSD on it:
# mdmfs -L -s 1024m md /mnt
/dev/md101: 1024.0MB (2097152 sectors) block size 16384, fragment size 2048
using 6 cylinder groups of 183.72MB, 11758 blks, 23552 inodes.
with soft updates
super-block backups (for fsck -b #) at:
160, 376416, 752672, 1128928, 1505184, 1881440
# make installworld installkernel distribution DESTDIR=/mnt SRCCONF=/dev/null \
__MAKE_CONF=/dev/null -DLOADER_ZFS_SUPPORT=YES
Remember which device was used when creating the ram disk (in my case its /dev/md101) – You’ll need it later.

Create a dump file and copy it to the USB stick:
# umount /mnt
# dump -0 -f - /dev/md101 > freebsd.img
# cp freebsd.img /media
# umount /mnt
# umount /media
# mdconfig -d -u 101

On the victim new FreeBSD system boot with the USB stick and log in. Figure out which disk you’ll use for ZFS. In my case it is /dev/da0.

Slice it up and create a layout:
# fdisk -BI /dev/da0
# bsdlabel -Bw /dev/da0s1
# bsdlabel -e /dev/da0s1
The last command will open ‘vi’ – edit the file so it looks something like this:

# /dev/da0s1:
8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
a:    256M    16      unused        0     0
c:    *       0       unused        0     0         # "raw" part, don't edit
d:    *       *       unused
The first slice(a) will be used for /boot while the rest of the disk (d) will be ZFS.

Format the a-slice and create the new zpool:
# newfs -U /dev/da0s1a
# zpool create data /dev/da0s1d

Create the ZFS layout (YMMV, but do not create a seperate /etc system):
# zfs create data/ROOT
# zfs create data/ROOT/usr
# zfs create data/ROOT/usr/home
# zfs create -o compression=on  data/ROOT/usr/src
# zfs create -o compression=on  data/ROOT/usr/ports
# zfs create -o compression=off data/ROOT/usr/ports/packages
# zfs create -o compression=off data/ROOT/usr/distfiles
# zfs create -o compression=on  data/ROOT/tmp
# zfs create data/ROOT/var
# zfs create -o compression=off data/ROOT/var/db
# zfs create -o compression=on data/ROOT/var/log
# zfs create -o compression=on data/ROOT/var/mail
And some polishing:
# cd /data/ROOT
# chmod 1777 /data/ROOT/tmp
# ln -s usr/home /data/ROOT/home

Prepare the a-slice for boot
# mkdir /data/ROOT/bootdir
# mount /dev/da0s1a /data/ROOT/bootdir
# mkdir /data/ROOT/bootdir/boot
# ln -s bootdir/boot /data/boot

Edit /data/ROOT/boot/loader.conf and add:
vfs.root.mountfrom="zfs:data/ROOT"
zfs_load="YES"

Make the data/ROOT bootable:
zpool set bootfs=data/ROOT data
Restore the image created earlier:
cd /data/ROOT
cat /freebsd.img | restore -rvf -
Chroot to make some adjustments:
chroot /data/ROOT
mount -t devfs devfs /dev

Edit /etc/rc.conf
ifconfig_re0="DHCP" #or whatever your machine have
hostname=pony
sshd_enable=TRUE
zfs_enable=TRUE #important!

Edit /etc/fstab
/dev/da0s1a     /boot   ufs     rw      0       0

create new user and assign password for him and root
# pw user add joe -m -G wheel
# passwd joe
Changing local password for joe
New Password:
Retype New Password:
# passwd root
Changing local password for root
New Password:
Retype New Password:

Exit chroot
# unmount /dev
# exit

Unmount ZFS and bootdir to make some final adjustments
# umount /data/ROOT/boot
# zfs umount -a
# zfs set mountpoint=legacy data/ROOT
# zfs set mountpoint=/tmp data/ROOT/tmp
# zfs set mountpoint=/usr data/ROOT/usr
# zfs set mountpoint=/var data/ROOT/var

Copy the latest zpool.cache to the a-slice:
# mount /dev/da0s1a /mnt
# cp /boot/zfs/zpool.cache /mnt/zfs
reboot and have a lot of fun!