Installing FreeBSD on a mirrored ZFS root pool using 4k sector disks, AHCI and GPT

For those who like a challenge.

You'll need development software for this. The following guide can not be completed using FreeBSD 8.2-RELEASE. I used FreeBSD 8.2-STABLE snapshot 201105 However, if you're willing to drop the 4k sector part, 8.2-RELEASE will do fine.

Advantages of this setup, a list of acronyms

ZFS: Most advanced filesystem on the planet and the only one that can prevent bit rot.
AHCI: Hot swap and native command queuing
GPT: Because the MSDOS era is over and we might want to use large disks
4k sectors: Because hard disk manufacterers are lying bastards *

Disadvantages of this setup

You will use in-development software, it WORKSFORME but is not well tested

Prequisites

FreeBSD 8.2-STABLE snapshot (DVD or USB)
Access to the internet during installation
AHCI set in your BIOS (optional)

Creating the filesystems

Starting the FIXIT environment

Boot from the USB stick or DVD
At the loader menu, press 6 to escape to a loader prompt
Load required modules, then continue booting:
OK load ahci
OK load zfs
OK load geom_nop
OK boot
In sysinstall(8), set your keyboard language and then choose Fixit from the main menu, specifying the appropriate device to run the environment from (DVD or USB)

Partitioning and installing bootcode with gpart

I'll use ada0 and ada1 in all examples. When you don't have AHCI, replace these by your drives (ad6 for example). Check the output of dmesg for daX or adX.

Use empty drives. Erase existing partition tables with dd(1):
# dd if=/dev/null of=/dev/ada0
# dd if=/dev/null of=/dev/ada1

create the GPT

# gpart create -s gpt ada0
# gpart create -s gpt ada1

Create the partitions

About 4k-drives:
4K-drives: 1 physical sector = 4 Kilobytes
512b-drives: 1 phyiscal sector = 512 Bytes
Start the first partion on sector 2048. All following partions should by start on a sector that you can devide by 8.
If you don't need the 4k sectors, leave -b 2048out when creating the swap space

# gpart add -s 64K -t freebsd-boot ada0
# gpart add -b 2048 -s 4G -t freebsd-swap -l swap0 ada0
# gpart add -t freebsd-zfs -l disk0 ada0

Repeat for the second disk
# gpart add -s 64K -t freebsd-boot ada1
# gpart add -b 2048 -s 4G -t freebsd-swap -l swap1 ada1
# gpart add -t freebsd-zfs -l disk1 ada1

gpart show -l will let you check you the current setup

Downloading patched files (optional)

This is only needed when you use 4k sectors

Activate the network

# dhclient -l /tmp/leases igb0 (Your interface may be different)
# echo "nameserver 192.168.1.1" > /etc/resolv.conf (Use your own nameserver)

Download the necessary files

# fetch http://people.freebsd.org/~pjd/zfsboot/gptzfsboot
# fetch http://people.freebsd.org/~pjd/zfsboot/pmbr
# fetch http://people.freebsd.org/~pjd/zfsboot/zfsloader

Install the patched boot code

# gpart bootcode -b ./pmbr -p ./gptzfsboot -i 1 ada0
# gpart bootcode -b ./pmbr -p ./gptzfsboot -i 1 ada1

When not needing the patched boot code, use pmbr and gptzfsboot from /dist/boot/

Create directory to store the zpool.cache file

# mkdir /boot/zfs

Create a mirrored zpool

# gnop create -S 4096 gpt/disk0
# zpool create rpool mirror gpt/disk0.nop gpt/disk1

Check sector size of the zpool with
# zdb | grep ashift
If the value of ashift is 12, it worked.

Remove the .nop extension from the device node
# zpool export rpool
# gnop destroy gpt/disk0.nop
# zpool import -o mountpoint=none rpool

Check the status of the zpool
# zpool status

Installing a basic system

Creating a ZFS hierarchy

# zfs create rpool/ROOT
# zfs create -o mountpoint=/mnt rpool/ROOT/freebsd
# zfs create -o mountpoint=/home rpool/HOME

# zfs create -o atime=off -o compression=on -o exec=on -o setuid=off rpool/ROOT/freebsd/tmp
# chmod 1777 /mnt/tmp

# zfs create -o atime=off rpool/ROOT/freebsd/usr
# zfs create rpool/ROOT/freebsd/usr/local
# zfs create rpool/ROOT/freebsd/usr/src
# zfs create -o compression=on -o setuid=off rpool/ROOT/freebsd/usr/ports
# zfs create -o compression=off -o exec=off rpool/ROOT/freebsd/usr/ports/distfiles
# zfs create -o compression=off -o exec=off rpool/ROOT/freebsd/usr/ports/packages

# zfs create -o atime=off rpool/ROOT/freebsd/var
# zfs create -o compression=on -o exec=off -o setuid=off rpool/ROOT/freebsd/var/crash
# zfs create -o compression=on -o exec=off -o setuid=off rpool/ROOT/freebsd/var/log
# zfs create -o compression=gzip -o exec=off -o setuid=off rpool/ROOT/freebsd/var/mail
# zfs create -o exec=off -o setuid=off rpool/ROOT/freebsd/var/db
# zfs create -o compression=on -o exec=on rpool/ROOT/freebsd/var/db/pkg
# zfs create -o exec=off -o setuid=off rpool/ROOT/freebsd/var/run
# zfs create -o exec=off -o setuid=off -o readonly=on rpool/ROOT/freebsd/var/empty

# zfs create -o atime=off -o compression=on -o exec=on -o setuid=off rpool/ROOT/freebsd/var/tmp
# chmod 1777 /mnt/var/tmp

Extract FreeBSD to the ZFS filesystems

# export DESTDIR=/mnt
# cd /dist/8.2-STABLE
# for dir in base catpages dict doc info lib32 manpages; do (cd $dir; ./install.sh); done
# cd kernels
# ./install.sh GENERIC
# cd ../src
# ./install.sh all

Copy GENERIC kernel

# cd /mnt/boot
# cp -Rpv GENERIC/* kernel/

Copy patched gptzfsboot

# cp -Rv ./gptzfsboot /mnt/boot/

Edit loader.conf

# vi /mnt/boot/loader.conf

ahci_load="YES" zfs_load="YES" vfs.root.mountfrom="zfs:rpool/ROOT/freebsd" kern.maxfiles="20480" (for samba)

Edit rc.conf

Use your own settings. Only the zfs_enable=yes is important in the example below

# vi /mnt/etc/rc.conf

hostname="leto.local.alvin.be" ifconfig_re0="inet 192.168.1.3 netmask 255.255.255.0" defaultrouter="192.168.1.1" keymap="us.iso" zfs_enable="YES"

Set up fstab

# vi /mnt/etc/fstab

/dev/gpt/swap0 none swap sw 0 0 /dev/gpt/swap0 none swap sw 0 0

Copy the zpool cache

# cp /boot/zfs/zpool.cache /mnt/boot/zfs/zpool.cache

Export LD_LIBRARY_PATH

# export LD_LIBRARY_PATH=/dist/lib

Unmount ZFS filesystems and prepare mountpoints

# cd
# zfs umount -a

# zfs set mountpoint=legacy rpool/ROOT/freebsd
# zfs set mountpoint=/usr rpool/ROOT/freebsd/usr
# zfs set mountpoint=/var rpool/ROOT/freebsd/var
# zfs set mountpoint=/tmp rpool/ROOT/freebsd/tmp
# zfs set mountpoint=/home rpool/HOME

Set bootfs property

# zpool set bootfs=rpool/ROOT/freebsd rpool

Reboot

Exit the Fixit environment and reboot
# reboot

Finalize

[add password, timezone, services,...]