2007.09.16 - 20:46:23 PDT
Long story, buckle up.
So I'm building a new server. Nothing fancy, but eventually I'd like this box to take over for the server that's running this site. So I'm trying to actually do stuff up right, have real redundancy, harden the server some, etc. The box has been sitting around for a while just minus HDDs and PSU, and those I got. I picked up a couple identical small disks and want to do mirroring for redundancy. This site has been hobbling along on a wing and a prayer, and (fingers crossed) no data loss yet. Yet. I'd like to keep it that way. I also decided I'm going to give FreeBSD a shot. Supposedly it has great performance under load, is way stable, and a host of other good things. Now, I don't know crap about BSD and that's actually a big part of why I'm using it. userlame likes to learn.
Parts arrived and I got to putting things together when, genius that I am, I toasted the motherboard. Oops. Ok, so now I'm waiting on a new mobo. While I'm waiting, I decided I'd do some test installing and BSD learning and such on a VM. Well I've finally gotten the install part down now, and that's what this post is about. The actual install is easy as pie. Add mirroring into the mix, and it gets a bit ugly.
So. I learned about these "slices" and "partitions" BSD uses -- it's much different than anything I've dealt with before -- and I kind of have a handle on what's going on. I decided I'm going to have my disk layout as such: each disk with a single slice and partitions laid out like so:
/ (root) - mirrored between drives
swap - striped between drives
/tmp - striped
/usr - mirrored
/var - mirrored
/var/log - mirrored
/home - mirrored
Getting through the install process is easy like I said. I can very quickly get one disk partitioned and do a minimal install that takes about 20 seconds. But where do I go from here once I have one disk layed out correctly (how to mirror)? After monkeying around for hours on end, here's how I did it:
Note: The way I do this is likely very wrong and I don't advocate anyone follow these directions. Really, I don't know what I'm doing wrt BSD.
First, I installed everything to one disk. Then, I just ran through the exact same install process on the other disk. I found this was the easiest way to get things partitioned and correct bsdlabels on each disk slice. At this point the disk looks like this:
ad0s1a - /
ad0s1b - swap
ad0s1d - /var
ad0s1e - /tmp
ad0s1f - /usr
ad0s1g - /var/log
ad0s1h - /home
And the other disk is exactly the same, just ad1 instead of ad0. Now I ran into a chicken and egg problem. I couldn't get gmirror to label (or insert) a mounted partition. I set sysctl kern.geom.debugflags=16 and that didn't help. I also couldn't use gmirror when booted from the cd (different version of some sort missing a lot of functionality it seemed). I tried about a katrillion things before I came up with a solution.
First, I booted to ad0 which gave me a running system with all the above partitions. Then I got gmirror into the mix with "gmirror load". Then, for each mirror I wanted (root, var, usr, varlog, home) I labeled the second disk's partitions. For example the root partition was "gmirror label -v -b round-robin root /dev/ad1s1a" and /usr was "gmirror label -v -b round-robin usr /dev/ad1s1f". After doing this for each partition I had a device in /dev/mirror for each label name - such as /dev/mirror/root.
Next, I wiped those mirrors and layed out a new filesystem on each of them like "newfs -O2 -U /dev/mirror/usr". Next, I mounted the new mirror filesystems into /mnt/labelname such as "mkdir /mnt/varlog" then "mount /dev/mirror/varlog /mnt/varlog". After all the mirrors were mounted, I dumped the live filesystems into them like "dump -L -0 -f- /home | (cd /mnt/home && restore -r -v -f-)". After dumping each one, I unmounted the filesystem and removed the /mnt/labelname directory to be sure things went smooth.
One special note: while I had /mnt/root mounted and after running "dump -L -0 -f- / | (cd /mnt/root && restore -r -v -f-)" I needed to edit /etc/fstab and /boot/loader.conf before rebooting, so I edited these on both the live filesystem and the mnt'd filesystem. In /etc/fstab (and /mnt/root/etc/fstab), I changed everything from the real partitions (such as /dev/ad0s1a and /dev/ad0s1g) to the mirrored partitions (such as /dev/mirror/root and /dev/mirror/varlog). In /boot/loader.conf (and /mnt/root/boot/loader.conf - both empty files) I simply added the line geom_mirror_load="YES". I finished umounting /mnt/root (it was the last one I did), and rebooted. Sure enough, on rebooting I was running on my mirrored filesystems! :
/dev/mirror/root on / (ufs, local)
devfs on /dev (devfs, local)
/dev/mirror/home on /home (ufs, local, soft-updates)
/dev/ad0s1e on /tmp (ufs, local, soft-updates) [Still on real disk; we'll get there]
/dev/mirror/usr on /usr (ufs, local, soft-updates)
/dev/mirror/var on /var (ufs, local, soft-updates)
/dev/mirror/varlog on /var/log (ufs, local, soft-updates)
But hold on, they aren't really mirrored yet. They're still only running on one disk. Now that the partitions on ad0s1 aren't being mounted however, it's a simple matter to add those partitions to the mirror. Examples: "gmirror insert -v root /dev/ad0s1a" and "gmirror insert -v home /dev/ad0s1h". And ta-da that's it! A mirrored filesystem! (I don't have [code] tags so this formatting will be nasty):
# gmirror status
Name Status Components
mirror/root COMPLETE ad0s1a
ad1s1a
mirror/var COMPLETE ad0s1d
ad1s1d
mirror/usr COMPLETE ad0s1f
ad1s1f
mirror/varlog COMPLETE ad0s1g
ad1s1g
mirror/home COMPLETE ad0s1h
ad1s1h
I know I mentioned something about doing striping earlier though, so let's get to that. Once I had the mirroring stuff under my belt, this wasn't so tough. Let's do the swap first. We can't use any partitions that are in use so first we have to turn swap off. No problem. "swapoff -a". Then we just create the stripe - "gstripe load" to turn it on and "gstripe label -v swap /dev/ad0s1b /dev/ad1s1b" and that's it! "swapon /dev/stripe/swap" and we're in business. Don't forget to update /etc/fstab with the new swap device as well. /tmp isn't much harder. Just a "umount /tmp" followed by "gstripe label -v tmp /dev/ad0s1e /dev/ad1s1e" and that's done. Change that in /etc/fstab as well. Just one more thing to do - update /boot/loader.conf to load striping as well. Add the line (only contains our 1 line from earlier now) geom_stripe_load="YES" and that's it.
A reboot later, and I'm running with striped swap and /tmp, and the rest of my filesystems are mirrored. And it only took the better part of 9 hours to get there!
So I'm building a new server. Nothing fancy, but eventually I'd like this box to take over for the server that's running this site. So I'm trying to actually do stuff up right, have real redundancy, harden the server some, etc. The box has been sitting around for a while just minus HDDs and PSU, and those I got. I picked up a couple identical small disks and want to do mirroring for redundancy. This site has been hobbling along on a wing and a prayer, and (fingers crossed) no data loss yet. Yet. I'd like to keep it that way. I also decided I'm going to give FreeBSD a shot. Supposedly it has great performance under load, is way stable, and a host of other good things. Now, I don't know crap about BSD and that's actually a big part of why I'm using it. userlame likes to learn.
Parts arrived and I got to putting things together when, genius that I am, I toasted the motherboard. Oops. Ok, so now I'm waiting on a new mobo. While I'm waiting, I decided I'd do some test installing and BSD learning and such on a VM. Well I've finally gotten the install part down now, and that's what this post is about. The actual install is easy as pie. Add mirroring into the mix, and it gets a bit ugly.
So. I learned about these "slices" and "partitions" BSD uses -- it's much different than anything I've dealt with before -- and I kind of have a handle on what's going on. I decided I'm going to have my disk layout as such: each disk with a single slice and partitions laid out like so:
/ (root) - mirrored between drives
swap - striped between drives
/tmp - striped
/usr - mirrored
/var - mirrored
/var/log - mirrored
/home - mirrored
Getting through the install process is easy like I said. I can very quickly get one disk partitioned and do a minimal install that takes about 20 seconds. But where do I go from here once I have one disk layed out correctly (how to mirror)? After monkeying around for hours on end, here's how I did it:
Note: The way I do this is likely very wrong and I don't advocate anyone follow these directions. Really, I don't know what I'm doing wrt BSD.
First, I installed everything to one disk. Then, I just ran through the exact same install process on the other disk. I found this was the easiest way to get things partitioned and correct bsdlabels on each disk slice. At this point the disk looks like this:
ad0s1a - /
ad0s1b - swap
ad0s1d - /var
ad0s1e - /tmp
ad0s1f - /usr
ad0s1g - /var/log
ad0s1h - /home
And the other disk is exactly the same, just ad1 instead of ad0. Now I ran into a chicken and egg problem. I couldn't get gmirror to label (or insert) a mounted partition. I set sysctl kern.geom.debugflags=16 and that didn't help. I also couldn't use gmirror when booted from the cd (different version of some sort missing a lot of functionality it seemed). I tried about a katrillion things before I came up with a solution.
First, I booted to ad0 which gave me a running system with all the above partitions. Then I got gmirror into the mix with "gmirror load". Then, for each mirror I wanted (root, var, usr, varlog, home) I labeled the second disk's partitions. For example the root partition was "gmirror label -v -b round-robin root /dev/ad1s1a" and /usr was "gmirror label -v -b round-robin usr /dev/ad1s1f". After doing this for each partition I had a device in /dev/mirror for each label name - such as /dev/mirror/root.
Next, I wiped those mirrors and layed out a new filesystem on each of them like "newfs -O2 -U /dev/mirror/usr". Next, I mounted the new mirror filesystems into /mnt/labelname such as "mkdir /mnt/varlog" then "mount /dev/mirror/varlog /mnt/varlog". After all the mirrors were mounted, I dumped the live filesystems into them like "dump -L -0 -f- /home | (cd /mnt/home && restore -r -v -f-)". After dumping each one, I unmounted the filesystem and removed the /mnt/labelname directory to be sure things went smooth.
One special note: while I had /mnt/root mounted and after running "dump -L -0 -f- / | (cd /mnt/root && restore -r -v -f-)" I needed to edit /etc/fstab and /boot/loader.conf before rebooting, so I edited these on both the live filesystem and the mnt'd filesystem. In /etc/fstab (and /mnt/root/etc/fstab), I changed everything from the real partitions (such as /dev/ad0s1a and /dev/ad0s1g) to the mirrored partitions (such as /dev/mirror/root and /dev/mirror/varlog). In /boot/loader.conf (and /mnt/root/boot/loader.conf - both empty files) I simply added the line geom_mirror_load="YES". I finished umounting /mnt/root (it was the last one I did), and rebooted. Sure enough, on rebooting I was running on my mirrored filesystems! :
/dev/mirror/root on / (ufs, local)
devfs on /dev (devfs, local)
/dev/mirror/home on /home (ufs, local, soft-updates)
/dev/ad0s1e on /tmp (ufs, local, soft-updates) [Still on real disk; we'll get there]
/dev/mirror/usr on /usr (ufs, local, soft-updates)
/dev/mirror/var on /var (ufs, local, soft-updates)
/dev/mirror/varlog on /var/log (ufs, local, soft-updates)
But hold on, they aren't really mirrored yet. They're still only running on one disk. Now that the partitions on ad0s1 aren't being mounted however, it's a simple matter to add those partitions to the mirror. Examples: "gmirror insert -v root /dev/ad0s1a" and "gmirror insert -v home /dev/ad0s1h". And ta-da that's it! A mirrored filesystem! (I don't have [code] tags so this formatting will be nasty):
# gmirror status
Name Status Components
mirror/root COMPLETE ad0s1a
ad1s1a
mirror/var COMPLETE ad0s1d
ad1s1d
mirror/usr COMPLETE ad0s1f
ad1s1f
mirror/varlog COMPLETE ad0s1g
ad1s1g
mirror/home COMPLETE ad0s1h
ad1s1h
I know I mentioned something about doing striping earlier though, so let's get to that. Once I had the mirroring stuff under my belt, this wasn't so tough. Let's do the swap first. We can't use any partitions that are in use so first we have to turn swap off. No problem. "swapoff -a". Then we just create the stripe - "gstripe load" to turn it on and "gstripe label -v swap /dev/ad0s1b /dev/ad1s1b" and that's it! "swapon /dev/stripe/swap" and we're in business. Don't forget to update /etc/fstab with the new swap device as well. /tmp isn't much harder. Just a "umount /tmp" followed by "gstripe label -v tmp /dev/ad0s1e /dev/ad1s1e" and that's done. Change that in /etc/fstab as well. Just one more thing to do - update /boot/loader.conf to load striping as well. Add the line (only contains our 1 line from earlier now) geom_stripe_load="YES" and that's it.
A reboot later, and I'm running with striped swap and /tmp, and the rest of my filesystems are mirrored. And it only took the better part of 9 hours to get there!