Openfiler install on large disk + failover setup + usb install
Posted on 12:42, May 5th, 2009 by Many Ayromlou
Okay so I’ve been playing around with openfiler for the past couple of months. We’re trying to setup a Student homedirectory NAS device and have a mirror machine that would take over if our primary dies. Our machines are hand built 13-TB NAS servers using 16 x 1TB Seagate disks and a 16 channel sata2 raid controller from 3Ware. There are several problems that one needs to overcome in this type of setup so I will try to cover it, bit by bit as I finish confirming it at work. As I said we’re using a Super Micro case and motherboard (Dual Quadcore Xeon) and we’ve stuffed a 16 channel 3ware 9650 controller in there. The first issue we had was with hardware and the fact that we had some screwy new firmware on the controller that was not working nicely with our 16 x 1TB seagate drives. We downgraded the firmware and got the machine to post. Then we created a (roughly) 14 TB container in raid-6 mode (16 drives, less 2). We further devided up the space into a 20GB boot partition (using the bios setting in 3ware bios) and a giant (rougly) 13TB partition that will hold our student data. The 20 GB partition will later on hold our swap space and non essential (frequently updated) folders under /var (lock, log, etc.)
We have physically 2 separate machines that are exact copies of each other hardware-wise. The plan initiallly was to use DRBD and heartbeat service to create a High availability NAS cluster, but since we are tying to authenticate (for smb) with our Windows system, we could not get that configuration working (and frankly I still don’t trust DRBD, as good as it is). So we decided to create two USB sticks images. One for master and another for slave. The master will be a machine enrolled into our Active directory domain and the slave will be a passive (private) rsync server. The master USB image is configured with all the AD stuff and two interfaces. One interface serves as the NAS and another runs rsync against our slave/rsync server. When/If the master fails (ie: motherboard failure) beyond recognition, we simply plug the master USB stick into our slave machine and reboot it. Since the machines are exact copies of one another the (old slave) will now be master and once the (old) master is fixed, it will become the new slave/rsync server. Real simple.
So here is Chapter one – How do you get Openfiler 2.3 to boot off a USB stick:
Before you start you’ll need the following:
Insert your USB stick, and boot from the OpenFiler 2.3 installation CD. At the boot prompt, type expert (for text mode type expert text, I used graphical mode). Manually configure your partitions. I just had one 2Gb partition (ext2) on /. I used ext2 since it has no journal and won’t constantly write to the USB stick. No Swap partition at this point. After the install I noticed that something between 600 and 700 Mb was used for the system, so you might be able to use about 200-300Mb for swap if really needed (however, I doubt the use for a swap partition, as USB storage is really slow). The installer will breeze through to the end. Note that it is realllyyy slow. It took more than an hour on my config. Reboot at the end and get the OF2.3 CD booting again in rescue mode by typing “linux rescue” at the prompt. Once you’re at the prompt mount the USB stick manually (fdisk -l might help as it will print out info about all the disks). My USB stick was /dev/sdc, hence the commands below:
Now you’ve got the partition mounted and your shell chrooted to the root of the USB stick. We next copy the initrd on the USB stick into a temporary directory (on the stick) and uncompress it so we can modify it. You need to do this so that grub can initialize the bootloader ram disk off the USB stick (ie: makes OF installation bootable from USB).
At this point we need to edit the “init” file (text file containing kernel module listings that are required during boot). I used vi to do this, not sure if there is another editor available during rescue mode. Find the line containing “insmod /lib/sd_mod.ko” and insert the following snippet under it:
Save the file and follow along with the following commands to physically copy the appropriate kernel modules to the temp directory.
IMPORTANT – Now adjust grub config (/boot/grub/grub.conf) to reflect the change to initrd filename. You should also repeat this on kernel upgrades (but then again, never touch a working system ;)).
More than likely it’s a no go, since the installer did not install grub properly. Now take out your Ubuntu (or other favourite rescue CD) out and boot from it. Don’t use the OF2.3 CD in rescue mode…..IT DOES NOT WORK. Once booted, mount the USB stick on the system and use the following commands to re-install grub:
Reboot and you should be good to go (you will get a couple of Errors during boot about modules already loaded stuff…..ignore). At some point you do want to move some of those auxiliary directories (/tmp/ /var/log /var/lock and others) and swap file off the stick and onto the 20GB portion of our raid-6 we prepped earlier on. Below you find the fdisk -l listing of that “logical disk” (/dev/sdb in our system):
Device Boot Start End Blocks Id System
here is a breakdown of what goes where (/dev/sdb6 is obviously swap which was prepared with “mkswap” command):
You need to make the above changes to your USB stick’s /etc/fstab, but before rebooting you need to use “cp -a” command to copy all the folders from the appropriate location on the USB stick to the above partitions (by mounting the partitions temporarily/one-at-a-time), just to make sure no process would go crazy if it didn’t find the lock directory (or cache, run, etc.).
Next we want to make four copies of this stick. You can use a Mac or Win (using rawrite) or better yet Linux. It’s important that the stick your copying is not booted. Use the Ubuntu/whatever CD you used ealier and boot it into rescue mode. Go to command line and use “dd” command to create three more copies of the stick you just preped.
Two copies (one for safe keeping) will become your Master USB sticks to boot the machine in Master mode (as described earlier in this article). The other two copies (one for safe keeping) will become your Slave sticks.
These notes have nothing to do with the installation. I’m just putting them down here for safe keeping. Only use these if you’re in trouble.
– If you want to create a “Home Share” and you don’t get the “Make Home Share” button on the interface, something has gone wrong with one of the xml config files. No worries, find and edit the file /opt/openfiler/etc/homespath.xml . Inside it will look something like this:
– If you have upgraded to a Windows 2008 R2 (Win2k8 r2) AD domain and you’re getting authentication errors when accessing your openfiler shares (although everything was working fine under R1) like the ones below:
Nov 16 08:42:02 openfiler winbindd: [2009/11/16 08:42:02, 0] rpc_client/cli_pipe.c:rpc_api_pipe(789)
Could not receive trustdoms
then your problem (more than likely) is the version of Samba that comes with openfiler 2.3. You need to upgrade to 3.4.5. Run “conary updateall” or do “System Update” from the interface, let it update everything and reboot your machine. Once your machine is back up, leave the AD domain and rejoin it and everything should be fine.
– If you’re having problems accessing a samba share you just created on your brand new openfiler, you might want to check the following. Lets say you have a Volume Group called “bigvg” and a Volume inside that called “studentvol” where you have a share called “test”. If you’re having problems accessing the share by just using something like
– Couple of useful commands for Samba troubleshooting…..
– There is another issue with this master/slave setup and that is UID/GID synchronization for samba. This comes into play since we’re rsyncing our files from master to slave. This process also transfers their respective UID/GID to the slave machine. If the master fails, our procedure is to turn if off and reboot the slave using the masters USB stick. This works, but all those rsync’ed UID/GID’s will not match when the slave machine is booted using the masters USB stick (samba voodoo that translates windows UID/GID’s to linux UID/GID is kinda random)…..UNLESS YOU DO THE FOLLOWING (taken from Samba How-To):
Please note that the update version of SAMBA that gets installed after you do “conary updateall” (see above) has a option for this under “Advance” tab of the Accounts section.
5 Responses to “Openfiler install on large disk + failover setup + usb install”
Leave a Reply
You must be logged in to post a comment.