jLuger.de - FreeBSDBackup

In the article BackupSystem I have presented a virtual server that stores data encrypted and does deduplication. The system works as expected but has a very bad I/O performance. It makes about 5GB per hour, yes hour. I suspect the zfs-fuse (userspace file system) to be slowing down the hdd throughput. With the release of FreeBSD 9.0 there is now an open source OS that supports zfs with deduplication natively (in kernel).

In this article I will show how to set up a similar system like that with Ubuntu server. I will not focus on the theoretical design decision nor on facts that are valid for both systems. The focus is on differences between FreeBSD and Ubuntu. I will also expect some good Linux knowledge.

In order to start, create a new virtual machine. The OS type is FreeBSD, as RAM you should give at least 1024 MB, and for the HDD 1,5 TB dynamically allocated is fine.

I have used the amd64-DVD of FreeBSD for installation. With this medium I had to change the chipset of the machine from PIIX3 to ICH9 or booting would fail. Of course don't forget to mount the installation image before starting the machine.

Before I continue I have to mention that the installation of FreeBSD with zfs as filsystem is based on the article from http://astralblue.livejournal.com/372471.html. The only issue was that the article was for FreeBSD 8.0 and doesn't work out of the box with 9.0. Thanks to http://namor.userpage.fu-berlin.de/howto_fbsd9_encrypted_ufs.html I was able to adjust it.

Now start the machine and select install when the dialog appears.

Continue installation until partitioning is required.

Then select shell.

For an explanation of the following commands see http://astralblue.livejournal.com/372471.html Continue installation (type exit for that)

When adding a user don't forget to set an UID. Ubuntu starts with 1000 while FreeBSD uses 1001 as initial value. Ensure that the Linux user and the FreeBSD user have the same ID or you won't get access to your files when using rsync for import on FreeBSD.

Add your user to group wheel when you want to get root with su.
Select No when asked to do manual configuration. You will be chrooted in the new installation but don't reboot. See the next step.

In "Complete" dialog select "Live CD" to make the final modifications.

User is root without a password. Note that FreeBSD still has a root account with password.

For an explanation of the following commands see http://astralblue.livejournal.com/372471.html Remove DVD from CD/DVD drive.

Change Network connection from NAT to Bridge. See the article BackupSystem for details.

When starting FreeBSD it may seem to hang on starting ssh. This isn't the ssh server but sendmail failing to get a FQDN for your hostname. Open /etc/hosts and add you hostname at the end of the localhost entries. See http://forums.fedoraforum.org/archive/index.php/t-85365.html.

In order to set up rsync type pkg_add -r rsync as root.In order to start it on boot add rsyncd_enable="YES" to the file /etc/rc.conf. The configuration is the same like on Ubuntu except that the config file is under /usr/local/etc/rsyncd.conf.

Installing dovecot is done via pkg_add -r dovecot as root. Put dovecot_enable="YES" in /etc/rc.conf and proceed with the configuration as done under Ubuntu. But then there are two little more things to do. First remove managesieve from the protocols. The dovecot.conf (also under /usr/local/etc) should contain two entries like this after the edit:
#protocols = imap pop3 imaps pop3s managesieve
protocols = imap pop3 imaps pop3s
The second issue is that FreeBSD doesn't automatically generate ssl key/cert files. To go on without them requires a lot of configuration and you will end up with an unsecure system. The generation of the files is described at http://ebalaskas.gr/wiki/dovecot/DovecotSSL?v=169m. The command:
openssl req -new -x509 -nodes -out /etc/ssl/certs/dovecot.pem -keyout /etc/ssl/private/dovecot.pem -days 1825 -newkey rsa:4096

That's it with the configuration. I have done a little performance test with importing the same files from my HDD that I had also imported into the Ubuntu server. I've got up to about 35GB per hour. That is a huge rise in performance and if you want to go productive with that one there are some improvements. First I have used that same disk for input and output and secondly it was on a virtual system. Using a real server and writing data on another HDD as from where you get your data should speed up the whole thing much more. But 35GB per hour is fine for me.


Update:
Please see also FreeBSDBackup revised and FreeBSDBackup Update 2017.