Archive for ‘rsync’ Category
Posted on 14:38, August 15th, 2012 by Many Ayromlou
Here are some quick tip(s) for copying a ton of files between unixy machines really fast. You’re probably thinking “why not use rsync?”…..well rsync can be miserably slow if your source or destination cpu is underpowered. You can always do a rsync after these commands to make 100% certain that everything checks out, but try using one of these methods for the initial copy:
Once your favourite process (above) is done you can do a quick rsync to tie up any loose ends.
Rsync will now fly through the filesystem as 99.9% of the time, 99.9% of the files on the destination are good. And as always make sure you understand the commands before you use them…..and keep backups just in case :-).
While the world was busy trying to figure out how to sync their palm pilots to Windows 98 (or was it 95) Andrew Tridgell was working on his thesis……Rsync. The endall-beall in the world of syncing. You see, if you’ve ever tried any syncing program before you’ll appreciate the ease with which rsync works. You’ll also see the efficiency of it’s algorithm in dealing with changes/updates, and for that you can thank Andrew. I’m just gonna show you how to use this excellent tool along with ssh (another must have/must use free software utility) to setup automated secure backups between two different platforms.
For the purpose of this article I’m using a PowerMac G5 with a 1TB Firewire disk hanging off it as my backup server. As my client I’m using the departmental research server (yes, forget that it’s a server, in this exercise it’s a CLIENT).
So first we need to make sure we have the tools: ssh and rsync. They are already part of my distribution (SuSE Linux 10.1 on the client and OSX 10.5.1 on the server) so I didn’t have to install anything. You want to make sure that your rsync uses protocol version 2.x.x on both sides. (type: rsync –version on your machines to see the protocol version).
Note: You can also grab RsyncX for older OSX versions and/or cwRsync for windows.
Step 1. Is to remove the interactivity that happens during an ssh session (ie: password prompt). To do this we must generate passphraseless keys that we can initially manually exchange between our machines. So in our case we want to grant access to our client machine — via ssh — from the server. Now here is were things get a bit OS specific. Under OSX there is no root account (atleast not one you can log into) by default. In my setup I want to allow email@example.com (OSX machine) to have access to firstname.lastname@example.org (Linux machine), so here is what I had to do.
On the OSX machine (server or machine initiating the rsync session) login as the standard user/admin and issue the following commands:
Now you need to transfer the id_dsa.pub key file to your client somehow. I do this just using ssh again. If you want to be totally super secure you can stick it on a usb key and walk it over to your client machine. So still on the server (OSX) machine I do the following (still in roots account, so be careful):
Now, to verify, while still logged into root on the server (OSX) try to ssh to your client machine (using root’s account on client). If everything works you should be able to log into root’s account on the client (Linux machine) without a password prompt. If you’re prompted for password, stop here, and redo/recheck your procedure.
Step 2. Pheew…..You made it, that was the hard part….the rest is pretty simple. Now we just need to test to make sure things are running smoothly and once that’s done we can automate everything using a cron job (Step 3.). So lets start our test run. For this I’ve chosen to backup the /etc directory on the server (lots of small files, nothing earth shattering) and my chosen path on the server is /Volumes/TERADISK. This is where OSX mounts my firewire drive (your milage may vary). I’ve created a folder on TERADISK called FULLBACKUP, which I use to have rsync house my backups. So to test issue the following command as root (yes still, so be carefull) on the server:
Step 3. Well we’ve reached the end. All you have to do now is to use crontab -e command while still logged in as root on the server to add rsync entries for folders you want synced between the client and the server. Note that I do NOT backup absolutely everything off the client, only specific folders (and their subfolders ofcourse). I’m pretty sure that rsync will get confused if you try to sync things like your dev directory from the linux client. In my cron I have the following:
That’s it. Hopefully this has been useful…..