Back Up with tar over ssh & rsync over ssh

1)

Shuffling files between servers is simple with scp:

[root@host]# scp some-archive.tgz server:/

Or even copying many files at once:

[root@host]# scp server:/usr/local/etc/* .

2)

But scp isn't designed to traverse subdirectories and preserve ownership and permissions. Fortunately, tar is one of the very early (and IMHO, most brilliant) design decisions in ssh to make it behave exactly as any other standard Unix command. When it is used to execute commands without an interactive login session, ssh simply accepts data on STDIN and prints the results to STDOUT. Think of any pipeline involving ssh as an easy portal to the machine you're connecting to. For example, suppose you want to backup all of the home directories on one server to an archive on another:

[root@host]# tar zcvf - /home | ssh server "cat > kk-homes.tgz"

Or even write a compressed archive directly to a tape drive on the remote machine:

[root@host]# tar zcvf - /var/named/data | ssh host "cat > /dev/tape"

Suppose you wanted to just make a copy of a directory structure from one machine directly into the filesystem of another. In this example, we have a working Apache on the local machine but a broken copy on the remote side. Let's get the two in sync:

[root@host]# cd /usr/local

[root@host:/usr/local]# tar zcf - apache/ | ssh pacman "cd /usr/local; mv apache apache.bak; tar zpxvf -"

This moves /usr/local/apache/ on pacman to /usr/local/apache.bak/, then creates an exact copy of /usr/local/apache/ from host, preserving permissions and the entire directory structure. You can experiment with using compression on both ends or not (with the z flag to tar), as performance will depend on the processing speed of both machines, the speed (and utilization) of the network, and whether you're already using compression in ssh.

Finally, let's assume that you have a large archive on the local machine and want to restore it to the remote side without having to copy it there first (suppose it's really huge, and you have enough space for the extracted copy, but not enough for a copy of the archive as well):

[root@host]# ssh kkb "cd /usr/local/pacland; tar zpvxf -" < really-big-archive.tgz

Or alternately, from the other direction:

root@host:/usr/local/pacland]# ssh kkbs "cat really-big-archive.tgz" | tar zpvxf -

If you encounter problems with archives created or extracted on the remote end, check to make sure that nothing is written to the terminal in your ~/.bashrc on the remote machine. If you like to run /usr/games/fortune or some other program that writes to your terminal, it's a better idea to keep it in ~/.bash_profile or ~/.bash_login than in ~/.bashrc, because you're only interested in seeing what fortune has to say when there is an actual human being logging in and definitely not when remote commands are executed as part of a pipeline. You can still set environment variables or run any other command you like in ~/.bashrc, as long as those commands are guaranteed never to print anything to STDOUT or STDERR.

Using ssh keys to eliminate the need for passwords makes slinging around arbitrary chunks of the filesystem even easier (and easily scriptable in cron, if you're so inclined).

3)

While tar over ssh is ideal for making remote copies of parts of a filesystem, rsync is even better suited for keeping the filesystem in sync between two machines. Typically, tar is used for the initial copy, and rsync is used to pick up whatever has changed since the last copy. This is because tar tends to be faster than rsync when none of the destination files exist, but rsync is much faster than tar when there are only a few differences between the two filesystems.

To run an rsync over ssh, pass it the -e switch, like this:

[root@host]# rsync -ave ssh kkraj:/home/ftp/pub/ /home/ftp/pub/

Notice the trailing / on the file spec from the source side (on kkraj.) On the source specification, a trailing / tells rsync to copy the contents of the directory, but not the directory itself. To include the directory as the top level of whatever is being copied, leave off the /:

[root@host]# rsync -ave ssh bkn:/home/six .

This will keep a copy of the ~root/six/ directory on village in sync with whatever is present on bkn:/home/six/.

By default, rsync will only copy files and directories, but not remove them from the destination copy when they are removed from the source. To keep the copies exact, include the -- delete flag:

[root@host]# rsync -ave ssh -- delete kkraj:~one/reports .

Now when old reports are removed from ~one/reports/ on kkraj, they're also removed from ~six/public_html/reports/ on jammer, every time this command is run. If you run a command like this in cron, leave off the v switch. This will keep the output quiet (unless rsync has a problem running, in which case you'll receive an email with the error output).

Using ssh as your transport for rsync traffic has the advantage of encrypting the data over the network and also takes advantage of any trust relationships you already have established using ssh client keys. For keeping large, complex directory structures in sync between two machines (especially when there are only a few differences between them), rsync is a very handy (and fast) tool to have at your disposal.