rsync

Introduction

rsync stands for remote sync and it is a tool to synchronize to directories being them in the same host or to sync a remote with a local server.

Any time you need to copy files from one directory to another or from one server to another and copy only files that have changed, then you can use rsync, keep in mind that both computers need to have rsync application installed

How rsync works?

rsync checks the filesize and date-time of the files in the origin and destination, if the differ then the file from the origin is transferred and copied to the destination. With this algorithm rsync may miss small changes, and will not send files that should be sent.

If you enter the option –checksum then rsync will perform checksums on local and remote systems and define which file should be copied from origin to destination.

Now that files to be sent are defined rsync needs to define which part of the file has changed, on the destination server two checksums are performed MD5 hash and rolling hash, the results are send to the origing server, on the origin system the file is also divided in chunks and rolling hash is performed, then the MD5 hash is performed and the sender determines which sections of the file have changed and that information is sent to the destination server, it also sends the destination server the information on where to insert or remove the changes so both copies are now identical.

When used locally rsync has access to both directories, but when used to sync two different servers, rsycn relies on ssh to transfer data between both machines.

Syntax

    options source
        |     |
        v     v
rsync -av /home/user1/ user@remoteserver:/home/user
  ^                          ^
  |                          |
command                 destination

• Command: Is the rsync command itself
• Options: All the rsync options comes here, the commonly used are -avrd and –delete, I will explain later each of them
• Origin: Are the origin files, can be local or remote files
• Destination: Is the remote directory, and it can be in the local filesystem or on a remote one

Simple examples

Before we can start we need to learn one very useful option –dry-run, this option will run the command, but will make no changes to files, it will just output what the command will do, if that is what you want, you can run it without the –dry-run option

rsync -avz --dry-run /sourcefolder user@host:/destinationfolder/

Sync two local folders

As I said before you can use rsync to sync two directories on the same machine:

rsync -av /home/user1/my-app/ /home/user1/my-app-bak/

Sync files from local to remote

To sync two directories being the local one the origin and the remote the destination.

rsync -av /home/user1/my-app/ user@server1:/home/user1/my-app/

Sync files from remote to local

To sync two directories being the remote one the origin and the local one the destination

rsync -av user1@server1:/home/user1/my-app/ /home/user1/my-app/

More examples

Compress files while transferring

If you want to compress to make transfer faster and use less bandwitdh use -z

rsync -avz localfolder/ user1@host:/remotefolder/

Let's explain the option switches

-a

This is equivalent to -rlptgoD. It implies recursive but preserve file attributions

-v

This option means verbose, and increses the amount of info displayed on the screen, useful when debugging

-z

This option compresses the files before sending to the destination system, it is usefull to speed up the transmission

Keep two directories in sync

If what you want is to have two directories in sync, this is both with the same information rsync is your best bet, and you can use the command with this options

rsync -cazv --delete /localfolder/ user@remote:/remotefolder/

This command will keep files in localfolder and remotefolder with the same info, let's explain the option switches

-c

Checksum, this option makes rsync to use checksum instead of just filesize and date to determine which files are going to be transmited

--delete

Deletes the files that exists in destination and not in origin, this in order to have both folders with exactly the same files

Exclude some files

If you want rsync to exclude some files from being sync you can use the option swich --exclude like these examples shows

Exclude a single file

rsync -avz --exclude 'file' source/ destination/

Exclude a type of files

rsync -avz --exclude '*.typ' source/ destination/

Exclude a folder

rsync -avz --exclude 'folder' source/ destination/

Exclude multiple files or folders

rsync -avz --exclude '*.file_type' --exclude 'folder' source/ destination/

Set the max file-size to be transferred

rsync -avz --max-size="250k" /sourcefolder/ user@server:/destinationfolder/

With the –max-size option you set the maximum size of the files to be synced, in the example we can see that the max size is set to 250 Kilo bytes, so no files bigger that that will be considered by rsync.

Delete source files after sync

If you want to delete the files from origin system after copy them.

rsync -zv --remove-source-files /sourcefolder/ user@host:/destinationfolder/

Set the speed limit for transfer

If you do not want to overload the Internet conection, you can limit the bandwidth

rsync --bwlimit=100k -avz /sourcefolder/ user@host:/destinationserver/

Specify ssh as the protocol to use

If you want to force rsync to use ssh use the -e option

rsync -avz -e ssh /sourcefolder/ user@host:/destinationserver/

Change the port for ssh connection

If you want to use another port instead of 22 for ssh connection

rsync -avz -e 'ssh -p 4000' /sourcefolder/ user@host:/destinationfolder/

Conclusion

As you can see there are a lot of options when using rsync, rsync is a great tool and to learn how to use it will help you maintaining servers.