Top ↑

Synchronise your files rsync-style with FTP

In this tutorial, I will show you how to synchronise (upstream) local files with an FTP server. It will focus primarily on how to do this in the context of putting a website in a live environment.

To simplify things, I've provided a really simple script to use. I must thank the people at Server Fault for the basis of this script. It's nicely customisable and easy to understand, which is nice too.

The script

#!/bin/bash 
HOST="ftp.microphonecheese.com"
USER="ogblog"
PASS="SeCREtPa5S"
LCD="/home/james/Projects/Web/AwesomeSite"
RCD="/"
lftp -c "set ftp:list-options -a;
open ftp://$USER:$PASS@$HOST; 
lcd $LCD;
cd $RCD;
mirror --reverse \
       --delete \
       --verbose \
       --exclude-glob config.xml \
       --exclude-glob .htaccess \
       --exclude-glob thumbnails/ \
       --exclude-glob fullsize/ \
       --exclude-glob *~ \
       --exclude-glob _*/ \
       --exclude-glob _*"

Explanation

First we have the hashbang at the top: #!/bin/bash tells whatever is running the script to run it with /bin/bash, i.e. the default Linux shell.

Now, the meat. The lines beginning with HOST, USER and the like define variables. This is where you change the script to your needs. For example, if your server is at ftp.purplemoose.org, the second line of the script would read HOST="ftp.purplemoose.org". The rest should be obvious, apart from perhaps LCD and RCD. These are the local directory and remote directory respecively.

The line after the variable declarations executes lftp, the Linux FTP command line tool. The stuff in quotes is an argument string passed to lftp. You don't need to worry about everything in it, except the last --exclude-glob lines. These define what files or folders lftp will ignore.

The examples above will ignore the files config.xml, .htaccess, all backup files (e.g. home.php~) and all files with a preceding underscore (_NOTES for example). The point, you may ask, of this is to not upload irrelevant files that might cause conflicts, or are disturbingly large. To exclude a folder, the pattern must have a trailing / (slash) after it, or it will be treated as a file. Folder: --exclude-glob stuff/, file: --exclude-glob stuff.

A final note to make is --delete will remove files that are no longer in the local directory; we are mirroring the local folder. --verbose simply shows what's going on, which is handy.

Examples

Here are a few examples to get you on your way.

Connect to host, transfer everything

#!/bin/bash 
HOST="ftp.microphonecheese.com"
USER="ogblog"
PASS="SeCREtPa5S"
LCD="/home/james/Projects/Web/AwesomeSite"
RCD="/"
lftp -c "set ftp:list-options -a;
open ftp://$USER:$PASS@$HOST; 
lcd $LCD;
cd $RCD;
mirror --reverse \
       --delete \
       --verbose"

Connect to host, transfer everything but .htaccess to /public_html

Most shared hosting sites will put your files in a public_html folder. This example will exclude the .htaccess file as you might have a different setup on your local host.

#!/bin/bash 
HOST="ftp.microphonecheese.com"
USER="ogblog"
PASS="SeCREtPa5S"
LCD="/home/james/Projects/Web/AwesomeSite"
RCD="/public_html"
lftp -c "set ftp:list-options -a;
open ftp://\$USER:\$PASS@\$HOST; 
lcd \$LCD;
cd \$RCD;
mirror --reverse \
       --delete \
       --verbose \
       --exclude-blob .htaccess"

Exclude all prefixed files with a few directories

#!/bin/bash 
HOST="ftp.microphonecheese.com"
USER="ogblog"
PASS="SeCREtPa5S"
LCD="/home/james/Projects/Web/AwesomeSite"
RCD="/public_html"
lftp -c "set ftp:list-options -a;
open ftp://\$USER:\$PASS@\$HOST; 
lcd \$LCD;
cd \$RCD;
mirror --reverse \
       --delete \
       --verbose \
       --exclude-blob *~ \
       --exclude-blob *.bkp \
       --exclude-block staging/"

And there we have it

Hopefully this tutorial has been helpful in some way or another. If you still don't get the syntax of the filtering commands, please leave a comment and I'll see what I can do.