If you haven't heard of btsync, it is a free product from Bittorrent (the company) which is based on bittorrent (the protocol) for not only sharing files, but keeping them in sync as well. Read more here.

For several years now I have had various servers and other hosts distributed around the globe; Virtual Private Servers in USA, Europe and Australia, physical servers in family businesses and community group buildings as well as various testing and development boxes at home. To handle system backups (as distinct from user backups), I have been using a hosted service that provides disk space for backups via rsync over SSH, and it has worked very well at a reasonable price... But it's still a price, and now I have btsync :D

All these systems around the globe, most of which have ample disk space free, and many with very good (100mbit+) connectivity to the internet... Hmmm!!

The new world order

My system now uses btsync to maintain a common shared backup folder, syned to each device. So each device hosts a copy of every other hosts backup, as well as it's own. There are several advantages here:

  • Each device now just has to write it's own backup to a local path; no more (relatively) complex transferring to remote systems with SSH keys etc.
  • Same goes for restoring files in the event of configuration error or admin error (oops); just restore what I need from the local disk.
  • I have multiple, geographically distributed copies of all backups.
  • Making use of otherwise wasted disk space.
  • Keeping my data on trusted hosts, rather than shipping it off to a third party (in the US no less!) where I don't know what's happening to it.

There are some potential pitfalls of course:

  • Hosts with low-end connectivity (eg, ADSL) suffer a little bit downloading backups for all the other hosts.
  • Security could be a concern since you're (by design) replicating your data all over the place, sometimes onto boxes with relatively low physical security.

These downsides are outweighed by the benefits and are acceptable risk to me. Your results may vary.

So, how is it done?

First, we need btsync! btsync is released as pre-compiled self-contained binaries for x86 and x64, as well as PowerPC and ARM for use on devices such as RaspberryPi or your preferred NAS (I use Synology).

This command will download the latest version for x64 platform and extract it to /usr/local/bin/btsync then set correct ownership and permissions:

   curl http://btsync.s3-website-us-east-1.amazonaws.com/btsync_x64.tar.gz | tar xvzf - -C /usr/local/bin/ btsync
   chown root:root /usr/local/bin/btsync
   chmod 755 /usr/local/bin/btsync

Now we can create a configuration file. You can generate a sample file using btsync itself:

   btsync --dump-sample-config > ~/.btsync.conf

I use a configuration that looks a little like this:

   { 
     "device_name": "usa.ca.MyHost",
     "listening_port" : 0,
     "storage_path" : "/etc/btsync/sync",
     "check_for_updates" : false,
     "use_upnp" : false,
     "download_limit" : 0,
     "upload_limit" : 0,
     "shared_folders" :
     [
       {
         "secret" : "GetYourOwn!!",
         "dir" : "/mnt/btsync",
         "use_relay_server" : true,
         "use_tracker" : true, 
         "use_dht" : false,
         "search_lan" : false,
         "use_sync_trash" : false
       }
     ]
   }

I suggest you generate a sample file which includes plenty of comments to explain all of this; The most important parts are setting an appropriate "device_name", creating and setting a common "secret" (see the next paragraph), and setting the path to sync ("dir") which in this example is /mnt/btsync

A note about the "secret". The secret is a 160-bit unique key that identifies your shared folder. All your hosts must have the SAME secret for the shared folder, but you also need to protect your secret since anyone with the key will be able to join the swarm and sync with you, accessing your backups, and putting their own in the mix!

Now you have btsync, a config file and a folder to sync, you're ready to start btsync:

   btsync --config /etc/btsync/.btsync.conf

That's it; btsync will start doing it's thing, looking for other hosts with the same shared folder (based on the secret) and synchronizing with them. You of course need to setup something to write your backups into the shared folder; I use and recommend rdiff-backup, but you can use anything you like such a tar, rsnapshot, rdup, duplicity or dar to name a few.