I've used SyncThing for many years, and for the most part I've been happy. I use it to synchronize the contents of directories between various machines of my own, and (occasionally) to share directories with others (although this often doesn't work out).
But recently I experienced catastrophic data loss (see the last paragraph of this post) through the use of SyncThing, and while I should have known better, it was severe enough that I decided to revisit my needs and consider alternatives.
As is usually the case, there is an existing unix tool that has been around for ages that is perfect for the job, and this time it is rsync.
To understand why rsync is the perfect fit for me it helps to understand what when wrong with SyncThing. I use SyncThing to synchronize my Documents directory to another machine on my LAN that is dedicated to backup duties. This machine serves no other purpose and has been happily humming-along for a couple years, but a couple weeks ago it suddenly disappeared from SyncThing's console and when I looked into it, it had rebooted and the external drive attached to it didn't mount. Since it was unmounted I ran a quick fsck which found some errors and repaired them. After this the disk mounted without complaint so for good measure I rebooted the machine to make sure it mounted automatically at boot. After the reboot everything looked good and the machine reappeared in SyncThing's console on my other machines, so I considered the matter resolved.
A week or so later I went looking for some files in a subdirectory of my Documents directory and couldn't find them. I assumed that I had moved something and did a search, but still the files were missing. Alarmed, I poked-around in some other directories and found more things missing, but no obvious pattern as to what was gone and what wasn't.
I'm not exactly sure when it dawned on me, but as I was thinking about what automated process could modify files in this directory I thought about SyncThing. After ruling out everything else I realized what had happened.
When I fsck'd the external disk on the backup machine, damaged files were removed from the filesystem (as expected), but as unexpected, when I mounted the disk and restarted SyncThing, it synchronized these deletes to my other machines, deleting the original copies of the files. Since I was using SyncThing as a means of backup, there are no other copies of these files to recover and the data is gone forever.
This is of course my fault. I'm sure there is a way to configure SyncThing so that deletes only flow one-way, and I shouldn't treat SyncThing as a backup (or at least disaster recovery) system, but after this experience I didn't want this to happen again so I decided this was the end of my relationship with SyncThing. Instead, what I want is something that only does what I explicitly tell it to do, and the first thing that came to mind was rsync.
I've used rsync extensively for various things but I never considered it for this sort of thing. Most of the time I've used rsync for one-time copies, migrations, etc. But when I thought about what I actually use SyncThing for, it seemed like a great fit so I decided to see if I could replace SyncThing with rsync.
My first task was to immediately create a backup, and since I didn't have a trustworthy local machine ready I looked-around for "someone else's computer" I could use. What I found was rsync.net, and it was exactly what I was hoping for. rsync.net is a simple, no-nonsense service that gives you somewhere to rsync stuff to for a reasonable, predictable price. I bought more than enough storage to backup my machines for about $15/month and was backing things up with about 10 minutes.
(of course I wasn't finished backing-up that soon thanks to my pathetic 4mbps upload from AT&T)
Once the initial backup is complete I can schedule the same command to run daily using crontab -e and inserting a line like this:
0 2 * * * /usr/bin/rsync -avH /home/jason username@subdomain.rsync.net:jjg-workstation/
This will back everything in my home directory up to rsync.net. If I delete files locally, they will not get deleted from the backup, and if I somehow delete them from the backup (you can ssh into your rsync.net host and do things like this), they will get replaced the next time the backup runs.
Now that I was on the road to having a real backup it was time to replace the rest of my SyncThing-based workflows. Next on the list was synchronizing FLAC files from my CD ripping machine over to my main laptop. In addition to ripping discs I do a lot of verification, corrections and organizing on this machine and I want these changes reflected when the files are sync'd over to my laptop. Here's the cron job I used for that:
0 0 * * * /usr/bin/rsync -a --delete /home/jason/CompactDisc jason@jjg-workstation.local:/home/jason/
Pretty much the same as the first one with the addition of the --delete flag, which tells rsync to remove files from the destination when they are deleted from the source. This way when I move FLAC files from the "incoming" directory to the "processed" directory I don't end up with multiple copies building-up in the incoming directory on my laptop.
Around this time I started thinking about other things I want to sync and the perennial problem of getting photos from my iPhone to my Linux laptop came to mind. Is there a way I could use rsync for this too? You bet your ass there is!
I found an app called PhotoBackup by Acrosync which can do just that. I had to setup rsyncd on my laptop in order for it to work, but once that was done it works quickly and flawlessly to get photos onto my laptop without copying the same files over and over, duplication or complicated interfaces.
This is a problem I've been trying to solve for years, and just searching for "rsync" on the app store solved it.
Note: On current iOS photos are stored in a proprietary format that needs to be converted to something more common before most tools (and the web) can work with it. There are linux tools for performing this conversion, but my friend raster pointed me to an even better solution: there is a setting in iOS which will stop using the proprietary format and use jpeg instead. This doesn't fix photos you've already taken, but all new photos will be immediately usable after being sync'd, without conversion.
Found under Settings > Camera > Formats > Most Compatible
After solving these problems with rsync I turned to a number of other SyncThing directories I use to share files between various machines in the lab (for example, the old T500 that runs the laser). In these cases the solution was a lot like syncing the CD ripping machine but I increased the frequency as in the case of the laser I want these files to be available in the time it takes me to walk from my workbench to the laser, mill, etc. I'm experimenting with running these jobs once a minute to see what sort of impact this has on system load, etc. If it's negligible I might speed them up even more, but I can usually wait at least a minute if that's as fast as I can safely go.
The final frontier for replacing SyncThing with rsync is sharing directories with others. I don't have a perfect answer for this yet, most of the ideas require some mildly-complicated setup (to be fair, so did SyncThing), but I'm going to try a few things out and once I have something that works well I'll write-up a post dedicated to that specific use case.
