Subversion Backup-Restore

So we are are now mostly migrated from CVS to Subversion. And we are (mostly) happy.

We have around 150,000 revisions split on two repositories (125,000 and 25,000 respectively). Performance is fair - much better than CVS in any event. We run it on Windows 2k3 using some virtual server claiming 2 x 2.4GHz Xenons and 3GB of RAM.

The "mostly-happy" stems from the trouble people experience now they have started using branching (the trouble appearing around the time of merging). And the few weird things we see in the Eclipse Subversive plugin. Still, much better than CVS IMHO.


Anyway, this blog entry is about our backup-restore strategy for Subversion.

We have two servers; the primary in active use, and a secondary that is used for daily backup restores.

Every afternoon at 16:00 a backup script (verify-repositories.rb) runs on the primary. It first validates the integrity of the repositories (this takes around 10 hours!). If the repositories are sound, a hot-backup is run while Subversion remains available. This nightly backup is copied to the secondary server.

In addition to the nightly (full) backups, each commit to Subversion is stored as individual dump files on the primary server (via the commit hooks, so we do not want a dependency on the secondary server at this point). These are basically incremental backup files.

Every morning at 05:00, the secondary server runs the restore script (restore_backup.rb) which does a full restore of the backup set from the same night. It then applies all incremental backups available (those not included in the full backup). This takes around six hours in total.


That is, the combined full and incremental backups are restored so that the primary and secondary servers match state sometime around lunch time. At this time, a mail is sent to our administration list containing status and instructions for how to switch from the primary to the secondary server. So the manual for a crash recovery is readily available.

In theory we should have no more than half a day of downtime in case of a crash in the morning (where we would have to wait for the secondary to complete its restore). A crash in the afternoon should (in theory) only cost us the 15 minutes it takes to switch the DNS entry.

Worst case, a new full restore will take around a day (if the previous automatic restore cannot be used for some reason).

This will work for a good while. But eventually, as the repositories and thus backup+restore time grows, I will have to add a third server.

For now, I am pretty happy with the setup though. I hope the description and scripts can inspire your Subversion backup/restore scheme.


The backup and restore scripts are placed in the Buildmeister repository at Kenai.