Backup 2.0

Since and b/c we are currently on a maintenance- and backup-spree, trouble arose when dealing w/ the gluster distributed-replicated setup involving four rootservers. After ranting a bit about the incompatibility of gluster e.g. in wheezy and jessy, we received a tip (thanks rpw!) and decided to start using git-annex from now on. Note: To run a really fresh version (5.2), using the neurodebian repo is a good option.

Create Backup Data

As with every backup, it is ideal to have a spare b0x to host all the initial backup data and to function as an initial place to sort and clean-up stuff. As w/ every backup, it is best practice to always keep local backups until we really have a fully functioning, redundant backup. So, in our case, we get most of the backup data via rsync from two other hosts, and some more via scp. It is also a good idea to create a listing of all included files for later comparison, e.g. by doing

tree -ah | tee -a TREE

Initialize local repo

When we have all the data we want in place, we can continue by

git init
Initialized empty Git repository in /home/d1g/ANNEX/.git/
git annex init "NWS4"
init NWS4 ok
(recording state in git...)
git annex direct

Now, files should be added:

git annex add . (takes quite some time to create the initial checksums)
git commit -m "Initial git-annex NWSx REPO"

Initialize remote repo

On all of the remote nodes, the preparation should be something like

mkdir ANNEX
cd ANNEX
git init
git annex init "NWS3"
git annex direct

Add remote repo

git remote add nws4 ssh://nws4/home/d1g/ANNEX

Synchronize data

Now we can start to synchronize the data in between all involved repos, first copying from the remote location:

git annex sync --content

or, if we want to be more specific:

git annex copy VMB2015 --from-nws4

Keep data up to date

git annex watch

If annex watch is not running, and If now one host sends a new file via

git annex add .
git annex sync --content

we can use nws4 to "spread" it to all the other hosts automatically, if we want - after having added them all as remotes - by issuing the exact same command there. For common things that could possibly go wrong, this page may contain some useful information.