Cassandra: links to snapshots in one place

Use rsync with '-L' to copy elsewhere.

#!/usr/bin/ruby

require 'fileutils'

mybasedir = '/var/lib/cass'
mydatadir = mybasedir + '/data'
mysnapshotlinksdir = mybasedir + '/snapshots'
FileUtils.remove_dir(mysnapshotlinksdir) if File.exists?(mysnapshotlinksdir)
Dir.chdir mydatadir
mysnapshotdirs = Dir.glob("**/*/")
mysnapshotdirs = mysnapshotdirs.grep(/snapshots\/.+/)
#p mysnapshotdirs

mysnapshots = {}
mysnapshotdirs.each do |mydir|
    mysnapshot = {}
    myparts = mydir.split('/')
    mysnapshot['keyspace'] = myparts[0]
    mysnapshot['cf'] = myparts[1]
    mysnapshot['tag'] = myparts[3]
    mysnapshot['linkdir'] = [mysnapshotlinksdir, mysnapshot['tag'], mysnapshot['keyspace']].join('/')
    mysnapshot['link'] = [mysnapshotlinksdir, mysnapshot['tag'], mysnapshot['keyspace'], mysnapshot['cf']].join('/')
    mysnapshot['target'] = [mydatadir, mydir].join('/')
    #p mysnapshot
    FileUtils.mkpath mysnapshot['linkdir'] unless File.exists?(mysnapshot['linkdir'])
    File.symlink(mysnapshot['target'], mysnapshot['link']) unless File.symlink?(mysnapshot['link'])
end

tsunami-udp: faster than rsync

1. build
1. sudo apt-get install git gcc
2. sudo apt-get install automake autoconf
3. git clone git://github.com/rriley/tsunami-udp.git
4. cd tsunami-udp
5. ./recompile.sh
6. sudo make install
2. run
1. you'll need a port open to allow direct connection from client to server
1. unfortunately, this doesn't work through NAT firewalls alone
2. firewall / port forwarding
1. to server, TCP, 46224 by default
2. to client, UDP, 46224 by default
2. start up server
1. tsunamid myfile.gz
3. connect with client
1. tsunami set rate 5M connect myserver.domain.com get myfile.gz
2. it will flood your connection if you dont set rate properly
3. documentation
1. http://tsunami-udp.cvs.sourceforge.net/viewvc/tsunami-udp/docs/USAGE.txt
2. splits files automatically
3. allows wildcards when running server and client commands, "*", namely
1. client will auto-find all files served, one after the next
2. use forward-slash, i.e. get \*, for client command 
1. so bash doesn't intrepret the asterisk
4. undocumented
1. doesn't do subdirectories, better tar that up and have plenty of disk space

Specify ssh key when using rsync

WARNING: don't use ~ and don't use double quotes.

rsync -av -e 'ssh -i /home/me/.ssh/id_rsa_other' root@logging.gumby.com:/remotedir/ /localdir/

Also, some alternative port:

rsync -av -e 'ssh -p 2221' root@logging.gumby.com:/remotedir/ /localdir/

rsync: include only these

1. rsync -avP --include=*/ [set of includes] --exclude=* 
1. for example
1. rsync -n -avP --include=*/ --include=*.dat  --include=*.idx --exclude=* /informatica/ /backup/informatica/

From: http://masstransmit.com/garage_blog/rsync-quirks/

Quick start howto for divish on Debian

1. Prep
1. Here, there is one backup server and two client servers that need to be backed up
2. Make sure root user on backup server can ssh to mybox01.mydomain.net and mybox02.mydomain.net without password
1. Later, much later, see online for better, more secure ways
3. install dirvish and rsync on server, rsync on clients
1. apt-get install dirvish
2. apt-get install rsync
2. Clients
1. The below are just example directories, use your own, the ones you want backed up
2. mkdir -p /data/backups/
3. mkdir -p /data/backups/etc
4. mkdir -p /data/backups/var/log
5. rsync -av /etc/ /data/backups/etc/
6. rsync -av /opt/ /data/backups/opt/
7. Make a cron job to do the rsyncs above nightly
3. Server
1. mkdir -p /backup/dirvish/mybox01/dirvish
2. mkdir -p /backup/dirvish/mybox02/dirvish
3. vi /backup/dirvish/mybox01/dirvish/default.conf
1. get contents below
4. vi /backup/dirvish/mybox02/dirvish/default.conf
1. get contents below
5. dirvish --vault mybox01 --init
6. dirvish --vault mybox02 --init
4. Verify
1. Backed up files should now be under /backups/dirvish on backup server
1. tree /backup/ -d -L 3
2. find /backup/dirvish -ls | less
5. Tell dirvish to do nightly pull
1. vi  /etc/dirvish/master.conf.mybackup
1. get contents below
6. Tomorrow, verify the pull worked, and next week too
7. Exclusion and expire options
1. Research options one can add to master.conf.mybackup

Server files

/etc/dirvish/master.conf

bank:
    /backup/dirvish

Exclude:
    lost+found/

/etc/dirvish/master.conf.mybackup

Runall:
    mybox01

    mybox02

/backup/dirvish/mybox01/dirvish/default.conf

client: mybox01.mydomain.net
tree: /data/backups

xdev: true
index: gzip

/backup/dirvish/mybox02/dirvish/default.conf

client: mybox02.mydomain.net
tree: /data/backups

xdev: true
index: gzip