Cassandra: links to snapshots in one place

Use rsync with '-L' to copy elsewhere.

#!/usr/bin/ruby

require 'fileutils'

mybasedir = '/var/lib/cass'
mydatadir = mybasedir + '/data'
mysnapshotlinksdir = mybasedir + '/snapshots'
FileUtils.remove_dir(mysnapshotlinksdir) if File.exists?(mysnapshotlinksdir)
Dir.chdir mydatadir
mysnapshotdirs = Dir.glob("**/*/")
mysnapshotdirs = mysnapshotdirs.grep(/snapshots\/.+/)
#p mysnapshotdirs

mysnapshots = {}
mysnapshotdirs.each do |mydir|
    mysnapshot = {}
    myparts = mydir.split('/')
    mysnapshot['keyspace'] = myparts[0]
    mysnapshot['cf'] = myparts[1]
    mysnapshot['tag'] = myparts[3]
    mysnapshot['linkdir'] = [mysnapshotlinksdir, mysnapshot['tag'], mysnapshot['keyspace']].join('/')
    mysnapshot['link'] = [mysnapshotlinksdir, mysnapshot['tag'], mysnapshot['keyspace'], mysnapshot['cf']].join('/')
    mysnapshot['target'] = [mydatadir, mydir].join('/')
    #p mysnapshot
    FileUtils.mkpath mysnapshot['linkdir'] unless File.exists?(mysnapshot['linkdir'])
    File.symlink(mysnapshot['target'], mysnapshot['link']) unless File.symlink?(mysnapshot['link'])
end

cassandra sstable version in filename

Look in the filename of the sstable, you'll see some letters between hyphens/dashes, ie "-"; these are the version of your sstable, see below for possible values.

See: https://github.com/QwertyManiac/cassandra-cdh4/blob/master/src/java/org/apache/cassandra/io/sstable/Descriptor.java

Btw, cassandra "support" loves to send you to look at code, so download the code and get familiar with the basic structure (somehow).

        public static final Version LEGACY = new Version("a"); // "pre-history"
        // b (0.7.0): added version to sstable filenames
        // c (0.7.0): bloom filter component computes hashes over raw key bytes instead of strings
        // d (0.7.0): row size in data component becomes a long instead of int
        // e (0.7.0): stores undecorated keys in data and index components
        // f (0.7.0): switched bloom filter implementations in data component
        // g (0.8): tracks flushed-at context in metadata component
        // h (1.0): tracks max client timestamp in metadata component
        // hb (1.0.3): records compression ration in metadata component
        // hc (1.0.4): records partitioner in metadata component
        // hd (1.0.10): includes row tombstones in maxtimestamp
        // he (1.1.3): includes ancestors generation in metadata component
        // hf (1.1.6): marker that replay position corresponds to 1.1.5+ millis-based id (see CASSANDRA-4782)
        // ia (1.2.0): column indexes are promoted to the index file
        // records estimated histogram of deletion times in tombstones
        // bloom filter (keys and columns) upgraded to Murmur3
        // ib (1.2.1): tracks min client timestamp in metadata component

apache-cassandra11 conflicts with apache-cassandra11-1.1.11-1.noarch

The Error:

Running rpm_check_debug

ERROR with rpm_check_debug vs depsolve:

apache-cassandra11 conflicts with apache-cassandra11-1.1.11-1.noarch

Solution:

1. wget http://rpm.riptano.com/community/noarch/apache-cassandra11-1.1.11-1.noarch.rpm
2. rpm -i apache-cassandra11-1.1.11-1.noarch.rpm --nodeps

This might be necessary before step 2, but make sure you back up your data first, even this should not delete it, you never know:

1. yum remove apache-cassandra11

Double looping with bash

Neat:

1. for ITEM in $(find /cassandra/data -type d -name snapshots)
1. do for DIR in $(find ${ITEM} -maxdepth 1 -mindepth 1 -type d -mtime -1)
1. do echo $ITEM $DIR
2. done
2. done

Cassandra in 30 seconds

1. writes 
1. writes entries directly to disk without checking if they already exist
2. does fancy indexing of entries
3. returns a write "OK" to the writing client after a quorum of nodes have confirmed
2. reads
1. tries to return the newest entry when client does a read
2. has methods to eventually get the newest entry to return even if old ones still around
3. replication
1. stores entries to multiple nodes if replication is turned on
4. deletes
1. doesn't offically delete, just marks dead entries with a "tombstone"
2. compaction is what gets rid of old versions of entries and dead entries
5. balancing
1. automatically fills in data holes if a node disappears
2. automatically spreads data if new nodes are added
6. resurrection
1. 3-nodes: X, Y, Z, all replicate all data
2. server X goes down
3. delete goes to Y and Z for key A
4. Y and Z are "compacted"
1. i.e., redundant keys & tombstones cleaned up / removed
2. key A is completely gone as far as  Y and Z know
5. X comes up and has value for key A
6. A is back! resurrected from the dead! life sucks.
7. NOTE: if Y and Z didn't have tombstones removed, they would have had a date that was more recent than X's key A entry, so they would have invalidated X's key A. But, they are gone after a compaction or cleanup.

JMX ports to open in firewall for jconsole to Cassandra

1. Port 7199
1. Used for about a dozen packets when JMX connection first made
1. A handshake of sorts
2. Probably sets up the agreement on which high port to connect to, used below
1. Similar to SIP
2. Similar to old FTP
2. Not used again after initial handshake
2. Port range 55000 to 55999
1. To see these packets, on JVM server
1. tcpdump -nn ! port 22 and host <jconsole client IP> (not literal, replace this)
3. If jconsole starts showing graphs, you are connected

To run jconsole directly on the server via VNC, see this article: http://sysadminandnetworking.blogspot.com/

Tricks and Tips

1. If you don't want to expose 1000 ports to the world for some reason
1. Open all ports on firewall in front of JVM server
2. On JVM server: tcpdump -nn ! port 22 and host <jconsole client IP>
3. Start jconsole connection on client machine
4. Watch to see which port JVM server is trying to reach jconsole client via
5. Close all but that port in the firewall, will be between 55000-55999
2. Do a local experiment to a local JVM JMX-able application if unsure of good jconsole connection result
3. Get your external IP from where you are running jconsole client
1. CLI: curl http://ipaddr.me
2. Or web browser: http://ipaddr.me