JMX ports to open in firewall for jconsole to Cassandra

1. Port 7199
1. Used for about a dozen packets when JMX connection first made
1. A handshake of sorts
2. Probably sets up the agreement on which high port to connect to, used below
1. Similar to SIP
2. Similar to old FTP
2. Not used again after initial handshake
2. Port range 55000 to 55999
1. To see these packets, on JVM server
1. tcpdump -nn ! port 22 and host <jconsole client IP> (not literal, replace this)
3. If jconsole starts showing graphs, you are connected

To run jconsole directly on the server via VNC, see this article: http://sysadminandnetworking.blogspot.com/

Tricks and Tips

1. If you don't want to expose 1000 ports to the world for some reason
1. Open all ports on firewall in front of JVM server
2. On JVM server: tcpdump -nn ! port 22 and host <jconsole client IP>
3. Start jconsole connection on client machine
4. Watch to see which port JVM server is trying to reach jconsole client via
5. Close all but that port in the firewall, will be between 55000-55999
2. Do a local experiment to a local JVM JMX-able application if unsure of good jconsole connection result
3. Get your external IP from where you are running jconsole client
1. CLI: curl http://ipaddr.me
2. Or web browser: http://ipaddr.me

Quick start how-to graphite base install on Debian

UPDATE: Also see Latest Graphite on Amazon Linux at AWS.

This only gets graphite working on your local Linux box. Left to user to translate to remote server installation thereafter.

NOTE: Do everything as root user

1. make sure apache2 is installed and working with wsgi
1. apt-get install apache2 -y
2. apt-get install libapache2-mod-wsgi -y
3. a2enmod wsgi
4. wsgi needs a place for its sockets
1. mkdir /etc/apache2/run
2. mkdir /var/run/wsgi
3. chmod 777 /etc/apache2/run /var/run/wsgi
4. this seems undocumented, thanks!
2. Install dependencies
1. apt-get install -y libapache2-mod-wsgi python-twisted python-memcache python-pysqlite2 python-simplejson
2. apt-get install -y python2.6 python-pip python-cairo python-django python-django-tagging
3. Install graphite elements
1. mkdir -p /root/graphite-install
2. cd /root/graphite-install
3. git clone https://github.com/graphite-project/graphite-web.git
4. git clone https://github.com/graphite-project/carbon.git
5. git clone https://github.com/graphite-project/whisper.git
6. git clone https://github.com/graphite-project/ceres
7. cd /root/graphite-install/whisper
8. git checkout 0.9.x
9. python setup.py install
10. cd /root/graphite-install/ceres
11. python setup.py install
    
12. cd /root/graphite-install/carbon
13. git checkout 0.9.x
14. python setup.py install
    
15. cd /root/graphite-install/graphite-web
16. git checkout 0.9.x
17. python check-dependencies.py
1. fix any missing dependencies
2. ignore warnings if certain you don't need a feature
18. python setup.py install
4. Setup configuration files and permissions
1. cp -v /opt/graphite/conf/carbon.conf.example /opt/graphite/conf/carbon.conf
2. cp -v /opt/graphite/conf/storage-schemas.conf.example /opt/graphite/conf/storage-schemas.conf
3. Create Django database
1. cp -v /opt/graphite/webapp/graphite/local_settings.py.example /opt/graphite/webapp/graphite/local_settings.py
1. Edit this file by uncommenting the entire DATABASES section
1. Otherwise next commands may fail with bizarre error
2. cd /opt/graphite/webapp/graphite
3. python manage.py syncdb
1. Make sure NO errors
2. Create a user/pass you'll never forget
4. chown -Rv www-data:www-data /opt/graphite/storage/
1. EXTREMELY IMPORTANT
5. Apache
1. Add below Apache configuration as a virtual host (details not covered here)
1. Or do this and struggle to make it work
1. cp -v /opt/graphite/examples/example-graphite-vhost.conf /etc/apache2/sites-available/graphite
2. Every "directory" and "location" in the virtual hosts file need these entries, or you'll get permission denied errors
1. apache versions before 2.4
1. Order deny,allow
2. Allow from all
2. apache versions 2.4 and after
1. Require all granted
3. a2ensite graphite
4. cp -v /opt/graphite/conf/graphite.wsgi.example /opt/graphite/conf/graphite.wsgi
5. apache2ctl -S
1. Check all is OK
6. service apache2 stop
7. service apache2 start
8. If you have trouble check permissions under this directory and/or run this again
1. chown -Rv www-data:www-data /opt/graphite/storage/
2. Seems some files are created by root that should be created by www-data user maybe
6. Start carbon daemon
1. cd /opt/graphite
2. /opt/graphite/bin/carbon-cache.py start
1. verify with:
1. lsof -nP -i :2003
2. TASK: automate this to start on reboot somehow
7. Add to /etc/hosts
1. 127.0.1.3 graphite
8. tail -F /opt/graphite/storage/log/webapp/error.log
9. python /opt/graphite/examples/example-client.py
10. Hit http://graphite
1. Should show initial graphite interface
11. View results
1. Click down into "system" -> loadavg_5min
2. Find "Select Recent Data" icon in upper-left toolbar
1. Set to 10 mins
3. You should see lines appearing as script runs and feed data to Graphite via Carbon
12. Check out giraffe once you get some stats to shove in it
1. https://github.com/kenhub/giraffe
2. Just dump the files you get from the git clone into a directory under you default Apache install and tweedle with dashboard.js until you see something
13. Try this a few times on the CLI of your graphite server every few minutes
1. echo "system.logs.changed_last10 `find /var/log -mmin -10 | wc -l` `date +%s`" | nc -w 1 localhost 2003
2. Try similar commands to pass in other stats
1. echo must output a single integer.

WSGISocketPrefix /var/run/wsgi

<VirtualHost *:80>
    ServerName graphite
    DocumentRoot "/opt/graphite/webapp"
    ErrorLog /opt/graphite/storage/log/webapp/error.log
    CustomLog /opt/graphite/storage/log/webapp/access.log common
    
    WSGIDaemonProcess graphite processes=5 threads=5 display-name='%{GROUP}' inactivity-timeout=120
    WSGIProcessGroup graphite
    WSGIApplicationGroup %{GLOBAL}
    WSGIImportScript /opt/graphite/conf/graphite.wsgi process-group=graphite application-group=%{GLOBAL}
    WSGIScriptAlias / /opt/graphite/conf/graphite.wsgi 

    Alias /content/ /opt/graphite/webapp/content/
    <Location "/content/">
        SetHandler None
    </Location>
    
    Alias /media/ "@DJANGO_ROOT@/contrib/admin/media/"
    <Location "/media/">
        SetHandler None
    </Location>
    
    <Directory /opt/graphite/conf/>
        Order deny,allow
        Allow from all
    </Directory>
</VirtualHost>

No http on port 2812 for monit

FAIL!

set httpd port 2812 and use the address localhost
  allow localhost

1. Problem
1. monit doesn't listen on port 2812
1. even though it is clearly in the config file 
2. Possible issues
1. Is there a "delay" set in your config?
1. monit won't listen until this delay has passed
3. Solution
1. Wait, usually a minute by default, and check the port again
2. Get rid of the delay in the config and restart monit