Quick start how-to graphite base install on Debian

UPDATE: Also see Latest Graphite on Amazon Linux at AWS.

This only gets graphite working on your local Linux box. Left to user to translate to remote server installation thereafter.

NOTE: Do everything as root user

1. make sure apache2 is installed and working with wsgi
1. apt-get install apache2 -y
2. apt-get install libapache2-mod-wsgi -y
3. a2enmod wsgi
4. wsgi needs a place for its sockets
1. mkdir /etc/apache2/run
2. mkdir /var/run/wsgi
3. chmod 777 /etc/apache2/run /var/run/wsgi
4. this seems undocumented, thanks!
2. Install dependencies
1. apt-get install -y libapache2-mod-wsgi python-twisted python-memcache python-pysqlite2 python-simplejson
2. apt-get install -y python2.6 python-pip python-cairo python-django python-django-tagging
3. Install graphite elements
1. mkdir -p /root/graphite-install
2. cd /root/graphite-install
3. git clone https://github.com/graphite-project/graphite-web.git
4. git clone https://github.com/graphite-project/carbon.git
5. git clone https://github.com/graphite-project/whisper.git
6. git clone https://github.com/graphite-project/ceres
7. cd /root/graphite-install/whisper
8. git checkout 0.9.x
9. python setup.py install
10. cd /root/graphite-install/ceres
11. python setup.py install
    
12. cd /root/graphite-install/carbon
13. git checkout 0.9.x
14. python setup.py install
    
15. cd /root/graphite-install/graphite-web
16. git checkout 0.9.x
17. python check-dependencies.py
1. fix any missing dependencies
2. ignore warnings if certain you don't need a feature
18. python setup.py install
4. Setup configuration files and permissions
1. cp -v /opt/graphite/conf/carbon.conf.example /opt/graphite/conf/carbon.conf
2. cp -v /opt/graphite/conf/storage-schemas.conf.example /opt/graphite/conf/storage-schemas.conf
3. Create Django database
1. cp -v /opt/graphite/webapp/graphite/local_settings.py.example /opt/graphite/webapp/graphite/local_settings.py
1. Edit this file by uncommenting the entire DATABASES section
1. Otherwise next commands may fail with bizarre error
2. cd /opt/graphite/webapp/graphite
3. python manage.py syncdb
1. Make sure NO errors
2. Create a user/pass you'll never forget
4. chown -Rv www-data:www-data /opt/graphite/storage/
1. EXTREMELY IMPORTANT
5. Apache
1. Add below Apache configuration as a virtual host (details not covered here)
1. Or do this and struggle to make it work
1. cp -v /opt/graphite/examples/example-graphite-vhost.conf /etc/apache2/sites-available/graphite
2. Every "directory" and "location" in the virtual hosts file need these entries, or you'll get permission denied errors
1. apache versions before 2.4
1. Order deny,allow
2. Allow from all
2. apache versions 2.4 and after
1. Require all granted
3. a2ensite graphite
4. cp -v /opt/graphite/conf/graphite.wsgi.example /opt/graphite/conf/graphite.wsgi
5. apache2ctl -S
1. Check all is OK
6. service apache2 stop
7. service apache2 start
8. If you have trouble check permissions under this directory and/or run this again
1. chown -Rv www-data:www-data /opt/graphite/storage/
2. Seems some files are created by root that should be created by www-data user maybe
6. Start carbon daemon
1. cd /opt/graphite
2. /opt/graphite/bin/carbon-cache.py start
1. verify with:
1. lsof -nP -i :2003
2. TASK: automate this to start on reboot somehow
7. Add to /etc/hosts
1. 127.0.1.3 graphite
8. tail -F /opt/graphite/storage/log/webapp/error.log
9. python /opt/graphite/examples/example-client.py
10. Hit http://graphite
1. Should show initial graphite interface
11. View results
1. Click down into "system" -> loadavg_5min
2. Find "Select Recent Data" icon in upper-left toolbar
1. Set to 10 mins
3. You should see lines appearing as script runs and feed data to Graphite via Carbon
12. Check out giraffe once you get some stats to shove in it
1. https://github.com/kenhub/giraffe
2. Just dump the files you get from the git clone into a directory under you default Apache install and tweedle with dashboard.js until you see something
13. Try this a few times on the CLI of your graphite server every few minutes
1. echo "system.logs.changed_last10 `find /var/log -mmin -10 | wc -l` `date +%s`" | nc -w 1 localhost 2003
2. Try similar commands to pass in other stats
1. echo must output a single integer.

WSGISocketPrefix /var/run/wsgi

<VirtualHost *:80>
    ServerName graphite
    DocumentRoot "/opt/graphite/webapp"
    ErrorLog /opt/graphite/storage/log/webapp/error.log
    CustomLog /opt/graphite/storage/log/webapp/access.log common
    
    WSGIDaemonProcess graphite processes=5 threads=5 display-name='%{GROUP}' inactivity-timeout=120
    WSGIProcessGroup graphite
    WSGIApplicationGroup %{GLOBAL}
    WSGIImportScript /opt/graphite/conf/graphite.wsgi process-group=graphite application-group=%{GLOBAL}
    WSGIScriptAlias / /opt/graphite/conf/graphite.wsgi 

    Alias /content/ /opt/graphite/webapp/content/
    <Location "/content/">
        SetHandler None
    </Location>
    
    Alias /media/ "@DJANGO_ROOT@/contrib/admin/media/"
    <Location "/media/">
        SetHandler None
    </Location>
    
    <Directory /opt/graphite/conf/>
        Order deny,allow
        Allow from all
    </Directory>
</VirtualHost>

Debugging Munin loaning graphs locally

1. Problems
1. Munin is a pain to debug remotely -- on prod -- when doing custom "loaning" graphs
1. Varnish gets in the way
2. (might not work, test yourself) fast-cgi doesn't work with new Munin 2.0 dynamic graph generation very well, just comment it out in the Apache config
2. These issues combined leads to a complete nightmare of caching and having to wait for graphs to be regenerated so you can see your changes
2. Troubleshooting
1. Try this link for perm checks
1. http://munin-monitoring.org/wiki/CgiHowto
2. IMPORTANT: ust turn this on in monit.conf manually, since debian turns it off, and who knows who else
1. graph_strategy cgi
3. Solution
1. Grab /var/lib/munin from production server
2. Install Munin 2.x or greater on your local box
3. Comment out the Munin files under /etc/cron* whatever/whereever, so your server doesn't try to update any of the files under /var/lib/munin
4. Move your local copy of /var/lib/munin aside
5. Move the production version of /var/lib/munin into place on your local system
6. Copy perms of your original /var/lib/munin to new one
7. Grab the prod server version of /etc/munin/munin.conf
8. Move your local copy of /etc/munin/munin.conf aside, rename something you'll remember
9. Move prod server version /etc/munin/munin.conf in place on your local box
10. Use "munin-html" to regenerate html pages as you make changes to your munin definitions in /etc/munin/munin.conf
1. basically these commands, but, for details, see http://blog.loftninjas.org/2010/04/08/an-evening-with-munin-graph-aggregation/
1. sudo su - munin -s /bin/bash
2. /usr/share/munin/munin-html --debug
2. might work / might not
11. Hit munin locally through your web browser; if you're lucky, all the prod info/graphs appear normally
12. Now, you can update /etc/munin/munin.conf as you like and graph changes and errors will show up instantly
13. Tweak URL to hit graphs you know the name of but munin-html failed to find for you
    
4. Long-term
1. Refresh the data from prod every 48 to 72 hours so your graph data doesn't fall off the chart
1. Since your local box is not updating data, all rrd data will be blank from the time your grab it from prod's /var/lib/munin