Latest graphite on Amazon Linux at AWS

NOTE: someone broke master branch, so reverting to 0.9.x

1. yum update -y;reboot
2. yum -y groupinstall "Development Tools"
3. yum -y install git-core python-pip mlocate
4. yum -y install python-rrdtool pycairo-devel
5. mkdir -p /root/graphite-install
6. cd /root/graphite-install
7. git clone https://github.com/graphite-project/carbon.git
8. git clone https://github.com/graphite-project/whisper.git
9. git clone https://github.com/graphite-project/ceres
10. git clone https://github.com/graphite-project/graphite-web.git
11. cd /root/graphite-install/whisper
12. git checkout 0.9.x
13. python setup.py install
14. cd /root/graphite-install/ceres
15. # only has master branch, but it works with others
16. pip install -r requirements.txt
17. python setup.py install
18. cd /root/graphite-install/carbon
19. git checkout 0.9.x
20. pip install -r requirements.txt
21. python setup.py install
22. cd /root/graphite-install/graphite-web
23. git checkout 0.9.x
24. yum install libffi-devel -y # some developer broke things, add this
25. pip install -r requirements.txt
26. python check-dependencies.py
27. python setup.py install 
28. cp -v /opt/graphite/conf/carbon.conf.example /opt/graphite/conf/carbon.conf
29. cp -v /opt/graphite/conf/storage-schemas.conf.example /opt/graphite/conf/storage-schemas.conf
30. cp -v /opt/graphite/conf/storage-aggregation.conf.example /opt/graphite/conf/storage-aggregation.conf
31. cp -v /opt/graphite/webapp/graphite/local_settings.py.example /opt/graphite/webapp/graphite/local_settings.py
32. cp -v /opt/graphite/conf/graphite.wsgi.example /opt/graphite/conf/graphite.wsgi
33. vi /opt/graphite/webapp/graphite/local_settings.py
1. set SECRET_KEY param
2. uncomment DATABASE section
34. add these lines to /root/.bash_profile
1. export GRAPHITE_ROOT=/opt/graphite
2. export PYTHONPATH=$GRAPHITE_ROOT/webapp:$GRAPHITE_ROOT/whisper
35. source /root/.bash_profile
36. django-admin.py syncdb --settings=graphite.settings
37. yum -y install httpd24 mod24_wsgi
38. mkdir /etc/httpd/vhosts.d
39. add to end of /etc/httpd/conf/httpd.conf
1. IncludeOptional vhosts.d/*.conf
40. create /etc/httpd/vhosts.d/graphite.conf with below
41. mkdir /var/run/wsgi;chmod -v 777 /var/run/wsgi
42. apachectl -S
1. check for errors
43. service httpd start
44. /opt/graphite/bin/carbon-cache.py start
45. find /opt/graphite/storage -type d -exec chmod -v 777 {} \;
46. find /opt/graphite/storage -type f -exec chmod -v 666 {} \;
47. echo '127.0.1.3 graphite' >> /etc/hosts
48. curl -I http://graphite
49. on local machine
1. make same /etc/hosts entry
2. port forward 127.0.1.3:8081 to remote-server:80
3. curl -I http://graphite:8081
50. END

WSGISocketPrefix /var/run/wsgi

<VirtualHost *:80>
    ServerName graphite
    DocumentRoot "/opt/graphite/webapp"
    ErrorLog logs/webapp_error.log
    CustomLog logs/webapp_access.log common

    WSGIDaemonProcess graphite processes=5 threads=5 display-name='%{GROUP}' inactivity-timeout=120
    WSGIProcessGroup graphite
    WSGIApplicationGroup %{GLOBAL}
    WSGIImportScript /opt/graphite/conf/graphite.wsgi process-group=graphite application-group=%{GLOBAL}
    WSGIScriptAlias / /opt/graphite/conf/graphite.wsgi

    Alias /content/ /opt/graphite/webapp/content/
    <Location "/content/">
        SetHandler None
        Require all granted
    </Location>
  
    Alias /media/ "@DJANGO_ROOT@/contrib/admin/media/"
    <Location "/media/">
        SetHandler None
        Require all granted
    </Location>
 
    <Directory /opt/graphite/conf/>
        Require all granted
    </Directory>
</VirtualHost>

Debugging Munin loaning graphs locally

1. Problems
1. Munin is a pain to debug remotely -- on prod -- when doing custom "loaning" graphs
1. Varnish gets in the way
2. (might not work, test yourself) fast-cgi doesn't work with new Munin 2.0 dynamic graph generation very well, just comment it out in the Apache config
2. These issues combined leads to a complete nightmare of caching and having to wait for graphs to be regenerated so you can see your changes
2. Troubleshooting
1. Try this link for perm checks
1. http://munin-monitoring.org/wiki/CgiHowto
2. IMPORTANT: ust turn this on in monit.conf manually, since debian turns it off, and who knows who else
1. graph_strategy cgi
3. Solution
1. Grab /var/lib/munin from production server
2. Install Munin 2.x or greater on your local box
3. Comment out the Munin files under /etc/cron* whatever/whereever, so your server doesn't try to update any of the files under /var/lib/munin
4. Move your local copy of /var/lib/munin aside
5. Move the production version of /var/lib/munin into place on your local system
6. Copy perms of your original /var/lib/munin to new one
7. Grab the prod server version of /etc/munin/munin.conf
8. Move your local copy of /etc/munin/munin.conf aside, rename something you'll remember
9. Move prod server version /etc/munin/munin.conf in place on your local box
10. Use "munin-html" to regenerate html pages as you make changes to your munin definitions in /etc/munin/munin.conf
1. basically these commands, but, for details, see http://blog.loftninjas.org/2010/04/08/an-evening-with-munin-graph-aggregation/
1. sudo su - munin -s /bin/bash
2. /usr/share/munin/munin-html --debug
2. might work / might not
11. Hit munin locally through your web browser; if you're lucky, all the prod info/graphs appear normally
12. Now, you can update /etc/munin/munin.conf as you like and graph changes and errors will show up instantly
13. Tweak URL to hit graphs you know the name of but munin-html failed to find for you
    
4. Long-term
1. Refresh the data from prod every 48 to 72 hours so your graph data doesn't fall off the chart
1. Since your local box is not updating data, all rrd data will be blank from the time your grab it from prod's /var/lib/munin