Monday 28 July 2014

Change graphite's default dashboard graph colors

  1. cp -v /opt/graphite/conf/graphTemplates.conf.example /opt/graphite/conf/graphTemplates.conf
  2. vi /opt/graphite/conf/graphTemplates.conf
  3. change '[default]' section to '[uggs]'
  4. change another section, e.g. '[solarized-dark]', to '[default]'
  5. reload dashboard and you should see changed colors

Wednesday 23 July 2014

Ganglia on Amazon Linux (and other RedHat derivatives)

The below is for setting up unicast, not multicast. AWS does not support multicast networking.

Key concepts: gmond daemons run on every server and use C code to collect the server's stats; this data is stored in local memory. Multiple gmonds can send their data on to one central gmond to hold, call this a gmond "bank"; this "bank" also uses only memory to store the server stats. gmetad comes along and collects the data from the gmond "banks" and stores in it rrds, these are files; the web interface uses these rrd files, and it usually runs on the same server as gmetad.

Cluster name: cluster name is key in grouping data and getting it from gmond to gmond and then on to gmetad. data_source is the way gmetad find the "banks"; and, by the way, you can have redundant "banks" for one cluster data_source.

Getting rid of multicast settings: comment out all references to multicast: bind_hostname, mcast_join, bind. Comment them all out.

UDP vs TCP, and port 8649: port 8649 is the default. UDP traffic on port 8649 is used for gmond intercommunication. And TCP traffic on port 8649 is used by the gmetad daemon to pull data from all your gmond "banks". Run "tcpdump -i any -nn port 8649" ALL THE TIME ON EVERY SERVER in a separate terminal when debugging.

Source: http://www.admin-magazine.com/HPC/Articles/Monitoring-HPC-Systems
  1. mkdir ganglia_rpms
  2. cd ganglia_rpms/
  3. wget http://vuksan.com/centos/RPMS-6/x86_64/ganglia-gmond-3.6.0-1.x86_64.rpm
  4. wget http://vuksan.com/centos/RPMS-6/x86_64/libganglia-3.6.0-1.x86_64.rpm
  5. wget http://vuksan.com/centos/RPMS-6/x86_64/ganglia-debuginfo-3.6.0-1.x86_64.rpm
  6. wget http://vuksan.com/centos/RPMS-6/x86_64/ganglia-devel-3.6.0-1.x86_64.rpm
  7. wget http://vuksan.com/centos/RPMS-6/x86_64/ganglia-gmetad-3.6.0-1.x86_64.rpm
  8. wget http://vuksan.com/centos/RPMS-6/x86_64/ganglia-gmond-modules-python-3.6.0-1.x86_64.rpm
  9. wget http://vuksan.com/centos/RPMS-6/x86_64/libconfuse-2.6-2.el6.rf.x86_64.rpm
  10. wget http://vuksan.com/centos/RPMS-6/x86_64/libconfuse-devel-2.6-2.el6.rf.x86_64.rpm
  11. yum localinstall ganglia-*.rpm lib*.rpm
  12. yum install httpd
  13. yum install php
  14. cd
  15. mkdir ganglia-web-dev
  16. cd ganglia-web-dev/
  17. wget http://downloads.sourceforge.net/project/ganglia/ganglia-web/3.5.12/ganglia-web-3.5.12.tar.gz
  18. tar zxvf ganglia-web-3.5.12.tar.gz
  19. cd ganglia-web-3.5.12
  20. vi Makefile 
    1. GDESTDIR = /var/www/html/ganglia
    2. APACHE_USER = apache
  21. make install
  22. cp apache.conf /etc/httpd/conf.d/ganglia.conf
  23. vi /etc/httpd/conf.d/ganglia.conf 
    1. /usr/share/ganglia-webfrontend -> /var/www/html/ganglia
  24. service httpd start
  25. setenforce 0 
    1. not needed on Amazon Linux
  26. vi /etc/ganglia/gmond.conf
  27. vi /etc/ganglia/gmetad.conf
  28. service gmond start
  29. service gmetad start

Tuesday 8 July 2014

Latest graphite on Amazon Linux at AWS

NOTE: someone broke master branch, so reverting to 0.9.x
  1. yum update -y;reboot
  2. yum -y groupinstall "Development Tools"
  3. yum -y install git-core python-pip mlocate
  4. yum -y install python-rrdtool pycairo-devel
  5. mkdir -p /root/graphite-install
  6. cd /root/graphite-install
  7. git clone https://github.com/graphite-project/carbon.git
  8. git clone https://github.com/graphite-project/whisper.git
  9. git clone https://github.com/graphite-project/ceres
  10. git clone https://github.com/graphite-project/graphite-web.git
  11. cd /root/graphite-install/whisper
  12. git checkout 0.9.x
  13. python setup.py install
  14. cd /root/graphite-install/ceres
  15. # only has master branch, but it works with others
  16. pip install -r requirements.txt
  17. python setup.py install
  18. cd /root/graphite-install/carbon
  19. git checkout 0.9.x
  20. pip install -r requirements.txt
  21. python setup.py install
  22. cd /root/graphite-install/graphite-web
  23. git checkout 0.9.x
  24. yum install libffi-devel -y # some developer broke things, add this
  25. pip install -r requirements.txt
  26. python check-dependencies.py
  27. python setup.py install 
  28. cp -v /opt/graphite/conf/carbon.conf.example /opt/graphite/conf/carbon.conf
  29. cp -v /opt/graphite/conf/storage-schemas.conf.example /opt/graphite/conf/storage-schemas.conf
  30. cp -v /opt/graphite/conf/storage-aggregation.conf.example /opt/graphite/conf/storage-aggregation.conf
  31. cp -v /opt/graphite/webapp/graphite/local_settings.py.example /opt/graphite/webapp/graphite/local_settings.py
  32. cp -v /opt/graphite/conf/graphite.wsgi.example /opt/graphite/conf/graphite.wsgi
  33. vi /opt/graphite/webapp/graphite/local_settings.py
    1. set SECRET_KEY param
    2. uncomment DATABASE section
  34. add these lines to /root/.bash_profile
    1. export GRAPHITE_ROOT=/opt/graphite
    2. export PYTHONPATH=$GRAPHITE_ROOT/webapp:$GRAPHITE_ROOT/whisper
  35. source /root/.bash_profile
  36. django-admin.py syncdb --settings=graphite.settings
  37. yum -y install httpd24 mod24_wsgi
  38. mkdir /etc/httpd/vhosts.d
  39. add to end of /etc/httpd/conf/httpd.conf
    1. IncludeOptional vhosts.d/*.conf
  40. create /etc/httpd/vhosts.d/graphite.conf with below
  41. mkdir /var/run/wsgi;chmod -v 777 /var/run/wsgi
  42. apachectl -S
    1. check for errors
  43. service httpd start
  44. /opt/graphite/bin/carbon-cache.py start
  45. find /opt/graphite/storage -type d -exec chmod -v 777 {} \;
  46. find /opt/graphite/storage -type f -exec chmod -v 666 {} \;
  47. echo '127.0.1.3 graphite' >> /etc/hosts
  48. curl -I http://graphite
  49. on local machine
    1. make same /etc/hosts entry
    2. port forward 127.0.1.3:8081 to remote-server:80
    3. curl -I http://graphite:8081
  50. END

WSGISocketPrefix /var/run/wsgi

<VirtualHost *:80>
    ServerName graphite
    DocumentRoot "/opt/graphite/webapp"
    ErrorLog logs/webapp_error.log
    CustomLog logs/webapp_access.log common

    WSGIDaemonProcess graphite processes=5 threads=5 display-name='%{GROUP}' inactivity-timeout=120
    WSGIProcessGroup graphite
    WSGIApplicationGroup %{GLOBAL}
    WSGIImportScript /opt/graphite/conf/graphite.wsgi process-group=graphite application-group=%{GLOBAL}
    WSGIScriptAlias / /opt/graphite/conf/graphite.wsgi

    Alias /content/ /opt/graphite/webapp/content/
    <Location "/content/">
        SetHandler None
        Require all granted
    </Location>
  
    Alias /media/ "@DJANGO_ROOT@/contrib/admin/media/"
    <Location "/media/">
        SetHandler None
        Require all granted
    </Location>
 
    <Directory /opt/graphite/conf/>
        Require all granted
    </Directory>
</VirtualHost>


Interview questions: 2020-12

Terraform provider vs provisioner Load balancing Network Load Balancer vs Application Load Balancer  Networking Layer 1 vs Layer 4 haproxy u...