Saturday 15 December 2012

OpenVPN on EC2/AWS


EC2 Instance
  1. Allow UDP to port 1194 under the instance's security group
    1. to the world 
    2. or just your IP network
Install and configure OpenVPN instance/server
  1. apt-get install openvpn
  2. cd /etc/openvpn
  3. openvpn --genkey --secret my.key
  4. Put code below into /etc/openvpn/server.conf
  5. Run: openvpn --config /etc/openvpn/server.conf
    1. Leave running while taking remaining steps
  6. Allow NAT
    1. modprobe iptable_nat
    2. echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward
    3. iptables -t nat -A POSTROUTING -s 192.168.2.0/24 -o eth0 -j MASQUERADE
  7. Checks for the paranoid
    1. lsof -nP -i
    2. lsof -c openvpn

port 1194
proto udp
dev tun
secret /etc/openvpn/my.key
ifconfig 192.168.2.1 192.168.2.2
keepalive 10 120
comp-lzo
persist-key
persist-tun
status server-tcp.log
verb 3

  1. If on Mac, use tunnelblick to open a file like myopenvpn.ovpn with below code
    1. Replace hostname with yours
    2. You'll need the my.key file in the same directory as the myopenvpn.ovpn file
  2. On Linux
    1. Put below in /etc/openvpn/client.conf
    2. Put my.key in /etc/openvpn as well
    3. Run: openvpn --config /etc/openvpn/client.conf
    4. Leave running
  3. For all operating systems
    1. When/if you see this in the client connection logs/messages, you've made it
      1. "Initialization Sequence Completed"
  4. Confirm
    1. You can ping 192.168.2.1, the openvpn server/instance via the ssh tunnel
    2. You can ssh into the openvpn server/instance via 192.168.2.1
    3. You have a new network interface with 192.168.2.2 defined on it
    4. Goto: http://ipaddr.me
      1. Does it look like your IP?
      2. If uncertain, use "whois" command to find out
dev tun
proto udp
remote ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com 1194
resolv-retry infinite
nobind
secret my.key
ifconfig 192.168.2.2 192.168.2.1
comp-lzo
verb 3
dhcp-option DNS 172.16.0.23
redirect-gateway def1

Friday 2 November 2012

Create a git server

Git

Create a git server

  1. apt-get install git
  2. adduser git
  3. Become the git user and add team's ssh keys to /home/git/.ssh/authorized_keys
  4. Change the git user's shell in /etc/passwd to /usr/bin/git-shell or to whichever path it is otherwise located
  5. mkdir /opt/git
  6. chown -Rv git.git /opt/git
  7. chmod -Rv g+ws /opt/git
  8. cd /opt/git
  9. mkdir myproject.git
  10. cd myproject.git
  11. git init --bare --shared
  12. See addition to /opt/git/myproject.git/config below
  13. mv /opt/git/myproject.git/hooks/post-update.sample /opt/git/myproject.git/hooks/post-update
  14. On your local computer do:
    1. mkdir ~/tmp
    2. cd ~/tmp
    3. git config --global user.name "Your Full Name"
    4. git config --global user.email your.name@abc.net
    5. git init
    6. git add .
    7. git commit -m 'initial commit'
    8. git remote add origin git@git.yourserver.com:/opt/git/myproject.git
    9. git push origin master
      1. NOTE: if you get an error that origin already exists, do "git remote rm origin" before push
  15. Debug the set up, this guide probably has a few mistakes
Add this to /opt/git/myproject.git/config
[http]
    receivepack = true

Add apache functionality

  1. apt-get install apache2
  2. addgroup www-data git
  3. a2enmod auth_basic
  4. a2enmod authnz_ldap
  5. Add virtual host definition below
  6. service apache2 restart
The virtual host file is something like this, but you need to fix the "XXXX" part, best luck:
<VirtualHost *:80>
 ServerName git.abc.net
 DocumentRoot /opt/git
 ErrorLog ${APACHE_LOG_DIR}/git_error.log
 CustomLog ${APACHE_LOG_DIR}/git_access.log combined
 
 SetEnv GIT_PROJECT_ROOT /opt/git
 SetEnv GIT_HTTP_EXPORT_ALL
 ScriptAlias / /usr/lib/git-core/git-http-backend/
 AliasMatch ^/(.*/objects/[0-9a-f]{2}/[0-9a-f]{38})$ /opt/git/$1
 AliasMatch ^/(.*/objects/pack/pack-[0-9a-f]{40}.(pack|idx))$ /opt/git/$1
 ScriptAliasMatch \
  "(?x)^/(.*/(HEAD | \
  info/refs | \
  objects/(info/[^/]+ | \
  [0-9a-f]{2}/[0-9a-f]{38} | \
  pack/pack-[0-9a-f]{40}\.(pack|idx)) | \
  git-(upload|receive)-pack))$" \
  /usr/lib/git-core/git-http-backend/$1
 <LocationMatch ".*">
  AuthType Basic
  AuthName "XXXX"
  AuthLDAPURL ldap://bananas:389/XXXX
  AuthBasicProvider ldap
  AuthLDAPBindDN "XXXX"
  AuthLDAPBindPassword XXXX
  require valid-user
 </LocationMatch>
</VirtualHost>

Thursday 11 October 2012

Quote: ACK packets and Quality of Service (qos)

"When you download, your computer needs to send (upload) ACK packets. These are basically saying "yep, I got that part of the download OK". If the computer you are downloading from detects that an ACK has not been received, it assumes that the data was not received and sends it again. The rate at which ACKs are sent back is also used to help determine the maximum speed that you can download data at, so it's important that ACKs get sent as soon as possible and don't get dropped in order to keep your downloads flowing fast. Also, repeatedly dropped ACKs can result in dropped connections, web page time-outs etc."

Monday 8 October 2012

sed with the capital E option on the Linux CLI

No, you are not crazy, sed on Linux has a secret, BSD compatible option, -E that is equivalent to -r.

Change default text editor for crontab to vim

sudo update-alternatives --config editor

Thursday 4 October 2012

ARP mechanism

  1. Computers Matterhorn and Washington are in an office
    1. connected to each other on the office local area network by 
      1. Ethernet cables and 
      2. network switches, 
    2. with no intervening gateways or routers
  2. Matterhorn wants to send a packet to Washington
    1. Through other means, it determines that Washington's IP address is 192.168.0.55. 
    2. In order to send the message, it also needs to know Washington's MAC address.
      1. First, Matterhorn uses a cached ARP table to look up 192.168.0.55 for any existing records of Washington's MAC address (00:eb:24:b2:05:ac). 
        1. If the MAC address is found,
          1. it sends the IP packet on the link layer to address 00:eb:24:b2:05:ac via the local network cabling. 
        2. If the cache did not produce a result for 192.168.0.55, 
          1. Matterhorn has to send a broadcast ARP message (destination FF:FF:FF:FF:FF:FF) requesting an answer for 192.168.0.55. 
          2. Washington responds with its MAC address (00:eb:24:b2:05:ac). 
            1. Washington may insert an entry for Matterhorn into its own ARP table for future use.
          3. The response information is cached in Matterhorn's ARP table and 
  3. the message can now be sent.

Tuesday 2 October 2012

No http on port 2812 for monit


FAIL!

set httpd port 2812 and use the address localhost
  allow localhost

  1. Problem
    1. monit doesn't listen on port 2812
      1. even though it is clearly in the config file 
  2. Possible issues
    1. Is there a "delay" set in your config?
      1. monit won't listen until this delay has passed
  3. Solution
    1. Wait, usually a minute by default, and check the port again
    2. Get rid of the delay in the config and restart monit

Saturday 29 September 2012

VLAN on Linux CLI


  1. Turn off all "Network Manager" type processes and applications
    1. Comment out /etc/network/interfaces entries even
  2. Do all commands as the root user
  3. Monitoring changes
    1. In one terminal, run this command and watch it as you execute the commands below to see what is changing
      1. sudo watch -d 'ip addr;echo =====;ip link;echo =====;ip route'
  4. Add VLAN
    1. ip link add link eth0 name eth0.20 type vlan id 20
    2. ip link set dev eth0 up
    3. ip link set dev eth0.20 up
    4. ip addr add 192.168.20.190/24 dev eth0.20
    5. ip route add default via 192.168.20.1
    6. For any other VLANs, change the "20" in the commands above to the desired VLAN, e.g.
      1. ip link add link eth0 name eth0.100 type vlan id 100
  5. Delete VLAN
    1. ip link delete eth0.20
  6. Abstract commands
    1. Create
      1. ip addr add IP/NETMASK dev INTERFACE.VID
      2. ip link set dev INTERFACE.VID up
      3. ip addr add 192.168.100.101/24 dev eth0.100
      4. ip link set dev eth0.100 up
    2. Destroy
      1. ip link set dev INTERFACE.VID down
      2. ip link set dev eth0.100 down
      3. ip link delete INTERFACE.VID
      4. ip link delete eth0.100
  7. Notes
    1. Network addr does not have to match the VLAN name
    2. VLAN name is arbitrary, you can call it "joe" if you'd like
    3. http://www.linuxjournal.com/article/7268
      1. "Trunks using the 802.1q protocol work by adding a 4-byte VLAN identifier to each frame"
      2. "When a switch receives a tagged unicast frame, it looks up the outgoing port using both the destination MAC address and the VLAN identifier"
      3. "When a broadcast frame is received, it is flooded out to all active ports participating in that VLAN"

Monday 17 September 2012

Anti-window, anti-desktop Linux desktop

  1. Install base debian with NO additional packages
  2. apt-get install xorg openbox obmenu slim terminator firefox wicd-gtk
  3. reboot
  4. Log in to openbox via slim
  5. Right-click on blank desktop and open default terminal
  6. Run obmenu
  7. Edit ~/.config/openbox/menu.xml to add things you like
  8. Set up network including wireless using early post on this blog
    1. It is not easy
    2. but nice to know once you learn

Debugging pfsense firewall rules clearly and easily

  1. Status -> System logs -> Settings
    1. Make sure Log packets blocked by the default rule is not checked
    2. Check Show log entries in reverse order
    3. Increase to 500 Number of log entries to show
  2.  Status -> System logs -> Settings -> Firewall 
    1. Dynamic View
      1. You don't have to hit refresh
    2. Normal View
      1. Make sure to hit refresh if you expect a rule was triggered by your or others actions
  3. Firewall -> Rules
    1. Under the interface(s) you want to debug
      1. Create a default deny rule at the end of the rule list
        1. Choose Log packets that are handled by this rule
        2. Give the rule a very unique name
      2. For other rules you want to debug
        1. Choose Log packets that are handled by this rule
        2. Give the rule a very unique name
  4. Hang out on Status -> System logs -> Firewall -> Dynamic View
    1. Tweak rules until you see the result you desire
      1. Packets blocked that should be blocked
      2. Packets allowed that should be allowed
    2. Click on the red/green blocked/accepted icons
      1. Will show a pop-up for the rule triggered, showing the unique name you gave to the rule
        1. If necessary, go back and give more unique names to rules to distinguish them from one another

VLANs with iproute on CLI

Set up VLAN "20" on eth0
  1. Base
    1. ip link add link eth0 name eth0.20 type vlan id 20
    2. ip link set dev eth0 up
    3. ip link set dev eth0.20 up
  2. Manually assign
    1. ip addr add 192.168.20.100/24 dev eth0.20
    2. ip route add 192.168.20.0/24 dev eth0.20
    3. ip route add default via 192.168.20.1
Delete
  1. ip route delete 192.168.20.0/24 dev eth0.20
  2. ip link set dev eth0.20 down
  3. ip link delete eth0.20
Notes: "name" may be anything you wish, but then in subsequent commands, you must use it, which can be a bit hard to spot. Take a look at "eth0.20", and try changing it to a random string for testing.

Tuesday 11 September 2012

pfsense under Virtualbox

This is very brief. Not completely complete. An outline. A hint.
  1. Download pfsense iso
  2. Download Ubuntu Desktop iso
  3. Create a VM for the pfsense
    1. two interfaces
        1. Adaper 1
        2. Enable Network Adapter
        3. NAT
      1. Adaper 2
        1. Enable Network Adapter
        2. Internal Network
          1. intnet
    2. add the pfsense iso as a Storage
      1. Empty
      2. On the far right, there is a disk icon with a drop-down arrow
      3. Choose a virtual CD/DVD disk file...
      4. Select the pfsense iso on your local drive
    3. OK
  4. Boot the pfsense and hit "I" (capital eye) at the appropriate time to install pfsense to the VM's hard drive
  5. em0 and em1 will be the interfaces to use 
    1. not found automatically in my case
  6. Rest of pfsense install not covered here
  7. Install Ubuntu as normal
    1. Adapter 1
      1. use intnet
  8. Start up Ubuntu desktop and log into 192.168.1.1or some similar gateway IP
Your host computer will not be able to hit the pfsense because your host is considered "outside the firewall". Hence, the trick to use an "inside man", an "inside computer" to reach the pfsense.

NOTE: Did you get stuck, blocked, lost? Let me know on which step, thanks.

Add Virtualbox "Guest Additions" to Ubuntu Desktop 12.04

  1. The VM must be running to install the Guest Additions, so start up the VM
  2. Once the VM window shows the booted instance, find the "Devices" menu option at the very top of the encapsulating window (this is hard to make clear/clarify) but don't do anything with it yet.
  3. Make sure you are logged into your user's desktop on the VM
  4. On the VM, make sure your user has root privileges (not covered here), also called "sudo" privileges
  5. Choose Devices -> Install Guest Additions
  6. Confirm all prompts to download and mount the virtual CD from Virtualbox, this may take a while to download/complete
  7. Run the install when prompted
  8. Reboot the VM
  9. Send issues

Saturday 8 September 2012

mac delivery mechanism


  1. your computer has a default gateway set 
    1. can figure out packet is not destined for the local network
    2. will use the MAC address of the default gateway 
  2. default gateway receives the layer-2 frame
    1. will see that the MAC address matches it's own
    2. will un-encapsulate the data link frame and 
    3. pass the data part up to the network layer
  3. at the network layer, layer 3
    1. will see that the destination IP address does not match it's own
      1. this is a packet that is supposed to be routed
    2. will look in it's routing table for the closest match
      1. which interface to send the packet out on
  4. will create a new data link frame addressed to the next hop
    1. data portion of this frame sent out the appropriate interface
  5. process will continue at each router along the way

Assume that none of the info in already cached in an ARP table on any of the machines or routers 
  1. your computer wants to send some data to a computer on network 3
    1. your computer will create an IP packet addressed to 200.0.3.2
    2. your computer will send out an ARP request for the default gateway's MAC address 
    3. on receiving the MAC address, your computer will send out the IP packet
      1. encapsulated within a data link frame that is addressed to the MAC address of router a's interface on network 1
  2. router a will receive this frame
    1. send the data portion up to the network layer 
    2. at the network layer, router a will see that the packet is not addressed to router a
    3. router a will look in it's routing table to find out where to send the packet
    4. routing table will show that network 3 is reachable via network 2
    5. routing table will also show the IP address for the next hop is 200.0.2.2
    6. router a will send out an ARP request onto network 2 asking for router b's MAC address
    7. on receiving MAC, router a will send the IP packet 
      1. encapsulated in a data link frame addressed to router b's MAC address
  3. router b receives this frame it will do the same thing that router a did
    1. will send the IP packet up to the network layer
    2. see that the packet is not addressed to router b 
    3. will then look up in it's routing table for the closest match and see that it is directly connected to network 3
      1. here isn't a next hop router to send it to. 
    4. will send out an ARP request to learn the MAC address for 200.0.3.2
    5. when MAC is received, router b will send out the IP packet
      1.  encapsulated within a data link frame that is addressed to the MAC address of the destination computer
  4. destination computer will see that the data link frame is addressed to it
    1. will pass the IP packet to the network layer. 
    2. at the network layer, the IP address will also match that of the computer
    3. the data from the IP packet will be passed up to the transport layer

Friday 7 September 2012

CLI random break time generator


  1. echo $((RANDOM%20+1)) | xargs -i utimer -c {}m && xfe
  2. export HIGH=$((RANDOM%20+10));for i in `seq 1 ${HIGH}`;do echo $i:${HIGH};sleep 60;done

Monday 3 September 2012

Cleaning up configuration files of removed Debian packages

If you want to rid yourself of those annoying packages showing up with a "rc" in the first field of "dpkg -l", do this as root, with prejudice:
dpkg -l | egrep '^rc ' | awk '{print $2}' | xargs dpkg -P
Let the good times roll!

Debugging Munin loaning graphs locally


  1. Problems
    1. Munin is a pain to debug remotely -- on prod -- when doing custom "loaning" graphs
      1. Varnish gets in the way
      2. (might not work, test yourself) fast-cgi doesn't work with new Munin 2.0 dynamic graph generation very well, just comment it out in the Apache config
    2. These issues combined leads to a complete nightmare of caching and having to wait for graphs to be regenerated so you can see your changes
  2. Troubleshooting
    1. Try this link for perm checks
      1. http://munin-monitoring.org/wiki/CgiHowto
    2. IMPORTANT: ust turn this on in monit.conf manually, since debian turns it off, and who knows who else
      1. graph_strategy cgi
  3. Solution
    1. Grab /var/lib/munin from production server
    2. Install Munin 2.x or greater on your local box
    3. Comment out the Munin files under /etc/cron* whatever/whereever, so your server doesn't try to update any of the files under /var/lib/munin
    4. Move your local copy of /var/lib/munin aside
    5. Move the production version of /var/lib/munin into place on your local system
    6. Copy perms of your original /var/lib/munin to new one
    7. Grab the prod server version of /etc/munin/munin.conf
    8. Move your local copy of /etc/munin/munin.conf aside, rename something you'll remember
    9. Move prod server version /etc/munin/munin.conf in place on your local box
    10. Use "munin-html" to regenerate html pages as you make changes to your munin definitions in /etc/munin/munin.conf
      1. basically these commands, but, for details, see http://blog.loftninjas.org/2010/04/08/an-evening-with-munin-graph-aggregation/
        1. sudo su - munin -s /bin/bash
        2. /usr/share/munin/munin-html --debug
      2. might work / might not
    11. Hit munin locally through your web browser; if you're lucky, all the prod info/graphs appear normally
    12. Now, you can update /etc/munin/munin.conf as you like and graph changes and errors will show up instantly
    13. Tweak URL to hit graphs you know the name of but munin-html failed to find for you
  4. Long-term
    1. Refresh the data from prod every 48 to 72 hours so your graph data doesn't fall off the chart
      1. Since your local box is not updating data, all rrd data will be blank from the time your grab it from prod's /var/lib/munin

Wireless with dead-simple Debian install

I had to do these steps to get wireless working with dead-simple install of Debian.
  1. Most commands done as root user
  2. There may be missing steps, let me know, it was a mess to get working, and I don't recall all steps perfectly
Steps:
  1. run this the whole time as root to see changes as they occur, or not
    1. watch -d 'ip addr;echo =====;ip link;echo =====;ip route'
  2. install broadcom drivers for my network card
    1. Broadcom Corporation BCM4313 802.11b/g/n Wireless LAN Controller
    2. /etc/apt/sources.list
      1. deb [arch=amd64,i386] ftp://ftp.fu-berlin.de/pub/unix/linux/mirrors/debian/ wheezy non-free
    3. aptitude update
    4. aptitude install firmware-brcm80211
  3. add your user to netdev group and restart X
    1. netdev:x:113:yourusername
  4. load network card kernel modules into the kernel
    1. modprobe brcmsmac
      1. don't use the "-r" option, only works to reload, not initial load
    2. /etc/init.d/dbus reload
      1. no idea what this does or if necessary
  5. apt-get install wireless-tools 
  6. verify
    1. lspci -v
      1. shows kernel module used, or not if failed to load, see last line per section for kernel module loaded for that device
    2. lsmod | grep brc
      1. shows list of kernel modules loaded
      2. else, they failed to load
    3. ip link
      1. see if there is a wlan0
    4. iwlist scan
      1. returns wireless networks in the area
  7. apt-get install apt-get install wicd-gtk wpasupplicant
  8. run wicd-gtk and try to connect that way, otherwise, do below
  9. /etc/network/interfaces
    1. #auto wlan0 # comment this out unless you want device started on boot
    2. iface wlan0 inet dhcp
    3. wpa-ssid YOURSSID
    4. wpa-psk YOURWIFIPASSWORD
    5. gateway YOURGATEWAYIP
  10. ifup wlan0

Interview questions: 2020-12

Terraform provider vs provisioner Load balancing Network Load Balancer vs Application Load Balancer  Networking Layer 1 vs Layer 4 haproxy u...