Wednesday 29 May 2013

Direct ssh to a server via proxy using putty/plink on Windows


  1. Make sure seamless ssh keys are setup to your bastion server for your username
    1. Not covered here
    2. See: http://www.ualberta.ca/CNS/RESEARCH/LinuxClusters/pka-putty.html
  2. Session -> Host Name -> mytargetserver.mydomain.com
  3. Connection -> Proxy
    1. Proxy Type -> Local
    2. Telnet command, or local proxy command 
      1. c:/program files (x86)/putty/plink.exe myproxy.mydomain.com -l myusername -agent -nc %host:%port
        1. adjust this path to plink.exe to match your local setup
          1. hint: install the complete putty install package, not just putty
  4. Tunnels
    1.   L8081 mytargetserver.mydomain.com:8081
Hint: always hit "Save", no matter what you do, or however inconvenient it was designed to be.

Another example: plink -L 127.0.0.1:1433:mysqlserver.com:1433 admin@google.com -i myprivkeyfile

Friday 24 May 2013

Show progress during dd copy

kill -USR1  <pid of dd>

Thursday 23 May 2013

In-memory page states and kscand


  1. kscand task
    1. periodically sweeps through all the pages in memory
    2. notes "last access time"
      1. was accessed?
        1. increments page's age counter
      2. wasn't accessed?
        1. decrements page's age counter
      3. age counter at zero
        1. move page to inactive dirty state
In-memory page states
  1. free
    1. begin in this state
    2. not being used
    3. available for allocation, i.e. made active
  2. active
    1. allocated
    2. actively in use
  3. inactive dirty
    1. has fallen into disuse
    2. candidate for removal from main memory
  4. inactive laundered
    1. interim state
    2. contents are being moved to disk
      1. when disk I/O operation complete
        1. moved to the inactive clean state
      2. if, during the disk operation, the page is accessed
        1. moved back into the active state
  5. inactive clean
    1. laundering succeeded, i.e. contents in sync with copy on disk
    2. may be 
      1. deallocated
      2. overwritten
Taken: http://www.redhat.com/magazine/001nov04/features/vm/

Wednesday 22 May 2013

LVM crypt disks on Linux/AWS



  1. dd if=/dev/urandom of=/keys/xvdm.key bs=1024 count=4
  2. dd if=/dev/urandom of=/keys/xvdn.key bs=1024 count=4
  3. cryptsetup --verbose -y luksFormat /dev/xvdm /keys/xvdm.key
  4. cryptsetup --verbose -y luksFormat /dev/xvdn /keys/xvdn.key
  5. cryptsetup luksOpen /dev/xvdm cryptm --key-file /etc/xvdm.key
  6. cryptsetup luksOpen /dev/xvdn cryptn --key-file /etc/xvdn.key
  7. pvcreate /dev/mapper/cryptm /dev/mapper/cryptn
  8. Add entries to /etc/crypttab for reboots and test somehow
    1. cryptm /dev/xvdm /etc/xvdm.key luks
    2. cryptn /dev/xvdn /etc/xvdn.key luks
Complete LVM setup and add entries to /etc/fstab.

Hint: don't make one, single typo...ever.

Thursday 16 May 2013

Double looping with bash

Neat:
  1. for ITEM in $(find /cassandra/data -type d -name snapshots)
    1. do for DIR in $(find ${ITEM} -maxdepth 1 -mindepth 1 -type d -mtime -1)
      1. do echo $ITEM $DIR
    2. done
  2. done

Tuesday 14 May 2013

Confluence: Lock wait timeout exceeded; try restarting transaction

WARNING! Atlassian themselves recommend STRONGLY against this procedure. If any action, take the action that shows you which table is locking, DO NOT DELETE anything unless you are 100% confident you can reverse your deletions. DO NOT DELETE, DO NOT DELETE!

Seeing this?

2013-05-14 16:39:55,581 ERROR [QuartzScheduler_Worker-1] [sf.hibernate.util.JDBCExceptionReporter] logExceptions Lock wait timeout exceeded; try restarting transaction
2013-05-14 16:39:55,581 ERROR [QuartzScheduler_Worker-1] [sf.hibernate.impl.SessionImpl] execute Could not synchronize database state with session


The first is actually reported from MySQL itself, the second from Hibernate, which wraps databases for Java apps.


If you are desperate, try deleting all rows from mysql's crowd.cwd_membership table after backing it up, worked for me, syncs started working again in under 16ms.
  1. mysqldump crowd | bzip2 -c > /mnt/dump_crowd_`date +%Y%m%d`.sql.bz2
  2. mysql crowd -e 'delete from cwd_membership'
If that doesn't help, try deleting old users from any confluence groups that are still in your LDAP dir, be brutal. While your at it, delete old users period from LDAP.

To spot the problem table, this might help, if another table is your problem:
  1. watch "mysql -e 'show processlist'"
  2. Then, run LDAP sync update via admin web GUI, and watch to see which table is locking
Other things you might be seeing in your logs if you have this issue:

"batch failed falling back to individual processing java.lang.RuntimeException: could not flush session"
"Error occurred while refreshing the cache for directory"
"synchroniseCache full synchronisation for directory [ XXXX ] starting"
"could not insert: [com.atlassian.crowd.embedded.hibernate2.HibernateMembership#YYYYY]"
"Lock wait timeout exceeded; try restarting transaction"
"Could not synchronize database state with session"
"could not flush session"


Monday 13 May 2013

Put stuff on your Nexus 4


  1. apt-get install gmtp
  2. Make sure your "Storage" is in MTP mode
P.S. Or, if you have access to a Mac: "Android File Transfer"

Saturday 11 May 2013

Tuesday 7 May 2013

EC2 server to VPC private instance via VPC NAT instance


  1. iptables -t nat -A PREROUTING -s 23.23.23.23/32 -d 10.0.0.254/32 -i eth0 -p tcp -m tcp --sport 1024:65535 --dport 3306 -j DNAT --to-destination 10.0.12.10:3306
    1. 23.23.23.23 is your external server's public IP address
    2. 10.0.0.254 is your VPC NAT instance's IP address in the public subnet
    3. 10.0.12.10 is the VPC IP address of your server in a private subnet
    4. 3306 is the port your service is listening on

Monday 6 May 2013

ec2-create-image: attached EBS volumes are snapshot and mapped

"ec2-create-image does snapshot the attached EBS volumes and add a block device mapping for those snapshots in the created AMI"
Taken: https://forums.aws.amazon.com/message.jspa?messageID=211674

Nicer settings for cssh: terminal_font, terminal_size, terminal_args

.clusterssh/config

  1. terminal_font=5x8
  2. terminal_size=140x48
  3. terminal_args=-fg green
  4. auto_close=1

Sunday 5 May 2013

Slow SSH: one possible solution, set "useDNS" to "no"


  1. In sshd_config on the targer server, set "useDNS" to "no", and restart sshd

Friday 3 May 2013

mysqldump between two servers over ssh


  1. set up ssh keys so server1 user can ssh to a server2
  2. set $HOME/.my.cnf so both users can get into respective mysql cli without passwords
    1. see below for sample file
  3. create the new, empty database on server2, receiving server
  4. from server1
    1. mysqldump mydatabase | ssh server2 mysql mydatabase

# $HOME/.my.cnf
[client]
password=myusersmysqlpassword

Openfire: use your 3rd-party, signed SSL cert

PLEASE LET ME KNOW IF YOU HAVE FIXES FOR THIS WITH LATEST VERSIONS
  1. default keytool password is "changeit"
    1. use it for all password prompts
    2. works 99%
    3. if it doesn't work, ask around, poke around
  2. Get keytool command in your PATH
  3. Use Openfire's web interface to "generate self-signed certificates"
    1. NOTE: "import a signed certificate and its private key"
      1. broken
        1. says certs were loaded in green, but shows no result in "Server Certificates" list
      2. whole reason for this post
  4. find existing keystores on your chat server
    1. nice updatedb
    2. locate keystore
    3. locate truststore
    4. here, we'll assume /opt/openfire/resources/security
  5. list the "domain" Openfire used for the "generate self-signed certificates" action above
    1. keytool -list -v -keystore /opt/openfire/resources/security/keystore | grep rsa
      1. e.g.: Alias name: my.domain.com_rsa
    2. remember this for a later step
  6. load your CAs root cert into the truststore
    1. first, see if it is there
      1. keytool -list -v -keystore /opt/openfire/resources/security/truststore | grep "Issuer:"
    2. if not, download it from your CA, and
      1. keytool -import -alias myCAsRootCertAlias -file myCAsRootCert.crt -keystore /opt/openfire/resources/security/truststore
      2. verify
  7. create a p12 with your key, cert and CA's cert
    1. openssl pkcs12 -export -in myCert.crt -inkey myKey.key -out myP12.p12 -name my.domain.com_rsa -CAfile myCAsCert.crt -caname root
  8. dump it to a new keystore
    1. keytool -importkeystore -deststorepass changeit -destkeypass changeit -destkeystore mykeystore -srckeystore myP12.p12 -srcstoretype PKCS12 -srcstorepass changeit -alias my.domain.com_rsa
  9. cp -v /opt/openfire/resources/security/keystore /opt/openfire/resources/security/keystore_2013xxyy
  10. cp -v mykeystore /opt/openfire/resources/security/keystore
  11. restart openfire

Interview questions: 2020-12

Terraform provider vs provisioner Load balancing Network Load Balancer vs Application Load Balancer  Networking Layer 1 vs Layer 4 haproxy u...