Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Agile Testing - Grig Gheorghiu
Syndicate content
Did anybody say webscale?
Updated: 9 hours 18 min ago

Triggering Jenkins jobs remotely via git post-commit hooks

Wed, 03/18/2015 - 23:49
Assume you want a Jenkins job (for example a job that deploys code or a job that runs integration tests) to run automatically every time you commit code via git. One way to do this would be to configure Github to access a webhook exposed by Jenkins, but this is tricky to do when your Jenkins instance is not exposed to the world.

One way I found to achieve this is to trigger Jenkins job remotely via a local git post-commit hook. There were several steps I had to take:

1) Create a Jenkins user to be used by remote curl commands -- let's call it user1 with password password1.

2) Configure a given Jenkins job -- let's call it JOB-NUMBER1 -- to allow remote builds to be triggered. If you go to the Jenkins configuration page for that job, you'll see a checkbox under the Build Triggers section called "Trigger builds remotely (e.g. from scripts)". Check that checkbox and also specify a random string as the Authentication Token -- let's say it is mytoken1.

3) Try to trigger a remote build for JOB-NUMBER1 by using a curl command similar to this one:

curl --user 'user1:password1' -X POST "http://jenkins.mycompany.com:8080/job/JOB-NUMBER1/build" --data token=mytoken1 --data delay=0sec

If the Jenkins build is parameterized, you need to specify each parameter in the curl command, even if those parameters have default values specified in the Jenkins job definition. Let's say you have 2 parameters, TARGET_HOST and TARGET_USER. Then the curl command looks something like this:

curl --user 'user1:password1' -X POST "http://jenkins.mycompany.com:8080/job/JOB-NUMBER1/build" --data token=mytoken1 --data delay=0sec --data-urlencode json='{"parameter": [{"name":"TARGET_HOST", "value":"myhost1.mycompany.com"}, {"name":"TARGET_USER", "value":"mytargetuser1"}]}'

When you run these curl commands, you should see JOB-NUMBER1 being triggered instanly in the Jenkins dashboard.

Note: if you get an error similar to "HTTP ERROR 403 No valid crumb was included in the request" it means that you have "Prevent Cross Site Request Forgery exploits" checked on the Jenkins "Configure Global Security" page. You need to uncheck that option. Since you're most probably not exposing your Jenkins instance to the world, that should be fine.

 4) Create git post-commit hook. To do this, you need to create a file called post-commit in your local .git/hooks directory under the repository from which you want to trigger the Jenkins job. The post-commit file is a regular bash script:

#!/bin/bash

curl --user 'user1:password1' -X POST "http://jenkins.mycompany.com:8080/job/JOB-NUMBER1/build" --data token=mytoken1 --data delay=0sec

Don't forget to make the post-commit file executabe: chmod 755 post-commit

At this point, whenever you commit code in this repository, you should see the Jenkins job being triggered instantly.

Sending Windows logs to Papertrail with nxlog

Thu, 02/26/2015 - 01:04
I am revisiting Papertrail as a log aggregation tool. It's really easy to send Linux logs to Papertrail via syslog or rsyslog or syslog-ng (see this article on how to configure syslog with TLS) but to send Windows logs you need to jump through some hoops.

Papertrail recommends nxlog as their Windows log management tool of choice, so that's what I used. This Papertrail article explains how to install and configure nxlog on Windows (I recommend enabling TLS).  The nxlog.conf template file provided by Papertrail will send Windows Event logs over. I also wanted to send application-specific logs, so here's what I did:

1) Add an Input section to nxlog.conf for each directory containing the files you want to send to Papertrail. For example, if one of your applications logs to C:\MyApp1\logs and your log files end with .log, you could have this input section:

# Monitor MyApp1 log files 
START_ANGLE_BRACKET Input MyApp1 END_ANGLE_BRACKET
 Module im_file
 File 'C:\\MyApp1\\logs\\*.log' 
 Exec $Message = $raw_event; 
 Exec if $Message =~ /GET \/ping/ drop(); 
 Exec if file_name() =~ /.*\\(.*)/ $SourceName = $1; 
 SavePos TRUE 
 Recursive TRUE 
START_ANGLE_BRACKET /Input END_ANGLE_BRACKET

Some observations:

  • Blogger doesn't like angle brackets so replace START_ANGLE_BRACKET with < and END_ANGLE_BRACKET with >
  • The name MyApp1 is the name of this Input section
  • The File statement points to the location and name of the log files
  • The first Exec statement saves the log line under consideration as the variable $Message
  • The second Exec statement drops messages that contain a specific regular expression, in my case just 'GET /ping' -- which happens to be health checks from the load balancer that pollute the logs; you can replace this with any regular expression that will filter out log lines you don't want sent to Papertrail
  • The next few statements were in the sample Input stanza from the template nxlog.conf file so I just left them there
2) Add more Input sections, one for each log location (i.e. multiple log files under a given directory) that you want to send to Papertrail. You need to give each Input section a unique name (e.g. MyApp1 above).
3) Add a Route section for the Input sections defined previously. If you defined 2 Input sections MyApp1 and MyApp2, your Route section would look something like:

START_ANGLE_BRACKET  Route 2 END_ANGLE_BRACKET
Path MyApp1, MyApp2=> filewatcher_transformer => syslogoutSTART_ANGLE_BRACKET /Route END_ANGLE_BRACKET
The filewatcher_transformer section was already included in the sample nxlog.conf file from Papertrail. The Route section above says that the files processed by the 2 Input paths MyApp1 and MyApp2 will be processed through the statements defined in the filewatcher_transformer section, then will be sent to Papertrail by virtue of being processed through the statements defined in the syslogout section.
At this point, if you restart the nxlog service on your Windows box, you should start seeing log entries from your application(s) flowing into the Papertrail console.

Setting up an OpenVPN server inside an AWS VPC

Sat, 01/31/2015 - 23:56
My goal in this post is to show how to set up an OpenVPN server within an AWS VPC so that clients can connect to EC2 instances via a VPN connection. It's fairly well documented how to configure a site-to-site VPN with an AWS VPC gateway, but articles talking about client-based VPN connections into a VPC are harder to find.

== OpenVPN server setup ==

Let's start with setting up the OpenVPN server. I launched a new Ubuntu 14.04 instance in our VPC and I downloaded the latest openvpn source code via:

wget http://swupdate.openvpn.org/community/releases/openvpn-2.3.6.tar.gz

In order for the 'configure' step to succeed, I also had to install the following Ubuntu packages:

apt-get install build-essential openssl libssl-dev lzop liblzo2-dev libpam-dev

I then ran the usual commands inside the openvpn-2.3.6 directory:

./configure; make; sudo make install

At this point I proceeded to set up my own Certificate Authority (CA), per the OpenVPN HOWTO guide.  As it turned out, I needed the easy-rsa helper scripts on the server running openvpn. I got them from github:

git clone https://github.com/OpenVPN/easy-rsa.git

To generate the master CA certificate & key, I did the following:

cd ~/easy-rsa/easyrsa3
cp vars.example vars

- edited vars file and set these variables with the proper values for my organization:
set_var EASYRSA_REQ_COUNTRY
set_var EASYRSA_REQ_PROVINCE
set_var EASYRSA_REQ_CITY
set_var EASYRSA_REQ_ORG
set_var EASYRSA_REQ_EMAIL
set_var EASYRSA_REQ_OU

./easyrsa init-pki
(this will create the correct directory structure)
./easyrsa build-ca
(this will use the info specified in the vars file above)

To generate the OpenVPN server certificate and key, I ran:

./easyrsa build-server-full server
(I was prompted for a password for the server key)

To generate an OpenVPN client certificate and key for user myuser, I ran:

./easyrsa  build-client-full myuser
(I was prompted for a password for the client key)

The next step was to generate the Diffie Hellman (DH) parameters for the server by running:

./easyrsa gen-dh

I was ready at this point to configure the OpenVPN server.

I created a directory called /etc/openvpn and copied the pki directory under ~/easy-rsa/easyrsa3 to /etc/openvpn. I also copied the sample server configuration file ~/openvpn-2.3.6/sample/sample-config-files/server.conf to /etc/openvpn.

I edited /etc/openvpn/server.conf and specified the following:

ca /etc/openvpn/pki/ca.crt
cert /etc/openvpn/pki/issued/server.crt
key /etc/openvpn/pki/private/server.key  # This file should be kept secret
dh /etc/openvpn/pki/dh.pem

server 10.9.0.0 255.255.255.0
ifconfig-pool-persist /etc/openvpn/ipp.txt

push "route 172.30.0.0 255.255.0.0"

The first block specifies the location of the CA certificate, the server key and certificate, and the DH certificate.

The 'server' parameter specifies a new subnet from which both the OpenVPN server and the OpenVPN clients connecting to the server will get their IP addresses. I set it to 10.9.0.0/24. The client IP allocations will be saved in the ipp.txt file, as specified in the ifconfig-pool-persist parameter.

One of the most important options, which I missed when I initially configured the server, is the 'push route' one. This makes the specified subnet (i.e. the instances in the VPC that you want to get to via the OpenVPN server) available to the clients connecting to the OpenVPN server without the need to create static routes on the clients. In my case, all the EC2 instances in the VPC are on the 172.30.0.0/16 subnet, so that's what I specified above.

Two more very important steps are needed on the OpenVPN server. It took me quite a while to find them so I hope you will be spared the pain.

The first step was to turn on IP forwarding on the server:

- uncomment the following line in /etc/sysctl.conf:
net.ipv4.ip_forward=1

- run
sysctl -p

The final step in the configuration of the OpenVPN server was to make it do NAT via itpables masquerading (thanks to rbgeek's blog post for these last two critical steps):

- run
iptables -t nat -A POSTROUTING -s 10.9.0.0/24 -o eth0 -j MASQUERADE

- also add the above line to /etc/rc.local so it gets run on reboot

Now all that's needed on the server is to actually run openvpn. You can run it in the foreground for troubleshooting purposes via:

openvpn /etc/openvpn/server.conf

Once everything works, run it in daemon mode via:

openvpn --daemon --config /etc/openvpn/server.conf

You will be prompted for the server key password when you start up openvpn. Haven't looked yet on how to run the server in a fully automated way.

Almost forgot to specify that you need to allow incoming traffic to UDP port 1194 in the AWS security group where your OpenVPN server belongs. Also allow traffic from that security group to the security groups of the EC2 instances that you actually want to reach over the OpenVPN tunnel.

== OpenVPN client setup ==

This is on a Mac OSX Mavericks client, but I'm sure it's similar for other clients.

Install tuntap
- download tuntap_20150118.tar.gz from http://tuntaposx.sourceforge.net
- untar and install tuntap_20150118.pkg

Install lzo
wget http://www.oberhumer.com/opensource/lzo/download/lzo-2.06.tar.gz
tar xvfz lzo-2.06.tar.gz
cd lzo-2.06
./configure; make; sudo make install

Install openvpn
- download openvpn-2.3.6.tar.gz from http://openvpn.net/index.php/open-source/downloads.htmltar xvf openvpn-2.3.6.tar
cd openvpn-2.3.6
./configure; make; sudo make install

At this point ‘openvpn --help’ should work.
The next step for the client setup is to copy the CA certificate ca.crt, and the client key and certificate (myuser.key and myuser.crt) from the OpenVPN server to the local client. I created an openvpn directory under my home directory on my Mac and dropped ca.crt in ~/openvpn/pki, myuser.key in ~/openvpn/pki/private and myuser.crt in ~/openvpn/pki/issued. I also copied the sample file ~/openvpn-2.3.6/sample/sample-config-files/client.conf to ~/openvpn and specified the following parameters in that file:
remote EXTERNAL_IP_ADDRESS_OF_OPENVPN_SERVER 1194
ca /Users/myuseropenvpn/pki/ca.crtcert /Users/myuser/openvpn/pki/issued/myuser.crtkey /Users/myuser/openvpn/pki/private/myuser.key
Then I started up the OpenVPN client via:
sudo openvpn ~/openvpn/client.conf(at this point I was prompted for the password for myuser.key)
To verify that the OpenVPN tunnel is up and running, I ping-ed the internal IP address of the OpenVPN server (in my case it was 10.9.0.1 on the internal subnet I specified in server.conf), as well as the internal IPs of various EC2 instances behind the OpenVPN server. Finally, I ssh-ed into those internal IP addresses and declared victory.
That's about it. Hope this helps!

UPDATE: I discovered in the mean time a very good Mac OSX GUI tool for managing client OpenVPN connections: Tunnelblick. All it took was importing the client.conf file mentioned above.

10 technologies that impressed me in 2014

Wed, 12/31/2014 - 07:26
Some of these have been new to me, some are old friends that I came to appreicate more. In alphabetical order:

  1. Ansible
  2. Bitcoin
  3. Consul
  4. Docker
  5. Golang
  6. HAProxy
  7. nginx
  8. OpenStack
  9. Sysdig
  10. Varnish

Dynamic DNS updates with nsupdate (new and improved!)

Wed, 12/17/2014 - 23:14
I blogged about this topic before. This post shows a slightly different way of using nsupdate remotely against a DNS server running BIND 9 in order to programatically update DNS records. The scenario I am describing here involves an Ubuntu 12.04 DNS server running BIND 9 and an Ubuntu 12.04 client running nsupdate against the DNS server.

1) Run ddns-confgen and specify /dev/urandom as the source of randomness and the name of the zone file you want to dynamically update via nsupdate:

$ ddns-confgen -r /dev/urandom -z myzone.com

# To activate this key, place the following in named.conf, and
# in a separate keyfile on the system or systems from which nsupdate
# will be run:
key "ddns-key.myzone.com" {
algorithm hmac-sha256;
secret "1D1niZqRvT8pNDgyrJcuCiykOQCHUL33k8ZYzmQYe/0=";
};

# Then, in the "zone" definition statement for "myzone.com",
# place an "update-policy" statement like this one, adjusted as
# needed for your preferred permissions:
update-policy {
 grant ddns-key.myzone.com zonesub ANY;
};

# After the keyfile has been placed, the following command will
# execute nsupdate using this key:
nsupdate -k <keyfile>

2) Follow the instructions in the output of ddns-keygen (above). I actually named the key just ddns-key, since I was going to use it for all the zones on my DNS server. So I added this stanza to /etc/bind/named.conf on the DNS server:

key "ddns-key" {
algorithm hmac-sha256;
secret "1D1niZqRvT8pNDgyrJcuCiykOQCHUL33k8ZYzmQYe/0=";
};

3) Allow updates when the key ddns-key is used. In my case, I added the allow-update line below to all zones that I wanted to dynamically update, not only to myzone.com:

zone "myzone.com" {
        type master;
        file "/etc/bind/zones/myzone.com.db";
allow-update { key "ddns-key"; };
};

At this point I also restarted the bind9 service on my DNS server.

4) On the client box, create a text file containing nsupdate commands to be sent to the DNS server. In the example below, I want to dynamically add both an A record and a reverse DNS PTR record:

$ cat update_dns1.txt
server dns1.mycompany.com
debug yes
zone myzone.com
update add testnsupdate1.myzone.com 3600 A 10.10.2.221
show
send
zone 2.10.10.in-addr.arpa
update add 221.2.10.10.in-addr.arpa 3600 PTR testnsupdate1.myzone.com
show
send

Still on the client box, create a file containing the stanza with the DDNS key generated in step 1:

$ cat ddns-key.txt
key "ddns-key" {
algorithm hmac-sha256;
secret "Wxp1uJv3SHT+R9rx96o6342KKNnjW8hjJTyxK2HYufg=";
};

5) Run nsupdate and feed it both the update_dns1.txt file containing the commands, and the ddns-key.txt file:

$ nsupdate -k ddns-key.txt -v update_dns1.txt

You should see some fairly verbose output, since the command file specifies 'debug yes'. At the same time, tail /var/log/syslog on the DNS server and make sure there are no errors.

In my case, there were some hurdles I had to overcome on the DNS server. The first one was that apparmor was installed and it wasn't allowing the creation of the journal files used to keep track of DDNS records. I saw lines like these in /var/log/syslog:

Dec 16 11:22:59 dns1 kernel: [49671335.189689] type=1400 audit(1418757779.712:12): apparmor="DENIED" operation="mknod" parent=1 profile="/usr/sbin/named" name="/etc/bind/zones/myzone.com.db.jnl" pid=31154 comm="named" requested_mask="c" denied_mask="c" fsuid=107 ouid=107
Dec 16 11:22:59 dns1 kernel: [49671335.306304] type=1400 audit(1418757779.828:13): apparmor="DENIED" operation="mknod" parent=1 profile="/usr/sbin/named" name="/etc/bind/zones/rev.2.10.10.in-addr.arpa.jnl" pid=31153 comm="named" requested_mask="c" denied_mask="c" fsuid=107 ouid=107

To get past this issue, I disabled apparmor for named:

# ln -s /etc/apparmor.d/usr.sbin.named /etc/apparmor.d/disable/
# service apparmor restart

The next issue was an OS permission denied (nothing to do with apparmor) when trying to create the journal files in /etc/bind/zones:

Dec 16 11:30:54 dns1 named[32640]: /etc/bind/zones/myzone.com.db.jnl: create: permission denied
Dec 16 11:30:54 dns named[32640]: /etc/bind/zones/rev.2.0.10.in-addr.arpa.jnl: create: permission denied

I got past this issue by running

# chown -R bind:bind /etc/bind/zones

At this point everything worked as expected.