Warning: Table './devblogsdb/cache_page' is marked as crashed and last (automatic?) repair failed query: SELECT data, created, headers, expire, serialized FROM cache_page WHERE cid = 'http://www.softdevblogs.com/?q=aggregator/sources/17' in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc on line 135

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 729

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 730

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 731

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 732
Software Development Blogs: Programming, Software Testing, Agile, Project Management
Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Agile Testing - Grig Gheorghiu
warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/common.inc on line 153.
Syndicate content
Did anybody say webscale?
Updated: 50 min 51 sec ago

LDAP server setup and client authentication

Fri, 04/15/2016 - 19:24
We recently bought at work a CloudBees Jenkins Enterprise license and I wanted to tie the user accounts to a directory service. I first tried to set up Jenkins authentication via the AWS Directory Service, hoping it will be pretty much like talking to an Active Directory server. That proved to be impossible to set up, at least for me. I also tried to have an LDAP proxy server talking to the AWS Directory Service and have Jenkins authenticate against the LDAP proxy. No dice. I ended up setting up a good old-fashioned LDAP server and managed to get Jenkins working with it. Here are some of my notes.

OpenLDAP server setup
I followed this excellent guide from Digital Ocean. The server was an Ubuntu 14.04 EC2 instance in my case. What follows in terms of the server setup is taken almost verbatim from the DO guide.

Set the hostname

# hostnamectl set-hostname my-ldap-server

Edit /etc/hosts and make sure this entry exists:
LOCAL_IP_ADDRESS my-ldap-server.mycompany.com my-ldap-server
(it makes a difference that the FQDN is the first entry in the line above!)

Make sure the following types of names are returned when you run hostname with different options:

# hostnamemy-ldap-server
# hostname -fmy-ldap-server.mycompany.com
# hostname -dmycompany.com

Install slapd

# apt-get install slapd ldap-utils# dpkg-reconfigure slapd
(here you specify the LDAP admin password)

Install the SSL Components

# apt-get install gnutls-bin ssl-cert

Create the CA Template
# mkdir /etc/ssl/templates
# vi /etc/ssl/templates/ca_server.conf# cat /etc/ssl/templates/ca_server.confcn = LDAP Server CAcacert_signing_key

Create the LDAP Service Template

# vi /etc/ssl/templates/ldap_server.conf# cat /etc/ssl/templates/ldap_server.conforganization = "My Company"cn = my-ldap-server.mycompany.comtls_www_serverencryption_keysigning_keyexpiration_days = 3650

Create the CA Key and Certificate

# certtool -p --outfile /etc/ssl/private/ca_server.key# certtool -s --load-privkey /etc/ssl/private/ca_server.key --template /etc/ssl/templates/ca_server.conf --outfile /etc/ssl/certs/ca_server.pem
Create the LDAP Service Key and Certificate

# certtool -p --sec-param high --outfile /etc/ssl/private/ldap_server.key# certtool -c --load-privkey /etc/ssl/private/ldap_server.key --load-ca-certificate /etc/ssl/certs/ca_server.pem --load-ca-privkey /etc/ssl/private/ca_server.key --template /etc/ssl/templates/ldap_server.conf --outfile /etc/ssl/certs/ldap_server.pem

Give OpenLDAP Access to the LDAP Server Key

# usermod -aG ssl-cert openldap# chown :ssl-cert /etc/ssl/private/ldap_server.key# chmod 640 /etc/ssl/private/ldap_server.key

Configure OpenLDAP to Use the Certificate and Keys
IMPORTANT NOTE: in modern versions of slapd, configuring the server is not done via slapd.conf anymore. Instead, you put together ldif files and run LDAP client utilities such as ldapmodify against the local server. The Distinguished Name of the entity you want to modify in terms of configuration is generally dn: cn=config but it can also be the LDAP database dn: olcDatabase={1}hdb,cn=config.
# vi addcerts.ldif# cat addcerts.ldifdn: cn=configchangetype: modifyadd: olcTLSCACertificateFileolcTLSCACertificateFile: /etc/ssl/certs/ca_server.pem-add: olcTLSCertificateFileolcTLSCertificateFile: /etc/ssl/certs/ldap_server.pem-add: olcTLSCertificateKeyFileolcTLSCertificateKeyFile: /etc/ssl/private/ldap_server.key

# ldapmodify -H ldapi:// -Y EXTERNAL -f addcerts.ldif# service slapd force-reload# cp /etc/ssl/certs/ca_server.pem /etc/ldap/ca_certs.pem# vi /etc/ldap/ldap.conf
* set TLS_CACERT to following:TLS_CACERT /etc/ldap/ca_certs.pem
# ldapwhoami -H ldap:// -x -ZZAnonymous

Force Connections to Use TLS
Change olcSecurity attribute to include 'tls=1':

# vi forcetls.ldif# cat forcetls.ldifdn: olcDatabase={1}hdb,cn=configchangetype: modifyadd: olcSecurityolcSecurity: tls=1

# ldapmodify -H ldapi:// -Y EXTERNAL -f forcetls.ldif# service slapd force-reload# ldapsearch -H ldap:// -x -b "dc=mycompany,dc=com" -LLL dn(shouldn’t work)
# ldapsearch -H ldap:// -x -b "dc=mycompany,dc=com" -LLL -Z dn(should work)

Disallow anonymous bind
Create user binduser to be used for LDAP searches:

# vi binduser.ldif# cat binduser.ldifdn: cn=binduser,dc=mycompany,dc=comobjectClass: topobjectClass: accountobjectClass: posixAccountobjectClass: shadowAccountcn: binduseruid: binduseruidNumber: 2000gidNumber: 200homeDirectory: /home/binduserloginShell: /bin/bashgecos: suseruserPassword: {crypt}xshadowLastChange: -1shadowMax: -1shadowWarning: -1

# ldapadd -x -W -D "cn=admin,dc=mycompany,dc=com" -Z -f binduser.ldifEnter LDAP Password:adding new entry "cn=binduser,dc=mycompany,dc=com"
Change olcDissalows attribute to include bind_anon:

# vi disallow_anon_bind.ldif# cat disallow_anon_bind.ldifdn: cn=configchangetype: modifyadd: olcDisallowsolcDisallows: bind_anon

# ldapmodify -H ldapi:// -Y EXTERNAL -ZZ -f disallow_anon_bind.ldif# service slapd force-reload
Also disable anonymous access to frontend:

# vi disable_anon_frontend.ldif# cat disable_anon_frontend.ldifdn: olcDatabase={-1}frontend,cn=configchangetype: modifyadd: olcRequiresolcRequires: authc

# ldapmodify -H ldapi:// -Y EXTERNAL -f disable_anon_frontend.ldif# service slapd force-reload

Create organizational units and users
Create helper scripts:

# cat add_ldap_ldif.sh


ldapadd -x -w adminpassword -D "cn=admin,dc=mycompany,dc=com" -Z -f $LDIF

# cat modify_ldap_ldif.sh#!/bin/bash


ldapmodify -x -w adminpassword -D "cn=admin,dc=mycompany,dc=com" -Z -f $LDIF

# cat set_ldap_pass.sh#!/bin/bash


ldappasswd -s $PASS -w adminpassword -D "cn=admin,dc=mycompany,dc=com" -x "uid=$USER,ou=users,dc=mycompany,dc=com" -Z
Create ‘mypeople’ organizational unit:

# cat add_ou_mypeople.ldifdn: ou=mypeople,dc=mycompany,dc=comobjectclass: organizationalunitou: usersdescription: all users
# ./add_ldap_ldif.sh add_ou_mypeople.ldif
Create 'groups' organizational unit:

# cat add_ou_groups.ldifdn: ou=groups,dc=mycompany,dc=comobjectclass: organizationalunitou: groupsdescription: all groups

# ./add_ldap_ldif.sh add_ou_groups.ldif
Create users (note the shadow attributes set to -1, which means they will be ignored):

# cat add_user_myuser.ldifdn: uid=myuser,ou=mypeople,dc=mycompany,dc=comobjectClass: topobjectClass: accountobjectClass: posixAccountobjectClass: shadowAccountcn: myuseruid: myuseruidNumber: 2001gidNumber: 201homeDirectory: /home/myuserloginShell: /bin/bashgecos: myuseruserPassword: {crypt}xshadowLastChange: -1shadowMax: -1shadowWarning: -1
# ./add_ldap_ldif.sh add_user_myuser.ldif# ./set_ldap_pass.sh myuser MYPASS

Enable LDAPS
In /etc/default/slapd set:

SLAPD_SERVICES="ldap:/// ldaps:/// ldapi:///"

Enable debugging
This was a life saver when it came to troubleshooting connection issues from clients such as Jenkins or other Linux boxes. To enable full debug output, set olcLogLevel to -1:

# cat enable_debugging.ldifdn: cn=configchangetype: modifyadd: olcLogLevelolcLogLevel: -1
# ldapadd -H ldapi:// -Y EXTERNAL -f enable_debugging.ldif
# service slapd force-reload

Configuring Jenkins LDAP authentication
Verify LDAPS connectivity from Jenkins to LDAP server
In my case, the Jenkins server is in the same VPC and subnet as the LDAP server, so I added an /etc/hosts entry on the Jenkins box pointing to the FQDN of the LDAP server so it can hit its internal IP address:

IP_ADDRESS_OF_LDAP_SERVER my-ldap-server.mycompany.com
I verified that port 636 (used by LDAPS) on the LDAP server is reachable from the Jenkins server:
# telnet my-ldap-server.mycompany.com 636Trying IP_ADDRESS_OF_LDAP_SERVER...Connected to my-ldap-server.mycompany.com.Escape character is '^]'.
Set up LDAPS client on Jenkins server (StartTLSdoes not work w/ Jenkins LDAP plugin!)
# apt-get install ldap-utils
IMPORTANT: Copy over /etc/ssl/certs/ca_server.pem from LDAP server as /etc/ldap/ca_certs.pem on Jenkins server and then:
# vi /etc/ldap/ldap.confset:TLS_CACERT /etc/ldap/ca_certs.pem
Add LDAP certificates to Java keystore used by Jenkins
As user jenkins:
$ mkdir .keystore$ cp /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/security/cacerts .keystore/(you may need to customize the above line in terms of the path to the cacerts file -- it is the one under your JAVA_HOME)

$ keytool --keystore /var/lib/jenkins/.keystore/cacerts --import --alias my-ldap-server.mycompany.com:636 --file /etc/ldap/ca_certs.pemEnter keystore password: changeitOwner: CN=LDAP Server CAIssuer: CN=LDAP Server CASerial number: 570bddb0Valid from: Mon Apr 11 17:24:00 UTC 2016 until: Tue Apr 11 17:24:00 UTC 2017Certificate fingerprints:....Extensions:....
Trust this certificate? [no]:  yesCertificate was added to keystore
In /etc/default/jenkins, set JAVA_ARGS to:
JAVA_ARGS="-Djava.awt.headless=true -Djavax.net.ssl.trustStore=/var/lib/jenkins/.keystore/cacerts -Djavax.net.ssl.trustStorePassword=changeit"  
As root, restart jenkins:

# service jenkins restart
Jenkins settings for LDAP plugin
This took me a while to get right. The trick was to set the rootDN to dc=mycompany, dc=com and the userSearchBase to ou=mypeople (or to whatever name you gave to your users' organizational unit). I also tried to get LDAP groups to work but wasn't very successful.
Here is the LDAP section in /var/lib/jenkins/config.xml:
 <securityRealm class="hudson.security.LDAPSecurityRealm" plugin="ldap@1.11">    <server>ldaps://my-ldap-server.mycompany.com:636</server>    <rootDN>dc=mycompany,dc=com</rootDN>    <inhibitInferRootDN>true</inhibitInferRootDN>    <userSearchBase>ou=mypeople</userSearchBase>    <userSearch>uid={0}</userSearch> <groupSearchBase>ou=groups</groupSearchBase> <groupMembershipStrategy class="jenkins.security.plugins.ldap.FromGroupSearchLDAPGroupMembershipStrategy"> <filter>member={0}</filter> </groupMembershipStrategy>    <managerDN>cn=binduser,dc=mycompany,dc=com</managerDN>    <managerPasswordSecret>JGeIGFZwjipl6hJNefTzCwClRcLqYWEUNmnXlC3AOXI=</managerPasswordSecret>    <disableMailAddressResolver>false</disableMailAddressResolver>    <displayNameAttributeName>displayname</displayNameAttributeName>    <mailAddressAttributeName>mail</mailAddressAttributeName>    <userIdStrategy class="jenkins.model.IdStrategy$CaseInsensitive"/>    <groupIdStrategy class="jenkins.model.IdStrategy$CaseInsensitive"/>

At this point, I was able to create users on the LDAP server and have them log in to Jenkins. With CloudBees Jenkins Enterprise, I was also able to use the Role-Based Access Control and Folder plugins in order to create project-specific folders and folder-specific groups specifying various roles. For example, a folder MyProjectNumber1 would have a Developers group defined inside it, as well as an Administrators group and a Readers group. These groups would be associated with fine-grained roles that only allow certain Jenkins operations for each group.
I tried to have these groups read by Jenkins from the LDAP server, but was unsuccessful. Instead, I had to populate the folder-specific groups in Jenkins with user names that were at least still defined in LDAP.  So that was half a win. Still waiting to see if I can define the groups in LDAP, but for now this is a workaround that works for me.
Allowing users to change their LDAP password
This was again a seemingly easy task but turned out to be pretty complicated. I set up another small EC2 instance to act as a jumpbox for users who want to change their LDAP password.
The jumpbox is in the same VPC and subnet as the LDAP server, so I added an /etc/hosts entry on the jumpbox pointing to the FQDN of the LDAP server so it can hit its internal IP address:

IP_ADDRESS_OF_LDAP_SERVER my-ldap-server.mycompany.com
I verified that port 636 (used by LDAPS) on the LDAP server is reachable from the jumpbox:
# telnet my-ldap-server.mycompany.com 636Trying IP_ADDRESS_OF_LDAP_SERVER...Connected to my-ldap-server.mycompany.com.Escape character is '^]'.
# apt-get install ldap-utils
IMPORTANT: Copy over /etc/ssl/certs/ca_server.pem from LDAP server as /etc/ldap/ca_certs.pem on the jumpbox and then:
# vi /etc/ldap/ldap.confset:TLS_CACERT /etc/ldap/ca_certs.pem
Next, I followed this LDAP Client Authentication guide from the Ubuntu documentation.
# apt-get install ldap-auth-client nscd
Here I had to answer the setup questions on LDAP server FQDN, admin DN and password, and bind user DN and password. 
# auth-client-config -t nss -p lac_ldap
I edited /etc/auth-client-config/profile.d/ldap-auth-config and set:
[lac_ldap]nss_passwd=passwd: ldap filesnss_group=group: ldap filesnss_shadow=shadow: ldap filesnss_netgroup=netgroup: nis
I edited /etc/ldap.conf and made sure the following entries were there:
base dc=mycompany,dc=comuri ldaps://my-ldap-server.mycompany.combinddn cn=binduser,mycompany,dc=combindpw BINDUSERPASSrootbinddn cn=admin,mycompany,dc=comport 636ssl ontls_cacertfile /etc/ldap/ca_certs.pemtls_cacertdir /etc/ssl/certs
I allowed password-based ssh logins to the jumpbox by editing /etc/ssh/sshd_config and setting:
PasswordAuthentication yes
# service ssh restart

IMPORTANT: On the LDAP server, I had to allow users to change their own password by adding this ACL:
# cat set_userpassword_acl.ldif
dn: olcDatabase={1}hdb,cn=configchangetype: modifyadd: olcAccessolcAccess: {0}to attrs=userpassword by dn="cn=admin,dc=mycompany,dc=com" write by self write by anonymous auth by users none
# ldapmodify -H ldapi:// -Y EXTERNAL -f set_userpassword_acl.ldif

At this point, users were able to log in via ssh to the jumpbox using a pre-set LDAP password, and change their LDAP password by using the regular Unix 'passwd' command.
I am still fine-tuning the LDAP setup on all fronts: LDAP server, LDAP client jumpbox and Jenkis server. The setup I have so far allows me to have a single sign-on account for users to log in to Jenkins. Some of my next steps is to use the same user LDAP accounts  for authentication and access control into MySQL and other services.

Joining an EC2 Windows instance to an AWS Directory Service domain

Wed, 04/06/2016 - 21:53
I've been struggling with the existing documentation on how to join an EC2 instance running Windows Server 2012 to an AWS Directory Service domain, so I am hastening to jot down some notes on how I got it to work.

1) Create an AWS Directory Service domain

There is good documentation on doing this. I chose the Microsoft Active Directory option.

A few notes on the creation of an AWS Directory Service:

  • I created a new VPC with 2 subnets for the Directory Service usage
  • I made sure each subnet has an Internet gateway associated so that it can be reachable from the outside
During the creation of the Directory Service, you'll be asked to specify an administrator-type user name and password. Make sure you remember what you specified there because you'll need this info in a subsequent step. Also make note of the DNS server IP addresses that were set during the Directory Service creation.
2) Create an IAM role to be associated with the EC2 Windows instance
  • the IAM role needs to be associated with the AmazonEC2RoleforSSM and AmazonSSMFullAccess policies
  • the IAM role also needs to have a trust relationship with ec2.amazonaws.com

3) Launch EC2 Windows instance associated with the AWS Directory Service domain
I chose the Windows_Server-2012-R2_RTM-English-64Bit-Base-2016.03.09 AMI.  In Step 3 of the AWS launch instance wizard ("Configure instance details") I made sure I specified the following:
  • Network: the VPC created in step 1) above
  • Subnet: one of the 2 subnets created in step 1) above
  • Domain join directory: the directory name for the Directory Service created in step 1) above
  • IAM role: the IAM role created in step 2) above
4) Connect to EC2 Windows instance via RDP
First get the administrator password via the AWS console (you'll need to paste the contents of the private key corresponding to the EC2 key you used when launching the Windows instance). Then connect to the Windows instance as the local administrator user.
Verify that you see the fully qualified domain name of your Directory Service (whatever you indicated in step 1) as the domain of the Windows instance (in Server Manager -> Local Server). If you don't, something went wrong with joining the domain during the instance launch. You can see the system log of that instance in the AWS console by selecting the instance, then going to Actions->Instance Settings->Get System Log. For example, in one of my failed attempts to get all of this working I saw errors related to the IAM role I was using, which at the time didn't have the correct SSM policies attached.
If the Windows instance is correctly joined to the domain, you need to install the Active Directory management tools in order to actually manage the AWS Directory Service. Here is a Rackspace article I found with good instructions.

5) Log in to the EC2 Windows instance as the domain admin to manage AD
After the EC2 Windows instance was rebooted, I managed to log in via RDP as my.aws.directory.fqdn\myusername (where both of these values are the ones chosen in Step 1 above) with the password also chosen in Step 1. At this point I was able to use the Active Directory management tools to add new AD users etc.
Here are some other good resources I found:

Running a static website with Hugo on Google Cloud Storage

Fri, 02/12/2016 - 21:29
I've played a bit with Hugo, the static web site generator written in golang that has been getting a lot of good press lately. At the suggestion of my colleague Warren Runk, I also experimented with hosting the static files generated by Hugo on Google Cloud Storage (GCS). That way there is no need for launching any instances that would serve those files. You can achieve this by using AWS S3 as well of course.

Notes on GCS setup
You first need to sign up for a Google Cloud Platform (GCP) account. You get a 30-day free trial with a new account. Once you are logged into the Google Cloud console, you need to create a new project. Let's call it my-gcs-hugo-project.

You need to also create a bucket in GCS. If you want to serve your site automatically out of this bucket, you need to give the bucket the same name as your site. Let's assume you call the bucket hugotest.mydomain.com. You will have to verify that you own mydomain.com either by creating a special CNAME in the DNS zone file for mydomain.com pointing to google.com, or by adding a special META tag to the HTML file served at hugotest.mydomain.com (you can achieve the latter by temporarily CNAME-ing hugotest to www.mydomain.com and adding the HEAD tag to the home page for www).

If you need to automate deployments to GCS, it's a good idea to create a GCP Service Account. Click on the 'hamburger' menu in the upper left of the GCP console, then go to Permissions, then Service Accounts. Create a new service account and download its private key in JSON format (the key will be called something like my-gcs-hugo-project-a37b5acd7bc5.json.

Let's say your service account is called my-gcp-service-account1. The account will automatically be assigned an email address similar to my-gcp-service-account1@my-gcs-hugo-project.iam.gserviceaccount.com.

I wanted to be able to deploy the static files generated by Hugo to GCS using Jenkins. So I followed these steps on the Jenkins server as the user running the Jenkins process (user jenkins in my case):

1) Installed the Google Cloud SDK

$ wget https://dl.google.com/dl/cloudsdk/channels/rapid/google-cloud-sdk.tar.gz$ tar xvfz google-cloud-sdk.tar.gz$ cd google-cloud-sdk/$ ./install.sh
- source .bashrc
$ which gcloud

2) Copied the service account's private key my-gcs-hugo-project-a37b5acd7bc5.json to the .ssh directory of the jenkins user.

3) Activated the service account using the gcloud command-line utility (still as user jenkins)

$ gcloud auth activate-service-account --key-file .ssh/my-gcs-hugo-project-a37b5acd7bc5.jsonActivated service account credentials for: [my-gcp-service-account1@my-gcs-hugo-project.iam.gserviceaccount.com]
4) Set the current GCP project to my-gcs-hugo-project

$ gcloud config set project my-gcs-hugo-project
$ gcloud config listYour active configuration is: [default]
[core]account = my-gcp-service-account1@my-gcs-hugo-project.iam.gserviceaccount.comdisable_usage_reporting = True
project = my-gcs-hugo-project
5) Configured GCS via the gsutil command-line utility (this may actually be redundant since we already configured the project with gcloud, but I leave it here in case you encounter issues with using just gcloud)

$ gsutil config -eIt looks like you are trying to run "/var/lib/jenkins/google-cloud-sdk/bin/bootstrapping/gsutil.py config".The "config" command is no longer needed with the Cloud SDK.To authenticate, run: gcloud auth loginReally run this command? (y/N) yBacking up existing config file "/var/lib/jenkins/.boto" to "/var/lib/jenkins/.boto.bak"...This command will create a boto config file at /var/lib/jenkins/.botocontaining your credentials, based on your responses to the followingquestions.What is the full path to your private key file? /var/lib/jenkins/.ssh/my-gcs-hugo-project-a37b5acd7bc5.json
Please navigate your browser to https://cloud.google.com/console#/project,then find the project you will use, and copy the Project ID string from thesecond column. Older projects do not have Project ID strings. For such projects,click the project and then copy the Project Number listed under that project.
What is your project-id? my-gcs-hugo-project
Boto config file "/var/lib/jenkins/.boto" created. If you need to usea proxy to access the Internet please see the instructions in that
6) Added the service account created above as an Owner for the bucket hugotest.mydomain.com

7) Copied a test file from the local file system of the Jenkins server to the bucket hugotest.mydomain.com  (still logged in as user jenkins), then listed all files in the bucket, then removed the test file

$ gsutil cp test.go gs://hugotest.mydomain.com/Copying file://test.go [Content-Type=application/octet-stream]...Uploading   gs://hugotest.mydomain.com/test.go:             951 B/951 B
$ gsutil ls gs://hugotest.mydomain.com/gs://hugotest.mydomain.com/test.go
$ gsutil rm gs://hugotest.mydomain.com/test.goRemoving gs://hugotest.mydomain.com/test.go...
8) Created a Jenkins job for uploading all static files for a given website to GCS
Assuming all these static files are checked in to GitHub, the Jenkins job will first check them out, then do something like this (where TARGET is the value selected from a Jenkins multiple-choice dropdown for this job):
# upload all filee and disable caching (for testing purposes)gsutil -h "Cache-Control:private" cp -r * gs://$BUCKETNAME/
# set read permissions for allUsersfor file in `find . -type f`; do    # remove first dot from file name    file=${file#"."}    gsutil acl ch -u allUsers:R gs://${BUCKETNAME}${file}done
The first gsutil command does a recursive copy (cp -r *) of all files to the bucket. This will preserve the directory structure of the website. For testing purposes, the gsutil command also sets the Cache-Control header on all files to private, which tells browsers not to cache the files.
The second gsutil command is executed for each object in the bucket, and it sets the ACL on that object so that the object has Read (R) permissions for allUsers (by default only owners and other specifically assigned users have Read permissions). This is because we want to serve a public website out of our GCS bucket.
At this point, you should be able to hit hugotest.mydomain.com in a browser and see your static site in all its glory.
Notes on Hugo setup
I've only dabbled in Hugo in the last couple of weeks, so these are very introductory-type notes.
Installing Hugo on OSX and creating a new Hugo site
$ brew update && brew install hugo$ mkdir hugo-sites$ cd hugo-sites$ hugo new site hugotest.mydomain.com$ git clone --recursive https://github.com/spf13/hugoThemes themes$ cd hugotest.mydomain.com$ ln -s ../themes .
At this point you have a skeleton directory structure created by Hugo (via the hugo new site command) under the directory hugotest.mydomain.com:
$ lsarchetypes  config.toml content     data        layouts     static themes
(note that we symlinked the themes directory into the hugotest.mydomain.com directory to avoid duplication)

Configuring your Hugo site and choosing a theme
One file you will need to pay a lot of attention to is the site configuration file config.toml. The default content of this file is deceptively simple:
$ cat config.tomlbaseurl = "http://replace-this-with-your-hugo-site.com/"languageCode = "en-us"title = "My New Hugo Site"
Before you do anything more, you need to decide on a theme for your site. Browse the Hugo Themes page and find something you like. Let's assume you choose the Casper theme. You will need to become familiar with the customizations that the theme offers. Here are some customizations I made in config.toml, going by the examples on the Casper theme web page:
$ cat config.tomlbaseurl = "http://hugotest.mydomain.com/"languageCode = "en-us"title = "My Speedy Test Site"newContentEditor = "vim"
theme = "casper"canonifyurls = true
[params]  description = "Serving static sites at the speed of light"  cover = "images/header.jpg"  logo = "images/mylogo.png"  # set true if you are not proud of using Hugo (true will hide the footer note "Proudly published with HUGO.....")  hideHUGOSupport = false
#  author = "Valère JEANTET"#  authorlocation = "Paris, France"#  authorwebsite = "http://vjeantet.fr"#  bio= "my bio"#  googleAnalyticsUserID = "UA-79101-12"#  # Optional RSS-Link, if not provided it defaults to the standard index.xml#  RSSLink = "http://feeds.feedburner.com/..."#  githubName = "vjeantet"#  twitterName = "vjeantet"  # facebookName = ""  # linkedinName = ""
I left most of the Casper-specific options commented out and only specified a cover image, a logo and a description. 
Creating a new page
If you want blog-style posts to appear on your home page, create a new page with Hugo under a directory called post (some themes want this directory to be named post and others want it posts, so check what the theme expects). 
Let's assume you want to create a page caled hello-world.md (I haven't even mentioned this so far, but Hugo deals by default with Markdown pages, so you will need to brush up a bit on our Markdown skills). You would run:
$ hugo new post/hello-world.md
This creates the post directory under the content directory, creates a file called hello-world.md in content/post, and opens up the file for editing in the editor you specified as the value for newContentEditor in config.toml (vim in my case). The default contents of the md file are specific to the theme you used. For Casper, here is what I get by default:
+++author = ""comments = truedate = "2016-02-12T11:54:32-08:00"draft = falseimage = ""menu = ""share = trueslug = "post-title"tags = ["tag1", "tag2"]title = "hello world"

Now add some content to that file and save it. Note that the draft property is set to false by the Casper theme. Other themes set it to true, in which case it would not be published by Hugo by default. The slug property is set by Casper to "post-title" by default. I changed it to "hello-world". I also changed the tags list to only contain one tag I called "blog".
At this point, you can run the hugo command by itself, and it will take the files it finds under content, static, and its other subdirectories, turn them into html/js/css/font files and save it in a directory called public:
$ hugo0 draft content0 future content1 pages created3 paginator pages created1 tags created0 categories createdin 55 ms
$ find publicpublicpublic/404.htmlpublic/csspublic/css/nav.csspublic/css/screen.csspublic/fontspublic/fonts/example.htmlpublic/fonts/genericons.csspublic/fonts/Genericons.eotpublic/fonts/Genericons.svgpublic/fonts/Genericons.ttfpublic/fonts/Genericons.woffpublic/index.htmlpublic/index.xmlpublic/jspublic/js/index.jspublic/js/jquery.fitvids.jspublic/js/jquery.jspublic/pagepublic/page/1public/page/1/index.htmlpublic/postpublic/post/hello-worldpublic/post/hello-world/index.htmlpublic/post/index.htmlpublic/post/index.xmlpublic/post/pagepublic/post/page/1public/post/page/1/index.htmlpublic/sitemap.xmlpublic/tagspublic/tags/blogpublic/tags/blog/index.htmlpublic/tags/blog/index.xmlpublic/tags/blog/pagepublic/tags/blog/page/1public/tags/blog/page/1/index.html
That's quite a number of files and directories created by hugo. Most of it is boilerplate coming from the theme. Our hello-world.md file was turned into a directory called hello-world under public/post, with an index.html file dropped in it. Note that the Casper theme names the hello-world directory after the slug property in the hello-world.md file.
Serving the site locally with Hugo
Hugo makes it very easy to check your site locally. Just run

$ hugo server
0 draft content
0 future content
1 pages created
3 paginator pages created
1 tags created
0 categories created
in 35 ms
Watching for changes in /Users/grig.gheorghiu/mycode/hugo-sites/hugotest.mydomain.com/{data,content,layouts,static,themes}
Serving pages from memory
Web Server is available at http://localhost:1313/ (bind address
Press Ctrl+C to stop

Now if you browse to http://localhost:1313 you should see something similar to this:

Not bad for a few minutes of work.

For other types of content, such as static pages not displayed on the home page, you can create Markdown files in a pages directory:

$ hugo new pages/static1.md
author = ""
comments = true
date = "2016-02-12T12:24:26-08:00"
draft = false
image = ""
menu = "main"
share = true
slug = "static1"
tags = ["tag1", "tag2"]
title = "static1"


Static page 1.

Note that the menu property value is "main" in this case. This tells the Casper theme to create a link to this page in the main drop-down menu available on the home page.

If you run hugo server again, you should see something the menu available in the upper right corner, and a link to static1 when you click on the menu:

To deploy your site to GCS, S3 or regular servers, you need to upload the files and directories under the public directory. It's that simple.

I'll stop here with my Hugo notes. DigitalOcean has a great tutorial on installing and running Hugo on Ubuntu 14.04.

Some notes on Ansible playbooks and roles

Thu, 02/11/2016 - 23:09
Some quick notes I jotted down while documenting our Ansible setup. Maybe they will be helpful for people new to Ansible.
Ansible playbooks and roles
Playbooks are YAML files that specify which roles are applied to hosts of certain type.
Example: api-servers.yml
$ cat api-servers.yml---
- hosts: api  sudo: yes  roles:    - base    - tuning    - postfix    - monitoring    - nginx    - api    - logstash-forwarder
This says that for each host in the api group we will run tasks defined in the roles listed above.
Example of a role: the base role is one that (in our case) is applied to all hosts. Here is its directory/file structure:
An Ansible role has the following important sub-directories:
defaults - contains the main.yml file which defines default values for variables used throughout other role files; note that the role’s files are checked in to GitHub, so these values shouldn’t contain secrets such as passwords, API keys etc. For those types of variables, use group_vars or host_vars files which will be discussed below.
files - contains static files that are copied over by ansible tasks to remote hosts
handlers - contains the main.yml file which defines actions such as stopping/starting/restarting services such as nginx, rsyslog etc.
meta - metadata about the role; things like author, description etc.
tasks - the meat and potatoes of ansible, contains one or more files that specify the actions to be taken on the host that is being configured; the main.yml file contains all the other files that get executed
Here are 2 examples of task files, one for configuring rsyslog to send logs to Papertrail and the other for installing the newrelic agent:
$ cat tasks/papertrail.yml- name: copy papertrail pem certificate file to /etc  copy: >    src=rsyslog/{{item}}    dest=/etc/{{item}}  with_items:    - papertrail-bundle.pem
- name: copy rsyslog config files for papertrail integration  copy: >    src=rsyslog/{{item}}    dest=/etc/rsyslog.d/{{item}}  with_items:    - 50-default.conf    - 60-papertrail.conf  notify:     - restart rsyslog
$ cat tasks/newrelic.yml- name: copy newrelic debian package  copy: >    src=newrelic/{{newrelic_deb_pkg}}    dest=/opt/{{newrelic_deb_pkg}}
- name: install newrelic debian package  apt: deb=/opt/{{newrelic_deb_pkg}}
- name: configure newrelic with proper license key  template: >    src=nrsysmond.cfg.j2    dest=/etc/newrelic/nrsysmond.cfg    owner=newrelic    group=newrelic    mode=0640  notify:     - restart newrelic
templates - contains Jinja2 templates with variables that get their values from defaults/main.yml or from group_vars or host_vars files. One special variable that we use (and is not defined in these files, but instead is predefined by Ansible) is inventory_hostname which points to the hostname of the target being configured. For example, here is the template for a hostname file which will be dropped into /etc/hostname on the target:
$ cat roles/base/templates/hostname.j2{{ inventory_hostname }}
Once you have a playbook and a role, there are a few more files you need to take care of:
  • hosts/myhosts - this is an INI-type file which defines groups of hosts. For example the following snippet of this file defines 2 groups called api and magento.

[api]api01 ansible_ssh_host=api01.mydomain.coapi02 ansible_ssh_host=api02.mydomain.co
[magento]mgto ansible_ssh_host=mgto.mydomain.co
The api-servers.yml playbook file referenced at the beginning of this document sets the hosts variable to the api group, so all Ansible tasks will get run against the hosts included in that group. In the hosts/myhosts file above, these hosts are api01 and api02.
  • group_vars/somegroupname - this is where variables with ‘secret’ values get defined for a specific group called somegroupname. The group_vars directory is not checked into GitHub. somegroupname needs to exactly correspond to the group defined in hosts/myhosts.

$ cat group_vars/apises_smtp_endpoint: email-smtp.us-west-2.amazonaws.comses_smtp_port: 587ses_smtp_username: some_usernameses_smtp_password: some_passworddatadog_api_key: some_api_key. . . other variables (DB credentials etc)

  • host_vars/somehostname - this is where variables with ‘secret’ values get defined for a specific host called somehostname. The host_vars directory is not checked into GitHub. somehostname needs to exactly correspond to a host defined in hosts/myhosts.

$ cat host_vars/api02insert_sample_data: false
This overrides the insert_sample_data variable and sets it to false only for the host called api02. This could also be used for differentiating between a DB master and slave for example.
Tying it all together
First you need to have ansible installed on your local machine. I used:
$ pip install ansible
To execute a playbook for a given hosts file against all api server, you would run:
$ ansible-playbook -i hosts/myhosts api-servers.yml
The name that ties together the hosts/myhosts file, the api-servers.yml file and the group_vars/groupname file is in this case api.
You need to make sure you have the desired values for that group in these 3 files:
  • hosts/myhosts: make sure you have the desired hosts under the [api] group
  • api-server.yml: make sure you have the desired roles for hosts in the api group
  • group_vars/api: make sure you have the desired values for variables that will be applied to the hosts in the api group
Launching a new api instance in EC2I blogged about this here.Updating an existing api instance
Make sure the instance hostname is the only hostname in the [api] group in the hosts/myhosts file. Then run:
$ ansible-playbook -i hosts/myhosts api-servers.yml

Setting up a mailinator-like test mail server with postfix and MailHog

Wed, 02/10/2016 - 01:09
The purpose of this exercise is to set up a mailinator-style mail server under our control. If you haven't used mailinator, it's a free service which provides an easy way to test sending email to random recipients. If you send mail to somebody189@mailinator.com and then go to
https://mailinator.com/inbox.jsp?to=somebody189, you will see a mailbox associated with that user, and any incoming email messages destined for that user. It's a handy way to also sign up for services that send confirmation emails.

I have been playing with MailHog, which is a mail server written in Golang for exactly the same purpose as mailinator. In fact, MailHog can happily intercept ANY recipient at ANY mail domain, provided it is set up properly. In my case, I didn't want to expose MailHog on port 25 externally, because that is a recipe for spam. Instead, I wanted to set up a regular postfix server for mydomain.com, then set up a catch-all user which will receive mail destined for anyuser@maildomain.com, and finally send all that mail to MailHog via procmail. Quite a few moving parts, but I got it to work and I am hastening to jot down my notes before I forget how I did it.
The nice thing about MailHog is that it provides a Web UI where you can eyeball the email messages you sent, including in raw format, and it also provides a JSON API which allows you to list messages and search for specific terms within messages. This last feature is very useful for end-to-end testing of your application's email sending capabilities.

I set up everything on a Google Cloud Engine instance running Ubuntu 14.04.
  • instance name: mailhog-mydomain-com
  • DNS/IP: mailhog.mydomain.com /
Install go 1.5.3 from source

First install the binaries for go 1.4, then compile go 1.5.
# apt-get update
# apt-get install build-essential git mercurial bazaar unzip
# cd /root
# wget https://storage.googleapis.com/golang/go1.4.3.linux-amd64.tar.gz
# tar xvfz go1.4.3.linux-amd64.tar.gz
# mv go go1.4
# git clone https://go.googlesource.com/go
# cd go
# git checkout go1.5.3
# cd src
# ./all.bash

# mkdir /opt/gocode Edit /root/.bashrc and add:
export GOPATH=/opt/gocode
export PATH=$PATH:/root/go/bin:$GOPATH/bin

then source /root/.bashrc.

# go version
go version go1.5.3 linux/amd64

Set up postfix

Install postfix and mailutils

# apt-get install postfix mailutils

- specified System mail name as mydomain.com

Set up catch-all mail user (local Unix user)

# adduser catchall
Edit /etc/aliases and replace content with the lines below.

# cat /etc/aliases

# See man 5 aliases for format
mailer-daemon: postmaster
postmaster: root
nobody: root
hostmaster: root
usenet: root
news: root
webmaster: root
www: root
ftp: root
abuse: root
root: catchall

# newaliasesEdit /etc/postfix/main.cf and add lines:

luser_relay = catchall
local_recipient_maps =

Restart postfix:

# service postfix restart

Use Google Cloud Platform Web UI to add firewall rule called allow-smtp for the “default” network associated with the mailhog-mydomain-com instance. The rule allows incoming traffic from everywhere to port tcp:25.
Set up DNS

Add A record for mailhog.mydomain.com pointing to

Add MX record for catchallpayments.com pointing to mailhog.mydomain.com.

Test the incoming mail setup

Send mail to catchall@mydomain.com from gmail.

Run mail utility on GCE instance as user catchall:

catchall@mailhog-mydomain-com:~$ mail

"/var/mail/catchall": 1 message 1 new

>N 1 Some Body     Tue Feb 9 00:23 52/2595 test from gmail

? 1

Return-Path: <some.body@gmail.com>
X-Original-To: catchall@mydomain.com
Delivered-To: catchall@mydomain.com

Send mail to random user which doesn’t exist locally catchall333@mydomain.com and verify that user catchall receives it:

catchall@mailhog-mydomain-com:~$ mail

"/var/mail/catchall": 1 message 1 new

>N 1 Some Body     Tue Feb 9 18:32 52/2702 test 3

? 1

Return-Path: <some.body@gmail.com>
X-Original-To: catchall333@mydomain.com
Delivered-To: catchall333@mydomain.com

Install and configure MailHog

Get MailHog

# go get github.com/mailhog/MailHog - this will drop several binaries in /opt/gocode/bin, including mhsendmail and MailHog (for reference, the code for mhsendmail is here)

# which MailHog

# which mhsendmail
Configure HTTP basic authentication

MailHog supports HTTP basic authentication via a file similar to .htpasswd. It uses bcrypt for password (see more details here). The MailHog binary can also generate passwords with bcrypt.

I created a password with MailHog:

# MailHog bcrypt somepassword

Then I created a file called .mailhogrc in /root and specified a user called mailhogapi with the password generated above:

# cat /root/.mailhogrc

Create upstart init file for MailHog

I specified the port MailHog listens on (I chose the same port as its default which is 1025) and the filed used for HTTP basic auth.

# cat /etc/init/mailhog.conf
# MailHog Test SMTP Server (Upstart unit)
description "MailHog Test SMTP Server"
start on (local-filesystems and net-device-up IFACE!=lo)
stop on runlevel [06]

exec /opt/gocode/bin/MailHog -smtp-bind-addr -auth-file /root/.mailhogrc
respawn limit 10 10
kill timeout 10
See more command line options for MailHog in this doc.

Start mailhog service

# start mailhog
mailhog start/running, process 25458

# ps -efd|grep Mail
root 7782 1 0 22:04 ? 00:00:00 /opt/gocode/bin/MailHog -smtp-bind-addr -auth-file /root/.mailhogrc

At this point MailHog is listening for SMTP messages on port 1025. It also provides a Web UI on default UI port 8025 and a JSON API also on port 8025.

Install procmail and configure it for user catchall

This is so messages addressed to user catchall (which again is our catch-all user) can get processed by a script via procmail.

# apt-get install procmail

Add this line to /etc/postfix/main.cf:

mailbox_command = /usr/bin/procmail -a "$EXTENSION" DEFAULT=$HOME/Maildir/ MAILDIR=$HOME/Maildir

(this will send all messages to procmail instead of individual user mailboxes)

Then su as user catchall and create .procmailrc file in its home directory:

catchall@mailhog-mydomain-com:~$ cat .procmailrc
| /opt/gocode/bin/mhsendmail --smtp-addr="localhost:1025"

This tells procmail to pipe the incoming mail message to mhsendmail, which will format it properly and pass it to port 1025, where MailHog is listening.

Test end-to-end

Use Google Cloud Platform Web UI to add firewall rule called allow-mailhog-ui for the “default” network associated with the mailhog-mydomain-com instance. The rule allows incoming traffic from everywhere to tcp:8025 (where the MailHog UI server listens). It’s OK to allow traffic to port 8025 from everywhere because it is protected via HTTP basic auth.

The MailHog UI is at http://mailhog.mydomain.com:8025

Any email sent to xyz@mydomain.com should appear in the MailHog Inbox.

By default, MailHog stores incoming messages in memory. Restarting MailHog (via ‘restart mailhog’ at the cmdline) will remove all messages.

MailHog also supports MongoDB as a persistent storage backend for incoming messages (exercise left to the reader.)

Use the MailHog JSON API to verify messages

List all messages:

$ curl --user 'mailhogapi:somepassword' -H "Content-Type: application/json" "http://mailhog.mydomain.com:8025/api/v2/messages"

Search messages for specific terms (for example for the recipient’s email):

$ curl -i --user 'mailhogapi:somepassword' -H "Content-Type: application/json" "http://mailhog.mydomain.com:8025/api/v2/search?kind=containing&query=test1%40mydomain.com"

See the MailHog API v2 docs here.

That's it, hope it makes your email sending testing more fun!

Setting up Jenkins to run headless Selenium tests in Docker containers

Sat, 02/06/2016 - 01:40
This is the third post in a series on running headless Selenium WebDriver tests. Here are the first two posts:
  1. Running Selenium WebDriver tests using Firefox headless mode on Ubuntu
  2. Running headless Selenium WebDriver tests in Docker containers
In this post I will show how to add the final piece to this workflow, namely how to fully automate the execution of Selenium-based WebDriver tests running Firefox in headless mode in Docker containers. I will use Jenkins for this example, but the same applies to other continuous integration systems.
1) Install docker-engine on the server running Jenkins (I covered this in my post #2 above)
2) Add the jenkins user to the docker group so that Jenkins can run the docker command-line tool in order to communicate with the docker daemon. Remember to restart Jenkins after doing this.
3) Go through the rest of the workflow in my post above ("Running headless Selenium WebDriver tests in Docker containers") and make sure you can run all the commands in that post from the command line of the server running Jenkins.
4) Create a directory structure for your Selenium WebDriver tests (mine are written in Python). 
I have a directory called selenium-docker which contains a directory called tests, under which I put all my Python WebDriver tests named sel_wd_*.py. I also  have a simple shell script I named run_selenium_tests.sh which does the following:
TARGET=$1 # e.g. someotherdomain.example.com (if not specified, the default is somedomain.example.com)
for f in `ls tests/sel_wd_*.py`; do    echo Running $f against $TARGET    python $f $TARGETdone
My selenium-docker directory also contains the xvfb.init file I need for starting up Xvfb in the container, and finally it contains this Dockerfile:
FROM ubuntu:trusty
RUN echo "deb http://ppa.launchpad.net/mozillateam/firefox-next/ubuntu trusty main" > /etc/apt/sources.list.d//mozillateam-firefox-next-trusty.listRUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys CE49EC21RUN apt-get updateRUN apt-get install -y firefox xvfb python-pipRUN pip install seleniumRUN mkdir -p /root/selenium/tests
ADD tests /root/selenium/testsADD run_all_selenium_tests.sh /root/selenium
ADD xvfb.init /etc/init.d/xvfbRUN chmod +x /etc/init.d/xvfbRUN update-rc.d xvfb defaults
ENV TARGET=somedomain.example.com
CMD (service xvfb start; export DISPLAY=:10; cd /root/selenium; ./run_all_selenium_tests.sh $TARGET)
I explained what this Dockerfile achieves in the 2nd post referenced above. The ADD instructions will copy all the files in the tests directory to the directory called /root/selenium/tests, and will copy run_all_selenium_tests.sh to /root/selenium. The ENV variable TARGET represents the URL against which we want to run our Selenium tests. It is set by default to somedomain.example.com, and is used as the first argument when running run_all_selenium_tests.sh in the CMD instruction.
At this point, I checked in the selenium-docker directory and all files and directories under it into a Github repository I will call 'devops'.
5) Create a new Jenkins project (I usually create a New Item and copy it from an existing project).
I specified that the build is parameterized and I indicated a choice parameter called TARGET_HOST with a few host/domain names that I want to test. I also specified Git as the Source Code Management type, and I indicated the URL of the devops repository on Github. Most of the action of course happens in the Jenkins build step, which in my case is of type "Execute shell". Here it is:
set +e
cd $WORKSPACE/selenium-docker
# build the image out of the Dockerfile in the current directory/usr/bin/docker build -t $IMAGE_NAME .
# run a container based on the imageCONTAINER_ID=`/usr/bin/docker run -d -e "TARGET=$TARGET_HOST" $IMAGE_NAME`
echo CONTAINER_ID=$CONTAINER_ID  # while the container is still running, sleep and check logs; repeat every 40 secwhile [ $? -eq 0 ];do  sleep 40  /usr/bin/docker logs $CONTAINER_ID  /usr/bin/docker ps | grep $IMAGE_NAMEdone
# docker logs sends errors to stderr so we need to save its output to a file first/usr/bin/docker logs $CONTAINER_ID > d.out 2>&1
# remove the container so they don't keep accumulatingdocker rm $CONTAINER_ID
# mark jenkins build as failed if log output contains FAILEDgrep "FAILED" d.out
if [[ $? -eq 0 ]]; then    rm d.out   exit 1else  rm d.out  exit 0fi
Some notes:
  • it is recommended that you specify #!/bin/bash as the 1st line of your script, to make sure that bash is the shell that is being used
  • use set +e if you want the Jenkins shell script to continue after hitting a non-zero return code (the default behavior is for the script to stop on the first line it encounters an error and for the build to be marked as failed; subsequent lines won't get executed, resulting in much pulling of hair)
  • the Jenkins script will build a new image every time it runs, so that we make sure we have updated Selenium scripts in place
  • when running the container via docker run, we specify -e "TARGET=$TARGET_HOST" as an extra command line argument. This will override the ENV variable named TARGET in the Dockerfile with the value received from the Jenkins multiple choice dropdown for TARGET_HOST
  • the main part of the shell script stays in a while loop that checks for the return code of "/usr/bin/docker ps | grep $IMAGE_NAME". This is so we wait for all the Selenium tests to finish, at which point docker ps will not show the container running anymore (you can still see the container by running docker ps -a)
  • once the tests finish, we save the stdout and stderr of the docker logs command for our container to a file (this is so we capture both stdout and stderr; at first I tried something like docker logs $CONTAINER_ID | grep FAILED but this was never successful, because it was grep-ing against stdout, and errors are sent to stderr)
  • we grep the file (d.out) for the string FAILED and if we find it, we exit with code 1, i.e. unsuccessful as far as Jenkins is concerned. If we don't find it, we exit successfully with code 0.