Warning: Table './devblogsdb/cache_page' is marked as crashed and last (automatic?) repair failed query: SELECT data, created, headers, expire, serialized FROM cache_page WHERE cid = 'http://www.softdevblogs.com/?q=aggregator/sources/17' in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc on line 135

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 729

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 730

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 731

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 732
Software Development Blogs: Programming, Software Testing, Agile, Project Management
Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Agile Testing - Grig Gheorghiu
warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/common.inc on line 153.
Syndicate content
Did anybody say webscale?
Updated: 9 hours 59 min ago

Using JMESPath queries with the AWS CLI

Wed, 07/13/2016 - 22:05
The AWS CLI, based on the boto3 Python library, is the recommended way of automating interactions with AWS. In this post I'll show some examples of more advanced AWS CLI usage using the query mechanism based on the JMESPath JSON query language.

Installing the AWS CLI tools is straightforward. On Ubuntu via apt-get:

# apt-get install awscli

Or via pip:

# apt-get install python-pip
# pip install awscli

The next step is to configure awscli by specifying the AWS Access Key ID and AWS Secret Access Key, as well as the default region and output format:

# aws configureAWS Access Key ID: your-aws-access-key-idAWS Secret Access Key: your-aws-secret-access-keyDefault region name [us-west-2]: us-west-2
Default output format [None]: json
The configure command creates a ~/.aws directory containing two files: config and credentials.
You can specify more than one pair of AWS keys by creating profiles in these files. For example, in ~/.aws/credentials you can have:
[profile profile1]AWS_ACCESS_KEY_ID=key1AWS_SECRET_ACCESS_KEY=secretkey1
[profile profile2]AWS_ACCESS_KEY_ID=key2AWS_SECRET_ACCESS_KEY=secretkey2
In ~/.aws/config you can have:[profile profile1]region = us-west-2
[profile profile2]region = us-east-1
You can specify a given profile when you run awscli:
# awscli --profile profile1
Let's assume you want to write a script using awscli that deletes EBS snapshots older than N days. Let's go through this one step at a time.
Here's how you can list all snapshots owned by you:
# aws ec2 describe-snapshots --profile profile1 --owner-id YOUR_AWS_ACCT_NUMBER --query "Snapshots[]"
Note the use of the --query option. It takes a parameter representing a JMESPath JSON query string. It's not trivial to figure out how to build these query strings, and I advise you to spend some time reading over the JMESPath tutorial and JMESPath examples.
In the example above, the query string is simply "Snapshots[]", which represents all the snapshots that are present in the AWS account associated with the profile profile1. The default output in our case is JSON, but you can specify --output text at the aws command line if you want to see each snapshot on its own line of text.
Let's assume that when you create the snapshots, you specify a description with contains PROD or STAGE for EBS volumes attached to production and stage EC2 instances respectively. If you want to only display snapshots containing the string PROD, you would do:
# aws ec2 describe-snapshots --profile profile1 --owner-id YOUR_AWS_ACCT_NUMBER--query "Snapshots[?contains(Description, \`PROD\`) == \`true\`]" --output text
The Snapshots[] array now contains a condition represented by the question mark ?. The condition uses the contains() function included in the JMESPath specification, and is applied against the Description field of each object in the Snapshots[] array, verifying that it contains the string PROD.  Note the use of backquotes surrounding the strings PROD and true in the condition. I spent some quality time troubleshooting my queries when I used single or double quotes with no avail. The backquotes also need to be escaped so that the shell doesn't interpret them as commands to be executed.
To restrict the PROD snapshots even further, to the ones older than say 7 days ago, you can do something like this:
DAYS=7TARGET_DATE=`date --date="$DAYS day ago" +%Y-%m-%d`
# aws ec2 describe-snapshots --profile profile1 --owner-id YOUR_AWS_ACCT_NUMBER--query "Snapshots[?contains(Description, \`PROD\`) == \`true\`]|[?StartTime < \`$TARGET_DATE\`]" --output text
Here I used the StartTime field of the objects in the Snapshots[] array and compared it against the target date. In this case, string comparison is good enough for the query to work.
In all the examples above, the aws command returned a subset of the Snapshots[] array and displayed all fields for each object in the array. If you wanted to display specific fields, let's say the ID, the start time and the description of each snapshot, you would run:
# aws ec2 describe-snapshots --profile profile1 --owner-id YOUR_AWS_ACCT_NUMBER--query "Snapshots[?contains(Description, \`PROD\`) == \`true\`]|[?StartTime < \`$TARGET_DATE\`].[SnapshotId,StartTime,Description]" --output text
To delete old snapshots, you can use the aws ec2 delete-snapshot command, which needs a snapshot ID as a parameter. You could use the command above to list only the SnapshotId for snapshots older than N days, then for each of these IDs, run something like this:
# aws ec2 delete-snapshot --profile profile1 --snapshot-id $id
All this is well and good when you run these commands interactively at the shell. However, I had no luck running them out of cron. The backquotes resulted in boto3 syntax errors. I had to do it the hard way, by listing all snapshots first, then going all in with sed and awk:
aws ec2 describe-snapshots --profile profile1 --owner-id YOUR_AWS_ACCT_NUMBER --output=text --query "Snapshots[].[SnapshotId,StartTime,Description]"  > $TMP_SNAPS
DAYS=7TARGET_DATE=`date --date="$DAYS day ago" +%Y-%m-%d`
cat $TMP_SNAPS | grep PROD | sed 's/T[0-9][0-9]:[0-9][0-9]:[0-9][0-9].000Z//' | awk -v target_date="$TARGET_DATE" '{if ($2 < target_date){print}}' > $TMP_PROD_SNAPS
echo PRODUCTION SNAPSHOTS OLDER THAN $DAYS DAYScat $TMP_PROD_SNAPS
for sid in `awk '{print $1}' $TMP_PROD_SNAPS` ; do echo Deleting PROD snapshot $sid aws ec2 delete-snapshot --profile $PROFILE --region $REGION --snapshot-id $siddone
Ugly, but it works out of cron. Hope it helps somebody out there.

July 14th 2016: I initially forgot to include this very good blog post from Joseph Lawson on advanced JMESPath usage with the AWS CLI.

More tips and tricks for running Gatling in Docker containers

Fri, 07/01/2016 - 18:11
This post is a continuation of my previous one on "Running Gatling tests in Docker containers via Jenkins". As I continued to set up Jenkins jobs to run Gatling tests, I found the need to separate those tests for different environments - development, staging and production. The initial example I showed contained a single setup, which is not suitable for multiple environments.

Here is my updated Gatling directory structure

gatling
gatling/conf
gatling/conf/production
gatling/conf/production/gatling.conf
gatling/conf/staging
gatling/conf/staging/gatling.conf
gatling/Dockerfile
gatling/results
gatling/user-files
gatling/user-files/data
gatling/user-files/data/production-urls.csv
gatling/user-files/data/staging-urls.csv
gatling/user-files/simulations
gatling/user-files/simulations/development
gatling/user-files/simulations/production
gatling/user-files/simulations/production/Simulation.scala
gatling/user-files/simulations/staging
gatling/user-files/simulations/staging/Simulation.scala

Note that I created a separate directory under simulations for each environment (development, staging, production), each with its own simulation files.

I also created a data directory under user-files, because that is the default location for CSV files used by Gatling feeders.

Most importantly, I created a separate configuration directory (staging, production) under gatling/conf, each directory containing its own customized gatling.conf file. I started by copying the gatling-defaults.conf file from GitHub to gatling/conf/staging/gatling.conf and gatling/conf/production/gatling.conf respectively.

Here is what I customized in staging/gatling.conf:

mute = true # When set to true, don't ask for simulation name nor run description
simulations = user-files/simulations/staging

I customized production/gatling.conf in a similar way:

mute = true # When set to true, don't ask for simulation name nor run description
simulations = user-files/simulations/production

Setting mute to true is important because without it, running Gatling in a Docker container was segfaulting while waiting for user input for the simulation ID:

Select simulation id (default is 'gatlingsimulation'). Accepted characters are a-z, A-Z, 0-9, - and _ Exception in thread "main" java.lang.NullPointerException at io.gatling.app.Selection$Selector.loop$1(Selection.scala:127) at io.gatling.app.Selection$Selector.askSimulationId(Selection.scala:135) at io.gatling.app.Selection$Selector.selection(Selection.scala:50) at io.gatling.app.Selection$.apply(Selection.scala:33) at io.gatling.app.Gatling.runIfNecessary(Gatling.scala:75) at io.gatling.app.Gatling.start(Gatling.scala:65) at io.gatling.app.Gatling$.start(Gatling.scala:57) at io.gatling.app.Gatling$.fromArgs(Gatling.scala:49) at io.gatling.app.Gatling$.main(Gatling.scala:43) at io.gatling.app.Gatling.main(Gatling.scala)

The other customization was to point the simulations attribute to the specific staging or production sub-directories.
Since the CSV files containing URLs to be load tested are also environment-specific, I modified the Simulation.scala files to take this into account. I also added 2 JAVA_OPTS variables that can be passed at runtime for HTTP basic authentication. Here is the new Crawl object (compare with the one from my previous post):
object Crawl {  val feeder = csv("staging-urls.csv").random
  val userName = System.getProperty("username")  val userPass = System.getProperty("password")
  val crawl = exec(feed(feeder)    .exec(http("${loc}")    .get("${loc}").basicAuth(userName, userPass)    ))}
One more thing is needed: to make Gatling use a specific configuration file instead of its default one, which is conf/gatling.conf. To do that, I set GATLING_CONF as an ENV variable in the Dockerfile, so it can be passed as a 'docker run' command line parameter. Here is the Dockerfile:
# Gatling is a highly capable load testing tool.## Documentation: http://gatling.io/docs/2.2.2/# Cheat sheet: http://gatling.io/#/cheat-sheet/2.2.2
FROM java:8-jdk-alpine
MAINTAINER Denis Vazhenin
# working directory for gatlingWORKDIR /opt
# gating versionENV GATLING_VERSION 2.2.2
# create directory for gatling installRUN mkdir -p gatling
# install gatlingRUN apk add --update wget && \  mkdir -p /tmp/downloads && \  wget -q -O /tmp/downloads/gatling-$GATLING_VERSION.zip \  https://repo1.maven.org/maven2/io/gatling/highcharts/gatling-charts-highcharts-bundle/$GATLING_VERSION/gatling-charts-highcharts-bundle-$GATLING_VERSION-bundle.zip && \  mkdir -p /tmp/archive && cd /tmp/archive && \  unzip /tmp/downloads/gatling-$GATLING_VERSION.zip && \  mv /tmp/archive/gatling-charts-highcharts-bundle-$GATLING_VERSION/* /opt/gatling/
# change context to gatling directoryWORKDIR  /opt/gatling
# set directories below to be mountable from hostVOLUME ["/opt/gatling/conf", "/opt/gatling/results", "/opt/gatling/user-files"]
# set environment variablesENV PATH /opt/gatling/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binENV GATLING_HOME /opt/gatlingENV GATLING_CONF /opt/gatling/confENV JAVA_OPTS ""
ENTRYPOINT ["gatling.sh"]
Finally, here is how I invoke 'docker run' to tie everything together:
docker run --rm -v ${WORKSPACE}/gatling/conf:/opt/gatling/conf -v ${WORKSPACE}/gatling/user-files:/opt/gatling/user-files -v ${WORKSPACE}/gatling/results:/opt/gatling/results -e GATLING_CONF="/opt/gatling/conf/staging" -e JAVA_OPTS="-Dusers=$USERS -Dduration=$DURATION -Dusername=myusername -Dpassword=mypass" /PATH/TO/DOCKER/REGISTRY/gatling
Note the GATLING_CONF parameter passed with -e with the value of /opt/gatling/conf/staging. Also note the username and password JAVA_OPTS parameters.

Happy load testing!

Running Gatling load tests in Docker containers via Jenkins

Wed, 06/29/2016 - 00:16
Gatling is a modern load testing tool written in Scala. As part of the Jenkins setup I am in charge of, I wanted to run load tests using Gatling against a collection of pages for a given website. Here are my notes on how I managed to do this.

Running Gatling as a Docker container locally

There is a Docker image already available on DockerHub, so you can simply pull down the image locally:


$ docker pull denvazh/gatling:2.2.2
Instructions on how to run a container based on this image are available on GitHub:
$ docker run -it --rm -v /home/core/gatling/conf:/opt/gatling/conf \-v /home/core/gatling/user-files:/opt/gatling/user-files \-v /home/core/gatling/results:/opt/gatling/results \ denvazh/gatling:2.2.2
Based on these instructions, I created a local directory called gatling, and under it I created 3 sub-directories: conf, results and user-files. I left the conf and results directories empty, and under user-files I created a simulations directory containing a Gatling load test scenario written in Scala. I also created a file in the user-files directory called urls.csv, containing a header named loc and a URL per line for each page that I want to load test.
Assuming the current directory is gatling, here are examples of these files:
$ cat user-files/urls.csvlochttps://my.website.comhttps://my.website.com/category1https://my.website.com/category2/product3
$ cat user-files/simulations/Simulation.scala
package my.gatling.simulation
import io.gatling.core.Predef._import io.gatling.http.Predef._import scala.concurrent.duration._
class GatlingSimulation extends Simulation {
  val httpConf = http    .baseURL("http://127.0.0.1")    .acceptHeader("text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")    .doNotTrackHeader("1")    .acceptLanguageHeader("en-US,en;q=0.5")    .acceptEncodingHeader("gzip, deflate")    .userAgentHeader("Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0")
  val scn1 = scenario("Scenario1")    .exec(Crawl.crawl)
  val userCount = Integer.getInteger("users", 1)  val durationInSeconds  = java.lang.Long.getLong("duration", 10L)  setUp(    scn1.inject(rampUsers(userCount) over (durationInSeconds seconds))  ).protocols(httpConf)}
object Crawl {
  val feeder = csv("/opt/gatling/user-files/urls.csv").random
  val crawl = exec(feed(feeder)    .exec(http("${loc}")    .get("${loc}")    ))}

I won't go through the different ways of writing Gatling load tests scenarios here. There are good instructions on the Gatling website -- see the Quickstart and the Advanced Tutorial. What the scenario above does is it reads the file urls.csv and randomly picks a URL from it, then runs a load test against that URL.
I do want to point out 2 variables in the above script:

  val userCount = Integer.getInteger("users", 1)  val durationInSeconds  = java.lang.Long.getLong("duration", 10L)
These variables specify the max number of users we want to ramp up to, and the duration of the ramp-up. They are used in the inject call:

scn1.inject(rampUsers(userCount) over (durationInSeconds seconds))

The special thing about these 2 variables is that they are read from JAVA_OPTS by Gatling. So if you have a -Dusers Java option and a -Dduration Java option, Gatling will know how to read them and how to set the userCount and durationInSeconds variables accordingly. This is a good thing, because it allows you to specify those numbers outside of Gatling, without hardcoding them in your simulation script. Here is more info on passing parameters via the command line to Gatling.

While pulling the Gatling docker image and running it is the simplest way to run Gatling, I prefer to understand what's going on in that image. I started off by getting the Dockerfile from GitHub:

$ cat Dockerfile
# Gatling is a highly capable load testing tool.## Documentation: http://gatling.io/docs/2.2.2/# Cheat sheet: http://gatling.io/#/cheat-sheet/2.2.2
FROM java:8-jdk-alpine
MAINTAINER Denis Vazhenin
# working directory for gatlingWORKDIR /opt
# gating versionENV GATLING_VERSION 2.2.2
# create directory for gatling installRUN mkdir -p gatling
# install gatlingRUN apk add --update wget && \  mkdir -p /tmp/downloads && \  wget -q -O /tmp/downloads/gatling-$GATLING_VERSION.zip \  https://repo1.maven.org/maven2/io/gatling/highcharts/gatling-charts-highcharts-bundle/$GATLING_VERSION/gatling-charts-highcharts-bundle-$GATLING_VERSION-bundle.zip && \  mkdir -p /tmp/archive && cd /tmp/archive && \  unzip /tmp/downloads/gatling-$GATLING_VERSION.zip && \  mv /tmp/archive/gatling-charts-highcharts-bundle-$GATLING_VERSION/* /opt/gatling/
# change context to gatling directoryWORKDIR  /opt/gatling
# set directories below to be mountable from hostVOLUME ["/opt/gatling/conf", "/opt/gatling/results", "/opt/gatling/user-files"]
# set environment variablesENV PATH /opt/gatling/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binENV GATLING_HOME /opt/gatling
ENTRYPOINT ["gatling.sh"]
I then added a way to pass JAVA_OPTS via an environment variable. I added this line after the ENV GATLING_HOME line:
ENV JAVA_OPTS ""
I dropped this Dockerfile in my gatling directory, then built a local Docker image off of it:
$ docker build -t gatling:local .
I  then invoked 'docker run' to launch a container based on this image, using the csv and simulation files from above. The current directory is still gatling.

$ docker run --rm -v `pwd`/conf:/opt/gatling/conf -v `pwd`/user-files:/opt/gatling/user-files -v `pwd`/results:/opt/gatling/results -e JAVA_OPTS="-Dusers=10 -Dduration=60" gatling:local -s MySimulationName

Note the -s flag which denotes a simulation name (which can be any string you want). If you don't specify this flag, the gatling.sh script which is the ENTRYPOINT in the container will wait for some user input and you will not be able to fully automate your load test.

Another thing to note is the use of JAVA_OPTS. In the example above, I pass -Dusers=10 and   -Dduration=60 as the two JAVA_OPTS parameters. The JAVA_OPTS variable itself is passed to 'docker run' via the -e option, which tells Docker to replace the default value for ENV JAVA_OPTS (which is "") with the value passed with -e.

Running Gatling as a Docker container from Jenkins

Once you have a working Gatling container locally, you can upload the Docker image built above to a private Docker registry. I used a private EC2 Container Registry (ECR).  

I also added the gatling directory and its sub-directories to a GitHub repository called devops.

In Jenkins, I created a new "Freestyle project" job with the following properties:

  • Parameterized build with 2 string parameters: USERS (default value 10) and DURATION in seconds (default value 60)
  • Git repository - add URL and credentials for the devops repository which contains the gatling files
  • An "Execute shell" build command similar to this one:
docker run --rm -v ${WORKSPACE}/gatling/conf:/opt/gatling/conf -v ${WORKSPACE}/gatling/user-files:/opt/gatling/user-files -v ${WORKSPACE}/gatling/results:/opt/gatling/results -e JAVA_OPTS="-Dusers=$USERS -Dduration=$DURATION"  /PATH/TO/DOCKER/REGISTRY/gatling -s MyLoadTest 


Note that we mount the gatling directories as Docker volumes, similarly to when we ran the Docker container locally, only this time we specify ${WORKSPACE} as the base directory. The 2 string parameters USERS and DURATION are passed as variables in JAVA_OPTS.
A nice thing about running Gatling via Jenkins is that the reports are available in the Workspace directory of the project. If you go to the Gatling project we created in Jenkins, click on Workspace, then on gatling, then results, you should see directories named gatlingsimulation-TIMESTAMP for each Gatling run. Each of these directories should have an index.html file, which will show you the Gatling report dashboard. Pretty neat.

Running Jenkins jobs in Docker containers

Fri, 06/17/2016 - 00:08
One of my main tasks at work is to configure Jenkins to act as a hub for all the deployment and automated testing jobs we run. We use CloudBees Jenkins Enterprise, mostly for its Role-Based Access Control plugin, which allows us to create one Jenkins folder per project/application and establish fine grained access control to that folder for groups of users. We also make heavy use of the Jenkins Enterprise Pipeline features (which I think are also available these days in the open source version).

Our Jenkins infrastructure is composed of a master node and several executor nodes which can run jobs in parallel if needed.

One pattern that my colleague Will Wright and I have decided upon is to run all Jenkins jobs as Docker containers. This way, we only need to install Docker Engine on the master node and the executor nodes. No need to install any project-specific pre-requisites or dependencies on every Jenkins node. All of these dependencies and pre-reqs are instead packaged in the Docker containers. It's a simple but powerful idea, that has worked very well for us. One of the nice things about this pattern is that you can keep adding various types of automated tests. If it can run from the command line, then it can run in a Docker container, which means you can run it from Jenkins!

I have seen this pattern discussed in multiple places recently, for example in this blog post about "Using Docker for a more flexible Jenkins".

Here are some examples of Jenkins jobs that we create for a given project/application:
  • a deployment job that runs Capistrano in its own Docker container, against targets in various environments (development, staging, production); this is a Pipeline script written in Groovy, which can call other jobs below
  • a Web UI testing job that runs the Selenium Python WebDriver and drives Firefox in headless mode (see my previous post on how to do this with Docker)
  • a JavaScript syntax checking job that runs JSHint against the application's JS files
  • an SSL scanner/checker that runs SSLyze against the application endpoints
We also run other types of tasks, such as running an AWS CLI command to perform certain actions, for example to invalidate a CloudFront resource. I am going to show here how we create a Docker image for one of these jobs, how we test it locally, and how we then integrate it in Jenkins.

I'll use as an example a simple Docker image that installs the AWS CLI package and runs a command when the container is invoked via 'docker run'.
I assume you have a local version of Docker installed. If you are on a Mac, you can use Docker Toolbox, or, if you are lucky and got access to it, you can use the native Docker for Mac. In any case,  I will assume that you have a local directory called awscli with the following Dockerfile in it:
FROM ubuntu:14.04
MAINTAINER You Yourself <you@example.com>
# disable interactive functionsARG DEBIAN_FRONTEND=noninteractiveENV AWS_ACCESS_KEY_ID=""ENV AWS_SECRET_ACCESS_KEY=""ENV AWS_COMMAND=""
RUN apt-get update && \    apt-get install -y python-pip && \    pip install awscli
WORKDIR /root
CMD (export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID; export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY; $AWS_COMMAND)
As I mentioned, this simply installs the awscli Python package via pip, then runs a command given as an environment variable when you invoke 'docker run'. It also uses two other environment variables that contain the AWS access key ID and secret access key. You don't want to hardcode these secrets in the Dockerfile and have them end up on GitHub.
The next step is to build an image based on this Dockerfile. I'll call the image awscli and I'll tag it as local:

$ docker build -t awscli:local .

Then you can run a container based on this image. The command line looks a bit complicated because I am passing (via the -e switch) the 3 environment variables discussed above:


$ docker run --rm -e AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=YOUR_SECRET_ACCESS_KEY -e AWS_COMMAND='aws cloudfront create-invalidation --distribution-id=abcdef --invalidation-batch Paths={Quantity=1,Items=[/media/*]},CallerReference=my-invalidation-123456' awscli:local
(where distribution-id needs to be the actual ID of your CloudFront distribution, and CallerReference needs to be unique per invalidation)

If all goes well, you should see the output of the 'aws cloudfront create-invalidation' command.

In our infrastructure, we have a special GitHub repository where we check in the various folders containing the Dockerfiles and any static files that need to be copied over to the Docker images. When we push the awscli directory to GitHub for example, we have a Jenkins job that will be notified of that commit and that will build the Docker image (similarly to how we did it locally with 'docker build'), then it will 'docker push' the image to a private AWS ECR repository we have.

Now let's assume we want to create a Jenkins job that will run this image as a container. First we define 2 secret credentials, specific to the Jenkins folder where we want to create the job (there are also global Jenkins credentials that can apply to all folders). These credentials are of type "Secret text" and contain the AWS access key ID and the AWS secret access key.

Then we create a new Jenkins job of type "Freestyle project" and call it cloudfront.invalidate. The build for this job is parameterized and contains 2 parameters: CF_ENVIRONMENT which is a drop-down containing the values "Staging" and "Production" referring to the CloudFront distribution we want to invalidate; and CF_RESOURCE, which is a text variable that needs to be set to the resource that needs to be invalidated (e.g. /media/*).

In the Build Environment section of the Jenkins job, we check "Use secret text(s) or file(s)" and add 2 Bindings, one for the first secret text credential containing the AWS access key ID, which we save in a variable called AWS_ACCESS_KEY_ID, and the other one for the second secret text credential containing the AWS secret access key, which we save in a variable called AWS_SECRET_ACCESS_KEY.

The Build section for this Jenkins job has a step of type "Execute shell" which uses the parameters and variables defined above and invokes 'docker run' using the path to the Docker image from our private ECR repository:

DISTRIBUTION_ID=MY_CF_DISTRIBUTION_ID_FOR_STAGING
if [ $CF_ENVIRONMENT == "PRODUCTION" ]; then
    DISTRIBUTION_ID=MY_CF_DISTRIBUTION_ID_FOR_PRODUCTION
fi

INVALIDATION_ID=jenkins-invalidation-`date +%Y%m%d%H%M%S`

COMMAND="aws cloudfront create-invalidation --distribution-id=$DISTRIBUTION_ID --invalidation-batch Paths={Quantity=1,Items=[$CF_RESOURCE]},CallerReference=$INVALIDATION_ID"

docker run --rm -e AWS_ACCESS_KEY_ID=$ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$SECRET_ACCESS_KEY -e AWS_COMMAND="$COMMAND" MY_PRIVATE_ECR_ID.dkr.ecr.us-west-2.amazonaws.com/awscli


When this job is run, the Docker image gets pulled down from AWS ECR, then a container based on the image is run and then removed upon completion (that's what --rm does, so that no old containers are left around).

I'll write another post soon with some more examples of Jenkins jobs that we run as Docker containers to do Selenium test, JSHint testing and SSLyze scanning.






Setting up AWS CloudFront for Magento

Thu, 05/26/2016 - 19:00
Here are some steps I jotted down for setting up AWS CloudFront as a CDN for the 3 asset directories that are used by Magento installations. I am assuming your Magento application servers are behind an ELB.


SSL certificate upload to AWS
Install aws command line utilities.
$ pip install awscli
Configure AWS credentials
Create IAM user and associate it with the IAMFullAccess policy. Run ‘aws configure’ and specify the user’s keys and the region.

Bring SSL key, certificate and intermediate certificate in current directory:
-rw-r--r-- 1 root root 4795 Apr 11 20:34 gd_bundle-g2-g1.crt-rw-r--r-- 1 root root 1830 Apr 11 20:34 wildcard.mydomain.com.crt-rw------- 1 root root 1675 Apr 11 20:34 wildcard.mydomain.com.key
Run following script for installing wildcard SSL certificate to be used in staging CloudFront setup:
$ cat add_ssl_cert_to_iam_for_prod_cloudfront.sh#!/bin/bash
aws iam upload-server-certificate --server-certificate-name WILDCARD_MYDOMAIN_COM_FOR_PROD_CF --certificate-body file://wildcard.mydomain.com.crt --private-key file://wildcard.mydomain.com.key --certificate-chain file://gd_bundle-g2-g1.crt --path /cloudfront/prod/

After uploading the SSL certificates, they will be available in drop-downs when configuring CloudFront for SSL.
Apache Cache-Control headers setup
  • Add these directives (modifying max-age accordingly) in all Apache vhosts, both for port 80 and for port 443
 <FilesMatch "\.(ico|pdf|flv|jpg|jpeg|png|gif|js|css|swf)$">        Header set Cache-Control "max-age=604800, public" </FilesMatch>
CloudFront setup
  • Origin: prod ELB (mydomain-production-lb-9321962155.us-west-2.elb.amazonaws.com)
  • Alternate domain name: cdn.mydomain.com\
  • SSL certificate: ID_OF_CERTIFICATE_UPLOADED_ABOVE
  • Custom SSL client support: Only Clients that Support Server Name Indication (SNI)
  • Domain name: eg7ac9k0fa3qwc.cloudfront.net
  • Behaviors
    • /media/* /skin/* /js/*
    • Viewer protocol policy: HTTP and HTTPS
    • Allowed HTTP methods: GET, HEAD
    • Forward headers: None
    • Object caching: Use origin cache headers
    • Forward cookies: None
    • Forward query strings: Yes
    • Smooth streaming: No
    • Restrict viewer access: No
    • Compress objects automatically: No

DNS setup
  • cdn.mydomain.com is a CNAME pointing to the CloudFront domain name above eg7ac9k0fa3qwc.cloudfront.net

Magento setup
This depends on the version of Magento you are running (1.x or 2.x), but you want to look for settings for the Base Skin URL, Base Media URL and Base Javascript URL, which are usually under System->Configuration->General-Web. You need to set them to point to the domain name you set up as a CNAME to CloudFront.

Base Skin URL: http://cdn.mydomain.com/skin
Base Media URL: http://cdn.mydomain.com/media
Base Javascript URL: http://cdn.mydomain.com/js
More in-depth Magento-specific instructions for integrating with CloudFront are available here.

LDAP server setup and client authentication

Fri, 04/15/2016 - 19:24
We recently bought at work a CloudBees Jenkins Enterprise license and I wanted to tie the user accounts to a directory service. I first tried to set up Jenkins authentication via the AWS Directory Service, hoping it will be pretty much like talking to an Active Directory server. That proved to be impossible to set up, at least for me. I also tried to have an LDAP proxy server talking to the AWS Directory Service and have Jenkins authenticate against the LDAP proxy. No dice. I ended up setting up a good old-fashioned LDAP server and managed to get Jenkins working with it. Here are some of my notes.

OpenLDAP server setup
I followed this excellent guide from Digital Ocean. The server was an Ubuntu 14.04 EC2 instance in my case. What follows in terms of the server setup is taken almost verbatim from the DO guide.

Set the hostname

# hostnamectl set-hostname my-ldap-server

Edit /etc/hosts and make sure this entry exists:
LOCAL_IP_ADDRESS my-ldap-server.mycompany.com my-ldap-server
(it makes a difference that the FQDN is the first entry in the line above!)

Make sure the following types of names are returned when you run hostname with different options:


# hostnamemy-ldap-server
# hostname -fmy-ldap-server.mycompany.com
# hostname -dmycompany.com

Install slapd

# apt-get install slapd ldap-utils# dpkg-reconfigure slapd
(here you specify the LDAP admin password)

Install the SSL Components

# apt-get install gnutls-bin ssl-cert

Create the CA Template
# mkdir /etc/ssl/templates
# vi /etc/ssl/templates/ca_server.conf# cat /etc/ssl/templates/ca_server.confcn = LDAP Server CAcacert_signing_key

Create the LDAP Service Template

# vi /etc/ssl/templates/ldap_server.conf# cat /etc/ssl/templates/ldap_server.conforganization = "My Company"cn = my-ldap-server.mycompany.comtls_www_serverencryption_keysigning_keyexpiration_days = 3650

Create the CA Key and Certificate

# certtool -p --outfile /etc/ssl/private/ca_server.key# certtool -s --load-privkey /etc/ssl/private/ca_server.key --template /etc/ssl/templates/ca_server.conf --outfile /etc/ssl/certs/ca_server.pem
Create the LDAP Service Key and Certificate

# certtool -p --sec-param high --outfile /etc/ssl/private/ldap_server.key# certtool -c --load-privkey /etc/ssl/private/ldap_server.key --load-ca-certificate /etc/ssl/certs/ca_server.pem --load-ca-privkey /etc/ssl/private/ca_server.key --template /etc/ssl/templates/ldap_server.conf --outfile /etc/ssl/certs/ldap_server.pem

Give OpenLDAP Access to the LDAP Server Key

# usermod -aG ssl-cert openldap# chown :ssl-cert /etc/ssl/private/ldap_server.key# chmod 640 /etc/ssl/private/ldap_server.key

Configure OpenLDAP to Use the Certificate and Keys
IMPORTANT NOTE: in modern versions of slapd, configuring the server is not done via slapd.conf anymore. Instead, you put together ldif files and run LDAP client utilities such as ldapmodify against the local server. The Distinguished Name of the entity you want to modify in terms of configuration is generally dn: cn=config but it can also be the LDAP database dn: olcDatabase={1}hdb,cn=config.
# vi addcerts.ldif# cat addcerts.ldifdn: cn=configchangetype: modifyadd: olcTLSCACertificateFileolcTLSCACertificateFile: /etc/ssl/certs/ca_server.pem-add: olcTLSCertificateFileolcTLSCertificateFile: /etc/ssl/certs/ldap_server.pem-add: olcTLSCertificateKeyFileolcTLSCertificateKeyFile: /etc/ssl/private/ldap_server.key

# ldapmodify -H ldapi:// -Y EXTERNAL -f addcerts.ldif# service slapd force-reload# cp /etc/ssl/certs/ca_server.pem /etc/ldap/ca_certs.pem# vi /etc/ldap/ldap.conf
* set TLS_CACERT to following:TLS_CACERT /etc/ldap/ca_certs.pem
# ldapwhoami -H ldap:// -x -ZZAnonymous

Force Connections to Use TLS
Change olcSecurity attribute to include 'tls=1':

# vi forcetls.ldif# cat forcetls.ldifdn: olcDatabase={1}hdb,cn=configchangetype: modifyadd: olcSecurityolcSecurity: tls=1

# ldapmodify -H ldapi:// -Y EXTERNAL -f forcetls.ldif# service slapd force-reload# ldapsearch -H ldap:// -x -b "dc=mycompany,dc=com" -LLL dn(shouldn’t work)
# ldapsearch -H ldap:// -x -b "dc=mycompany,dc=com" -LLL -Z dn(should work)

Disallow anonymous bind
Create user binduser to be used for LDAP searches:


# vi binduser.ldif# cat binduser.ldifdn: cn=binduser,dc=mycompany,dc=comobjectClass: topobjectClass: accountobjectClass: posixAccountobjectClass: shadowAccountcn: binduseruid: binduseruidNumber: 2000gidNumber: 200homeDirectory: /home/binduserloginShell: /bin/bashgecos: suseruserPassword: {crypt}xshadowLastChange: -1shadowMax: -1shadowWarning: -1

# ldapadd -x -W -D "cn=admin,dc=mycompany,dc=com" -Z -f binduser.ldifEnter LDAP Password:adding new entry "cn=binduser,dc=mycompany,dc=com"
Change olcDissalows attribute to include bind_anon:


# vi disallow_anon_bind.ldif# cat disallow_anon_bind.ldifdn: cn=configchangetype: modifyadd: olcDisallowsolcDisallows: bind_anon

# ldapmodify -H ldapi:// -Y EXTERNAL -ZZ -f disallow_anon_bind.ldif# service slapd force-reload
Also disable anonymous access to frontend:

# vi disable_anon_frontend.ldif# cat disable_anon_frontend.ldifdn: olcDatabase={-1}frontend,cn=configchangetype: modifyadd: olcRequiresolcRequires: authc

# ldapmodify -H ldapi:// -Y EXTERNAL -f disable_anon_frontend.ldif# service slapd force-reload

Create organizational units and users
Create helper scripts:

# cat add_ldap_ldif.sh
#!/bin/bash

LDIF=$1

ldapadd -x -w adminpassword -D "cn=admin,dc=mycompany,dc=com" -Z -f $LDIF

# cat modify_ldap_ldif.sh#!/bin/bash

LDIF=$1

ldapmodify -x -w adminpassword -D "cn=admin,dc=mycompany,dc=com" -Z -f $LDIF

# cat set_ldap_pass.sh#!/bin/bash

USER=$1PASS=$2

ldappasswd -s $PASS -w adminpassword -D "cn=admin,dc=mycompany,dc=com" -x "uid=$USER,ou=users,dc=mycompany,dc=com" -Z
Create ‘mypeople’ organizational unit:


# cat add_ou_mypeople.ldifdn: ou=mypeople,dc=mycompany,dc=comobjectclass: organizationalunitou: usersdescription: all users
# ./add_ldap_ldif.sh add_ou_mypeople.ldif
Create 'groups' organizational unit:


# cat add_ou_groups.ldifdn: ou=groups,dc=mycompany,dc=comobjectclass: organizationalunitou: groupsdescription: all groups

# ./add_ldap_ldif.sh add_ou_groups.ldif
Create users (note the shadow attributes set to -1, which means they will be ignored):


# cat add_user_myuser.ldifdn: uid=myuser,ou=mypeople,dc=mycompany,dc=comobjectClass: topobjectClass: accountobjectClass: posixAccountobjectClass: shadowAccountcn: myuseruid: myuseruidNumber: 2001gidNumber: 201homeDirectory: /home/myuserloginShell: /bin/bashgecos: myuseruserPassword: {crypt}xshadowLastChange: -1shadowMax: -1shadowWarning: -1
# ./add_ldap_ldif.sh add_user_myuser.ldif# ./set_ldap_pass.sh myuser MYPASS

Enable LDAPS
In /etc/default/slapd set:

SLAPD_SERVICES="ldap:/// ldaps:/// ldapi:///"

Enable debugging
This was a life saver when it came to troubleshooting connection issues from clients such as Jenkins or other Linux boxes. To enable full debug output, set olcLogLevel to -1:

# cat enable_debugging.ldifdn: cn=configchangetype: modifyadd: olcLogLevelolcLogLevel: -1
# ldapadd -H ldapi:// -Y EXTERNAL -f enable_debugging.ldif
# service slapd force-reload

Configuring Jenkins LDAP authentication
Verify LDAPS connectivity from Jenkins to LDAP server
In my case, the Jenkins server is in the same VPC and subnet as the LDAP server, so I added an /etc/hosts entry on the Jenkins box pointing to the FQDN of the LDAP server so it can hit its internal IP address:

IP_ADDRESS_OF_LDAP_SERVER my-ldap-server.mycompany.com
I verified that port 636 (used by LDAPS) on the LDAP server is reachable from the Jenkins server:
# telnet my-ldap-server.mycompany.com 636Trying IP_ADDRESS_OF_LDAP_SERVER...Connected to my-ldap-server.mycompany.com.Escape character is '^]'.
Set up LDAPS client on Jenkins server (StartTLSdoes not work w/ Jenkins LDAP plugin!)
# apt-get install ldap-utils
IMPORTANT: Copy over /etc/ssl/certs/ca_server.pem from LDAP server as /etc/ldap/ca_certs.pem on Jenkins server and then:
# vi /etc/ldap/ldap.confset:TLS_CACERT /etc/ldap/ca_certs.pem
Add LDAP certificates to Java keystore used by Jenkins
As user jenkins:
$ mkdir .keystore$ cp /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/security/cacerts .keystore/(you may need to customize the above line in terms of the path to the cacerts file -- it is the one under your JAVA_HOME)

$ keytool --keystore /var/lib/jenkins/.keystore/cacerts --import --alias my-ldap-server.mycompany.com:636 --file /etc/ldap/ca_certs.pemEnter keystore password: changeitOwner: CN=LDAP Server CAIssuer: CN=LDAP Server CASerial number: 570bddb0Valid from: Mon Apr 11 17:24:00 UTC 2016 until: Tue Apr 11 17:24:00 UTC 2017Certificate fingerprints:....Extensions:....
Trust this certificate? [no]:  yesCertificate was added to keystore
In /etc/default/jenkins, set JAVA_ARGS to:
JAVA_ARGS="-Djava.awt.headless=true -Djavax.net.ssl.trustStore=/var/lib/jenkins/.keystore/cacerts -Djavax.net.ssl.trustStorePassword=changeit"  
As root, restart jenkins:

# service jenkins restart
Jenkins settings for LDAP plugin
This took me a while to get right. The trick was to set the rootDN to dc=mycompany, dc=com and the userSearchBase to ou=mypeople (or to whatever name you gave to your users' organizational unit). I also tried to get LDAP groups to work but wasn't very successful.
Here is the LDAP section in /var/lib/jenkins/config.xml:
 <securityRealm class="hudson.security.LDAPSecurityRealm" plugin="ldap@1.11">    <server>ldaps://my-ldap-server.mycompany.com:636</server>    <rootDN>dc=mycompany,dc=com</rootDN>    <inhibitInferRootDN>true</inhibitInferRootDN>    <userSearchBase>ou=mypeople</userSearchBase>    <userSearch>uid={0}</userSearch> <groupSearchBase>ou=groups</groupSearchBase> <groupMembershipStrategy class="jenkins.security.plugins.ldap.FromGroupSearchLDAPGroupMembershipStrategy"> <filter>member={0}</filter> </groupMembershipStrategy>    <managerDN>cn=binduser,dc=mycompany,dc=com</managerDN>    <managerPasswordSecret>JGeIGFZwjipl6hJNefTzCwClRcLqYWEUNmnXlC3AOXI=</managerPasswordSecret>    <disableMailAddressResolver>false</disableMailAddressResolver>    <displayNameAttributeName>displayname</displayNameAttributeName>    <mailAddressAttributeName>mail</mailAddressAttributeName>    <userIdStrategy class="jenkins.model.IdStrategy$CaseInsensitive"/>    <groupIdStrategy class="jenkins.model.IdStrategy$CaseInsensitive"/>
 </securityRealm>

At this point, I was able to create users on the LDAP server and have them log in to Jenkins. With CloudBees Jenkins Enterprise, I was also able to use the Role-Based Access Control and Folder plugins in order to create project-specific folders and folder-specific groups specifying various roles. For example, a folder MyProjectNumber1 would have a Developers group defined inside it, as well as an Administrators group and a Readers group. These groups would be associated with fine-grained roles that only allow certain Jenkins operations for each group.
I tried to have these groups read by Jenkins from the LDAP server, but was unsuccessful. Instead, I had to populate the folder-specific groups in Jenkins with user names that were at least still defined in LDAP.  So that was half a win. Still waiting to see if I can define the groups in LDAP, but for now this is a workaround that works for me.
Allowing users to change their LDAP password
This was again a seemingly easy task but turned out to be pretty complicated. I set up another small EC2 instance to act as a jumpbox for users who want to change their LDAP password.
The jumpbox is in the same VPC and subnet as the LDAP server, so I added an /etc/hosts entry on the jumpbox pointing to the FQDN of the LDAP server so it can hit its internal IP address:

IP_ADDRESS_OF_LDAP_SERVER my-ldap-server.mycompany.com
I verified that port 636 (used by LDAPS) on the LDAP server is reachable from the jumpbox:
# telnet my-ldap-server.mycompany.com 636Trying IP_ADDRESS_OF_LDAP_SERVER...Connected to my-ldap-server.mycompany.com.Escape character is '^]'.
# apt-get install ldap-utils
IMPORTANT: Copy over /etc/ssl/certs/ca_server.pem from LDAP server as /etc/ldap/ca_certs.pem on the jumpbox and then:
# vi /etc/ldap/ldap.confset:TLS_CACERT /etc/ldap/ca_certs.pem
Next, I followed this LDAP Client Authentication guide from the Ubuntu documentation.
# apt-get install ldap-auth-client nscd
Here I had to answer the setup questions on LDAP server FQDN, admin DN and password, and bind user DN and password. 
# auth-client-config -t nss -p lac_ldap
I edited /etc/auth-client-config/profile.d/ldap-auth-config and set:
[lac_ldap]nss_passwd=passwd: ldap filesnss_group=group: ldap filesnss_shadow=shadow: ldap filesnss_netgroup=netgroup: nis
I edited /etc/ldap.conf and made sure the following entries were there:
base dc=mycompany,dc=comuri ldaps://my-ldap-server.mycompany.combinddn cn=binduser,mycompany,dc=combindpw BINDUSERPASSrootbinddn cn=admin,mycompany,dc=comport 636ssl ontls_cacertfile /etc/ldap/ca_certs.pemtls_cacertdir /etc/ssl/certs
I allowed password-based ssh logins to the jumpbox by editing /etc/ssh/sshd_config and setting:
PasswordAuthentication yes
# service ssh restart

IMPORTANT: On the LDAP server, I had to allow users to change their own password by adding this ACL:
# cat set_userpassword_acl.ldif
dn: olcDatabase={1}hdb,cn=configchangetype: modifyadd: olcAccessolcAccess: {0}to attrs=userpassword by dn="cn=admin,dc=mycompany,dc=com" write by self write by anonymous auth by users none
Then:
# ldapmodify -H ldapi:// -Y EXTERNAL -f set_userpassword_acl.ldif

At this point, users were able to log in via ssh to the jumpbox using a pre-set LDAP password, and change their LDAP password by using the regular Unix 'passwd' command.
I am still fine-tuning the LDAP setup on all fronts: LDAP server, LDAP client jumpbox and Jenkis server. The setup I have so far allows me to have a single sign-on account for users to log in to Jenkins. Some of my next steps is to use the same user LDAP accounts  for authentication and access control into MySQL and other services.

Joining an EC2 Windows instance to an AWS Directory Service domain

Wed, 04/06/2016 - 21:53
I've been struggling with the existing documentation on how to join an EC2 instance running Windows Server 2012 to an AWS Directory Service domain, so I am hastening to jot down some notes on how I got it to work.

1) Create an AWS Directory Service domain

There is good documentation on doing this. I chose the Microsoft Active Directory option.

A few notes on the creation of an AWS Directory Service:

  • I created a new VPC with 2 subnets for the Directory Service usage
  • I made sure each subnet has an Internet gateway associated so that it can be reachable from the outside
During the creation of the Directory Service, you'll be asked to specify an administrator-type user name and password. Make sure you remember what you specified there because you'll need this info in a subsequent step. Also make note of the DNS server IP addresses that were set during the Directory Service creation.
2) Create an IAM role to be associated with the EC2 Windows instance
  • the IAM role needs to be associated with the AmazonEC2RoleforSSM and AmazonSSMFullAccess policies
  • the IAM role also needs to have a trust relationship with ec2.amazonaws.com

3) Launch EC2 Windows instance associated with the AWS Directory Service domain
I chose the Windows_Server-2012-R2_RTM-English-64Bit-Base-2016.03.09 AMI.  In Step 3 of the AWS launch instance wizard ("Configure instance details") I made sure I specified the following:
  • Network: the VPC created in step 1) above
  • Subnet: one of the 2 subnets created in step 1) above
  • Domain join directory: the directory name for the Directory Service created in step 1) above
  • IAM role: the IAM role created in step 2) above
4) Connect to EC2 Windows instance via RDP
First get the administrator password via the AWS console (you'll need to paste the contents of the private key corresponding to the EC2 key you used when launching the Windows instance). Then connect to the Windows instance as the local administrator user.
Verify that you see the fully qualified domain name of your Directory Service (whatever you indicated in step 1) as the domain of the Windows instance (in Server Manager -> Local Server). If you don't, something went wrong with joining the domain during the instance launch. You can see the system log of that instance in the AWS console by selecting the instance, then going to Actions->Instance Settings->Get System Log. For example, in one of my failed attempts to get all of this working I saw errors related to the IAM role I was using, which at the time didn't have the correct SSM policies attached.
If the Windows instance is correctly joined to the domain, you need to install the Active Directory management tools in order to actually manage the AWS Directory Service. Here is a Rackspace article I found with good instructions.

5) Log in to the EC2 Windows instance as the domain admin to manage AD
After the EC2 Windows instance was rebooted, I managed to log in via RDP as my.aws.directory.fqdn\myusername (where both of these values are the ones chosen in Step 1 above) with the password also chosen in Step 1. At this point I was able to use the Active Directory management tools to add new AD users etc.
Here are some other good resources I found: