Warning: Table './devblogsdb/cache_page' is marked as crashed and last (automatic?) repair failed query: SELECT data, created, headers, expire, serialized FROM cache_page WHERE cid = 'http://www.softdevblogs.com/?q=aggregator/categories/4' in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc on line 135

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 729

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 730

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 731

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 732
Software Development Blogs: Programming, Software Testing, Agile, Project Management
Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Database
warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/common.inc on line 153.

Docker orchestration with Rancher

Agile Testing - Grig Gheorghiu - Fri, 09/09/2016 - 20:27
For the last month or so I've been experimenting with Rancher as the orchestration layer for Docker-based deployments. I've been pretty happy with it so far. Here are some of my notes and a few tips and tricks. I also recommend reading through the very good Rancher  documentation. In what follows I'll assume that the cluster management engine used by Rancher is its own engine called Cattle. Rancher also supports Kubernetes, Mesos and Docker Swarm.

Running the Rancher server

I provisioned an EC2 instance, installed Docker on it, then ran this command to launch the Rancher server as a Docker container (it will also get launched automatically if you reboot the EC2 instance):


# docker run -d --restart=always -p 8080:8080 rancher/server

Creating Rancher environments
It's important to think about the various environments you want to manage in Rancher. If you have multiple projects that you want to manage with Rancher, as well as multiple environments for your infrastructure, such as development, staging and production, I recommend you create a Rancher environment per project/infrastructure-environment combination, for example a Rancher environment called proj1dev, another one called proj1stage, another called proj1prod, and similarly for other projects: proj2dev, proj2stage, proj2prod etc.
Tip: Since all containers in the same Rancher environment can by default connect to all other containers in that Rancher environment, having a project/infrastructure-environment combination as detailed above will provide good isolation and security from one project to another, and from one infrastructure environment to another within the same project. I recommend you become familiar with Rancher environments by reading more about them in the documentation.
In what follows I'll assume the current environment is proj1dev.
Creating Rancher API key pairs
Within each environment, create an API key pair. Copy and paste the two keys (one access key and one secret access key) somewhere safe.

Adding Rancher hosts
Within each environment, you need to add Rancher hosts. They are the compute nodes that will run the various Docker containers that you will orchestrate with Rancher. In my case, I provisioned two hosts per environment as EC2 instances running Docker.
In the Rancher UI, when you go to Infrastructure  -> Hosts then click the Add Host button, you should see a docker run command that you can run on each host in order to launch the Rancher Agent on that host. Something like this:
# docker run -d --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/rancher:/var/lib/rancher rancher/agent:v1.0.2 http://your-rancher-server-name.example.com:8080/v1/scripts/5536854597A70149E388:1473267600000:rfQVqxXcvIPulNw72fUOQG66iGM
Note that you need to allow UDP ports 500 and 4500 from each Rancher host to/from any other host and to/from the Rancher server. This is because Rancher uses IPSec tunnels for inter-host communication. The Rancher hosts also need to talk to the Rancher server over port 8080 (or whatever port you have exposed for the Rancher server container).
Adding ECR registries
We use ECR as our Docker registry. Within each environment, I had to add our ECR registry. In the Rancher UI, I went to Infrastructure -> Registries, then clicked Add Registry and chose Custom as the registry type. In the attribute fields, I specified:
  • Address: my_ecr_registry_id.dkr.ecr.my_region.amazonaws.com
  • Email: none
  • Username: AWS
  • Password: the result of running these commands (you need to install and configure the awscli for this to work):
    • apt-get install python-pip; pip install awscli
    • aws configure (specify the keys for an IAM user allowed to access the ECR registry)
    • aws ecr get-login | cut -d ' ' -f 6

Application architecture
For this example I will consider an application composed of a Web application based on Apache/PHP running in 2 or more containers and mounting its shared files (configuration, media) over NFS. The Web app talks to a MySQL database server mounting its data files over NFS. The Web app containers are behind one or more instances of a Rancher load balancer, and the Rancher LB instances are fronted by an Amazon Elastic Load Balancer.
Rancher stacks
A 'stack' in Rancher corresponds to a set of services defined in a docker-compose YAML file. These services can also have Rancher-specific attributes (such as desired number of containers aka 'scale', health checks, etc) defined in a special rancher-compose YAML file. I'll show plenty of examples of these files in what follows. My stack naming convention will be projname-environment-stacktype, for example proj1-development-nfs, proj1--development-database etc.
Tip: Try to experiment with creating stacks in the Rancher UI, then either view or export their configurations via the stack settings button in the UI:

This was a life saver for me especially when it comes to lower-level stacks such as NFS or Rancher load balancers. Exporting the configuration will download a zip file containing two files: docker-compose.yml and rancher-compose.yml. It will save you from figuring out on your own the exact syntax you need to use in these files.
Creating an NFS stack
One of the advantages of using Rancher is that it offers an extensive catalog of services ready to be used within your infrastructure. One such service is Convoy NFS. To use it, I started out by going to the Catalog menu option in the Rancher UI, then selecting Convoy NFS. In the following screen I specified proj1-development-nfs as the stack name, as well as the NFS server's IP address and mount point.


Note that I had already set up an EC2 instance to act as an NFS server. I attached an EBS volume per project/environment. So in the example above, I exported a directory called /nfs/development/proj1.
After launching the NFS stack, you should see it in the Stacks screen in the Rancher UI. The stack will consist of 2 services, one called convoy-nfs and the other called convoy-nfs-storagepool:

Once the NFS stack is up and running, you can export its configuration as explained above.

To create or update a stack programmatically, I used the rancher-compose utility and wrapped it inside shell scripts. Here is an example of a shell script that calls rancher-compose to create an NFS stack:
$ cat rancher-nfssetup.sh#!/bin/bash

COMMAND=$@

rancher-compose -p proj1-development-nfs --url $RANCHER_URL --access-key $RANCHER_API_ACCESS_KEY --secret-key $RANCHER_API_SECRET_KEY --env-file .envvars --file docker-compose-nfssetup.yml --rancher-file rancher-compose.yml $COMMAND

Note that there is no command line option for the target Rancher environment. It suffices to use the Rancher API keys for a given environment in order to target that environment.

Here is the docker-compose file for this stack, which I obtained by exporting the stack configuration from the UI:
$ cat docker-compose-nfssetup.ymlconvoy-nfs-storagepool: labels: io.rancher.container.create_agent: 'true' command: - storagepool-agent image: rancher/convoy-agent:v0.9.0 volumes: - /var/run:/host/var/run - /run:/host/run convoy-nfs: labels: io.rancher.scheduler.global: 'true' io.rancher.container.create_agent: 'true' command: - volume-agent-nfs image: rancher/convoy-agent:v0.9.0 pid: host privileged: true volumes: - /lib/modules:/lib/modules:ro - /proc:/host/proc - /var/run:/host/var/run - /run:/host/run - /etc/docker/plugins:/etc/docker/plugins
Here is the portion of my rancher-compose.yml file that has to do with the NFS stack, again obtained by exporting the NFS stack configuration:
convoy-nfs-storagepool: scale: 1 health_check: port: 10241 interval: 2000 unhealthy_threshold: 3 strategy: recreate response_timeout: 2000 request_line: GET /healthcheck HTTP/1.0 healthy_threshold: 2 metadata: mount_dir: /nfs/development/proj1 nfs_server: 172.31.41.108 convoy-nfs: health_check: port: 10241 interval: 2000 unhealthy_threshold: 3 strategy: recreate response_timeout: 2000 request_line: GET /healthcheck HTTP/1.0 healthy_threshold: 2 metadata: mount_dir: /nfs/development/proj1 nfs_server: 172.31.41.108 mount_opts: ''

To create the NFS stack, all I need to do at this point is to call:

$ ./rancher-nfssetup.sh up

To inspect the logs for the stack, I can call:

$ ./rancher-nfssetup.sh logs

Note that I passed various arguments to the rancher-compose utility. Most of them are specified as environment variables. This allows me to add the bash script to version control without worrying about credentials, secrets etc. I also use the --env-file .envvars option, which allows me to define environment variables in the .envvars file and have them interpolated by rancher-compose in the various yml files it uses.
Creating volumes using the NFS stack
One of my goals was to attach NFS-based volumes to Docker containers in my infrastructure. To do this, I needed to create volumes in Rancher. One way to do it is to go to Infrastructure -> Storage in the Rancher UI, then go to the area corresponding to the NFS stack you want and click Add Volume, giving the volume a name and a description. Doing it manually is well and good, but I wanted to do it automatically, so I used another bash script around rancher-compose together with another docker-compose file:
$ cat rancher-volsetup.sh#!/bin/bash COMMAND=$@ rancher-compose -p proj1-development-volsetup --url $RANCHER_URL --access-key $RANCHER_API_ACCESS_KEY --secret-key $RANCHER_API_SECRET_KEY --env-file .envvars --file docker-compose-volsetup.yml --rancher-file rancher-compose.yml $COMMAND

$ cat docker-compose-volsetup.ymlvolsetup: image: ubuntu:14.04 labels: io.rancher.container.start_once: true volumes: - volMysqlData:/var/lib/mysql - volAppShared:/var/www/shared volume_driver: proj1-development-nfs
A few things to note in the docker-compose-volsetup.yml file:
  • I used the ubuntu:14.04 Docker image and I attached two volumes, one called volMysqlData and once called volAppSharedData. The first one will be mounted on the Docker container as /var/lib/mysql and the second one will be mounted as /var/www/shared. These are arbitrary paths, since my goal was just to create the volumes as Rancher resources.
  • I wanted the volsetup service to run once so that the volumes get created, then stop. For that, I used the special Rancher label io.rancher.container.start_once: true
  • I used as the volume_driver the NFS stack proj1-development-nfs I created above. This is important, because I want these volumes to be created within this NFS stack.
I used the following commands to create and start the proj1-development-volsetup stack, then to show its logs, and finally to shut it down and remove its containers, which are not needed anymore once the volumes get created: ./rancher-volsetup.sh up -d sleep 30 ./rancher-volsetup.sh logs ./rancher-volsetup.sh down ./rancher-volsetup.sh rm --force
I haven't figured out yet how to remove a Rancher stack programmatically, so for these 'helper' type stacks I had to use the Rancher UI to delete them.At this point, if you look in the /nfs/development/proj1 directory on the NFS server, you should see 2 directories with the same names as the volumes we created.
Creating a database stack
So far I haven't used any custom Docker images. For the database layer of my application, I will want to use a custom image which I will push to the Amazon ECR registry. I will use this image in a docker-compose file in order to set up and start the database in Rancher.
I have a directory called db containing the following Dockerfile:
$ cat Dockerfile
FROM percona

VOLUME /var/lib/mysql

COPY etc/mysql/my.cnf /etc/mysql/my.cnf
COPY scripts/db_setup.sh /usr/local/bin/db_setup.sh

I have a customized MySQL configuration file my.cnf (in my local directory db/etc/mysql) which gets copied to the Docker image as /etc/mysql.my.cnf. I also have a db_setup.sh bash script in my local directory db/scripts which gets copied to /usr/local/bin in the Docker image. In this script I grant rights to a MySQL user used by the Web app, and I also load a MySQL dump file if it exists:
$ cat scripts/db_setup.sh#!/bin/bash set -e host="$1" until mysql -h "$host" -uroot -p$MYSQL_ROOT_PASSWORD -e "SHOW DATABASES"; do >&2 echo "MySQL is unavailable - sleeping" sleep 1 done >&2 echo "MySQL is up - executing GRANT statement" mysql -h "$host" -uroot -p$MYSQL_ROOT_PASSWORD \ -e "GRANT ALL ON $MYSQL_DATABASE.* TO $MYSQL_USER@'%' IDENTIFIED BY \"$MYSQL_PASSWORD\"" >&2 echo "Starting to load SQL dump" mysql -h "$host" -uroot -p$MYSQL_ROOT_PASSWORD $MYSQL_DATABASE < /dbdump/$MYSQL_DUMP_FILE >&2 echo "Finished loading SQL dump"
Note that the database name, database user name and password, as well as the MySQL root password are all passed in environment variables.
To build this Docker image, I ran:
$ docker build -t my_ecr_registry_id.dkr.ecr.my_region.amazonaws.com/db:proj1-development .
Note that I tagged the image with the proj1-development tag.
To push this image to Amazon ECR, I first called:
$(aws get-login)
then:
$ docker push my_ecr_registry_id.dkr.ecr.my_region.amazonaws.com/db:proj1-development
To run the db_setup.sh script inside a Docker container in order to set up the database, I put together the following docker-compose file:
$ cat docker-compose-dbsetup.ymlECRCredentials:  environment:    AWS_REGION: $AWS_REGION    AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID    AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY  labels:    io.rancher.container.pull_image: always    io.rancher.container.create_agent: 'true'    io.rancher.container.agent.role: environment    io.rancher.container.start_once: true  tty: true  image: objectpartners/rancher-ecr-credentials  stdin_open: true
db:  image: my_ecr_registry_id.dkr.ecr.my_region.amazonaws.com/db:proj1-development  labels:    io.rancher.container.pull_image: always    io.rancher.scheduler.affinity:host_label: dbsetup=proj1  volumes:    - volMysqlData:/var/lib/mysql  volume_driver: proj1-development-nfs  environment:    - MYSQL_ROOT_PASSWORD=$MYSQL_ROOT_PASSWORD
dbsetup:  image: my_ecr_registry_id.dkr.ecr.my_region.amazonaws.com/db:proj1-development  labels:    io.rancher.container.pull_image: always    io.rancher.container.start_once: true    io.rancher.scheduler.affinity:host_label: dbsetup=proj1  command: /usr/local/bin/db_setup.sh db  links:    - db:db  volumes:    - volMysqlData:/var/lib/mysql    - /dbdump/proj1:/dbdump  volume_driver: proj1-development-nfs  environment:    - MYSQL_ROOT_PASSWORD=$MYSQL_ROOT_PASSWORD    - MYSQL_DATABASE=$MYSQL_DATABASE    - MYSQL_USER=$MYSQL_USER    - MYSQL_PASSWORD=$MYSQL_PASSWORD    - MYSQL_DUMP_FILE=$MYSQL_DUMP_FILE
A few things to note:
  • there are 3 services in this docker-compose file
    • a ECRCredentials service which connects to Amazon ECR and allows the ECR image db:proj1-development to be used by the other 2 services
    • a db service which runs a Docker container based on the db:proj1-development ECR image, and which launches a MySQL database with the root password set to the value of the MYSQL_ROOT_PASSWORD environment variable
    • a dbsetup service that also runs a Docker container based on the db:proj1-development ECR image, but instead of the default command, which would run MySQL, it runs the db_setup.sh script (specified in the command directive); this service also uses environment variables specifying the database to be loaded from the SQL dump file, as well as the user and password that will get grants to that database
  • the dbsetup service links to the db service via the links directive
  • the dbsetup service is a 'run once then stop' type of service, which is why it has the label io.rancher.container.start_once: true attached
  • both the db and the dbsetup service will run on a Rancher host with the label 'dbsetup=proj1'; this is because we want to load the SQL dump from a file that the dbsetup service can find
    • we will put this file on a specific Rancher host in a directory called /dbdump/proj1, which will then be mounted by the dbsetup container as /dbdump
    • the db_setup.sh script will then load the SQL file called MYSQL_DUMP_FILE from the /dbdump directory
    • this can also work if we'd just put the SQL file in the same NFS volume as the MySQL data files, but I wanted to experiment with host labels in this case
  • wherever NFS volumes are used, for example for volMysqlData, the volume_driver needs to be set to the proper NFS stack, proj1-development-nfs in this case
It goes without saying that mounting the MySQL data files from NFS is a potential performance bottleneck, so you probably wouldn't do this in production. I wanted to experiment with NFS in Rancher, and the performance I've seen in development and staging for some of our projects doesn't seem too bad.
To run a Rancher stack based on this docker-compose-dbsetup.yml file, I used this bash script:
$ cat rancher-dbsetup.sh#!/bin/bash
COMMAND=$@
rancher-compose -p proj1-development-dbsetup --url $RANCHER_URL --access-key $RANCHER_API_ACCESS_KEY --secret-key $RANCHER_API_SECRET_KEY --env-file .envvars --file docker-compose-dbsetup.yml --rancher-file rancher-compose.yml $COMMAND
Note that all environment variables referenced in the docker-compose-dbsetup.yml file are set in the .envvars file.
I wanted to run the proj1-development-dbsetup stack and then shut down its services once the dbsetup service completes.  I used these commands as part of a bash script:
./rancher-dbsetup.sh up -d
while :do        ./rancher-dbsetup.sh logs --lines "10" > dbsetup.log 2>&1        grep 'Finished loading SQL dump' dbsetup.log        result=$?        if [ $result -eq 0 ]; then            break        fi        echo Waiting 10 seconds for DB load to finish...        sleep 10done./rancher-dbsetup.sh logs./rancher-dbsetup.sh down./rancher-dbsetup.sh rm --force
Once the database is setup, I want to launch MySQL and keep it running so it can be used by the Web application. I have a separate docker-compose file for that:
$ cat docker-compose-dblaunch.ymlECRCredentials:  environment:    AWS_REGION: $AWS_REGION    AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID    AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY  labels:    io.rancher.container.pull_image: always    io.rancher.container.create_agent: 'true'    io.rancher.container.agent.role: environment    io.rancher.container.start_once: true  tty: true  image: objectpartners/rancher-ecr-credentials  stdin_open: true
db:  image: my_ecr_registry_id.dkr.ecr.my_region.amazonaws.com/db:proj1-development  labels:    io.rancher.container.pull_image: always  volumes:    - volMysqlData:/var/lib/mysql  volume_driver: proj1-development-nfs
The db service is similar to the one in the docker-compose-dbsetup.yml file. In this case the database is all set up, so we don't need anything except the NFS volume to mount the MySQL data files from.
As usual, I have a bash script that calls docker-compose in order to create a stack called proj1-development-database:
$ cat rancher-dblaunch.sh#!/bin/bash
COMMAND=$@
rancher-compose -p proj1-development-database --url $RANCHER_URL --access-key $RANCHER_API_ACCESS_KEY --secret-key $RANCHER_API_SECRET_KEY --env-file .envvars --file docker-compose-dblaunch.yml --rancher-file rancher-compose.yml $COMMAND
I call this script like this:
./rancher-dblaunch.sh up -d
At this point, the proj1-development-database stack is up and running and contains the db service running as a container on one of the Rancher hosts in the Rancher 'proj1dev' environment.
Creating a Web application stack

So far, I've been using either off-the-shelf or slightly customized Docker images. For the Web application stack I will be using more heavily customized images. The building block is a 'base' image whose Dockerfile contains directives for installing commonly used packages and for adding users.

Here is the Dockerfile for a 'base' image running Ubuntu 14.04:

FROM ubuntu:14.04

RUN apt-get update && \
    apt-get install -y ntp build-essential build-essential binutils zlib1g-dev \
                       git acl cronolog lzop unzip mcrypt expat xsltproc python-pip curl language-pack-en-base
RUN pip install awscli

RUN adduser --ui 501 --ingroup www-data --shell /bin/bash --home /home/myuser myuser
RUN mkdir /home/myuser/.ssh
COPY files/myuser_authorized_keys /home/myuser/.ssh/authorized_keys
RUN chown -R myuser:www-data /home/myuser/.ssh && \
    chmod 700 /home/myuser/.ssh && \
    chmod 600 /home/myuser/.ssh/authorized_keys 

When I built this image, I tagged it as my_ecr_registry_id.dkr.ecr.my_region.amazonaws.com/base:proj1-development.

Here is the Dockerfile for an image (based on the base image above) that installs Apache, PHP 5.6 (using a custom apt repository), RVM, Ruby and the compass gem:

FROM  my_ecr_registry_id.dkr.ecr.my_region.amazonaws.com/base:proj1-development

RUN export LC_ALL=en_US.UTF-8 && export LC_ALL=en_US.UTF-8 && export LANG=en_US.UTF-8 && \
        apt-get install -y mysql-client-5.6 software-properties-common && add-apt-repository ppa:ondrej/php5-5.6

RUN apt-get update && \
    apt-get install -y --allow-unauthenticated apache2 apache2-utils libapache2-mod-php5 \
                       php5 php5-mcrypt php5-curl php-pear php5-gd \
                       php5-dev php5-mysql php5-readline php5-xsl php5-xmlrpc php5-intl

# Install composer
RUN curl -sSL https://getcomposer.org/composer.phar -o /usr/bin/composer \
    && chmod +x /usr/bin/composer \
    && composer selfupdate

# Install rvm and compass gem for SASS image compilation

RUN curl https://raw.githubusercontent.com/rvm/rvm/master/binscripts/rvm-installer -o /tmp/rvm-installer.sh && \
        chmod 755 /tmp/rvm-installer.sh && \
        gpg --keyserver hkp://keys.gnupg.net --recv-keys D39DC0E3 && \
        /tmp/rvm-installer.sh stable --path /home/myuser/.rvm --auto-dotfiles --user-install && \
        /home/myuser/.rvm/bin/rvm get stable && \
        /home/myuser/.rvm/bin/rvm reload && \
        /home/myuser/.rvm/bin/rvm autolibs 3

RUN /home/myuser/.rvm/bin/rvm install ruby-2.2.2  && \
        /home/myuser/.rvm/bin/rvm alias create default ruby-2.2.2 && \
        /home/myuser/.rvm/wrappers/ruby-2.2.2/gem install bundler && \
        /home/myuser/.rvm/wrappers/ruby-2.2.2/gem install compass

COPY files/apache2-foreground /usr/local/bin/
EXPOSE 80
CMD ["apache2-foreground"]

When I built this image, I tagged it as  my_ecr_registry_id.dkr.ecr.my_region.amazonaws.com/apache-php:proj1-development

With these 2 images as building blocks, I put together 2 more images, one for building artifacts for the Web application, and one for launching it.

Here is the Dockerfile for an image that builds the artifacts for the Web application:

FROM my_ecr_registry_id.dkr.ecr.my_region.amazonaws.com/apache-php:proj1-development

ADD ./scripts/app_setup.sh /usr/local/bin/

The heavy lifting takes place in the app_setup.sh script. That's where you would do things such as pull a specified git branch from application repo on GitHub, then run composer (if it's a PHP app) or other build tools in order to generate the artifacts necessary for running the application. At the end of this script, I generate a tar.gz of the code + any artifacts and upload it to S3 so I can use it when I generate the Docker image for the Web app.

When I built this image, I tagged it as  my_ecr_registry_id.dkr.ecr.my_region.amazonaws.com/appsetup:proj1-development

To actually run a Docker container based on the appsetup image, I used this docker-compose file:

$ cat docker-compose-appsetup.yml
ECRCredentials:
  environment:
    AWS_REGION: $AWS_REGION
    AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
    AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
  labels:
    io.rancher.container.pull_image: always
    io.rancher.container.create_agent: 'true'
    io.rancher.container.agent.role: environment
    io.rancher.container.start_once: true
  tty: true
  image: objectpartners/rancher-ecr-credentials
  stdin_open: true

appsetup:
        image: my_ecr_registry_id.dkr.ecr.my_region.amazonaws.com/appsetup:proj1-development
  labels:
    io.rancher.container.pull_image: always
  command: /usr/local/bin/app_setup.sh
  external_links:
    - proj1-development-database/db:db
  volumes:
    - volAppShared:/var/www/shared
  volume_driver: proj1-development-nfs
  environment:
    - GIT_URL=$GIT_URL
    - GIT_BRANCH=$GIT_BRANCH
    - AWS_S3_REGION=$AWS_S3_REGION
    - AWS_S3_ACCESS_KEY_ID=$AWS_S3_ACCESS_KEY_ID
    - AWS_S3_SECRET_ACCESS_KEY=$AWS_S3_SECRET_ACCESS_KEY
    - AWS_S3_RELEASE_BUCKET=$AWS_S3_RELEASE_BUCKET
    - AWS_S3_RELEASE_FILENAME=$AWS_S3_RELEASE_FILENAME

Some things to note:
  • the command executed when a Docker container based on the appsetup service is launched is /usr/local/bin/app_setup.sh, as specified in the command directive
    • the app_setup.sh script runs commands that connect to the database, hence the need for the appsetup service to link to the MySQL database running in the proj1-development-database stack launched above; for that, I used the external_links directive
  • the appsetup service mounts an NFS volume (volAppShared) as /var/www/shared
    • the volume_driver needs to be proj1-development-nfs
    • before running the service, I created proper application configuration files under /nfs/development/proj1/volAppShared on the NFS server, specifying things such as the database server name (which needs to be 'db', since this is how the database container is linked as), the database name, user name and password, etc.
  • the appsetup service uses various environment variables referenced in the environment directive; it will pass these variables to the app_setup.sh script
To run the appsetup service, I used another bash script around the rancher-compose command:
$ cat rancher-appsetup.sh#!/bin/bash
COMMAND=$@
rancher-compose -p proj1-development-appsetup --url $RANCHER_URL --access-key $RANCHER_API_ACCESS_KEY --secret-key $RANCHER_API_SECRET_KEY --env-file .envvars --file docker-compose-appsetup.yml --rancher-file rancher-compose.yml $COMMAND

Tip: When using its Cattle cluster management engine, Rancher does not add services linked to each other as static entries in /etc/hosts on the containers. Instead, it provides an internal DNS service so that containers in the same environment can reach each other by DNS names as long as they link to each other in docker-compose files. If you go to a shell prompt inside a container, you can ping other containers by name even from one Rancher stack to another. For example, from a web container in the proj1-development-app stack you can ping a database container in the proj1-development-database stack linked in the docker-compose file as db and you would get back a name of the type db.proj1-development-app.rancher.internal.
Tip: There is no need to expose ports from containers within the same Rancher environment. I spent many hours troubleshooting issues related to ports and making sure ports are unique across stacks, only to realize that the internal ports that the services listen on (3306 for MySQL, 80 and 443 for Apache) are reachable from the other containers in the same Rancher environment. The only ports you need exposed to the external world in the architecture I am describing are the load balancer ports, as I'll describe below.
Here is the Dockerfile for an image that runs the Web application:
FROM my_ecr_registry_id.dkr.ecr.my_region.amazonaws.com/apache-php:proj1-development
# disable interactive functions
ARG DEBIAN_FRONTEND=noninteractive

RUN a2enmod headers \
&& a2enmod rewrite \
&& a2enmod ssl

RUN rm -rf /etc/apache2/ports.conf /etc/apache2/sites-enabled/*
ADD etc/apache2/sites-enabled /etc/apache2/sites-enabled
ADD etc/apache2/ports.conf /etc/apache2/ports.conf

ADD release /var/www/html/release
RUN chown -R myuser:www-data /var/www/html/release
This image is based on the apache-php image but adds Apache customizations, as well as the release directory obtained from the tar.gz file uploaded to S3 by the appsetup service.

When I built this image, I tagged it as  my_ecr_registry_id.dkr.ecr.my_region.amazonaws.com/app:proj1-development

Code deployment

My code deployment process is a bash script (which can be used standalone, or as part of a Jenkins job, or can be turned into a Jenkins pipeline) that first runs the appsetup service in order to generate a tar.gz of the code and artifacts, then downloads it from S3 and uses it as the local release directory to be copied into the app image. The script then pushes the app Docker image to Amazon ECR. The environment variables are either defined in an .envvars file or passed via Jenkins parameters. The script assumes that the Dockerfile for the app image is in the current directory, and that the etc directory structure used for the Apache files in the app image is also in the current directory (they are all checked into the project repository, so Jenkins will find them).

./rancher-appsetup.sh up -dsleep 20cp /dev/null appsetup.logwhile :do        ./rancher-appsetup.sh logs >> appsetup.log 2>&1        grep 'Restarting web server apache2' appsetup.log        result=$?        if [ $result -eq 0 ]; then            break        fi        echo Waiting 10 seconds for app code deployment to finish...        sleep 10done./rancher-appsetup.sh logs./rancher-appsetup.sh down./rancher-appsetup.sh rm --force
# download release.tar.gz from S3 and unpack it
set -a. .envvarsset +a
export AWS_ACCESS_KEY_ID=$AWS_S3_ACCESS_KEY_IDexport AWS_SECRET_ACCESS_KEY=$AWS_S3_SECRET_ACCESS_KEY
rm -rf $AWS_S3_RELEASE_FILENAME.tar.gz
aws s3 --region $AWS_S3_REGION ls s3://$AWS_S3_RELEASE_BUCKET/aws s3 --region $AWS_S3_REGION cp s3://$AWS_S3_RELEASE_BUCKET/$AWS_S3_RELEASE_FILENAME.tar.gz .
tar xfz $AWS_S3_RELEASE_FILENAME.tar.gz
# build app docker image and push it to ECR
cat << "EOF" > awscreds[default]aws_access_key_id=$AWS_ACCESS_KEY_IDaws_secret_access_key=$AWS_SECRET_ACCESS_KEYEOF
export AWS_SHARED_CREDENTIALS_FILE=./awscreds $(aws ecr --region=$AWS_REGION get-login)/usr/bin/docker build -t my_ecr_registry_id.dkr.ecr.my_region.amazonaws.com/app:proj1-development ./usr/bin/docker push my_ecr_registry_id.dkr.ecr.my_region.amazonaws.com/app:proj1-development

Launching the app service

At this point, the Docker image for the app service has been pushed to Amazon ECR, but the service itself hasn't been started. To do that, I use this docker-compose file:

$ cat docker-compose-app.yml
ECRCredentials:
  environment:
    AWS_REGION: $AWS_REGION
    AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
    AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
  labels:
    io.rancher.container.pull_image: always
    io.rancher.container.create_agent: 'true'
    io.rancher.container.agent.role: environment
    io.rancher.container.start_once: true
  tty: true
  image: objectpartners/rancher-ecr-credentials
  stdin_open: true

app:
  image: my_ecr_registry_id.dkr.ecr.my_region.amazonaws.com/app:proj1-development
  labels:
    io.rancher.container.pull_image: always
  external_links:
    - proj1-development-database/db:db
  volumes:
    - volAppShared:/var/www/shared
  volume_driver: proj1-development-nfs

Nothing very different about this file compare to the files I've shown so far. The app service mounts the volAppShared NFS volume as /var/www/shared, and links to the MySQL database service db already running in the proj1-development-database Rancher stack, giving it the name 'db'.

To run the app service, I use this bash script wrapping rancher-compose:

$ cat rancher-app.sh
#!/bin/bash

COMMAND=$@

rancher-compose -p proj1-development-app --url $RANCHER_URL --access-key $RANCHER_API_ACCESS_KEY --secret-key $RANCHER_API_SECRET_KEY --env-file .envvars --file docker-compose-app.yml --rancher-file rancher-compose.yml $COMMAND

Since the proj1-development-app stack may already be running with an old version of the app Docker image, I will invoke rancher-app.sh with the force-upgrade option of the rancher-compose command:

./rancher-app.sh up -d --force-upgrade --confirm-upgrade --pull --batch-size "1"

This will perform a rolling upgrade of the app service, by stopping the containers for the app service one at a time (as indicated by the batch-size parameter), then pulling the latest Docker image for the app service, and finally starting each container again. Speaking of 'containers' plural, you can indicate how many containers should run at all times for the app service by adding these lines to rancher-compose.yml:

app:
  scale: 2

In my case, I want 2 containers to run at all times. If you stop one container from the Rancher UI, you will see another one restarted automatically by Rancher in order to preserve the value specified for the 'scale' parameter.

Creating a load balancer stack

When I started to run load balancers in Rancher, I created them via the Rancher UI. I created a new stack, then added a load balancer service to it. It took me a while to figure out that I can then export the stack configuration and generate a docker-compose file and a rancher-compose snippet I can add to my main rancher-compose.yml file.

Here is the docker-compose file I use:

$ cat docker-compose-lbsetup.yml
lb:
  ports:
  - 8000:80
  - 8001:443
  external_links:
  - proj1-development-app/app:app
  labels:
    io.rancher.loadbalancer.ssl.ports: '8001'
    io.rancher.loadbalancer.target.proj1-development-app/app: proj1.dev.mydomain.com:8000=80,8001=443
  tty: true
  image: rancher/load-balancer-service
  stdin_open: true

The ports directive tell the load balancer which ports to expose externally and what ports to map them to. This example shows that port 8000 will be exposed externally and mapped to port 80 on the target service, and port 8001 will be exposed externally and mapped to port 443 on the target service.

The external_links directive tells the load balancer which service to load balance. In this example, it is the app service in the proj1-development-app stack.

The labels directive does layer 7 load balancing by allowing you to specify a domain name that you want to send to a specific port. In this example, I want to send HTTP requests coming on port 8000 for proj1.dev.mydomain.com to port 80 on the target containers for the app service, and HTTPS requests coming on port 8001 for the same proj1.dev.mydomain.com name to port 443 on the target containers.

I could have also added a new line under labels, specifying that I want requests for proj1-admin.dev.mydomain.com coming on port 8000 to be sent to a different port on the target containers, assuming that I had Apache configured to listen on that port. You can read more about the load balancing features available in Rancher in the documentation.

Here is the load balancer section in rancher-compose.yml:

lb:
  scale: 2
  load_balancer_config:
    haproxy_config: {}
  default_cert: proj1.dev.mydomain.com
  health_check:
    port: 42
    interval: 2000
    unhealthy_threshold: 3
    healthy_threshold: 2
    response_timeout: 2000

Note that there is a mention of a default_cert. This is an SSL key + cert that I uploaded to Rancher via the UI by going to Infrastructure -> Certificates and that I named proj1.dev.mydomain.com. The Rancher Catalog does contain an integration for Let's Encrypt but I haven't had a chance to test it yet (from the Rancher Catalog: "The Let's Encrypt Certificate Manager obtains a free (SAN) SSL Certificate from the Let's Encrypt CA and adds it to Rancher's certificate store. Once the certificate is created it is scheduled for auto-renewal 14-days before expiration. The renewed certificate is propagated to all applicable load balancer services.")

Note also that the scale value is 2, which means that there will be 2 containers for the lb service.

Tip: In the Rancher UI, you can open a shell into any container, or view the logs for any container by going to the Settings icon of that container, and choosing Execute Shell or View Logs:

Tip: Rancher load balancers are based on haproxy. You can open a shell into a container running for the lb service, then look at the haproxy configuration file in /etc/haproxy/haproxy.cfg. To troubleshoot haproxy issues, you can enable UDP logging in /etc/rsyslog.conf by removing the comments before the following 2 lines:

#$ModLoad imudp
#$UDPServerRun 514

then restarting the rsyslog service. Then you can restart the haproxy service and inspect its log file in /var/log/haproxy.log.
To run the lb service, I use this bash script:

$ cat rancher-lbsetup.sh
#!/bin/bash

COMMAND=$@

rancher-compose -p proj1-development-lb --url $RANCHER_URL --access-key $RANCHER_API_ACCESS_KEY --secret-key $RANCHER_API_SECRET_KEY --env-file .envvars --file docker-compose-lbsetup.yml --rancher-file rancher-compose.yml $COMMAND

I want to do a rolling upgrade of the lb service in case anything has changed, so I invoke the rancher-compose wrapper script in a similar way to the one for the app service:

./rancher-lbsetup.sh up -d --force-upgrade --confirm-upgrade --batch-size "1"

Putting it all together in Jenkins

First I created a GitHub repository with the following structure:

  • All docker-compose-*.yml files
  • The rancher-compose.yml file
  • All rancher-*.sh bash scripts wrapping the rancher-compose command
  • A directory for the base Docker image (containing its Dockerfile and any other files that need to go into that image)
  • A directory for the apache-php Docker image
  • A directory for the db Docker image
  • A directory for the appsetup Docker image
  • A Dockerfile in the current directory for the app Docker image
  • An etc directory in the current directory used by the Dockerfile for the app image

Each project/environment combination has a branch created in this GitHub repository. For example, for the proj1 development environment I would create a proj1dev branch which would then contain any customizations I need for this project -- usually stack names, Docker tags, Apache configuration files under the etc directory.

My end goal was to use Jenkins to drive the launching of the Rancher services and the deployment of the code. Eventually I will use a Jenkins Pipeline to string together the various steps of the workflow, but for now I have 5 individual Jenkins jobs which all check out the proj1dev branch of the GitHub repo above. The jobs contain shell-type build steps where I actually call the various rancher bash scripts around rancher-compose. The Jenkins jobs also take parameters corresponding to the environment variables used in the docker-compose files and in the rancher bash scripts. I also use the Credentials section in Jenkins to store any secrets such as the Rancher API keys, AWS keys, S3 keys, ECR keys etc. On the Jenkins master and executor nodes I installed the rancher and rancher-compose CLI utilities (I downloaded the rancher CLI from the footer of the Rancher UI).

Job #1 builds the Docker images discussed above: base, apache-php, db, and appsetup (but not the app image yet).

Job #2 runs rancher-nfssetup.sh and rancher-volsetup.sh in order to set up the NFS stack and the volumes used by the dbsetup, appsetup, db and app services.

Job #3 runs rancher-dbsetup.sh and rancher-dblaunch.sh in order to set up the database via the dbsetup service, then launch the db service.

At this point, everything is ready for deployment of the application.

Job #4 is the code deployment job. It runs the sequence of steps detailed in the Code Deployment section above.

Job #5 is the rolling upgrade job for the app service and the lb service. If those services have never been started before, they will get started. If they are already running, they will be upgraded in a rolling fashion, batch-size containers at a time as I detailed above.

When a new code release needs to be pushed to the proj1dev Rancher environment, I would just run job #4 followed by job #5. Obviously you can string these jobs together in a Jenkins Pipeline, which I intend to do next.

Some more Rancher tips and tricks









Exposing a private Amazon RDS instance with iptables NAT rules

Agile Testing - Grig Gheorghiu - Thu, 07/28/2016 - 17:43
I needed to expose a private Amazon MySQL RDS instance to a 3rd party SaaS tool. I tried several approaches and finally found one that seemed to work pretty well.

I ended up creating a small EC2 instance in the same VPC as the RDS instance, and applied these iptables NAT/masquerading rules to it, mapping local port 3307 to port 3306 on the RDS instance, whose internal IP address is in this case 172.16.11.2.

# cat iptables_tunnel_port_3307.sh
#!/bin/bash

iptables -F
iptables -F -t nat
iptables -X
iptables -t nat -A PREROUTING -p tcp --dport 3307 -j DNAT --to 172.16.11.2:3306
iptables -A FORWARD -p tcp -d 172.16.11.2 --dport 3306 -j ACCEPT
iptables -t nat -A OUTPUT -p tcp -o lo --dport 3307 -j DNAT --to 172.16.11.2:3306
iptables -t nat -A POSTROUTING  -j MASQUERADE

I also had to enable IP forwarding on the EC2 instance:

# sysctl net.ipv4.ip_forward
# sysctl -p

At this point, I was able to hit the external IP of the EC2 instance on port 3307, and get to the private RDS instance on port 3306. I was also able to attach the EC2 instance to an EC2 Security Group allowing the 3rd party SaaS tool IP addresses to access port 3307 on the EC2 instance.

My thanks to the people discussing a similar issue on this thread of LinuxQuestions. Without their discussion, I don't think I'd have been able to figure out a solution.

Using JMESPath queries with the AWS CLI

Agile Testing - Grig Gheorghiu - Wed, 07/13/2016 - 22:05
The AWS CLI, based on the boto3 Python library, is the recommended way of automating interactions with AWS. In this post I'll show some examples of more advanced AWS CLI usage using the query mechanism based on the JMESPath JSON query language.

Installing the AWS CLI tools is straightforward. On Ubuntu via apt-get:

# apt-get install awscli

Or via pip:

# apt-get install python-pip
# pip install awscli

The next step is to configure awscli by specifying the AWS Access Key ID and AWS Secret Access Key, as well as the default region and output format:

# aws configureAWS Access Key ID: your-aws-access-key-idAWS Secret Access Key: your-aws-secret-access-keyDefault region name [us-west-2]: us-west-2
Default output format [None]: json
The configure command creates a ~/.aws directory containing two files: config and credentials.
You can specify more than one pair of AWS keys by creating profiles in these files. For example, in ~/.aws/credentials you can have:
[profile profile1]AWS_ACCESS_KEY_ID=key1AWS_SECRET_ACCESS_KEY=secretkey1
[profile profile2]AWS_ACCESS_KEY_ID=key2AWS_SECRET_ACCESS_KEY=secretkey2
In ~/.aws/config you can have:[profile profile1]region = us-west-2
[profile profile2]region = us-east-1
You can specify a given profile when you run awscli:
# awscli --profile profile1
Let's assume you want to write a script using awscli that deletes EBS snapshots older than N days. Let's go through this one step at a time.
Here's how you can list all snapshots owned by you:
# aws ec2 describe-snapshots --profile profile1 --owner-id YOUR_AWS_ACCT_NUMBER --query "Snapshots[]"
Note the use of the --query option. It takes a parameter representing a JMESPath JSON query string. It's not trivial to figure out how to build these query strings, and I advise you to spend some time reading over the JMESPath tutorial and JMESPath examples.
In the example above, the query string is simply "Snapshots[]", which represents all the snapshots that are present in the AWS account associated with the profile profile1. The default output in our case is JSON, but you can specify --output text at the aws command line if you want to see each snapshot on its own line of text.
Let's assume that when you create the snapshots, you specify a description with contains PROD or STAGE for EBS volumes attached to production and stage EC2 instances respectively. If you want to only display snapshots containing the string PROD, you would do:
# aws ec2 describe-snapshots --profile profile1 --owner-id YOUR_AWS_ACCT_NUMBER--query "Snapshots[?contains(Description, \`PROD\`) == \`true\`]" --output text
The Snapshots[] array now contains a condition represented by the question mark ?. The condition uses the contains() function included in the JMESPath specification, and is applied against the Description field of each object in the Snapshots[] array, verifying that it contains the string PROD.  Note the use of backquotes surrounding the strings PROD and true in the condition. I spent some quality time troubleshooting my queries when I used single or double quotes with no avail. The backquotes also need to be escaped so that the shell doesn't interpret them as commands to be executed.
To restrict the PROD snapshots even further, to the ones older than say 7 days ago, you can do something like this:
DAYS=7TARGET_DATE=`date --date="$DAYS day ago" +%Y-%m-%d`
# aws ec2 describe-snapshots --profile profile1 --owner-id YOUR_AWS_ACCT_NUMBER--query "Snapshots[?contains(Description, \`PROD\`) == \`true\`]|[?StartTime < \`$TARGET_DATE\`]" --output text
Here I used the StartTime field of the objects in the Snapshots[] array and compared it against the target date. In this case, string comparison is good enough for the query to work.
In all the examples above, the aws command returned a subset of the Snapshots[] array and displayed all fields for each object in the array. If you wanted to display specific fields, let's say the ID, the start time and the description of each snapshot, you would run:
# aws ec2 describe-snapshots --profile profile1 --owner-id YOUR_AWS_ACCT_NUMBER--query "Snapshots[?contains(Description, \`PROD\`) == \`true\`]|[?StartTime < \`$TARGET_DATE\`].[SnapshotId,StartTime,Description]" --output text
To delete old snapshots, you can use the aws ec2 delete-snapshot command, which needs a snapshot ID as a parameter. You could use the command above to list only the SnapshotId for snapshots older than N days, then for each of these IDs, run something like this:
# aws ec2 delete-snapshot --profile profile1 --snapshot-id $id
All this is well and good when you run these commands interactively at the shell. However, I had no luck running them out of cron. The backquotes resulted in boto3 syntax errors. I had to do it the hard way, by listing all snapshots first, then going all in with sed and awk:
aws ec2 describe-snapshots --profile profile1 --owner-id YOUR_AWS_ACCT_NUMBER --output=text --query "Snapshots[].[SnapshotId,StartTime,Description]"  > $TMP_SNAPS
DAYS=7TARGET_DATE=`date --date="$DAYS day ago" +%Y-%m-%d`
cat $TMP_SNAPS | grep PROD | sed 's/T[0-9][0-9]:[0-9][0-9]:[0-9][0-9].000Z//' | awk -v target_date="$TARGET_DATE" '{if ($2 < target_date){print}}' > $TMP_PROD_SNAPS
echo PRODUCTION SNAPSHOTS OLDER THAN $DAYS DAYScat $TMP_PROD_SNAPS
for sid in `awk '{print $1}' $TMP_PROD_SNAPS` ; do echo Deleting PROD snapshot $sid aws ec2 delete-snapshot --profile $PROFILE --region $REGION --snapshot-id $siddone
Ugly, but it works out of cron. Hope it helps somebody out there.

July 14th 2016: I initially forgot to include this very good blog post from Joseph Lawson on advanced JMESPath usage with the AWS CLI.

More tips and tricks for running Gatling in Docker containers

Agile Testing - Grig Gheorghiu - Fri, 07/01/2016 - 18:11
This post is a continuation of my previous one on "Running Gatling tests in Docker containers via Jenkins". As I continued to set up Jenkins jobs to run Gatling tests, I found the need to separate those tests for different environments - development, staging and production. The initial example I showed contained a single setup, which is not suitable for multiple environments.

Here is my updated Gatling directory structure

gatling
gatling/conf
gatling/conf/production
gatling/conf/production/gatling.conf
gatling/conf/staging
gatling/conf/staging/gatling.conf
gatling/Dockerfile
gatling/results
gatling/user-files
gatling/user-files/data
gatling/user-files/data/production-urls.csv
gatling/user-files/data/staging-urls.csv
gatling/user-files/simulations
gatling/user-files/simulations/development
gatling/user-files/simulations/production
gatling/user-files/simulations/production/Simulation.scala
gatling/user-files/simulations/staging
gatling/user-files/simulations/staging/Simulation.scala

Note that I created a separate directory under simulations for each environment (development, staging, production), each with its own simulation files.

I also created a data directory under user-files, because that is the default location for CSV files used by Gatling feeders.

Most importantly, I created a separate configuration directory (staging, production) under gatling/conf, each directory containing its own customized gatling.conf file. I started by copying the gatling-defaults.conf file from GitHub to gatling/conf/staging/gatling.conf and gatling/conf/production/gatling.conf respectively.

Here is what I customized in staging/gatling.conf:

mute = true # When set to true, don't ask for simulation name nor run description
simulations = user-files/simulations/staging

I customized production/gatling.conf in a similar way:

mute = true # When set to true, don't ask for simulation name nor run description
simulations = user-files/simulations/production

Setting mute to true is important because without it, running Gatling in a Docker container was segfaulting while waiting for user input for the simulation ID:

Select simulation id (default is 'gatlingsimulation'). Accepted characters are a-z, A-Z, 0-9, - and _ Exception in thread "main" java.lang.NullPointerException at io.gatling.app.Selection$Selector.loop$1(Selection.scala:127) at io.gatling.app.Selection$Selector.askSimulationId(Selection.scala:135) at io.gatling.app.Selection$Selector.selection(Selection.scala:50) at io.gatling.app.Selection$.apply(Selection.scala:33) at io.gatling.app.Gatling.runIfNecessary(Gatling.scala:75) at io.gatling.app.Gatling.start(Gatling.scala:65) at io.gatling.app.Gatling$.start(Gatling.scala:57) at io.gatling.app.Gatling$.fromArgs(Gatling.scala:49) at io.gatling.app.Gatling$.main(Gatling.scala:43) at io.gatling.app.Gatling.main(Gatling.scala)

The other customization was to point the simulations attribute to the specific staging or production sub-directories.
Since the CSV files containing URLs to be load tested are also environment-specific, I modified the Simulation.scala files to take this into account. I also added 2 JAVA_OPTS variables that can be passed at runtime for HTTP basic authentication. Here is the new Crawl object (compare with the one from my previous post):
object Crawl {  val feeder = csv("staging-urls.csv").random
  val userName = System.getProperty("username")  val userPass = System.getProperty("password")
  val crawl = exec(feed(feeder)    .exec(http("${loc}")    .get("${loc}").basicAuth(userName, userPass)    ))}
One more thing is needed: to make Gatling use a specific configuration file instead of its default one, which is conf/gatling.conf. To do that, I set GATLING_CONF as an ENV variable in the Dockerfile, so it can be passed as a 'docker run' command line parameter. Here is the Dockerfile:
# Gatling is a highly capable load testing tool.## Documentation: http://gatling.io/docs/2.2.2/# Cheat sheet: http://gatling.io/#/cheat-sheet/2.2.2
FROM java:8-jdk-alpine
MAINTAINER Denis Vazhenin
# working directory for gatlingWORKDIR /opt
# gating versionENV GATLING_VERSION 2.2.2
# create directory for gatling installRUN mkdir -p gatling
# install gatlingRUN apk add --update wget && \  mkdir -p /tmp/downloads && \  wget -q -O /tmp/downloads/gatling-$GATLING_VERSION.zip \  https://repo1.maven.org/maven2/io/gatling/highcharts/gatling-charts-highcharts-bundle/$GATLING_VERSION/gatling-charts-highcharts-bundle-$GATLING_VERSION-bundle.zip && \  mkdir -p /tmp/archive && cd /tmp/archive && \  unzip /tmp/downloads/gatling-$GATLING_VERSION.zip && \  mv /tmp/archive/gatling-charts-highcharts-bundle-$GATLING_VERSION/* /opt/gatling/
# change context to gatling directoryWORKDIR  /opt/gatling
# set directories below to be mountable from hostVOLUME ["/opt/gatling/conf", "/opt/gatling/results", "/opt/gatling/user-files"]
# set environment variablesENV PATH /opt/gatling/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binENV GATLING_HOME /opt/gatlingENV GATLING_CONF /opt/gatling/confENV JAVA_OPTS ""
ENTRYPOINT ["gatling.sh"]
Finally, here is how I invoke 'docker run' to tie everything together:
docker run --rm -v ${WORKSPACE}/gatling/conf:/opt/gatling/conf -v ${WORKSPACE}/gatling/user-files:/opt/gatling/user-files -v ${WORKSPACE}/gatling/results:/opt/gatling/results -e GATLING_CONF="/opt/gatling/conf/staging" -e JAVA_OPTS="-Dusers=$USERS -Dduration=$DURATION -Dusername=myusername -Dpassword=mypass" /PATH/TO/DOCKER/REGISTRY/gatling
Note the GATLING_CONF parameter passed with -e with the value of /opt/gatling/conf/staging. Also note the username and password JAVA_OPTS parameters.

Happy load testing!

Running Gatling load tests in Docker containers via Jenkins

Agile Testing - Grig Gheorghiu - Wed, 06/29/2016 - 00:16
Gatling is a modern load testing tool written in Scala. As part of the Jenkins setup I am in charge of, I wanted to run load tests using Gatling against a collection of pages for a given website. Here are my notes on how I managed to do this.

Running Gatling as a Docker container locally

There is a Docker image already available on DockerHub, so you can simply pull down the image locally:


$ docker pull denvazh/gatling:2.2.2
Instructions on how to run a container based on this image are available on GitHub:
$ docker run -it --rm -v /home/core/gatling/conf:/opt/gatling/conf \-v /home/core/gatling/user-files:/opt/gatling/user-files \-v /home/core/gatling/results:/opt/gatling/results \ denvazh/gatling:2.2.2
Based on these instructions, I created a local directory called gatling, and under it I created 3 sub-directories: conf, results and user-files. I left the conf and results directories empty, and under user-files I created a simulations directory containing a Gatling load test scenario written in Scala. I also created a file in the user-files directory called urls.csv, containing a header named loc and a URL per line for each page that I want to load test.
Assuming the current directory is gatling, here are examples of these files:
$ cat user-files/urls.csvlochttps://my.website.comhttps://my.website.com/category1https://my.website.com/category2/product3
$ cat user-files/simulations/Simulation.scala
package my.gatling.simulation
import io.gatling.core.Predef._import io.gatling.http.Predef._import scala.concurrent.duration._
class GatlingSimulation extends Simulation {
  val httpConf = http    .baseURL("http://127.0.0.1")    .acceptHeader("text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")    .doNotTrackHeader("1")    .acceptLanguageHeader("en-US,en;q=0.5")    .acceptEncodingHeader("gzip, deflate")    .userAgentHeader("Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0")
  val scn1 = scenario("Scenario1")    .exec(Crawl.crawl)
  val userCount = Integer.getInteger("users", 1)  val durationInSeconds  = java.lang.Long.getLong("duration", 10L)  setUp(    scn1.inject(rampUsers(userCount) over (durationInSeconds seconds))  ).protocols(httpConf)}
object Crawl {
  val feeder = csv("/opt/gatling/user-files/urls.csv").random
  val crawl = exec(feed(feeder)    .exec(http("${loc}")    .get("${loc}")    ))}

I won't go through the different ways of writing Gatling load tests scenarios here. There are good instructions on the Gatling website -- see the Quickstart and the Advanced Tutorial. What the scenario above does is it reads the file urls.csv and randomly picks a URL from it, then runs a load test against that URL.
I do want to point out 2 variables in the above script:

  val userCount = Integer.getInteger("users", 1)  val durationInSeconds  = java.lang.Long.getLong("duration", 10L)
These variables specify the max number of users we want to ramp up to, and the duration of the ramp-up. They are used in the inject call:

scn1.inject(rampUsers(userCount) over (durationInSeconds seconds))

The special thing about these 2 variables is that they are read from JAVA_OPTS by Gatling. So if you have a -Dusers Java option and a -Dduration Java option, Gatling will know how to read them and how to set the userCount and durationInSeconds variables accordingly. This is a good thing, because it allows you to specify those numbers outside of Gatling, without hardcoding them in your simulation script. Here is more info on passing parameters via the command line to Gatling.

While pulling the Gatling docker image and running it is the simplest way to run Gatling, I prefer to understand what's going on in that image. I started off by getting the Dockerfile from GitHub:

$ cat Dockerfile
# Gatling is a highly capable load testing tool.## Documentation: http://gatling.io/docs/2.2.2/# Cheat sheet: http://gatling.io/#/cheat-sheet/2.2.2
FROM java:8-jdk-alpine
MAINTAINER Denis Vazhenin
# working directory for gatlingWORKDIR /opt
# gating versionENV GATLING_VERSION 2.2.2
# create directory for gatling installRUN mkdir -p gatling
# install gatlingRUN apk add --update wget && \  mkdir -p /tmp/downloads && \  wget -q -O /tmp/downloads/gatling-$GATLING_VERSION.zip \  https://repo1.maven.org/maven2/io/gatling/highcharts/gatling-charts-highcharts-bundle/$GATLING_VERSION/gatling-charts-highcharts-bundle-$GATLING_VERSION-bundle.zip && \  mkdir -p /tmp/archive && cd /tmp/archive && \  unzip /tmp/downloads/gatling-$GATLING_VERSION.zip && \  mv /tmp/archive/gatling-charts-highcharts-bundle-$GATLING_VERSION/* /opt/gatling/
# change context to gatling directoryWORKDIR  /opt/gatling
# set directories below to be mountable from hostVOLUME ["/opt/gatling/conf", "/opt/gatling/results", "/opt/gatling/user-files"]
# set environment variablesENV PATH /opt/gatling/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binENV GATLING_HOME /opt/gatling
ENTRYPOINT ["gatling.sh"]
I then added a way to pass JAVA_OPTS via an environment variable. I added this line after the ENV GATLING_HOME line:
ENV JAVA_OPTS ""
I dropped this Dockerfile in my gatling directory, then built a local Docker image off of it:
$ docker build -t gatling:local .
I  then invoked 'docker run' to launch a container based on this image, using the csv and simulation files from above. The current directory is still gatling.

$ docker run --rm -v `pwd`/conf:/opt/gatling/conf -v `pwd`/user-files:/opt/gatling/user-files -v `pwd`/results:/opt/gatling/results -e JAVA_OPTS="-Dusers=10 -Dduration=60" gatling:local -s MySimulationName

Note the -s flag which denotes a simulation name (which can be any string you want). If you don't specify this flag, the gatling.sh script which is the ENTRYPOINT in the container will wait for some user input and you will not be able to fully automate your load test.

Another thing to note is the use of JAVA_OPTS. In the example above, I pass -Dusers=10 and   -Dduration=60 as the two JAVA_OPTS parameters. The JAVA_OPTS variable itself is passed to 'docker run' via the -e option, which tells Docker to replace the default value for ENV JAVA_OPTS (which is "") with the value passed with -e.

Running Gatling as a Docker container from Jenkins

Once you have a working Gatling container locally, you can upload the Docker image built above to a private Docker registry. I used a private EC2 Container Registry (ECR).  

I also added the gatling directory and its sub-directories to a GitHub repository called devops.

In Jenkins, I created a new "Freestyle project" job with the following properties:

  • Parameterized build with 2 string parameters: USERS (default value 10) and DURATION in seconds (default value 60)
  • Git repository - add URL and credentials for the devops repository which contains the gatling files
  • An "Execute shell" build command similar to this one:
docker run --rm -v ${WORKSPACE}/gatling/conf:/opt/gatling/conf -v ${WORKSPACE}/gatling/user-files:/opt/gatling/user-files -v ${WORKSPACE}/gatling/results:/opt/gatling/results -e JAVA_OPTS="-Dusers=$USERS -Dduration=$DURATION"  /PATH/TO/DOCKER/REGISTRY/gatling -s MyLoadTest 


Note that we mount the gatling directories as Docker volumes, similarly to when we ran the Docker container locally, only this time we specify ${WORKSPACE} as the base directory. The 2 string parameters USERS and DURATION are passed as variables in JAVA_OPTS.
A nice thing about running Gatling via Jenkins is that the reports are available in the Workspace directory of the project. If you go to the Gatling project we created in Jenkins, click on Workspace, then on gatling, then results, you should see directories named gatlingsimulation-TIMESTAMP for each Gatling run. Each of these directories should have an index.html file, which will show you the Gatling report dashboard. Pretty neat.

Running Jenkins jobs in Docker containers

Agile Testing - Grig Gheorghiu - Fri, 06/17/2016 - 00:08
One of my main tasks at work is to configure Jenkins to act as a hub for all the deployment and automated testing jobs we run. We use CloudBees Jenkins Enterprise, mostly for its Role-Based Access Control plugin, which allows us to create one Jenkins folder per project/application and establish fine grained access control to that folder for groups of users. We also make heavy use of the Jenkins Enterprise Pipeline features (which I think are also available these days in the open source version).

Our Jenkins infrastructure is composed of a master node and several executor nodes which can run jobs in parallel if needed.

One pattern that my colleague Will Wright and I have decided upon is to run all Jenkins jobs as Docker containers. This way, we only need to install Docker Engine on the master node and the executor nodes. No need to install any project-specific pre-requisites or dependencies on every Jenkins node. All of these dependencies and pre-reqs are instead packaged in the Docker containers. It's a simple but powerful idea, that has worked very well for us. One of the nice things about this pattern is that you can keep adding various types of automated tests. If it can run from the command line, then it can run in a Docker container, which means you can run it from Jenkins!

I have seen this pattern discussed in multiple places recently, for example in this blog post about "Using Docker for a more flexible Jenkins".

Here are some examples of Jenkins jobs that we create for a given project/application:
  • a deployment job that runs Capistrano in its own Docker container, against targets in various environments (development, staging, production); this is a Pipeline script written in Groovy, which can call other jobs below
  • a Web UI testing job that runs the Selenium Python WebDriver and drives Firefox in headless mode (see my previous post on how to do this with Docker)
  • a JavaScript syntax checking job that runs JSHint against the application's JS files
  • an SSL scanner/checker that runs SSLyze against the application endpoints
We also run other types of tasks, such as running an AWS CLI command to perform certain actions, for example to invalidate a CloudFront resource. I am going to show here how we create a Docker image for one of these jobs, how we test it locally, and how we then integrate it in Jenkins.

I'll use as an example a simple Docker image that installs the AWS CLI package and runs a command when the container is invoked via 'docker run'.
I assume you have a local version of Docker installed. If you are on a Mac, you can use Docker Toolbox, or, if you are lucky and got access to it, you can use the native Docker for Mac. In any case,  I will assume that you have a local directory called awscli with the following Dockerfile in it:
FROM ubuntu:14.04
MAINTAINER You Yourself <you@example.com>
# disable interactive functionsARG DEBIAN_FRONTEND=noninteractiveENV AWS_ACCESS_KEY_ID=""ENV AWS_SECRET_ACCESS_KEY=""ENV AWS_COMMAND=""
RUN apt-get update && \    apt-get install -y python-pip && \    pip install awscli
WORKDIR /root
CMD (export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID; export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY; $AWS_COMMAND)
As I mentioned, this simply installs the awscli Python package via pip, then runs a command given as an environment variable when you invoke 'docker run'. It also uses two other environment variables that contain the AWS access key ID and secret access key. You don't want to hardcode these secrets in the Dockerfile and have them end up on GitHub.
The next step is to build an image based on this Dockerfile. I'll call the image awscli and I'll tag it as local:

$ docker build -t awscli:local .

Then you can run a container based on this image. The command line looks a bit complicated because I am passing (via the -e switch) the 3 environment variables discussed above:


$ docker run --rm -e AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=YOUR_SECRET_ACCESS_KEY -e AWS_COMMAND='aws cloudfront create-invalidation --distribution-id=abcdef --invalidation-batch Paths={Quantity=1,Items=[/media/*]},CallerReference=my-invalidation-123456' awscli:local
(where distribution-id needs to be the actual ID of your CloudFront distribution, and CallerReference needs to be unique per invalidation)

If all goes well, you should see the output of the 'aws cloudfront create-invalidation' command.

In our infrastructure, we have a special GitHub repository where we check in the various folders containing the Dockerfiles and any static files that need to be copied over to the Docker images. When we push the awscli directory to GitHub for example, we have a Jenkins job that will be notified of that commit and that will build the Docker image (similarly to how we did it locally with 'docker build'), then it will 'docker push' the image to a private AWS ECR repository we have.

Now let's assume we want to create a Jenkins job that will run this image as a container. First we define 2 secret credentials, specific to the Jenkins folder where we want to create the job (there are also global Jenkins credentials that can apply to all folders). These credentials are of type "Secret text" and contain the AWS access key ID and the AWS secret access key.

Then we create a new Jenkins job of type "Freestyle project" and call it cloudfront.invalidate. The build for this job is parameterized and contains 2 parameters: CF_ENVIRONMENT which is a drop-down containing the values "Staging" and "Production" referring to the CloudFront distribution we want to invalidate; and CF_RESOURCE, which is a text variable that needs to be set to the resource that needs to be invalidated (e.g. /media/*).

In the Build Environment section of the Jenkins job, we check "Use secret text(s) or file(s)" and add 2 Bindings, one for the first secret text credential containing the AWS access key ID, which we save in a variable called AWS_ACCESS_KEY_ID, and the other one for the second secret text credential containing the AWS secret access key, which we save in a variable called AWS_SECRET_ACCESS_KEY.

The Build section for this Jenkins job has a step of type "Execute shell" which uses the parameters and variables defined above and invokes 'docker run' using the path to the Docker image from our private ECR repository:

DISTRIBUTION_ID=MY_CF_DISTRIBUTION_ID_FOR_STAGING
if [ $CF_ENVIRONMENT == "PRODUCTION" ]; then
    DISTRIBUTION_ID=MY_CF_DISTRIBUTION_ID_FOR_PRODUCTION
fi

INVALIDATION_ID=jenkins-invalidation-`date +%Y%m%d%H%M%S`

COMMAND="aws cloudfront create-invalidation --distribution-id=$DISTRIBUTION_ID --invalidation-batch Paths={Quantity=1,Items=[$CF_RESOURCE]},CallerReference=$INVALIDATION_ID"

docker run --rm -e AWS_ACCESS_KEY_ID=$ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$SECRET_ACCESS_KEY -e AWS_COMMAND="$COMMAND" MY_PRIVATE_ECR_ID.dkr.ecr.us-west-2.amazonaws.com/awscli


When this job is run, the Docker image gets pulled down from AWS ECR, then a container based on the image is run and then removed upon completion (that's what --rm does, so that no old containers are left around).

I'll write another post soon with some more examples of Jenkins jobs that we run as Docker containers to do Selenium test, JSHint testing and SSLyze scanning.






Thu, 01/01/1970 - 01:00