Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Xebia Blog
Syndicate content
Updated: 14 hours 18 min ago

Marketing scrum vs IT scrum - a report published and presented at agile 2014

Fri, 07/25/2014 - 17:49

As we know, Scrum is the perfect framework for IT / software development projects to learn, adapt to change and deliver great software of value, faster.

But is Scrum also usable outside of software development? Can we apply similar or maybe even the same principals in other departments in the enterprise?

Yes, we can! And yes there are differences but there are also a lot of similarities.

We (Remco en Me)  successfully implemented Scrum in the marketing departments of two large companies: The ANWB and ING Bank. Both companies are now using Scrum for the development of new campaigns, their full commercial expressions and even at the product development level. They wanted a faster time to market, more ownership, and greater innovation. How did we approach and realized a transition with those goals in the marketing environment? And what are the results?

So when we are not delivering software but other things, how does Scrum change? Well, a great deal actually. The people working in these other departments are, in general, quite different to those in Software Development (and yes more than you would expect). This means coaches or change agents need to take another approach.

Since the people are different, it is possible to go faster or ‘deeper’ in certain areas. Entrepreneurial skills or ambitions are more present in marketing. This gives a sense of ‘act first apologize later’, taking ownership, a higher drive to succeed, and upfront and willing behavior. Scrumming here means thinking more about business goals and KPIs (how to go from department to scrumteam goals for example). After that the fun begins


I will be speaking about this topic at agile 2014. A great honor offcourse to be standing there. I will also attende the conference and therefor try to post some updates here.

To read more about this topic you can read my publication about marketing scrum. It has the extensive research paper I publisched about this story. Please feel free to give me comments and questions either about agile 2014 or the paper.

 

Enjoy reading the paper:

Marketing scrum vs IT scrum – two marketing case studies who now ‘act first and apologize later'

 

WorldCup 2014 Retrospective: The Magic about Mindset and Leadership

Sun, 07/13/2014 - 13:56

This weekend preparing this bjohan-cruijff-hollande-1974logpost, I ran into a brilliant quote from Johan Cruijff. At a conference a few years ago for the Dutch local government, he told a great story about a talented blind golfer, Ronald Boef he played golf with.  Despite his handicap, Ronald Boef played his best golf in difficult mental circumstances like playing balls over a big pond or consistent putting. The conclusion of Johan Cruijff: "Ronald doesn’t “see" the problems, he is only focussing on the next target. He thinks from a positive mindset".   I couldn’t agree more.  In my opinion, this is one of the fundamentals behind eXtreme Manufacturing (XM) and the reason why the Dutch team didn’t made it through the WorldCup finals.

Like many consultants, topsport is an inspiring source for me.  Almost every day I show or tell stories from great sport coaches like Marc Lammers or Johan Cruijjff.   Like every major sports event, also this WorldCup in Brasil contained some interesting lessons for me I wanted to share with you.

The Big Question: You can have the best individual team members but still not be able to perform.  Why?

 3rd Place Playoff - 2014 FIFA World Cup Brazil

Top Team Ingredient #1: Mindset

The defeat of Spain against the Netherlands, the glorious win of Germany over Brazil showed having fun, faith and determination pay off and a lack of these ingredients will bring you in a lot of trouble.   Until the penalty series of the semi-finals, the right side of this recipe also worked for the Dutch squad. Now, penalty series are for no one a fun exercise, which only leaves faith and determination.   Unlike the previous penalty series against Costa Rica, the Dutch team had no faith in their keeper as a penalty-killer which directly effected the teams determination. They became more hesitant and aware of what could happen when missing a penalty.  Yes, Ronald Boef probably would have taken the penalties better than the Dutch team did against Argentina.. ;-)

Top Team Ingredient #2: Leadership

Like Johan Cruijjf stated in the same video, the leader on the pitch should be 100% concentrated on every detail and also (in my words) be the natural leader of the team, coaching them in keeping the spirit up and giving them enough room “to grow".  Despite his great qualities as a football-player, as a captain Robin van Persie was obviously not the natural leader of the team. Arjan Robben was. The natural leadership of Arjan Robben in combination with his determination was an important reason why The Netherlands were able to regain their motivation and pull off a highly respected 3rd place in this WorldCup.

In my opinion, a high performing team should always have a natural leader.  The options:

  1. A formal leader with natural leadership qualities is the perfect combination.
  2. A formal leader without natural leadership qualities but able to delegate this to another team member is also okay.
  3. A formal leader without natural leadership qualities and ignoring don’t having this competence, is bad news for the team, the team’s environment but above all, for the formal leader himself.

For the new coach van the Dutch team, Guus Hiddink, it will be a challenge convincing Robin van Persie to step back as the 1st captain after nominating Arjan Robben.  Robin van Persie should keep one thing in mind here:  no one is doubting his qualities as a top world class striker.  As a natural leader however, he is not that world class.  Trying to be one is effecting his performance as a world class striker and that would in the end be a disappointment for his supporters but above all, for Robin van Persie himself.

What does this imply for Leadership within organizations?

Leadership, especially natural leadership, is crucial for having highly motivated and productive teams.  The team stays motivated and focussed on their goal.

How ever, a lot of employees are still instrumentally “nominated” to become a coach or manager without having any leadership skills.  In my opinion, natural leadership is something you can’t gain by nomination or just by learning it.  You can improve it, but there should be some basis of natural leadership.  Ignoring this can be even counter-productive: conflicts will arise, the spirit and productivity will go down.

Why Little's Law Works...Always

Fri, 07/11/2014 - 12:00

On the internet there is much information on Little's Law. It is described an explained in many places [Vac]. Recently, the conditions under which it is true got  attention [Ram11]. As will be explained in this blog the conditions under which the law is true are very mild. It will be shown that for teams working on backlog items virtually there are no conditions.

Why does it work? Under what conditions does it hold?

Airplane Folding

In the previous post (Applying Little's Law in Agile Games) I described how to calculate the quantities in Little's Law. As an example this was applied to the Agile game of folding airplanes consisting of 3 rounds of folding.

airplane-1

Let's look in more detail in which round an airplane was picked up and in which round it was completed. This is depicted in the following figure.

The horizontal axis shows the number of rounds. The vertical axis describes each airplane to fold. The picture is then interpreted as follows. Airplane no. 2 is picked up in round 1 and competed in the same round. It has a waiting time of 1 round. This is indicated at the right of the lowest shaded rectangle.
Airplane no. 8 was picked up in round 1 and finished in round 3. A waiting time of 3 rounds. Airplane no 12 (top most shaded area) was picked up in round 3 and unfinished. Up to round 3 a waiting time of 1 round.

The number 3, 5, and 10 denote the number of completed airplanes at the end of round 1, 2, and 3 respectively.

Waiting Times

The waiting times are determined by counting the number of 'cells' in a row.

The pictures show that we have 12 airplanes (12 'rows'), 3 completed in the first round, 2 more completed in the second round and 5 additionally  folded airplanes in the third and last round giving a total of 10 finished paper airplanes.

All twelve airplanes have waiting times of 1, 1, 1, 2, 2, 3, 3, 3, 3, 3, 3, and 1 respectively.

Work in Progress

In the figure below the number of airplanes picked up by the team in each round are indicated with red numbers above the round. new doc 6_2
In round 1 the team has taken up the folding of 11 airplanes (of which 3 are completed). In round 2 the team was folding 8 airplanes (of which 2 were competed) and in round 3 the team was folding 7 airplanes (of which it completed 5).

Work in progress is determined by counting the number of 'cells' in a column.

Little's Law.....Again

Now that we have determined the waiting times and amount of work in progress, let's calculate the average waiting time and average work in progress.

Average Waiting Time. This quantity we get by adding all waiting times and dividing by the number of items. This gives 26/12.

Average Work in Progress. This quantity is equal to (11+8+7)/3 = 26/3.

Average input rate. This is equal to 12 (the height of the third column) divided by 3 which gives 4.

Again we find that: Average Waiting Time = Average Work in Progress / Average input rate.

Why It Works

new doc 6_3 Little's Law works....always....because the average waiting times is got by adding the lengths of all the rows dividing by the number of rows, so it is proportional to the size of the shaded area in the picture to the right.

The average work in progress is got by adding the heights of the columns in the shaded area which is also proportional to the size of the shaded area.

Both the waiting time and work in progress relate to the size of the shaded area: one by adding the heights and the other by adding the rows. The proportionality corresponds to the average input rate.

Conditions

What assumptions did we make? None...well this is not exactly true. The only assumptions we make in this calculation:

  • We count discrete items
  • There are a finite number of rounds (or sprints)
  • Items enter and possibly leave the system.

That's it. It doesn't need to be stable, ageing (items having increasingly larger waiting times) is not a problem, prioritisation/scheduling of items (also known as queueing discipline), etc. Only the above assumptions need to be true.

Note: Especially the second condition is important, i.e. Little's Law is measured over a finite time interval. For infinite time interval additional conditions need to be fulfilled.

Note: When applying this to agile teams we always consider finite time intervals, e.g. 6 months, 1 year, 8 sprints, etc.

Conclusion

Little's Law is true because the average waiting time is proportional to the size of the shaded area (see figure) and the average work in progress is also proportional to the size of the same shaded area.

Only 3 basic conditions need to be met for Little's Law to be true.

References

[Vac] Little’s Law Part I, Dan Vacanti, http://corporatekanban.com/littles-law-part-i/

[Ram11] Little’s Law – It’s not about the numbers, Agile Ramblings, http://agileramblings.com/2012/12/11/littles-law-its-not-about-the-numbers/

Bootstrapping and monitoring multiple processes in Docker using monit

Thu, 07/10/2014 - 23:01

If you have every tried to start a docker container and keep it running, you must have encountered the problem that this is no easy task. Most stuff I like to start in container are things like http servers, application servers and various other middleware components which tend to have start scripts that daemonize the program. Starting a single process is a pain, starting multiple processes becomes nasty. My advise is to use monit to start all but the most simple Docker application containers! When I found monit while delving through the inner works Cloud Foundry, I was ecstatic about it! It was so elegant, small, fast, with a beautiful DSL that I thought it was the hottest thing since sliced bread! I was determined to blog it off the roof tops. Until.... I discovered that the first release dated from somewhere in 2002. So it was not hot and new; Clearly I had been lying under a UNIX rock for quite a while. This time, the time was right to write about it! Most of the middleware components I want to start in a docker container, have a habit to start the process, daemonize it and exit immediately, with the docker container on its tail. My first attempt to circumvent this while starting a tomcat server in Docker looked something like this:

/bin/bash -c "service tomcat7 start;while service tomcat7 status;do sleep 1;done

Quite horrific. Imaging the ugliness when you have to start multiple processes. A better solution is needed:  With the zabbix docker container  the problem was solved using simplevisor. As you can read in this post that was not a pleasant experience either. As I knew little about simplevisor and could not solve the problem, I put in an issue and  resorted to a plain installation. But a voice in my head started nagging: "Why don't you fix it and send a pull request?"  (actually, it was the voice of my colleague Arjan Molenaar). Then, I remembered from my earlier explorations to the inner workings of Cloud Foundry, a tool that would be really suitable for the job: monit. Why? It will:

  1. Give you a beautiful,readable specification file stating which processes to start
  2. Make sure that your processes will keep on running
  3. Deliver you a clean and crisp monitoring application
  4. Reduce all your Docker starts to a single command!

In the case of the Zabbix server there were seven processes to start: the zabbix server, agent, java agent, apache, mysql and sshd. In monit this looks as follows:

check process mysqld with pidfile /var/run/mysqld/mysqld.pid
        start program = "/sbin/service mysqld start"
        stop program = "/sbin/service mysqld stop"

check process zabbix-server with pidfile /var/run/zabbix/zabbix_server.pid
        start program = "/sbin/service zabbix-server start"
        stop program = "/sbin/service zabbix-server stop"
        depends on mysqld

check process zabbix-agent with pidfile /var/run/zabbix/zabbix_agentd.pid
        start program = "/sbin/service zabbix-agent start"
        stop program = "/sbin/service zabbix-agent stop"

check process zabbix-java-gateway with pidfile /var/run/zabbix/zabbix_java.pid
        start program = "/sbin/service zabbix-java-gateway start"
        stop program = "/sbin/service zabbix-java-gateway stop"

check process httpd with pidfile /var/run/httpd/httpd.pid
        start program = "/sbin/service httpd start"
        stop program = "/sbin/service httpd stop"
        depends on zabbix-server

check process sshd with pidfile /var/run/sshd.pid
        start program = "/sbin/service sshd start"
        stop program = "/sbin/service sshd stop"

Normally when you start monit it will start as a daemon. But fortunately, you can prevent this with the following configuration.

set init

Your Dockerfile CMD can now always look the same:

    monit -d 10 -Ic /etc/monitrc

Finally, by adding the following statement to the configuration you get an application to view the status of your container processes,

set httpd
     port 2812
     allow myuser:mypassword

After starting the container, surf to port 2812 and you will get a beautiful page showing the state of your processes and the ability to stop and restart them.

monit overview monit process control

Just delve into the documentation of monit and you will find much more features that will allow you to monitor network ports and files, start corrective actions and send out alerts.

Monit is true to its UNIX heritage: it is elegant and promotes an autonomous monitoring system. Monit is cool!

Applying Little's Law in Agile Games

Mon, 07/07/2014 - 21:20

Have you ever used Little's Law to explain that lower WiP (work in progress) limits lead to shorter cycle times? Ever tried to illustrate Little's Law in an Agile game and found it doesn't hold? Then read this blog to discover that it is exactly true in Agile games and how it really works.

Some time ago I gave a kanban workshop. Part of the workshop was a game of folding paper airplanes to illustrate flow. To illustrate Little's Law we determined the throughput, cycle time and work in progress. To my surprise the law didn't hold. Not even close. In this blog I want to share the insight into why it does work!

Introduction

It is well known that the average number of items in progress is proportional to the average cycle time of completed work items. The proportionality is the average input rate (or throughput rate) of work items. This relation is known as Little's Law. It was discovered by Little in the 1960s and has found many applications.

In kanban teams this relationship is often used to qualitatively argue that it is favourable for flow to have not too much work in parallel. To this end WiP (work in progress) limits are introduced. The smaller the WiP the smaller the average cycle time which means better flow.

A surprise to me was that it is exactly true and it remains true under very relaxed conditions.

Little's Law

In mathematical form the law is often stated as:

(1)

 \bar{N} = \lambda \bar{W} \bar{N} = \lambda \bar{W}

Here  \bar{N} \bar{N} is the average number of work items in progress at a certain time, and  \bar{W} \bar{W} is the average cycle time.  \lambda \lambda is the average input rate (new work items per unit of time). In stable systems this also equals the average throughput. In this case Little's Law is often (re)stated as

(2)

 \frac{\mathrm{Work\, in\, progress}}{\mathrm{Throughput}} = \mathrm{Cycle Time} \frac{\mathrm{Work\, in\, progress}}{\mathrm{Throughput}} = \mathrm{Cycle Time}

Conditions

In practise one considers Little's Law over a finite period of time, e.g. 6 months, 5 sprints, 3 rounds in an Agile game. Also in practise, teams work on backlog items which are discrete items. After the work is done this results in a new product increment.

Under the following conditions (1) is exact:

  • The system is observed over a finite period of time,
  • The system is a queueing system.

A queuing system is a system that consists of discrete items which arrive at a certain rate, receive service after which they depart.

Examples of a queueing system. An agile team works on backlog items. A kanban team that works on production incidents. A scrum team.

Agile Game

An often used game for explaining the importance of flow to team is the game of folding paper airplanes. Many forms of this games exist. See e.g. [Heintz11].

For this blog's purpose consider a team that folds air planes. The backlog is a stack of white paper. 3 Rounds of folding are done. Airplanes that are folded and fly at least 2 meters are considered done.

new doc 5_1-1

At the end of each round we will collect the following metrics:

  • number of completed airplanes
  • number of airplanes in progress and not yet finished.

The result of the the 3 rounds are shown at the right. At the end of round 1 Team A completed 3 airplanes and having 8 unfinished airplanes. Likewise, Team B finished 4 airplanes in round 3 giving a total of 12 finished airplanes and having 6 unfinished airplanes in progress.

The cycle time are got by writing the round number of the sheet of paper when starting to fold the airplane. When done, write the round number of the paper. The cycle time for one airplanes is got by subtracting the two and adding 1.

Calculating Little's Law

The way I was always calculating the number for work in progress, throughput and cycle time has been

  1. averaging cycle time for all completed airplanes,
  2. averaging the throughput over all rounds,
  3. averaging the work in progress over all rounds.

When calculated at the end of round 3, for Team A this amounts to:

  • Average work in progress = (8+6+2)/3 = 16/3,
  • Average throughput = 10 (completed airplanes)/3 = 10/3,
  • Average cycle time = 22/10 = 11/5

Using (2) above we get: 16/3 / (10/3) = 8/5. This is not equal to the average cycle time of 11/5. Not even close. How come?

The Truth

The interpretation of work in progress, throughput and cycle time I got from working with cumulative flow diagrams. There are many resources explaining these, see e.g. [Vega2011].

The key to the correct interpretation is choosing the time interval for which to measure the quantities  \bar{W} \bar{W} ,  \lambda \lambda , and  \bar{W} \bar{W} . Second, using the input rate instead of the throughput. Third, at the end of the time period include the unfinished items. Last, in calculating  \bar{N} \bar{N} consider all items that went through the system.

When we reinterpret the results for teams A and B we get

Team A

  • Average work in progress
    In round 1 3 airplanes were completed and left 8 unfinished; a total of 11 for work in progress (11 airplanes picked up as work)
    In round 2 the team completed 2 airplanes and have 6 unfinished; a total of 8
    In round 3 the team finished an additional 5 airplanes and left 2 uncompleted; a total of 7
    When measured over 3 rounds an average of (11+8+7)/3 = 26/3
  • Average input rate
    Using the input rate:
    In round 1 the team picked up 11 new airplanes
    In round 2 the team picked up no additional airplanes
    In round 3 one new airplanes was picked up.
    An average input rate of (11+0+1)/3 = 4 airplanes per round
  • Average cycle time
    At the end of the third round 2 airplanes are left in progress; one taken up in the third round having a waiting time of 1 and one left from the first round having waiting time of 3 rounds. A total waiting time of 22 + 3 + 1 = 26 rounds.
    Averaging over 12 airplanes we have an average cycle time of 26/12 = 13/6 rounds per airplane.

Dividing the average work in progress by the average input rate we get 26/3 divided by 4 = 26/12(!). This is exactly equal to the calculated average cycle time!

Team B

In a similar manner the reinterpreted results for team B are:

  • Average work in progress = (13+14+10)/3 = 37/3 airplanes,
  • Average input rate = (13+2+3)/3 = 6 airplanes per round,
  • Average cycle time = (27 (completed) + 10 (unfinished))/18 (airplanes) = 37/18 rounds per airplane

Again, dividing the average work in progress by the average input rate we get 37/18 rounds per airplane, which again is exactly equal to the average cycle time or waiting time!

Note: the cycle time of 10 days is built up by (a) 1 airplane from round 1 (cycle time of 3), 2 airplanes picked up in round 2 (total of 4 rounds), 3 airplanes picked up in round 3 (total of 3 rounds).

What About Cumulative Flow Diagrams?

Now that we understand how to calculate the quantities in Little's Law, we go back to cumulative flow diagrams. How come Little's Law works in this case.

In the case of teams that have collected data on cycle time, work in progress and throughput Little's Law work when done as explained in the section 'Calculating Little's Law' because:

  1. the teams are kept stable by having WiP limits on the left most column ("To Do"); then the throughput is more or less equal to the input rate,
  2. the team has completed a fairly large amount of work items in which case the waiting time of unfinished work items can be neglected,
  3. when measured over the (large part of the) value creation process, the completed items per time period can often be neglected in the calculation of the average work in progress.
Summary

Little's Law (1) holds under the conditions that (a) the system considered is a queueing system and (b) the observation or measurements are done over a finite time interval. It then holds independently of the stationaryness of the probability distributions, queuing discipline, emptiness of the system at the start and end of the time interval.

Calculate the quantities  \bar{N} \bar{N} ,  \lambda \lambda , and  \bar{W} \bar{W} as follows:

  • Average work in progress  \bar{N} \bar{N}
    For each time interval considered count the total amount of work in the system and add any items completed in that time interval.
  • Average cycle time  \bar{W} \bar{W}
    Sum the cycle times for all completed items and include the waiting time for unfinished items and divide by the total number of items.
  • Average input rate  \lambda \lambda
    Add the total number of items that entered the system and divide by the total number of time intervals.
References

[Little61] Little, J. D. C. 1961. A proof for the queuing formula: L = ãW . Oper. Res. 9(3) 383–387.

[Heintz11] John Heintz, June 2011, Agile Airplane Game, GistLabs, http://gistlabs.com/2011/06/agile-airplane-game/

[Vega11] Vega Information System Services, Inc., September 2011, Basics of Reading Cumulative Flow Diagrams, http://www.vissinc.com/2011/09/29/basics-of-reading-cumulative-flow-diagrams/

 

Create the smallest possible Docker container

Fri, 07/04/2014 - 21:59

When you are playing around with Docker, you quickly notice that you are downloading large numbers of megabytes as you use preconfigured containers. A simple Ubuntu container easily exceeds 200MB and as software is installed on top of it, the size increases. In some use cases, you do not need everything that comes with Ubuntu. For example, if you want to run a simple web server, written in Go, there is no need for any tool around that at all.

I have been searching for the smallest possible container to start with and found this one:

docker pull scratch

The scratch image is perfect. Literally perfect! It is elegant, small and fast. It does not contain any bugs, security leaks, slow code or technical debt. And that is because it is basically empty. Except for a bit of metadata added by Docker. In fact, you could have created this scratch image yourself with this command as described in the Docker documentation:

tar cv --files-from /dev/null | docker import - scratch

 

So that is it, the smallest possible Docker image. End of blog post!

... or is there something more we can say about this? For example, how do you use the scratch base image? It turns out this brings some challenges of its own.

Creating content for the scratch image

What can we run on an empty base image? An executable without dependencies. Do you have executables without dependencies?

I used to write code in Python, Java and JavaScript. Each of these languages/platforms require a runtime installed. Recently, I started looking into the Go (or GoLang if you prefer) platform. And it seems (spoiler alert) like Go is statically  linked. So I tried compiling a simple web server saying Hello World and running it within the scratch container. Here is the code for the Hello World web server:

package main

import (
	"fmt"
	"net/http"
)

func helloHandler(w http.ResponseWriter, r *http.Request) {
	fmt.Fprintln(w, "Hello World from Go in minimal Docker container")
}

func main() {
	http.HandleFunc("/", helloHandler)

	fmt.Println("Started, serving at 8080")
	err := http.ListenAndServe(":8080", nil)
	if err != nil {
		panic("ListenAndServe: " + err.Error())
	}
}

 

Obviously, I cannot compile my webserver inside the scratch container as there is no Go compiler in it. And as I am working on a Mac, I also cannot compile a Linux binary just like that. (Actually, it is possible to cross-compile GoLang sources to different platforms, but that is material for another blog post)

So I first need a Docker container with a Go compiler. Let's start simple:

docker run -ti google/golang /bin/bash

 

Inside this container, I can build the Go web server, which I have committed in a GitHub repository:

go get github.com/adriaandejonge/helloworld

 

The go get command is a variant of the go build command that allows fetching and building remote dependencies. You can start the resulting executable with:

$GOPATH/bin/helloworld

 

This works. But it is not what we want. We need the hello world container to run inside the scratch container. So, in fact, we need a Dockerfile saying:

FROM scratch
ADD bin/helloworld /helloworld
CMD ["/helloworld"]

 

and then start that. Unfortunately, the way we started the google/golang container, there is no way to build this Dockerfile. So first, we need a way to access Docker from within the container.

Calling Docker from within Docker

When you use Docker, sooner or later you run into the need to control Docker from within Docker. There are multiple ways to accomplish this. You could use recursion and run Docker inside Docker. However, that seems overly complex and again leads to large containers. You can also provide access to the Docker server outside the instance with a few additional command line options:

docker run -v /var/run/docker.sock:/var/run/docker.sock -v $(which docker):$(which docker) -ti google/golang /bin/bash

 

Before you continue, please rerun the Go compiler, as Docker forgot our previous compilation during the restart:

go get github.com/adriaandejonge/helloworld

 

When starting the container, the -v flag creates a volume inside the Docker container and allows you to provide a file from the Docker machine as input. The /var/run/docker.sock is the Unix socket that allows access to the Docker server. The $(which docker) part is a clever way to provide the path for the docker executable inside the container without hardcoding it. However, be careful when you use this command on an Apple when using boot2docker. If the docker executable is installed in a different location than it is installed in boot2docker's virtual machine, this results in a mismatch. It will be the executable inside the boot2docker virtual server that gets inserted into the container. So you may want to replace $(which docker) with /usr/local/bin/docker which is hardcoded. Similarly, if you run a different system, there is a chance that the /var/run/docker.sock has a different location and you need to adjust it accordingly.

Now you can use the Dockerfile inside the google/golang container in the $GOPATH directory, which points to /gopath in this example. Actually, I already checked this Dockerfile into GitHub. So you can copy it from the Go build directory to the desired location like this:

cp $GOPATH/src/github.com/adriaandejonge/helloworld/Dockerfile $GOPATH

 

You need to copy this as the compiled binary is now located in $GOPATH/bin and it is not possible to include files from parent directories when building a Dockerfile. So after copying, the next step is:

docker build -t adejonge/helloworld $GOPATH

 

And if all goes, well, Docker responds with something like:

Successfully built 6ff3fd5a381d

 

Which allows you to run the container:

docker run -ti --name hellobroken adejonge/helloworld

 

But unfortunately, now Docker responds with:

2014/07/02 17:06:48 no such file or directory

 

So what is going on? We have a statically linked executable inside a scratch container. Did we make a mistake?

As it turns out, Go does not statically link libraries. Or at least not all libraries. Under Linux, we can see the dynamically linked libraries for an executable with the ldd command:

ldd $GOPATH/bin/helloworld 

 

Which responds with:

linux-vdso.so.1 => (0x00007fff039fe000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f61df30f000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f61def84000)
/lib64/ld-linux-x86-64.so.2 (0x00007f61df530000)

 

So before we can run the Hello World webserver, we need to tell the Go compiler to actually do static linking.

Creating statically linked executables in Go

In order to create statically linked executables, we need to tell Go to use the cgo compiler rather than the go compiler. The command to do so is:

CGO_ENABLED=0 go get -a -ldflags '-s' github.com/adriaandejonge/helloworld

 

The CGO_ENABLED environment variable tells Go to use the cgo compiler rather than the go compiler. The -a flag tells Go to rebuild all dependencies. Otherwise you still end up with dynamically linked dependencies. And finally the -ldflags '-s' flag is a nice extra. It reduces the file size of the resulting executable by roughly 50%. You can also do this without the cgo compiler. The size reduction is a result from removing debug information.

Just to be sure, rerun the ldd command.

ldd $GOPATH/bin/helloworld 

 

It should now respond with:

not a dynamic executable

 

You can also rerun the steps for creating the Docker container around the executable from scratch:

docker build -t adejonge/helloworld $GOPATH

 

And if all goes well, Docker responds with something like:

Successfully built 6ff3fd5a381d

 

Which allows you to run the container:

docker run -ti --name helloworld adejonge/helloworld

 

And this time it should respond with:

Started, serving at 8080

 

Until so far, there were many manual steps and there is a lot of room for error. Let's exit from the google/golang container and continue from the surrounding machine:

<Press Ctrl-C>
exit

 

You can check the existence or absence of containers and images with:

docker ps -a
docker images -a

 

And you can do some cleaning of Docker with:

docker rm -f helloworld
docker rmi -f adejonge/helloworld

 

Creating a Docker container that creates a Docker container

The steps we took so far, we can also record in a Dockerfile and have Docker do the work for us:

FROM google/golang
RUN CGO_ENABLED=0 go get -a -ldflags '-s' github.com/adriaandejonge/helloworld
RUN cp /gopath/src/github.com/adriaandejonge/helloworld/Dockerfile /gopath
CMD docker build -t adejonge/helloworld gopath

 

I checked this Dockerfile into a separate GitHub repository called adriaandejonge/hellobuild. It can be built with this command:

docker build -t adejonge/hellobuild github.com/adriaandejonge/hellobuild

 

Providing the  -t flag names the image as adejonge/hellobuild and implicitly tags it as latest. These names make it easier for you to remove the image later on. Next,  you can create a container from this image while providing the flags that you have seen earlier in this post:

docker run -v /var/run/docker.sock:/var/run/docker.sock -v $(which docker):$(which docker) -ti --name hellobuild adejonge/hellobuild

 

Providing the --name hellobuild flag makes it easier to remove the container after running. In fact, you can do so right away, because after running this command, you already created the adejonge/helloworld image:

docker rm -f hellobuild
docker rmi -f adejonge/hellobuild

 

And now you can start a new container named helloworld based on the adejonge/helloworld image as you have done before:

docker run -ti --name helloworld adejonge/helloworld

 

Because all these steps are run from the same command line, without opening a bash shell inside a Docker container, you can add these steps to a bash script and run it automatically. For your convenience, I have added these bash scripts to the hellobuild GitHub repository.

Also, if you want to try the smallest possible Docker container running a Hello World web server without following all the steps described in this blog post, you can also use the pre-built image that I checked into the Docker Hub repository:

docker pull adejonge/helloworld

 

With docker images -a you can see that the size is 3.6MB. Of course, you can make it even smaller if you manage to create an executable that is smaller than the web server in Go that I wrote. In C or Assembly you may be able to do so. However, you can never make it smaller than the scratch image.

Dockerfiles as automated installation scripts

Thu, 07/03/2014 - 19:16

Dockerfiles are great and easily readable specifications for the installation and configuration of an application. It is terse, can be understood by anyone who understands UNIX commands, results in a testable product and can easily be turned into an automated installation script using a little awk'ward magic. Just in case you want to install the application in question on the good old fashioned way, without the Docker hassle :-)

In this case, we needed to experiment with the Codahale Metrics library and Zabbix. Instead of installing a complete Zabbix server, I googled for a docker container and was pleased to find a ready to run Zabbix server configuration created by Bernardo Gomez Palacio. . Unfortunately, the server stopped repeatedly after about 5 minutes due the simplevisor's impression that it was requested to stop. I could not figure out where this request was coming from, and as it was pretty persistent, I decided to install zabbix on a virtual box.

So I checked out the  docker-zabbix github project and found a ready to run Vagrant configuration to build the zabbix docker container itself (Cool!). The Dockerfile contained easily and readable instructions on how to install and configure Zabbix. But,  instead of copy-and-pasting the instructions to the command prompt, I cloned the project on the vagrant box and created the following awk script in order to execute the instructions in the Dockerfile directly on the running system.

/^ADD/ {
sub(/ADD/, "")
    cmd = "mkdir -p $(dirname " $2 ")"
    system(cmd)
    cmd = "cp " $0
    system(cmd)
}

/^RUN/ {
    sub(/RUN/, "")
    cmd = $0
    system(cmd)
}

After a few minutes, the image was properly configured. I just needed to run the database initialisation script (/start.sh) and ensured that all the services were started on reboot.

 cd /etc/init.d
for i in zabbix* httpd mysqld snmp* ; do
     chkconfig $i on
     service $i start
done

Even if you do not use Docker in production, Dockerfiles are a great improvement in the specifications of installation instructions!

How architecture enables kick ass teams (1): replication considered harmful?

Thu, 07/03/2014 - 11:51

At Xebia we regularly have discussions regarding Agile Architecture? What is it? What does it take? How should you organise this? Is it technical or organisational? And much more questions
 which I won’t be answering today. What I will do today is kick off a blog series covering subjects that are often part of these heated debates. In general what we strive for with Agile Architecture is an architecture that enables the organisation to keep moving fast and without IT be a limiting factor for realising changes. As you read this series you’ll start noticing one theme coming back over and over again: Autonomy. Sometimes we’ll be focussing on the architecture of systems, sometimes on the architecture of the organisation or teams, but autonomy is the overarching theme. And if you’re familiar with Conways Law it should be no surprise that there is a strong correlation between team and system structure. Having a structure of teams  that is completely different from your system landscape causes friction. We are convinced that striving for optimal team and system autonomy will lead to an organisation which is able to quickly adapt and respond to changes.

The first subject is replication of data, this is more a systems (landscape) issue and less of an organisational issue and definitely not the only one, more posts will follow.

We all have to deal with situations where:

  • consumers of a data retrieval service (e.g. customer account details) require this service to be highly available, or
  • compute intensive analysis must be done using the data in a system, or
  • data owned by a system must be searched in a way that is not (efficiently) supported by that system

These situations all impact the autonomy of the system owning the data.Is the system able to provide the it's functionality at the require quality level or do these external requirements lead to negative consequences on quality of the service provided or maintainability? Should these requirements be forced into the system or is another approach more appropriate?

Above examples all could be solved by replicating data into another system which is more suitable for meeting these requirements but 
 replication of data is considered to be harmful by some. Is it really? Often mentioned reasons not to replicate data are:

  • The replicated data will always be less accurate and timely than the original data
    True, and is this really a problem for the specific situation you’re dealing with? Sometimes you really need the latest version of a customer record, but in many situations it is no problem is the data is seconds, minutes or even hours old.
  • Business logic that interprets the data is implemented twice and needs to be maintained
    Yes, and you have to compare the costs of this against the benefits. As long as the benefits outweigh the costs, it is a good choice.  You can even consider to provide a library that is used in both systems.
  • System X is the authoritative source of the data and should be the only one that exposes it
    Agree, and keeping that system as the authoritative source is good practice and does not mean that there could be read only access to the same (replicated) data in other systems.

As you can see it is never a black and white decision, you’ll have to make a balanced decision and include benefits and costs of both alternatives. The gained autonomy and business benefits derived from this can easily outweigh the extra development, hosting and maintenance costs of replicating data.

A few concrete examples from my own experience:

We had a situation where a CRM system instance owned data which was also required in a 24x7 emergency support proces. The data was nicely exposed by a number of data retrieval services. At that organisation the CRM system deployment was such that most components were redundant, but during updates the system as a whole would still be down for several hours. Which was not acceptable given that the data was required in a 24x7 emergency support process. Making the CRM system deployment upgradable without downtime was not possible or would cost .
In this situation the costs of replicating the CRM system database to another datacenter using standard database features and having the data retrieval services access either that replicated database or the original database (as fall back) was much cheaper than trying to make CRM system itself high available. The replicated database would remain running accessible even when CRM system  got upgraded. Yes, we’re bypassing the CRM system business logic for interpreting the data, but for this situation the logic was so simple that the costs of reimplementing and maintaining this in a new lightweight service (separate from CRM system) were neglectable.

Another example is from a telecom provider that uses a chain of fulfilment systems in which it registered all network products sold to its customers (e.g. internet access, telephony, tv). Each product instance depends on instances registered in another system and if you drill down deep enough you’ll reach the physical network hardware ports on which it runs. The systems that registered all products used a relational model which was okay for registration. However, questions like “if this product instance breaks, which customers are impacted” were impossible to answer without overheating CPUs in those systems. By publishing all changes in the registrations to a separate system we could model the whole inventory of services as a network graph and easily do analysis on it without impacting the fulfilment systems. The fact that the data would be a (at most) a few seconds old was absolutely no problem.

And a last example is that sometimes you want to do a full (phonetic) text search through a subset of your domain model. Relational data models quickly get you into an unmaintainable situation. You’re SQL queries will require many tables, lot’s of inefficient “LIKE ‘%gold%’" and developers that have a hard time understanding what a query actually intended to do. Replicating the data to a search engine makes searching far easier and provides more possibilities for searches that are hard to realise in a relational database.

As you can see replication of data can increase autonomy of systems and teams and thereby make your system or system landscape and organisation more agile. I.e. you can realise new functionality faster and get it available for your users quicker because the coupling with other systems or teams is reduced.

In a next blog we'll discuss another subject that impacts team or system autonomy.

How combined Lean- and Agile practices will change the world as we know it

Tue, 07/01/2014 - 08:50

You might have attended this month at our presentation about eXtreme Manufacturing and the keynote of Nalden last week on XebiCon 2014. There are a few epic takeaways and additions I would like to share with you in this blogpost.

Epic TakeAway #1: The Learn, Unlearn and Relearn Cycle Like Nalden expressed in his inspiring keynote, one of the major things for him to be successful is being able to Learn, Unlearn and Relearn every time again. In my opinion, this will be the key ability for every successful company in the near future.  In fact, this is how nature evolutes: in the end, only the species who are able to adapt to changing circumstances will survive and evolute. This mechanism makes for example, most of the startups fail, but those who will survive, can be extremely disruptive for non-agile organizations.  Best example for this is of course Whatsapp.  Beating up the Telco Industry by almost destroying their whole businessmodel in only a few months. Learn more about disruptive innovation from one of my personal heroes, Harvard Professor Clayton Christensen.

Epic TakeAway #2: Unlearning Waterfall, Relearning Lean & Agile Globally, Waterfall is still the dominant method in companies and universities.  Waterfall has its origins more than 40 years ago. Times have changed. A lot. A new, successful and disruptive product could be there in only a matter of days instead of (many) years. Finally, things are changing. For example, the US Department of Defence has recently embraced Lean and Agile as mandatory practices, especially Scrum. Schools and universities are also more and more adopting the Agile way of working. Later more in this blogpost.

Epic TakeAway #3: Combined Lean- and Agile practices =  XM Lean practices arose in Japan in the 1980’s , mainly in the manufacturing industry, Toyota being the frontrunner here.  Agile practices like Scrum, were first introduced in the 1990’s by Ken Schwaber and Jeff Sutherland, these practices were mainly applied in the IT-industry. Until now, the manufacturing and IT world didn’t really joined forces combining Lean and Agile practices.  Until recently.  The WikiSpeed initiative of Joe Justice proved combining these practices result in a hyper-productive environment, where a 100 Mile/Gallon road legal sportscar could be developed in less than 3 months.  Out of this success eXtreme Manufacturing (XM) arose. Finally, a powerful combination of best practices from the manufacturing- and IT-world came together.

Epic TakeAway #4: Agile Mindset & Education fotoLike Sir Ken Robinson and Dan Pink already described in their famous TED-talks, the way most people are educated and rewarded, is not suitable anymore for modern times and even conflicts with the way we are born.  We learn by "failing", not by preventing it.  Failing in it’s essence should stimulate creativity to do things better next time, not be punished.  On the long run, failing (read: learning!) has more added value than short-term succes, for example by chasing milestones blindly. EduScrum in the Netherlands stimulates schools and universities to apply Scrum in their daily classes in order to stimulate creativity, happiness, self-reliantness and talent. The results of the schools joining these initiative are spectacular: happy students, less dropouts an significantly higher grades. For a prestigious project for the Delft University, Forze, the development of a hydrogen race car, the students are currently being trained and coached to apply Agile and Lean practices.  Also these results are more than promising. The Forze team is happier, more productive and more able to learn faster and better from setbacks.  Actually, they are taking the first steps of being anti-fragile.  Due too an intercession of the Forze team members themselves,  the current support of agile (Xebia) coaches is now planned being extended to the flagship of the Delft University:  the NUON solar team.

The Final Epic TakeAway In my opinion, we reached a tipping point in the way goals should be achieved.  Organizations are massively abandoning Waterfall and embracing Agile practices, like Scrum.  Adding Lean practices like Joe Justice did in his WikiSpeed project, makes Agile and Lean extremely powerful.  Yes, this will even make this world a much better place.  We cannot prevent nature disasters with this, but we can be anti-fragile.  We cannot prevent every epidemic, but we can respond in an XM-fashion on this by developing a vaccin in only days instead of years.  This brings me finally to the missing statement of the current Agile Manifesto:   We should Unlearn and Relearn before we Judge.  Dare to Dream like a little kid again. Unlearn your skepticism.  Companies like Boeing, Lockheed Martin and John Deere already did. Adopting XM speeded up their velocity in some cases with more than 7 times.

Mocking a REST backend for your AngularJS / Grunt web application

Thu, 06/26/2014 - 17:15

Anyone who ever developed a web application will know that a lot of time is spend in a browser to check if everything works as well and looks good. And you want to make sure it looks good in all possible situations. For a single-page application, build with a framework such as AngularJS, that gets all it's data from a REST backend this means you should verify your front-end against different responses from your backend. For a small application with primarily GET requests to display data, you might get away with testing against your real (development) backend. But for large and complex applications, you need to mock your backend.

In this post I'll go in to detail how you can solve this by mocking GET requests for an AngularJS web application that's built using Grunt.

In our current project, we're building a new mobile front-end for an existing web application. Very convenient since the backend already exists with all the REST services that we need. An even bigger convenience is that the team that built the existing web application also built an entire mock implementation of the backend. This mock implementation will give standard responses for every possible request. Great for our Protractor end-to-end tests! (Perhaps another post about that another day.) But this mock implementation is not so great for the non standard scenario's. Think of error messages, incomplete data, large numbers or a strange valuta. How can we make sure our UI displays these kind of cases correct? We usually cover all these cases in our unit tests, but sometimes you just want to see it right in front of you as well. So we started building a simple solution right inside our Grunt configuration.

To make this solution work, we need to make sure that all our REST requests go through the Grunt web server layer. Our web application is served by Grunt on localhost port 9000. This is the standard configuration that Yeoman generates (you really should use Yeoman to scaffold your project). Our development backend is also running on localhost, but on port 5000. In our web application we want to make all REST calls using the `/api` path so we need to rewrite all requests to http://localhost:9000/api to our backend: http://localhost:5000/api. We can do this by adding middleware in the connect:livereload configuration of our Gruntfile.

livereload: {
  options: {
    open: true,
    middleware: function (connect, options) {
      return [
        require('connect-modrewrite')(['^/api http://localhost:5000/api [P L]']),

        /* The lines below are generated by Yeoman */
        connect.static('.tmp'),
        connect().use(
          '/bower_components',
          connect.static('./bower_components')
        ),
        connect.static(appConfig.app)
      ];
    }
  }
},

Do the same for the connect:test section as well.

Since we're using 'connect-modrewrite' here, we'll have to add this to our project:

npm install connect-modrewrite --save-dev

With this configuration every request starting will http://localhost:9000/api will be passed on to http://localhost:5000/api so we can just use /api in our AngularJS application. Now that we have this working, we can write some custom middleware to mock some of our requests.

Let's say we have a GET request /api/user returning some JSON data:

{"id": 1, "name":"Bob"}

Now we'd like to see what happens with our application in case the name is missing:

{"id": 1}

It would be nice if we could send a simple POST request to change the response of all subsequent calls. Something like this:

curl -X POST -d '{"id": 1}' http://localhost:9000/mock/api/user

We prefixed the path that we want to mock with /mock in order to know when we should start mocking something. Let's see how we can implement this. In the same Gruntfile that contains our middleware configuration we add a new function that will help us mock our requests.

var mocks = [];
function captureMock() {
  return function (req, res, next) {

    // match on POST requests starting with /mock
    if (req.method === 'POST' && req.url.indexOf('/mock') === 0) {

      // everything after /mock is the path that we need to mock
      var path = req.url.substring(5);

      var body = '';
      req.on('data', function (data) {
        body += data;
      });
      req.on('end', function () {

        mocks[path] = body;

        res.writeHead(200);
        res.end();
      });
    } else {
      next();
    }
  };
}

And we need to add the above function to our middleware configuration:

middleware: function (connect, options) {
  return [
    captureMock(),
    require('connect-modrewrite')(['^/api http://localhost:5000/api [P L]']),

    connect.static('.tmp'),
    connect().use(
      '/bower_components',
      connect.static('./bower_components')
    ),
    connect.static(appConfig.app)
  ];
}

Our function will be called for each incoming request. It will capture each request starting with /mock as a request to define a mock request. Next it stores the body in the mocks variable with the path as key. So if we execute our curl POST request we end up with something like this in our mocks array:

mocks['/api/user'] = '{"id": 1}';

Next we need to actually return this data for requests to http://localhost:9000/api/user. Let's make a new function for that.

function mock() {
  return function (req, res, next) {
    var mockedResponse = mocks[req.url];
    if (mockedResponse) {
      res.writeHead(200);
      res.write(mockedResponse);
      res.end();
    } else {
      next();
    }
  };
}

And also add it to our middleware.

  ...
  captureMock(),
  mock(),
  require('connect-modrewrite')(['^/api http://localhost:5000/api [P L]']),
  ...

Great, we now have a simple mocking solution in just a few lines of code that allows us to send simple POST requests to our server with the requests we want to mock. However, it can only send status codes of 200 and it cannot differentiate between different HTTP methods like GET, PUT, POST and DELETE. Let's change our functions a bit to support that functionality as well.

 var mocks = {
  GET: {},
  PUT: {},
  POST: {},
  PATCH: {},
  DELETE: {}
};

function mock() {
  return function (req, res, next) {
    if (req.method === 'POST' && req.url.indexOf('/mock') === 0) {
      var path = req.url.substring(5);

      var body = '';
      req.on('data', function (data) {
        body += data;
      });
      req.on('end', function () {

        var headers = {
          'Content-Type': req.headers['content-type']
        };
        for (var key in req.headers) {
          if (req.headers.hasOwnProperty(key)) {
            if (key.indexOf('mock-header-') === 0) {
              headers[key.substring(12)] = req.headers[key];
            }
          }
        }

        mocks[req.headers['mock-method'] || 'GET'][path] = {
          body: body,
          responseCode: req.headers['mock-response'] || 200,
          headers: headers
        };

        res.writeHead(200);
        res.end();
      });
    }
  };
};

function mock() {
  return function (req, res, next) {
    var mockedResponse = mocks[req.method][req.url];
    if (mockedResponse) {
      res.writeHead(mockedResponse.responseCode, mockedResponse.headers);
      res.write(mockedResponse.body);
      res.end();
    } else {
      next();
    }
  };
}

We can now create more advanced mocks:

curl -X POST \
    -H "mock-method: DELETE" \
    -H "mock-response: 403" \
    -H "Content-type: application/json" \
    -H "mock-header-Last-Modified: Tue, 15 Nov 1994 12:45:26 GMT" \
    -d '{"error": "Not authorized"}' http://localhost:9000/mock/api/user

curl -D - -X DELETE http://localhost:9000/api/user
HTTP/1.1 403 Forbidden
Content-Type: application/json
last-modified: Tue, 15 Nov 1994 12:45:26 GMT
Date: Wed, 18 Jun 2014 13:39:30 GMT
Connection: keep-alive
Transfer-Encoding: chunked

{"error": "Not authorized"}

Since we thought this would be useful for other developers, we decided to make all this available as open source library on GitHub and NPM

To add this to your project, just install with npm:

npm install mock-rest-request --save-dev

And of course add it to your middleware configuration:

middleware: function (connect, options) {
  var mockRequests = require('mock-rest-request');
  return [
    mockRequests(),
    
    connect.static('.tmp'),
    connect().use(
      '/bower_components',
      connect.static('./bower_components')
    ),
    connect.static(appConfig.app)
  ];
}

How Agile accelerates your business

Wed, 06/25/2014 - 10:11

This drawing explains how agility accelerates your business. It is free to use and distribute. Should you have any questions regarding the subjects mentioned, feel free to get in touch.
Dia1

How to verify Web Service State in a Protractor Test

Sat, 06/21/2014 - 08:24

Sometimes it can be useful to verify the state of a web service in an end-to-end test. In my case, I was testing a web application that was using a third-party Javascript plugin that logged page views to a Rest service. I wanted to have some tests to verify that all our web pages did include the plugin, and that it was communicating with the Rest service properly when a new page was opened.
Because the webpages were written with AngularJS, Protractor was our framework of choice for our end-to-end test suite. But how to verify web service state in Protractor?

My first draft of a Protractor test looked like this:

var businessMonitoring = require('../util/businessMonitoring.js');
var wizard = require('./../pageobjects/wizard.js');

describe('Business Monitoring', function() {
  it('should log the page name of every page view in the wizard', function() {
    wizard.open();
    expect(wizard.activeStepNumber.getText()).toBe('1');

    // We opened the first page of the wizard and we expect it to have been logged
    expect(businessMonitoring.getMonitoredPageName()).toBe('/wizard/introduction');

    wizard.nextButton.click();
    expect(wizard.completeStep.getAttribute('class')).toContain('active');
    // We have clicked the ‘next’ button so the ‘completed’ page has opened, this should have // been logged as well
    expect(businessMonitoring.getMonitoredPageName()).toBe('/wizard/completed');
  });
});

The next thing I had to write was the businessMonitoring.js script, which should somehow make contact with the Rest service to verify that the correct page name was logged.
First I needed a simple plugin to make http requests. I found the 'request' npm package , which provides a simple API to make a http request like this:

var request = require('request');

var executeRequest = function(method, url) {
  var defer = protractor.promise.defer();
  
  // method can be ‘GET’, ‘POST’ or ‘PUT’
  request({uri: url, method: method, json: true}, function(error, response, body) {

    if (error || response.statusCode >= 400) {
      defer.reject({
        error : error,
        message : response
      });
    } else {
      defer.fulfill(body);
    }
  });

  // Return a promise so the caller can wait on it for the request to complete
  return defer.promise;
};

Then I completed the businessmonitoring.js script with a method that gets the last request from the Rest service, using the request plugin.
It looked like this:

var businessMonitoring = exports; 

< .. The request wrapper with the executeRequest method is included here, left out here for brevity ..>

businessMonitoring.getMonitoredPageName = function () {

    var defer = protractor.promise.defer();

    executeRequest('GET', 'lastRequest')  // Calls the method which was defined above
      .then(function success(data) {
        defer.fulfill(data,.url);
      }, function error(e) {
        defer.reject('Error when calling BusinessMonitoring web service: ' + e);
      });

    return defer.promise;
 };

It just fires a GET request to the Rest service to see which page was logged. It is an Ajax call so the result is not immediately available, so a promise is returned instead.
But when I plugged the script into my Protractor test, it didn't work.
I could see that the requests to the Rest service were done, but they were done immediately before any of my end-to-end tests were executed.
How come?

The reason is that Protractor uses the WebdriverJS framework to handle its control flow. Statements like expect(), which we use in our Protractor tests, don't execute their assertions immediately, but instead they put their assertions on a queue. WebdriverJS first fills the queue with all assertions and other statements from the test, and then it executes the commands on the queue. Click here for a more extensive explanation of the WebdriverJs control flow.

That means that all statements in Protractor tests need to return promises, otherwise they will execute immediately when Protractor is only building its test queue. And that's what happened with my first implementation of the businessMonitoring mock.
The solution is to let the getMonitoredPageName return its promise within another promise, like this:

var businessMonitoring = exports; 

businessMonitoring.getMonitoredPageName = function () {
  // Return a promise that will execute the rest call,
  // so that the call is only done when the controlflow queue is executed.
  var deferredExecutor = protractor.promise.defer();

  deferredExecutor.then(function() {
    var defer = protractor.promise.defer();

    executeRequest('GET', 'lastRequest')
      .then(function success(data) {
        defer.fulfill(data.url);
      }, function error(e) {
        defer.reject('Error when calling BusinessMonitoring mock: ' + e);
      });

    return defer.promise;
  });

  return deferredExecutor;
};

Protractor takes care of resolving all the promises, so the code in my Protractor test did not have to be changed.

Concordion without the JUnit Code

Fri, 06/20/2014 - 20:58

Concordion is a framework to support Behaviour Driven Design. It is based on JUnit to run tests and HTML enriched with a little Concordion syntax to call fixture methods and make assertions on test outcome. I won't describe Concordion because it is well documented here: http://concordion.org/.
Instead I'll describe a small utility class I've created to avoid code duplication. Concordion requires a JUnit class for each test. The utility I'll describe below allows you to run all Concordion tests without having a utility class for each test.

In Concordion you specify test cases and expected outcomes in a HTML file. Each HTML file is accompanied by a Java class of the same name that has the @RunWith(ConcordionRunner.class annotation. This Java class is comparable with Fitnesse's fixture classes. Here you can create methods that are used in the HTML file to connect the test case to the business code.

In my particular use case the team ended up writing lots of mostly empty Java files. The system under test was processing XML message files, so all the test needed to do was call a single method to hand the XML to the business code and then validate results. Each Java class was basically the same, except for its name.

To avoid this duplication I created a class that uses the JavaAssist library to generate a Java class on the fly and run it as a JUnit test. You can find my code on Github:

git clone git@github.com:jvermeir/concordionDemo

ConcordionRunner generates a class using a template. The template can be really simple like in my example where FixtureTemplate extends MyFixture. MyFixture holds all fixture code to connect the test to the application under test. This is where we would put all fixture code necessary to call a service using a XML message. In the example there's just the single getGreeting() method.
HelloWorldAgain.html is the actual Concordion test. It shows the call to getGreeting(), which is a method of MyFixture.
The dependencies are like this:
FixtureTemplate extends MyFixture
YourTest.html uses Concordion
Example uses ConcordionRunner uses JUnitCore
The example in Example.java shows how to use ConcordionRunner to execute a test. This could easily be extended to recursively go through a directory and execute all tests found. Note that Example writes the generated class to a file. This may help in troubleshooting but isn't really necessary.
Now it would be nice to adapt the Eclipse plugin to allow you to right click the HTML file and run it as a Concordion test without adding a unit test.

Deploying a Node.js app to Docker on CoreOS using Deis

Wed, 06/18/2014 - 17:00

The world of on-premise private PaaSes is changing rapidly. A few years ago, we were building on on-premise private PaaSes based upon the existing infrastructure and using Puppet as an automation tool to quickly provision new application servers.  We provided a self-service portal where development teams could get any type of server in any type of environment running within minutes.  We created a virtual server for each application to keep it manageable, which of course is quite resource intensive.

Since June 9th, Docker has been declared production ready, so this opens  the option of provisioning light weight containers to the teams instead of full virtual machine. This will increase the speed of provisioning even further while reducing the cost of creating a platform and minimising resource consumption.

To illustrate how easy life is becoming, we are going to deploy an original CloudFoundry node.js application to Docker on a CoreOS cluster. This hands-on experiment is based on MacOS, Vagrant and VirtualBox.

Step 1. Installing  etcdctl and fleetctl

Before you start, you need to install etcdctl and fleetctl on your host.  etcd is a nifty distributed key-value store while fleet manages the deployment of  (Docker) services to a CoreOS cluster.
$ brew install go etcdctl
$ git clone https://github.com/coreos/fleet.git
$ cd fleet && ./build && mv bin/fleetctl /usr/local/bin

 

Step 2. Install the Deis Command Line Client

To control the PaaS you need to install the Deis command line client:

$ brew install python
$ sudo pip install deis

Step 3. Build the platform

Deis  provides all the facilities for building, deployment and managing applications.

$ git clone https://github.com/deis/deis.git
$ cd deis
$ vagrant up

$ ssh-add ~/.vagrant.d/insecure_private_key
$ export DOCKER_HOST=tcp://172.17.8.100:4243
$ export FLEETCTL_TUNNEL=172.17.8.100
$ make pull

Step 4. Start the platform

Now all is set to start the platform:

$ make run

After this run has completed, you can see that the 7 components in the Deis Architecture have been started using the list-units command: The builder, the cache, the controller, the database, the logger, the registry and the router. This architecture looks quite similar to the architecture of CloudFoundry.

$ fleetctl list-units

UNIT STATE LOAD ACTIVE SUB DESC MACHINE deis-builder.service launched loaded active running deis-builder 79874bde.../172.17.8.100 deis-cache.service launched loaded active running deis-cache 79874bde.../172.17.8.100 deis-controller.service launched loaded active running deis-controller 79874bde.../172.17.8.100 deis-database.service launched loaded active running deis-database 79874bde.../172.17.8.100 deis-logger.service launched loaded active running deis-logger 79874bde.../172.17.8.100 deis-registry.service launched loaded active running deis-registry 79874bde.../172.17.8.100 deis-router.1.service launched loaded active running deis-router 79874bde.../172.17.8.100

Alternatively, you can inspect the result by looking inside the virtual machine:

$ vagrant ssh -c "docker ps"

Now we have our platform running, we can start using it!

Step 5. Register a new user to Deis and add the public ssh key

$ deis register 172.17.8.100:8000 \
     --username=mark \
     --password=goddesses \
     --email=mark.van.holsteijn@..com
$ deis keys:add ~/.ssh/id_rsa.pub

Step 6. Create a Cluster

Create a application cluster under the domain 'dev.172.17.8.100.xip.io'.  The --hosts specifies all hosts in the cluster: the only available host  at this moment in the cluster is 172.17.8.100.

$ deis clusters:create dev  dev.172.17.8.100.xip.io \
        --hosts=172.17.8.100 \
        --auth=~/.vagrant.d/insecure_private_key

Step 7. Get the app

We created a simple but effective  node.js application that show you what happens when you scale or push a new version of the application.

$ git clone git@github.com:mvanholsteijn/sample_nodejs_cf.git
$ deis apps:create appmon --cluster=dev
$ deis config:set RELEASE=deis-v1
$ git push deis master

Step 8. open your application

Voila! Your application is running. Now click on start monitoring.

$ deis apps:open 

you should see something like this:

app-mon-1

Step 9. scaling your application

To see scaling in action,  type the following command:

$ deis ps:scale web=4

It will start 3 new containers which will show up in the list.

app-mon-4

 

Step 10. upgrading your application

Now make a change to the application, for instance change the message to 'Greetings from Deis release' and push your change:

$ git commit -a -m "Message changed"
$ git  push deis master

After a while you will see the following on your monitor!

app-mon-8

 

Step 11. Looking on CoreOS

You can use  fleetctl again to look at the new services that have been added to the platform!

 

$ fleetctl list-units

UNIT STATE LOAD ACTIVE SUB DESC MACHINE app-mon_v7.web.1-announce.service launched loaded active running app-mon_v7.web.1 announce 79874bde.../172.17.8.100 app-mon_v7.web.1-log.service launched loaded active running app-mon_v7.web.1 log 79874bde.../172.17.8.100 app-mon_v7.web.1.service launched loaded active running app-mon_v7.web.1 79874bde.../172.17.8.100 app-mon_v7.web.2-announce.service launched loaded active running app-mon_v7.web.2 announce 79874bde.../172.17.8.100 app-mon_v7.web.2-log.service launched loaded active running app-mon_v7.web.2 log 79874bde.../172.17.8.100 app-mon_v7.web.2.service launched loaded active running app-mon_v7.web.2 79874bde.../172.17.8.100 app-mon_v7.web.3-announce.service launched loaded active running app-mon_v7.web.3 announce 79874bde.../172.17.8.100 app-mon_v7.web.3-log.service launched loaded active running app-mon_v7.web.3 log 79874bde.../172.17.8.100 app-mon_v7.web.3.service launched loaded active running app-mon_v7.web.3 79874bde.../172.17.8.100 app-mon_v7.web.4-announce.service launched loaded active running app-mon_v7.web.4 announce 79874bde.../172.17.8.100 app-mon_v7.web.4-log.service launched loaded active running app-mon_v7.web.4 log 79874bde.../172.17.8.100 app-mon_v7.web.4.service launched loaded active running app-mon_v7.web.4 79874bde.../172.17.8.100 deis-builder.service launched loaded active running deis-builder 79874bde.../172.17.8.100 deis-cache.service launched loaded active running deis-cache 79874bde.../172.17.8.100 deis-controller.service launched loaded active running deis-controller 79874bde.../172.17.8.100 deis-database.service launched loaded active running deis-database 79874bde.../172.17.8.100 deis-logger.service launched loaded active running deis-logger 79874bde.../172.17.8.100 deis-registry.service launched loaded active running deis-registry 79874bde.../172.17.8.100 deis-router.1.service launched loaded active running deis-router 79874bde.../172.17.8.100

 

Conclusion

Deis is a very simple and easy to use way to create a PaaS using Docker and CoreOS. The node.js application we created, could be deployed using Deis without a single modification. We will be diving into Deis and CoreOS in more details in following posts!

One Change at a Time

Tue, 06/17/2014 - 08:22

One of the twelve Agile principles states "At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behaviour accordingly" [1]. Many Agile teams have retrospectives every two weeks that result in concrete actions that are executed in the next time period (sprint). There are many types of tuning and adjustments that a team can do. Examples are actions that improve the flow of work, automation of tasks, team coorporation.

Is it good habit for retrospectives to focus on the same type of improvement or should the team alter the type of improvements that will be done? In this blog I will look into the effect of multiple consecutive actions that affect the flow of work.

The simulation is inspired by the GetKanban game [2].

An Experiment

Ideally one would have a set-up for an experiment in which two exactly equivalent teams are compared. One team that would perform consecutive actions to improve the flow of work. The other team would just make one adjustment to improve the flow, and focus subsequent improvements on other areas than flow. At the same time the flow is measured and verified after a certain period of time whether the first or the second team has achieved better results in terms of flow.

Such an experiment is in practise (very) difficult to perform. In this blog I will study the same experiment by making use of simulations.

Simulation

For the purpose of simulation I consider a team consisting of three specialists: 1 designer, 1 developer, and 1 tester. The team uses a kanban process to achieve flow. See the picture below for the begin situation.


foto

In the simulation, at the beginning of each working day it is determined how much work will be completed by the team. During the simulation the average cycle time is measured. The initial Work in Progress (WiP) limits are set to 3 for each column and is indicated by the red '3's in the picture.

The average amount of work done by the 'team' and the average effort of one work item are such that on average it takes one card about 5,5 days to complete.

At the end of each work day, cards are pulled into the next columns (if allowed by the WiP limits). The policy is to always pull in as much work as allowed so the columns are maximally filled. Furthermore, the backlog is assumed to always have enough user stories ready to be pulled into the 'design' column. This very much resembles developing a new product when the backlog is filled with more than enough stories.

The system starts with a clean board and all column empty. After letting the system run for 75 simulated work days, we will trigger a policy change. Particularly the WiP limit for the 'design' is increased from '3' to '5'. After this policy change the system runs for another 100 work days.

From the chart showing the average cycle time we will be able to study the effect of WiP limit changing adjustments.

Note:

The simulation assumes a simple uniform distribution for the amount of work done by the team and the effort assigned to a work item. I assume this is OK for the purpose of this blog. A consequence of this, is that the result probably can't be scaled. For instance, the situation in which a column in the picture above is a single scrum team is not applicable since a more complex probability distribution should be used instead of the uniform distribution.

Results

The picture below shows the result of running the experiment.

 

retro_length

 

After the start it takes the system little over 40 work days to reach the stable state of an average cycle time of about 24(*) days. This is the cycle time one would expect. Remember, the 'ready' column has a size of '3' and the other columns get work done. So, one would expect a cycle time of around 4 times 5,5 which equals 22 days which is close to 24.

At day 75 the WiP limit is changed. As can be inferred from the picture, the cycle time starts to rise only at day 100 (takes about 1 cycle time (24 days) to respond). The new stable state is reached at day 145 with an average cycle time of around 30(**) days. It takes 70 days(!) to reach the new equilibrium.

The chart shows the following interesting features:

  1. It takes roughly 2 times the (new) average cycle time to reach the equilibrium state,
  2. The response time (when one begins to notice an effect of the policy change) is about the length of the average cycle time.

(*) One can calculate (using transition matrices) the theoretical average cycle time for this system to be 24 days.

(**) Similar, the theoretical average cycle time of the system after policy change is 31 days.

 

Conclusion

In this blog we have seen that when a team makes adjustments that affect the flow, the system needs time to get to its new stable state. Until this state has been reached any new tuning of the flow is questionable. Simulations show that the time it takes to reach the new stable state is about 2 times the average cycle time.

For scrum teams that have 2-week sprints, the system may need about 2 months before new tuning of flow is effective. Meanwhile, the team can very well focus on other improvements, e.g. retrospectives that focus on the team aspect or collaboration with the team's environment.

Moreover, don't expect to see any changes in measurements of e.g. cycle time within the time period of the average cycle time after making a flow affecting change.

To summarise, after making flow affecting changes (e.g. increasing or decreasing WiP limits):

  • Let the system run for at least the duration of the average cycle time so it has time to respond to the change,
  • After it responds, notice the effect of the change,
  • If the effect is positive, let the system run for another duration of the average cycle time, to get to the new stable state,
  • If the effect is negative, do something else, e.g. go back to the old state, and remember that the system needs to respond to this as well!
References

[1] Agile manifesto, http://agilemanifesto.org/principles.html

[2] GetKanban, http://getkanban.com

Combining Salt with Docker

Sat, 06/14/2014 - 10:59

You could use Salt to build and run Docker containers but that is not how I use it here. This blogpost is about Docker containers that run Salt minions, which is just an experiment. The use case? Suppose you have several containers that run a particular piece of middleware, and this piece of middleware needs a security update, i.e. an OpenSSL hotfix. It is necessary to perform the update immediately.

 

The Dockerfile

In order to build a container you have to write down the container description in a file called Dockerfile. Here is the Dockerfile:

#-------
# Standard heading stuff

FROM centos
MAINTAINER No Reply noreply@xebia.com

# Do Salt install stuff and squeeze in a master.conf snippet that tells the minion
# to contact the master specified.

RUN rpm -Uvh http://ftp.linux.ncsu.edu/pub/epel/6/i386/epel-release-6-8.noarch.rpm
RUN yum install -y salt-minion --enablerepo=epel-testing
RUN [ ! -d /etc/salt/minion.d ] && mkdir /etc/salt/minion.d
ADD ./master.conf /etc/salt/minion.d/master.conf

# Run the Salt Minion and do not detach from the terminal.
# This is important because the Docker container will exit whenever
# the CMD process exits.

CMD /usr/bin/salt-minion
#-------

 

Build the image

Time to run the Dockerfile through docker. The command is:

$ docker build --rm=true -t salt-minion .

provided that you run this command in the directory where file Dockerfile and master.conf resides. Docker creates an image with tag ‘salt-minion’ and throws away all intermediate images after a successful build.

 

Run a container

The command is:

$ docker run -d salt-minion

and Docker returns:

aab154310ba6452ba2c686d15b1e3ca5fd85124d38c7935f1200d33b3a3e7ced

The Salt minion on the container is started and searches for a Salt master to connect to, defined by the configuration setting “master” in file /etc/salt/minion.d/master.conf. You might want to run the Salt master in “auto_accept” mode so that minion keys are accepted automatically. Docker assigns a container id to the running container. That is the magic key that docker reports as a result of the run command.

The following command shows the running container:

$ docker ps
CONTAINER ID        IMAGE                COMMAND                CREATED             STATUS              NAMES
273a6b77a8fa        salt-minion:latest   /bin/sh -c /etc/rc.l   3 seconds ago       Up 3 seconds        distracted_lumiere

 

Apply the hot fix
There you are: the Salt minion is controlled by your Salt master. Provided that you have a state module that contains the OpenSSL hot fix, you can now easily update all docker nodes to include the hotfix:

salt \* state.sls openssl-hotfix

That is all there is to it.

State of Web Development 2014

Fri, 06/06/2014 - 13:10

On the 5th of June, we (Lammert Westerhoff and Freek Wielstra) presented the State of Web Development 2014 at our yearly conference, XebiCon on the SS Rotterdam. It was a great success and we had some great feedback. We'd like to share the same presentation we gave there with everyone who couldn't make it to the conference in this blog post.

We started the presentation by showing an overwhelming tag cloud:

WebXebiCon.006

 

 

When you’re new to web development or when you haven’t done any web development for a long time, you’re likely to get lost in this jungle of web technologies. Don’t worry if you are overwhelmed with this, because that’s intentional. This tag cloud contains many languages and web frameworks that modern web applications consist of. But it also contains a number of development tools that are used during development. The goal of this post is to guide you through this jungle by explaining to you which technologies you can use to build a modern web application and which tools you need during the full development lifecycle.

The get a better understanding of which technologies and tools we should use, we should first look at bit at the evolution of a number of important technologies. Then we will understand why web development is quite complex these days and why we need tooling. Once that's clear, we can have a look at the available tooling.

The evolution of web applications

A traditional website consists of HTML, CSS and sometimes a little javascript. The HTML tells the browser the content that needs to be displayed. The CSS tells it how that content should be displayed and what it needs to look like. And JavaScript was mainly used to add small behaviour, like displaying a popup or showing an image carousel.

WebXebiCon.016

Of course, these static websites were hard to maintain. A change of content needed a change in the HTML file which then needed to be uploaded to the web server with FTP.

Luckily, there were a couple of technologies that solved this problem for us. Using a technology like PHP, JSP or ASP we could dynamically generate the websites at runtime, which were then sent to the browser. These websites usually were connected to a database. The database then stored the content instead of storing it directly in the HTML files. That way we could make changes to the website by changing the content stored in the database. This allowed us to already build more complex web applications that also stored user created content, like blogs or even entire web stores.

While we had the server side technologies that we needed to generate our websites, we needed new innovations on the client side as well. Especially because we entered the era of mobile devices, we needed new browser capabilities, like location services, browser storage and touch events. Therefore, new HTML and CSS standards were being developed that contain many features that make the web browsers much more powerful and richer, which allows us developers to build richer web applications for both mobile and desktop.

To make this a lot easier for everyone, Twitter released Bootstrap, which handles all the responsiveness automatically. It’s still being used by many responsive websites today.

With all the new powerful features of CSS3, the style sheets also became more complex and tedious. To overcome that problem, technologies like Sass and Less were developed. These are CSS preprocessors that offer new language features that make some of the more tedious styling easier to do. Both will produce CSS and both solve the same problems. Which one you should choose depends on your circumstances and personal flavour.

WebXebiCon.021

Users don’t only want web sites to work and look good on every device, they also want web sites to be more interactive. They want the web site to respond to their actions. For example, zoom into an image or drag an item into a shopping basket. Even though all this was already possible in JavaScript for a long time, jQuery made such things a lot easier. jQuery is a JavaScript library that makes Dom manipulation easier, especially for multiple browsers. With the popularity of jQuery, the web became overwhelmed with jQuery plugins that did all sort of things. Below we see a screenshot of the plugin registry. And of course a library for mobile had to be released.

WebXebiCon.026

One of the very important features of jQuery is that it makes it easy to make AJAX requests. AJAX stands for Asynchronous Javascript And XML. It allows the web browser to retrieve data from the backend without leaving the page or doing a page refresh. And only the data is transferred from the backend to the client, instead of entire generated pages. Once the data is retrieved, it is displayed dynamically using the DOM manipulation features of jQuery. And even though XML is in the name, nowadays JSON is often used instead of XML, which is a more lightweight and more flexible data format.

AJAX requests are also at the core of single web page applications (SPA).

WebXebiCon.027

These web applications load all the HTML, CSS and Javascript resources at once. Once the page is loaded, you won’t have any other pages loading or refreshing anymore. All the rendering and navigation is done at the client side without leaving the actual page. Of course the user might perceive that he’s navigating between different pages, but this is all handled on the client by Javascript.

The single page web applications are becoming more and more popular since they usually provide a better user experience. They rely heavily on the AJAX requests that we saw earlier to retrieve data from a server. The data that is retrieved by these request are one of the only types of communication between the client and server once the page is initially loaded.

Since with SPA’s, we move a lot of logic and complexity from the server to the client, it really comes in handy to use the framework that handles most of the complexity. Three of the most popular frameworks are Backbone.js, Ember.js and AngularJS. Which one you should choose depends a lot on your requirements, what you’re used to and personal preference. Though it is clear that AngularJS is the most popular one at this moment, has the most activity and the largest community. People seem to be moving from Backbone to Angular and Ember. We also use Angular ourselves and we’re happy with it. The Google Angular team is working on the next major version which should even be a big improvement. That’s why we think Angular is a good and safe choice.

From what we've seen in the previous paragraphs, it’s obvious that more and more functionality and complexity is moved from the server side to the client side. And by using frameworks and libraries such as Bootstrap, jQuery and Angular, we also increase our client side dependencies. This also means that our development focus and efforts are moving from the server to the client. It’s time to take front end development serious and to do it right. So let’s have a look at what this means for our development process.

Web Application Development

As we saw earlier, we start simple by creating a couple of HTML, CSS and Javascript files. Then we would like to take advantage of the power of Less or Sass for our stylesheets. And since we’re making a rich or single page web application, we’ll be making a lot of javascript files. And perhaps some CoffeeScript or Dart instead of just Javascript. For those who don’t know coffee script or Dart, these are alternative languages that both compile to Javascript. CoffeeScript is almost the same language as javascript but just with a slightly different syntax. Dart however is quite different is definitely worth checking out when you have a bit of spare time.

Of course we want to use many of the great third party libraries that we discussed earlier. And since those libraries often depend on other libraries, we’ll end up with a lot of external libraries.

WebXebiCon.033

With this picture we kind of get an idea which resources we have for our web application. But to just create these files, download the libraries and upload them to our web server is not enough. First of all, web browsers don’t understand Less, Sass, CoffeeScript or Dart. We need to produce something that the browser does understand. For Less and Sass that’s CSS and Coffee script and Dart need to be compiled to normal Javascript.

WebXebiCon.034

Now we’re at a stage that we could deploy these files to our web server, but still that’s not really the way we like to develop our apps. So we’re going to do a couple of extra steps. We will minify almost all of our files. This will grab all those files, do some smart tricks, and we’ll end up with much smaller files. This will safe the end user some download time, which is nice.

WebXebiCon.036

We also want to make sure that all of the files we made don’t have any syntax mistakes. That’s why we’ll run a syntax validator on all of those files.

And of course we need to test. So we write a lot of unit tests to cover all our JavaScript functionality. But we also write view tests to make sure that our views display the correct information to the user. With good decoupling we can run our unit tests and view tests in isolation, which is good since we can easily spot problems when one of our test fails and it makes it easy to maintain and refactor our code. But we also want to make sure everything works well together, which is why we also write a couple of end to end tests.

WebXebiCon.042

With all these steps in place we’re pretty confident that we have everything we need to make sure that we write quality code. But there is one more thing we need to worry about during development. And that’s dependency management. We want automatic downloads of the third party libraries that we use and their dependencies. And we want to specify their version as well, or specify a version range and get automatic updates within our range.

Now we’re completely happy with all the steps in our development process. But how do we do all this? Or better said, how do we automate all this? We need a build tool.

Let’s quickly recap the things we want our build tool to do. It needs to compile and minify our style sheets and scripts. After that it will take these files and together with the HTML build a distribution package that we can deploy to our production server.

And with every change in any of our files, we want to make sure that everything still works as we would expect. So we validate our files and run our tests. These are our quality assurance steps.

Like we just mentioned, we also want all of our dependencies to be managed automatically.

But there are even more things that we can do with modern build tools that we couldn’t even think of but are great for development. How about live watch and reload of our browser during development. We launch our app in a browser, change some logic or styling, and the web browser will refresh the page automatically to show our changes.

The web development community has been very innovative and came up with many more cool thing we can do with these tools. And you can even write your own build actions.

WebXebiCon.050

At this point we have a full understanding of what the web development process looks like. And by seeing this, we can say that front end development has become a first class citizen of software development. Now that we recognise the complexity of front end development and that we need tooling to manage this complexity, it time to have a deeper look at some of the most popular tools of this moment.

Modern tools

Most of the web development tools that are built these days are part of the Node.js ecosystem.

WebXebiCon.054

Node.js is a JavaScript engine to run JS outside a browser environment. It has powerful system libraries for basics like file and network access, all with a minimal footprint These characteristics make NodeJS a perfect platform to build server side applications and terminal applications.

Since it’s JavaScript, a language that web developers already know, it’s the perfect environment to build web development tools with.

The Node Package Manager makes it easy to install any registered Node.js tool or application with just a single command. It keeps an online registry of all available packages, and anyone can publish their own. The Node.js and NPM ecosystem have developed into an active and healthy developer community.

One of the most popular tools written in Node.js is Grunt. It is the standard build tool for front-end web projects. It’s a task based build tool, which means you can define a set of tasks that need to run for a production build, development build or a test run. Sounds pretty much what we need for our development process.

WebXebiCon.060

Since it’s plugin based, it has plugins for pretty much any task you want to run. That includes plugins for our compilation, minification, validation and testing steps, as well as dependency management. That makes it perfect for us.

Once you start grunt in development mode, it will launch its built-in webserver and host your web application. And with the right plugins, we get our live reload. The Grunt plugin will watch for any changes in our resources, perform necessary tasks and refresh the open browser page for us, so we can see the results of our work right away - drastically shortening the feedback loop compared to traditional web development.

Since Grunt is the standard and most popular build tool for web projects at this moment, it’s widely adopted and has a large and mature developer ecosystem.

But Grunt also has it’s drawbacks. Let’s have a look at Grunt’s configuration file, the Gruntfile.js. This file contains configuration for our tasks and the plugins that perform them.

WebXebiCon.062

With a lot of tasks, the Gruntfile really becomes a lot of configuration. The example below taken from our own project and it’s only about half of our configuration. No need to read what it says exactly, it’s just to give you an idea about the amount.

WebXebiCon.064

But the amount of configuration is not the only downside of Grunt. The other problem is that it works with temporary files and directories. For each task (depending on the task, of course), it will read a couple of files, transform or convert them into something else and write the output back to disk. The the next task will then use these new files as their input. This creates a chain of temporary files and directories. It’s not a huge problem, especially  but it certainly is not the fastest way of doing things.

That’s why Gulp.js, the streaming build system, was made. It’s a newer, alternative build tool that’s quickly gaining popularity. Gulp can do pretty much the same things as Grunt, so we don’t need to worry too much that we lose essential features of our development process. However, since Gulp is newer than Grunt, it doesn’t have as many plugins as Grunt yet. But since Gulp does the same as Grunt, but then better, it seems that many developers make the switch to Gulp and more and more plugins are made for Gulp as well.

WebXebiCon.065

So what’s the difference? Instead of requiring a lot of configuration for it’s tasks, Gulp instead uses “Code over configuration”. That means that you just tell the tool to do something in a certain way instead of configuring both what it should do and how it should do it. Much easier and much more direct. And resulting in far less code than Grunt’s configuration for doing the same.

WebXebiCon.066

It also solves Grunt’s problem of temporary files and directories since it’s stream based. Tasks will directly stream their output to the next task. This results in much faster build times.

WebXebiCon.067

So it seems that Gulp solves exactly the two problems we had with Grunt. So it’s to be expected that it will gain popularity and eventually pass Grunt. It’s just a matter of time until it reaches the same maturity that Grunt currently has.

We talked earlier about dependency management to manage our third party libraries. While Grunt and Gulp use the Node Package Manager directly to download it’s plugins and their dependencies, we still need to manage the dependencies for our app. To do that, we use Bower, a package manager for the web, by Twitter. Bower works similar as NPM in that you specify all your dependencies in a single configuration file. Bower will then download these libraries from the Internet. Since Grunt and Gulp have plugins for Bower, you don’t even need to worry about this, since it will make sure to download the files when you build your project or to download updates when needed - if your build system is properly configured.

WebXebiCon.068

So many tools and frameworks! Isn’t it too much work to setup and configure?
Yes; setting up a full web app development stack, configuring each plugin for your project, is a lot of development work which can take up days. Luckily there is a tool to solve that problem: Yeoman.

Yeoman is a project scaffolding tool that generates a basis for your project, using templates called Generators. Some of those generators will set up a web application project with a preconfigured build strait, containing everything we need for compilation, minification, validation, testing, dependency management and much more. It will also ask us if we want to use frameworks such as Bootstrap and AngularJS. This makes it an ideal solution to setup a new project and automated development process.

More recent generators will set up Gulp as the build tool, providing an alternative to Bower. This should help developers that aren’t familiar with Gulp yet to get up to speed quickly.

WebXebiCon.072

Let’s recap what we talked about in this post in a summary.

WebXebiCon.079

More

The full presentation slides will soon be available from the XebiCon website. Feel free to comment below with your feedback or reach out to us on Twitter.

Lammert Westerhoff @lwesterhoff

Freek Wielstra @frwielstra

Join XebiCon and meet your peers!

Wed, 05/21/2014 - 10:29

Xebia e-mail banner

Come to meet your peers @ XebiCon 2014! There are still tickets available. Please register at www.xebicon.nl

At XebiCon 2014, we share knowledge and insights on and about current issues and customer cases in the area of Agile, Software Arechitecture, Continuous Delivery and Modern Software Development (Mobile, Big Data, Craftsmanship).

This year we have the best keynote speakers:
André Kuipers: European Space Agency astronaut about Innovation
Nalden: Co-Founder WeTransfer about Business Acceleration

 

Wrestling with the Browser Event Queue

Sat, 05/17/2014 - 16:03

In this post I want to show some things I’ve learned recently about the browser event queue.
It all started with the following innocent looking problem:

I was building a single page web application, and on one form the user should have two options to enter a stock order.
The options were represented as radio buttons, each of them having its own input field. The radio buttons gave the user the choice: he could either enter a order amount, or a order quantity.
Both fields should have their own specific input validations which should be triggered when the user left the input field.

Here's the Html snippet:

  <form>
    <input type="radio" name="options" id="option_amount">Amount</input>
    <input id="option_mount_input"/>
    <input type="radio" name="options" id="option_quantity">Quantity</input>
    <input id="option_quantity_input"/>
  </form>

And the Javascript:

function validate_amount() {
    if(!$.isNumeric($("#option_amount_input").text())) {
      alert("Please enter a valid amount");
    }
}

$("#option_amount_input").on("blur", validate_amount);

All was working fine, except for one scenario: what if the user entered an amount in a wrong way, but then decided he wanted to enter a quantity so he clicked the quantity option instead.
In that case I wouldn’t want the validation for the amount to go off.

Here is my first attempt to fix this scenario:

var shouldValidateAmount = true;

function validate_amount() {
    setTimeout(delayedAmountValidation, 0);
}

function click_quantity() {
    shouldValidateAmount = false;
}

function click_amount() {
    shouldValidateAmount = true;
}

function delayedAmountValidation() {
    if(shouldValidateAmount) {
         if(!$.isNumeric($("#option_amount_input").text())) {
            alert("Please enter a valid amount");
         }
    }
}

$("#option_amount_input").on("blur", validate_amount);
$("#option_quantity").on("click", click_quantity);
$("#option_amount").on("click", click_amount);

The idea is that when the user leaves the amount field, the amount validation will be triggered, but after a timeout. The timeout is zero so the validation event is just put at the end of the event queue.
So before the validation will be executed, the click handler for the quantity radio option will be handled first which will then cancel any postponed validations. (or so I thought)

But this didn’t work. For some reason the postponed validation was still being executed before the click handler.
Only if I adjusted the timout delay until it was high enough, in my case that was 70ms, it would be executed after the click handler. But that would be a very brittle solution of course.

Here's a diagram showing how the event queue handled my solution:

event queue
So what to do about this? Time to look at the browser event queue in some more depth.

It turns out that browser considers certain groups of event as batches. Those are groups which the browser considers to be logically belonging together.
An example is a blur-event for one input combined with a focus-gained event for another input.
Timeouts generated from within a batch have to wait until all the events from their batch are handled, and only then get a chance to execute.

A blur-event and a click event on another element do apparently not belong to the same batch. Therefore my timeout was executed immediately after the focus-lost event was handled, and before the click event handler.

To listen to the focus event on the radio option was no solution of course, because just because it got focus didn’t necessarily mean it was seleced; the user could have tabbed there as well.
But it turns out that there are more events than just onFocus that belong to the same batch as blur; mouseDown and mouseEnter events!

Here is the final solution, which listens for a mouseDown event on the radio button:

event queue batch
And the code:

var shouldValidateAmount = true;

function validate_amount() {
    setTimeout(delayedAmountValidation, 0);
}

function click_quantity() {
    shouldValidateAmount = false;
}

function click_amount() {
    shouldValidateAmount = true;
}

function delayedAmountValidation() {
    if(shouldValidateAmount) {
         if(!$.isNumeric($("#option_amount_input").text())) {
            alert("Please enter a valid amount");
         }
    }
}

$("#option_amount_input").on("blur", validate_amount);
$("#option_quantity").on("mousedown", click_quantity);
$("#option_amount").on("mousedown", click_amount);

References:
Link to Mozilla doc on events and timing