Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Xebia Blog
Syndicate content
Updated: 14 hours 57 min ago

CocoaHeadsNL @ Xebia on September 16th

Thu, 08/28/2014 - 11:20

On Tuesday the 16th the Dutch CocoaHeads will be visiting us. It promises to be a great night for anybody doing iOS or OSX development. The night starts at 17:00, diner at 18:00.

If you are an iOS/OSX developer and like to meet fellow developers? Come join the CocoaHeads on september 16th at our office. More details are on the CocoaHeadsNL meetup page.

What is your next step in Continuous Delivery? Part 1

Wed, 08/27/2014 - 21:15

Continuous Delivery helps you deliver software faster, with better quality and at lower cost. Who doesn't want to delivery software faster, better and cheaper? I certainly want that!

No matter how good you are at Continuous Delivery, you can always do one step better. Even if you are as good as Google or Facebook, you can still do one step better. Myself included, I can do one step better.

But also if you are just getting started with Continuous Delivery, there is a feasible step to take you forward.

In this series, I describe a plan that helps you determine where you are right now and what your next step should be. To be complete, I'll start at the very beginning. I expect most of you have passed the first steps already.

The steps you already took

This is the first part in the series: What is your next step in Continuous Delivery? I'll start with three steps combined in a single post. This is because the great majority of you has gone through these steps already.

Step 0: Your very first lines of code

Do you remember the very first lines of code you wrote? Perhaps as a student or maybe before that as a teenager? Did you use version control? Did you bring it to a test environment before going to production? I know I did not.

None of us was born with an innate skills for delivering software in a certain way. However, many of us are taught a certain way of delivering software that still is a long way from Continuous Delivery.

Step 1: Version control

At some point during your study of career, you have been introduced to Version Control. I remember starting with CVS, migrating to Subversion and I am currently using Git. Each of these systems are an improvement over te previous one.

It is common to store the source code for your software in version control. Do you already have definitions or scripts for your infrastructure in version control? And for your automated acceptance tests or database schemas? In later steps, we'll get back to that.

Step 2: Release process

Your current release process may be far from Continuous Delivery. Despite appearances, your current release process is a useful step towards Continuous Delivery.

Even if you delivery to production less than twice a year, you are better off than a company that delivers their code unpredictably, untested and unmanaged. Or worse, a company that edits their code directly on a production machine.

In your delivery process, you have planning, control, a production-like testing environment, actual testing and maintenance after the go-live. The main difference with Continuous Delivery is the frequency and the amount of software that is released at the same time.

So yes, a release process is a productive step towards Continuous Delivery. Now let's see if we can optimize beyond this manual release process.

Step 3: Scripts

Imagine you have issues on your production server... Who do you go to for help? Do you have someone in mind?

Let me guess, you are thinking about a middle-aged guy who has been working at your organisation for 10+ years. Even if your organization is only 3 years old, I bet he's been working there for more than 10 years. Or at least, it seems like it.

My next guess is that this guy wrote some scripts to automate recurring tasks and make his life easier. Am I right?

These scripts are an important step towards Continuous Delivery. in fact, Continuous Delivery is all about automating repetitive tasks. The only thing that falls short is that these scripts are a one-man-initiative. It is a good initiative, but there is no strategy behind it and a lack of management support.

If you don't have this guy working for you, then you may have a bigger step to take when continuing towards the next step of Continuous Delivery. To successfully adopt Continuous Delivery on the long run, you are going to need someone like him.

Following steps

In the next parts, we will look at the following steps towards becoming world champion delivering software:

  • Step 4: Continuous Delivery
  • Step 5: Continuous Deployment
  • Step 6: "Hands-off"
  • Step 7: High Scalability

Stay tuned for the following posts.

Synchronize the Team

Tue, 08/26/2014 - 13:52

How can you, as a scrum master, improve the chances that the scrum team has a common vision and understanding of both the user story and the solution, from the start until the end of the sprint?   

The problem

The planning session is where the team should synchronize on understanding the user story and agree on how to build the solution. But there is no real validation that all the team members are on the same page. The team tends to dive into the technical details quite fast in order to identify and size the tasks. The technical details are often discussed by only a few team members and with little or no functional or business context. Once the team leaves the session, there is no guarantee that they remain synchronized when the sprint progresses. 

The only other team synchronization ritual, prescribed by the scrum process, is the daily scrum or stand-up. In most teams the daily scrum is as short as possible, avoiding semantic discussions. I also prefer the stand-ups to be short and sweet. So how can you or the team determine that the team is (still) synchronized?

Specify the story

In the planning session, after a story is considered ready enough be to pulled into the sprint, we start analyzing the story. This is the specification part, using a technique called ‚ÄėSpecification by Example‚Äô. The idea is to write testable functional specifications with actual examples. We decompose the story into specifications and define the conditions of failure and success with examples, so they can be tested.¬†Thinking of examples makes the specification more concrete and the interpretation of the requirements more specific.

Having the whole team work out the specifications and examples, helps the team to stay focussed on the functional part of the story longer and in more detail, before shifting mindsets to the development tasks.  Writing the specifications will also help to determine wether a story is ready enough. While the sprint progresses and all the tests are green, the story should be done for the part of building the functionality.

You can use a tool like Fitnesse  or Cucumber to write testable specifications. The tests are run against the actual code, so they provide an accurate view on the progress. When all the tests pass, the team has successfully created the functionality. In addition to the scrum board and burn down charts, the functional tests provide a good and accurate view on the sprint progress.

Design the solution

Once the story has been decomposed into clear and testable specifications we start creating a design on a whiteboard. The main goal is to create a shared visible understanding of the solution, so avoid (technical) details to prevent big up-front designs and loosing the involvement of the less technical members on the team. You can use whatever format works for your team (e.g. UML), but be sure it is comprehensible by everybody on the team.

The creation of the design, as an effort by the whole team, tends to sparks discussion. In stead of relying on the consistency of non-visible mental images in the heads of team members, there is a tangible image shared with everyone.

The whiteboard design will be a good starting point for refinement as the team gains insight during the sprint. The whiteboard should always be visible and within reach of the team during the sprint. Using a whiteboard makes it easy to adapt or complement the design. You’ll notice the team standing around the whiteboard or pointing to it in discussions quite often.

The design can be easily turned into a digital artefact by creating a photo copy of it. A digital copy can be valuable to anyone wanting to learn the system in the future. The design could also be used in the sprint demo, should the audience be interested in a technical overview.

Conclusion

The team now leaves the sprint planning with a set of functional tests and a whiteboard design. The tests are useful to validate and synchronize on the functional goals. The whiteboard designs are useful to validate and synchronize on the technical goals. The shared understanding of the team is more visible and can be validated, throughout the sprint. The team has become more transparent.

It might be a good practice to have the developers write the specification, and the testers or analysts draw the designs on the board. This is to provoke more communication, by getting the people out of their comfort zone and forcing them to ask more questions.

There are more compelling reasons to implement (or not) something like specification by design or to have the team make design overviews. But it also helps the team to stay on the same page, when there are visible and testable artefacts to rely on during the sprint.

Vert.x with core.async. Handling asynchronous workflows

Mon, 08/25/2014 - 12:00

Anyone who was written code that has to coordinate complex asynchronous workflows knows it can be a real pain, especially when you limit yourself to using only callbacks directly. Various tools have arisen to tackle these issues, like Reactive Extensions and Javascript promises.

Clojure's answer comes in the form of core.async: An implementation of CSP for both Clojure and Clojurescript. In this post I want to demonstrate how powerful core.async is under a variety of circumstances. The context will be writing a Vert.x event-handler.

Vert.x is a young, light-weight, polyglot, high-performance, event-driven application platform on top of the JVM. It has an actor-like concurrency model, where the coarse-grained actors (called verticles) can communicate over a distributed event bus. Although Vert.x is still quite young, it's sure to grow as a big player in the future of the reactive web.

Scenarios

The scenario is as follows. Our verticle registers a handler on some address and depends on 3 other verticles.

1. Composition

Imagine the new Mars rover got stuck against some Mars rock and we need to send it instructions to destroy the rock with its inbuilt laser. Also imagine that the controlling software is written with Vert.x. There is a single verticle responsible for handling the necessary steps:

  1. Use the sensor to locate the position of the rock
  2. Use the position to scan hardness of the rock
  3. Use the hardness to calibrate and fire the laser. Report back status
  4. Report success or failure to the main caller

As you can see, in each step we need the result of the previous step, meaning composition.
A straightforward callback-based approach would look something like this:

(ns example.verticle
  (:require [vertx.eventbus :as eb]))

(eb/on-message
  "console.laser"
  (fn [instructions]
    (let [reply-msg eb/*current-message*]
      (eb/send "rover.scope" (scope-msg instructions)
        (fn [coords]
          (eb/send "rover.sensor" (sensor-msg coords)
            (fn [data]
              (let [power (calibrate-laser data)]
                (eb/send "rover.laser" (laser-msg power)
                  (fn [status]
                    (eb/reply* reply-msg (parse-status status))))))))))))

A code structure quite typical of composed async functions. Now let's bring in core.async:

(ns example.verticle
  (:refer-clojure :exclude [send])
  (:require [ vertx.eventbus :as eb]
            [ clojure.core.async :refer [go chan put! <!]]))

(defn send [addr msg]
  (let [ch (chan 1)]
    (eb/send addr msg #(put! ch %))
    ch))

(eb/on-message
  "console.laser"
  (fn [instructions]
    (go (let [coords (<! (send "rover.scope" (scope-msg instructions)))
              data (<! (send "rover.sensor" (sensor-msg coords)))
              power (calibrate-laser data)
              status (<! (send "rover.laser" (laser-msg power)))]
          (eb/reply (parse-status status))))))

We created our own reusable send function which returns a channel on which the result of eb/send will be put. Apart from the 2. Concurrent requests

Another thing we might want to do is query different handlers concurrently. Although we can use composition, this is not very performant as we do not need to wait for reply from service-A in order to call service-B.

As a concrete example, imagine we need to collect atmospheric data about some geographical area in order to make a weather forecast. The data will include the temperature, humidity and wind speed which are requested from three different independent services. Once all three asynchronous requests return, we can create a forecast and reply to the main caller. But how do we know when the last callback is fired? We need to keep some memory (mutable state) which is updated when each of the callback fires and process the data when the last one returns.

core.async easily accommodates this scenario without adding extra mutable state for coordinations inside your handlers. The state is contained in the channel.

(eb/on-message
  "forecast.report"
  (fn [coords]
    (let [ch (chan 3)]
      (eb/send "temperature.service" coords #(put! ch {:temperature %}))
      (eb/send "humidity.service" coords #(put! ch {:humidity %}))
      (eb/send "wind-speed.service" coords #(put! ch {:wind-speed %}))
      (go (let [data (merge (<! ch) (<! ch) (<! ch))
                forecast (create-forecast data)]
            (eb/reply forecast))))))
3. Fastest response

Sometimes there are multiple services at your disposal providing similar functionality and you just want the fastest one. With just a small adjustment, we can make the previous code work for this scenario as well.

(eb/on-message
  "server.request"
  (fn [msg]
    (let [ch (chan 3)]
      (eb/send "service-A" msg #(put! ch %))
      (eb/send "service-B" msg #(put! ch %))
      (eb/send "service-C" msg #(put! ch %))
      (go (eb/reply (<! ch))))))

We just take the first result on the channel and ignore the other results. After the go block has replied, there are no more takers on the channel. The results from the services that were too late are still put on the channel, but after the request finished, there are no more references to it and the channel with the results can be garbage-collected.

4. Handling timeouts and choice with alts!

We can create timeout channels that close themselves after a specified amount of time. Closed channels can not be written to anymore, but any messages in the buffer can still be read. After that, every read will return nil.

One thing core.async provides that most other tools don't is choice. From the examples:

One killer feature for channels over queues is the ability to wait on many channels at the same time (like a socket select). This is done with `alts!!` (ordinary threads) or `alts!` in go blocks.

This, combined with timeout channels gives the ability to wait on a channel up a maximum amount of time before giving up. By adjusting example 2 a bit:

(eb/on-message
  "forecast.report"
  (fn [coords]
    (let [ch (chan)
          t-ch (timeout 3000)]
      (eb/send "temperature.service" coords #(put! ch {:temperature %}))
      (eb/send "humidity.service" coords #(put! ch {:humidity %}))
      (eb/send "wind-speed.service" coords #(put! ch {:wind-speed %}))
      (go-loop [n 3 data {}]
        (if (pos? n)
          (if-some [result (alts! [ch t-ch])]
            (recur (dec n) (merge data result))
            (eb/fail 408 "Request timed out"))
          (eb/reply (create-forecast data)))))))

This will do the same thing as before, but we will wait a total of 3s for the requests to finish, otherwise we reply with a timeout failure. Notice that we did not put the timeout parameter in the vert.x API call of eb/send. Having a first-class timeout channel allows us to coordinate these timeouts more more easily than adding timeout parameters and failure-callbacks.

Wrapping up

The above scenarios are clearly simplified to focus on the different workflows, but they should give you an idea on how to start using it in Vert.x.

Some questions that have arisen for me is whether core.async can play nicely with Vert.x, which was the original motivation for this blog post. Verticles are single-threaded by design, while core.async introduces background threads to dispatch go-blocks or state machine callbacks. Since the dispatched go-blocks carry the correct message context the functions eb/send, eb/reply, etc.. can be called from these go blocks and all goes well.

There is of course a lot more to core.async than is shown here. But that is a story for another blog.

Docker on a raspberry pi

Mon, 08/25/2014 - 07:11

This blog describes how easy it is to use docker in combination with a Raspberry Pi. Because of docker, deploying software to the Raspberry Pi is a piece of cake.

What is a raspberry pi?
The Raspberry Pi is a credit-card sized computer that plugs into your TV and a keyboard. It is a capable little computer which can be used in electronics projects and for many things that your desktop PC does, like spreadsheets, word-processing and games. It also plays high-definition video. A raspberry pi runs linux, has an ARM processor of 700 MHZ and internal memory of 512 MB. Last but not least, it only costs around  35 Euro.

A raspberry pi

A raspberry pi version B

Because of the price, size and performance, the raspberry pi is a step to the 'Internet of things' principle. With a raspberry pi it is possible to control and connect everything to everything. For instance, my home project which is an raspberry pi controlling a robot.

 

Raspberry Pi in action

What is docker?
Docker is an open platform for developers and sysadmins to build, ship and run distributed applications.¬†With Docker, developers can build any app in any language using any toolchain. ‚ÄúDockerized‚ÄĚ apps are completely portable and can run anywhere. A dockerized app contains the application, its environment, dependencies and even the OS.

Why combine docker and raspberry pi?
It is nice to work with a Raspberry Pi because it is a great platform to connect devices. Deploying anything however, is kind of a pain. With dockerized apps we can develop and test our application on our own home machine, when it works we can deploy it to the raspberry. We can do this without any pain or worries about corruption of the underlying operating system and tools. And last but not least, you can easily undo your tryouts.

What is better than I expected
First of all; it was relatively easy to install docker on the raspberry pi. When you use the Arch Linux operating system, docker is already part of the package manager! I expected to do a lot of cross-compiling of the docker application, because the raspberry pi uses an ARM-architecture (instead of the default x86 architecture), but someone did this already for me!

Second of all; there are a bunch of ready-to-use docker-images especially for the raspberry pi. To run dockerized applications on the raspberry pi you are depending on the base images. These base images must also support the ARM-architecture. For each situation there is an image, whether you want to run node.js, python, ruby or just java.

The worst thing that worried me was the performance of running virtualized software on a raspberry pi. But it all went well and I did not notice any performance reduce. Docker requires far less resources than running virtual machines. A docker proces runs straight on the host, giving native CPU performance. Using Docker requires a small overhead for memory and network.

What I don't like about docker on a raspberry pi
The slogan of docker to 'build, ship and run any app anywhere' is not entirely valid. You cannot develop your Dockerfile on your local machine and deploy the same application directly to your raspberry pi. This is because each dockerfile includes a core image. For running your application on your local machine, you need a x86-based docker-image. For your raspberry pi you need an ARM-based image. That is a pity, because this means you can only build your docker-image for your Raspberry Pi on the raspberry pi, which is slow.

I tried several things.

  1. I used the emulator QEMU to emulate the rasberry pi on a fast Macbook. But, because of the inefficiency of the emulation, it is just as slow as building your dockerfile on a raspberry pi.
  2. I tried cross-compiling. This wasn't possible, because the commands in your dockerfile are replayed on a running image and the running raspberry-pi image can only be run on ... a raspberry pi.

How to run a simple node.js application with docker on a raspberry pi  

Step 1: Installing Arch Linux
The first step is to install arch linux on an SD card for the raspberry pi. The preferred OS for the raspberry pi is a debian based OS: Raspbian, which is nicely configured to work with a raspberry pi. But in this case, the ArchLinux is better because we use the OS only to run docker on it. Arch Linux is a much smaller and a more barebone OS. The best way is by following the steps at http://archlinuxarm.org/platforms/armv6/raspberry-pi. In my case, I use version 3.12.20-4-ARCH. In addition to the tutorial:

  1. After downloading the image, install it on a sd-card by running the command:
    sudo dd if=path_of_your_image.img of=/dev/diskn bs=1m
  2. When there is no HDMI output at boot, remove the config.txt on the SD-card. It will magically work!
  3. Login using root / root.
  4. Arch Linux will use 2 GB by default. If you have a SD-card with a higher capacity you can resize it using the following steps http://gleenders.blogspot.nl/2014/03/raspberry-pi-resizing-sd-card-root.html

Step 2: Installing a wifi dongle
In my case I wanted to connect a wireless dongle to the raspberry pi, by following these simple steps

  1. Install the wireless tools:
        pacman -Syu
        pacman -S wireless_tool
        
  2. Setup the configuration, by running:
    wifi-menu
  3. Autostart the wifi with:
        netctl list
        netctl enable wlan0-[name]
    

Because the raspberry pi is now connected to the network you are able to SSH to it.

Step 3: Installing docker
The actual install of docker is relative easy. There is a docker version compatible with the ARM processor (that is used within the Raspberry Pi). This docker is part of the package manager of Arch Linux and the used version is 1.0.0. At the time of writing this blog docker release version 1.1.2. The missing features are

  1. Enhanced security for the LXC driver.
  2. .dockerignore support.
  3. Pause containers during docker commit.
  4. Add --tail to docker logs.

You will install docker and start is as a service on system boot by the commands:

pacman -S docker
systemctl enable docker
Installing docker with pacman

Installing docker with pacman

Step 4: Run a single nodejs application
After we've installed docker on the raspberry pi, we want to run a simple nodejs application. The application we will deploy is inspired on the nodejs web in the tutorial on the docker website: https://github.com/enokd/docker-node-hello/. This nodejs application prints a "hello world" to the console of the webbrowser. We have to change the dockerfile to:

# DOCKER-VERSION 1.0.0
FROM resin/rpi-raspbian

# install required packages
RUN apt-get update
RUN apt-get install -y wget dialog

# install nodejs
RUN wget http://node-arm.herokuapp.com/node_latest_armhf.deb
RUN dpkg -i node_latest_armhf.deb

COPY . /src
RUN cd /src; npm install

# run application
EXPOSE 8080
CMD ["node", "/src/index.js"]

And it works!

Screen Shot 2014-08-07 at 20.52.09

The webpage that runs in nodejs on a docker image on a raspberry pi

 

Just by running four little steps, you are able to use docker on your raspberry pi! Good luck!

 

Xebia IT Architects Innovation Day

Sat, 08/23/2014 - 17:51

Friday August 22nd was Xebia’s first Innovation Day. We spent a full day experimenting with technology. I helped organizing the day for XITA, Xebia’s IT Architects department (Hmm. Department doesn’t feel quite right to describe what we are, but anyway). Innovation days are intended to inspire as well as educate. We split up in small teams and focused on a particular technology. Below is as list of project teams:

‚ÄĘ Docker-izing enterprise software
‚ÄĘ Run a web application high-available across multiple CoreOS nodes using Kubernetes
‚ÄĘ Application architecture (team 1)
‚ÄĘ Application architecture (team 2)
‚ÄĘ Replace Puppet with Salt
‚ÄĘ Scale "infinitely" with Apache Mesos

In the coming weeks we will publish what we learned in separate blogs.

First Xebia Innovation Day

Help! Too Many Incidents! - Capacity Assignment Policy In Agile Teams

Wed, 08/20/2014 - 22:26

As an Agile coach, scrum master, product owner, or team member you probably have been in the situation before in which more work is thrown at the team than the team has capacity to resolve.

In case of work that is already known this basically is a scheduling problem of determining the optimal order that the team will complete the work so as to maximise the business value and outcome. This typically applies to the case that a team is working to build or extend a new product.

The other interesting case is e.g. operational teams that work on items that arrive in an ad hoc way. Examples include production incidents. Work arrives ad hoc and the product owner needs to allocate a certain capacity of the team to certain types of incidents. E.g. should the team work on database related issues, or on front-end related issues?

If the team has more than enough capacity the answer is easy: solve them all! This blog will show how to determine what capacity of the team is best allocated to what type of incident.

What are we trying to solve?

Before going into details, let's define what problem we want to solve.

Assume that the team recognises various types of incidents, e.g. database related, GUI related, perhaps some more. Each type of incident will have an associated average resolution time. Also, each type will arrive at the team at a certain rate, the input rate. E.g. database related incidents arrive 3 times per month, whereas GUI related incidents occur 4 times per week. Finally, each incident type will have different operational costs assigned to it. The effect of database related incidents might be that 30 users are unable to work. GUI related incidents e.g. affect only part of the application affecting a few users.

At any time, the team has a backlog of incidents to resolve. With this backlog an operational cost is concerned. This operational we want to minimise.

What makes this problem interesting is that we want to minimise this cost under the constraint of having limited number of resources, or capacity. The product owner may wish to deliberately ignore GUI type of incidents and let the team work on database related incidents. Or assign 20% of the capacity to GUI related and 80% of the available capacity to database related incidents?

Types of Work

For each type of work we define the input rate, production rate, cost rate, waiting time, and average resolution time:

 \lambda_i = \text{average input rate for type '$i$'}, \lambda_i = \text{average input rate for type '$i$'},

 C_i = \text{operational cost rate for type '$i$'}, C_i = \text{operational cost rate for type '$i$'},

 x_i = \text{average resolution time for type '$i$'}, x_i = \text{average resolution time for type '$i$'},

 w_i = \text{average waiting time for type '$i$'}, w_i = \text{average waiting time for type '$i$'},

 s_i = \text{average time spend in the system for type '$i$'}, s_i = \text{average time spend in the system for type '$i$'},

 \mu_i = \text{average production rate for type '$i$'} \mu_i = \text{average production rate for type '$i$'}

Some items get resolved and spend the time s_i = x_i + w_is_i = x_i + w_i in the system. Other items never get resolved and spend time  s_i = w_i s_i = w_i in the system.

In the previous blog Little's Law in 3D the average total operational cost is expressed as:

 \text{Average operational cost for type '$i$'} = \frac{1}{2} \lambda_i C_i \overline{S_i(S_i+T)} \text{Average operational cost for type '$i$'} = \frac{1}{2} \lambda_i C_i \overline{S_i(S_i+T)}

To get the goal cost we need to sum this for all work types 'i'.

System

The process for work items is that they enter the system (team) as soon as they are found or detected. When they are found these items will contribute immediately to the total operational cost. This stops as soon as they are resolved. For some the product owner decides that the team will start working on them. The point that the team start working on an item the waiting time w_iw_i is known and on average they spend a time x_ix_i before it is resolved.

As the team has limited resources, they cannot work on all the items. Over time the average time spent in the system will increase. As shown in the previous blog Why Little's Law Works...Always Little's Law still applies when we consider a finite time interval.

This process is depicted below:

new doc 13_2

 \overline{M} = \text{fixed team capacity}, \overline{M} = \text{fixed team capacity},

 \overline{M_i} = \text{team capacity allocated to working on problems type '$i$'}, \overline{M_i} = \text{team capacity allocated to working on problems type '$i$'},

 \overline{N} = \text{total number of items in the system} \overline{N} = \text{total number of items in the system}

The total number of items allowed in the 'green' area is restricted by the team's capacity. The team may set a WiP limit to enforce this. In contrast the number of items in the 'orange' area is not constrained: incidents flow into the system as they are found and leave the system only after they have been resolved.

Without going into the details, the total operational cost can be rewritten in terms of x_ix_i and w_iw_i:

(1)  \text{Average operational cost for type '$i$'} = \frac{1}{2} \lambda_i C_i \overline{w_i(w_i+T)} + \mu_i C_i \overline{x_i} \,\, \overline{w_i} + \frac{1}{2} \mu_i C_i \overline{x_i(x_i+T)} \text{Average operational cost for type '$i$'} = \frac{1}{2} \lambda_i C_i \overline{w_i(w_i+T)} + \mu_i C_i \overline{x_i} \,\, \overline{w_i} + \frac{1}{2} \mu_i C_i \overline{x_i(x_i+T)}

What are we trying to solve? Again.

Now that I have shown the system, defined exactly what I mean with the variables, I will refine what exactly we will be solving.

Find M_iM_i such that this will minimise (1) under the constraint that the team has a fixed and limited capacity.

Important note

The system we are considering is not stable. Therefore we need to be careful when applying and using Little's Law. To circumvent necessary conditions for Little's Law to hold, I will consider the average total operational cost over a finite time interval. This means that we will minimise the average of the cost over the time interval from start to a certain time. As the accumulated cost increases over time the average is not the same as the cost at the end of the time interval.

Note: For our optimisation problem to make sense the system needs to be unstable. For a stable system it follows from Little's Law that the average input rate for type i is equal to the average production rate for type 'i'. In case there is no optimisation since we cannot choose those to be different. The ability to choose them differently is the essence of our optimisation problem.

Little's Law

At this point Little's Law provides a few relations between the variables  M, M_i, N, w_i, x_i, \mu_i, \lambda_i M, M_i, N, w_i, x_i, \mu_i, \lambda_i . These relations we can use to find what values of M_iM_i will minimise the average total operational cost.

As described in the previous blog Little's Law in 3D Little's Law gives relations for the system as a whole, per work item type and for each subsystem. These relations are:

 \overline{N_i} = \lambda_i \,\, \overline{s_i} \overline{N_i} = \lambda_i \,\, \overline{s_i}

 \overline{N_i} - \overline{M_i} = \lambda_i \,\, \overline{w_i} \overline{N_i} - \overline{M_i} = \lambda_i \,\, \overline{w_i}

 \overline{M_i} = \mu_i \,\,\overline{x_i} \overline{M_i} = \mu_i \,\,\overline{x_i}

 M_1 + M_2 + ... = M M_1 + M_2 + ... = M

The latter relation is not derived from Little's Law but merely states that total capacity of the team is fixed.

Note that Little's Law also has given us relation (1) above.

Result

Again, without going into the very interesting details of the calculation I will just state the result and show how to use it to calculate the capacities to allocate to certain work item types.

First, for each work item type determine the product between the average input rate (\lambda_i\lambda_i) and the average resolution time (x_ix_i). The interpretation of this is the average number of new incidents arriving while the team works on resolving an item. Put the result in a row vector and name it 'V':

(2)  V = (\lambda_1 x_1, \lambda_2 x_2, ...) V = (\lambda_1 x_1, \lambda_2 x_2, ...)

Next, add all at the components of this vector and denote this by ||V||||V||.

Second, multiply the result of the previous step for each item by the quotient of the average resolution time (x_ix_i) and the cost rate (C_iC_i). Put the result in a row vector and name it 'W':

(3)  W = (\lambda_1 x_1 \frac{x_1}{C_1}, \lambda_2 x_2 \frac{x_2}{C_2}, ...) W = (\lambda_1 x_1 \frac{x_1}{C_1}, \lambda_2 x_2 \frac{x_2}{C_2}, ...)

Again, add all components of this row vector and call this ||W||||W||.

Then, the capacity to allocate to item of type 'k' is proportional to:

(4)  \frac{M_k}{M} \sim W_k - \frac{1}{M} (W_k ||V|| - V_k ||W||) \frac{M_k}{M} \sim W_k - \frac{1}{M} (W_k ||V|| - V_k ||W||)

Here, V_kV_k denotes the k-th component of the row vector 'V'. So, V_1V_1 is equal to \lambda_1 x_1\lambda_1 x_1. Likewise for W_kW_k.

Finally, because these should add up to 1, each of (4) is divided by the sum of all of them.

Example

If this seems complicated, let's do a real calculation and see how the formulas of the previous section are applied.

Two types of incidents

As a first example consider a team that collects data on all incidents and types of work. The data collected over time includes the resolution time, dates that the incident occurred and the date the issue was resolved. The product owner assigns a business value to each incident which corresponds to the cost rate of the incident which in this case is measured in the number of (business) uses affected. Any other means of assigning a cost rate will do also.

The team consist of 6 team members, so the team's capacity MM is equal to 12 where each member is allowed to work on a maximum of 2 incidents.

From their data they discover that they have 2 main types of incidents. See the so-called Cycle Time Histogram below.

new doc 13_9

The picture above shows two types of incidents, having typical average resolution times of around 2 days and 2 weeks. Analysis shows that these are related to the GUI and database components respectively. From their data the team determines that they have an average input rate of 6 per week and 2 per month respectively. The average cost rate for each type is 10 per day and 200 per day respectively.

That is, the database related issues have: \lambda = 2 \text{per month} = 2/20 = 1/10 \text{per day} \lambda = 2 \text{per month} = 2/20 = 1/10 \text{per day} ,  C = 200 \text{per day} C = 200 \text{per day} , and resolution time  x = 2 \text{weeks} = 10 \text{days} x = 2 \text{weeks} = 10 \text{days} . While the GUI related issues have:  \lambda = 6 \text{per week} = 6/5 \text{per day} \lambda = 6 \text{per week} = 6/5 \text{per day} ,  C = 10 \text{per day} C = 10 \text{per day} , and resolution time  x = 2 \text{days} x = 2 \text{days} .

The row vector 'V' becomes (product of \lambda\lambda and xx:

 V = (1/10 * 10, 6/5 * 2) = (1, 12/5) V = (1/10 * 10, 6/5 * 2) = (1, 12/5) ,   ||V|| = 1 + 12/5 = 17/5 ||V|| = 1 + 12/5 = 17/5

The row vector 'W' becomes:

 W = (1/10 * 10 * 10 / 200, 6/5 * 2 * 2 / 10) = (1/20, 12/25) W = (1/10 * 10 * 10 / 200, 6/5 * 2 * 2 / 10) = (1/20, 12/25) ,  ||W|| = 1/20 + 12/25 = 53/100 ||W|| = 1/20 + 12/25 = 53/100

Putting this together we obtain the result that a percentage of the team's capacity should be allocated to resolve database related issues that is equal to:

 M_\text{database}/M \sim 1/20 - 1/12 *(1/20 * 17/5 - 1 * 53/100) = 1/20 + 1/12 * 36/100 = 1/20 + 3/100 = 8/100 = 40/500 M_\text{database}/M \sim 1/20 - 1/12 *(1/20 * 17/5 - 1 * 53/100) = 1/20 + 1/12 * 36/100 = 1/20 + 3/100 = 8/100 = 40/500

and a percentage should be allocated to work on GUI related items that is

 M_\text{GUI}/M \sim 12/25 - 1/12 *(12/25 * 17/5 - 12/5 * 53/100) = 12/25 - 1/12 * 9/125 = 12/25 - 3/500 = 237/500 M_\text{GUI}/M \sim 12/25 - 1/12 *(12/25 * 17/5 - 12/5 * 53/100) = 12/25 - 1/12 * 9/125 = 12/25 - 3/500 = 237/500

Summing these two we get as the sum 277/500. This means that we allocate 40/237 ~ 16% and 237/277 ~ 84% of the team's capacity to database and GUI work items respectively.

Kanban teams may define a class of service to each of these incident types and put a WiP limit on the database related incident lane of 2 cards and a WiP limit of 10 to the number of cards in the GUI related lane.

Scrum teams may allocate part of the team's velocity to user stories related to database and GUI related items based on the percentages calculated above.

Conclusion

Starting with the expression for the average total operational cost I have shown that this leads to an interesting optimisation problem in which we ant to determine the optimal allocation of a team's capacity to different work item type in such a way that it will on average minimise the average total operation cost present in the system.

The division of the team's capacity over the various work item types is determined by the work item types' average input rate, resolution time, and cost rate and is proportional to

(4)  \frac{M_k}{M} \sim W_k - \frac{1}{M} (W_k ||V|| - V_k ||W||) \frac{M_k}{M} \sim W_k - \frac{1}{M} (W_k ||V|| - V_k ||W||)

The data needed to perform this calculation is easily gathered by teams. Teams may use a cycle time histogram to find appropriate work item types. See this article on control charts for more information.

 

BE Agile before you Become Agile

Wed, 08/20/2014 - 20:49

People dislike change. It disrupts our routines and we need to invest to adapt. We only go along if we understand why change is needed and how we benefit from it.
The key to intrinsic motivation is to experience the benefits of the change yourself, rather than having someone else explain it to you.

Agility is almost an acronym for change. It is critical to let people experience the benefits of Agility before asking them to buy into this new way of working. This post explains how to create a great Agile experience in a fun, simple, cost efficient and highly effective way. BEing agile, before BEcoming agile!

The concept of a ‚ÄúCompany Innovation Day‚ÄĚ

Have you seen this clip about Dan Pinks’ Drive? According to him, the key factors for more motivation and better performance are: autonomy, mastery and purpose.
If you have some scrum experience this might sound familiar, right? That is because these 3 things really tie in nicely with agile and scrum, for example:

Autonomy = being able to self-direct;
‚ÄĘ Let the team plan their own work
‚ÄĘ Let the team decide how to best solve problems

Mastery = learning, applying and mastering new skills and abilities, a.k.a. "get better at stuff";
‚ÄĘ Retrospect and improve
‚ÄĘ Learn, apply and master new skills to get achieve goals as a team.

Purpose = understanding necessity and being as effective as possible;
‚ÄĘ Write user stories that add value
‚ÄĘ Define sprint goals that tie in to product- and business goals.

In the clip, the company "Atlassian" is mentioned. This is the company that makes "JIRA", one of the most popular Agile support tools. Atlassian tries to facilitate autonomy, mastery and purpose by organizing one day per quarter of ‚Äúmanagement free‚ÄĚ innovation. They call it a ‚Äúship it day‚ÄĚ.

Now this is cool! According to Dan, their people had fun (most important), fixed a whole array of bugs and delivered new product ideas as well. They have to ship all this in one day, again showing similarities with the time boxed scrum approach. When I first saw this, I realized that this kind of fast delivery of value is pretty much something you would like to achieve with Agile Scrum too! Doing Scrum right would feel like a continuous series of ship it days.

My own experience with innovation days

Recently I organized an innovation day with a client (for tips see on how to organize yours, click here). We invited the whole department to volunteer. If you didn’t feel like it, you could just skip it and focus on sprint work. Next we promoted the day and this resulted in a growing list of ideas coming in.
Except for the framing of the day, the formation of ideas and teams was totally self-organized and also result driven as we asked for the expected result. Ultimately we had 20 initiatives to be completed in one day.
On the day itself, almost everyone joined in and people worked hard to achieve results at the end of the day.
The day ended in presenting the results and having pizzas. Only some ideas just missed the deadline, but most were finished including usable and fresh new stuff with direct business value. When looking at the photos of that day it struck me that 9 out of ten photos showed smiling faces. Sweet!

The first innovation day was concluded with an evaluation. In my opinion evaluation is essential, because this is the perfect moment discuss deeper lessons and insights. Questions like; ‚Äúhow can we create the innovation day energy levels during sprints‚ÄĚ, and ‚Äúhow can we utilize self-organizing abilities more‚ÄĚ are invaluable as they could lead to new insights, inspiration and experiments for day-to-day operations.

The value of an innovation day as a starting point for Agile

All in all, I think an innovation day is the perfect way to get people experiencing the power of Agile.
Doing the innovation day on ‚Äúday one‚ÄĚ offers huge benefits when added to standard stuff like training and games. This is because the context is real. You have a real goal, a real timebox and you need to self-organize to achieve the desired result.
People doing the work get to experience their potential and the power of doing stuff within a simplified context. Managers get to experience unleashing the human potential when they focus only on the context and environment for that day.
I can only imagine the amazement and renewed joy when people experience the possibilities coming from a strong waterfall setting. All that good stuff from just a one-day investment!

Conclusion

It would be great if you would start out an Agile change initiative with an innovation day. Get people enthusiastic and inspired (e.g. motivated for change) first and then tell them why it works and how we are going to apply the same principles in day-to-day operations. This will result in less friction and resistance and give people a better sense for where they are heading.

Do you want to start doing innovation days or do you want to share your experience, feel free to leave a comment below.

Agile 2014 ‚Äď speaking and attending; a summary

Tue, 08/19/2014 - 17:14

So Agile 2014 is over again… and what an interesting conference it was.

What did I find most rewarding? Meeting so many agile people! My first conclusion was that there were experts like us agile consultants or starting agile coaches, ScrumMasters and other people getting acquainted with our cool agile world. Another trend I noticed was the scaled agile movement. Everybody seems to be involved in that somehow. Some more successful than others; some more true to agile than others.

What I missed this year was the movement of scrum or agile outside IT although my talk about scrum for marketing had a lot of positive responses.  Everybody I talked to was interested in hearing more information about it.

There was a talk maybe even two about hardware agile but I did not found a lot of buzz around it. Maybe next year? I do feel that there is potential here. I believe Fullstack product development should be the future. Marketing and IT teams? Hardware and software teams?  Splitting these still sounds as design and developer teams to me.

But what a great conference it was. I met a lot of awesome people. Some just entering the agile world; some authors of books I had read which got me further in the agile movement. I talked to the guys from Spotify. The company which is unique in its agile adoption / maturity. And they don’t even think that they are there yet. But then again will somebody ever truly BE agile ..?

I met the guys from scrum.inc who developed a great new scaled framework. Awesome ideas on that subject and awesome potential to treat it as a community created open framework; keep your eyes open for that!

I attended some nice talks too; also some horrible ones. Or actually 1, which should never have been presented in a 90 minute slot in a conference like this. But lets get back to the nice stories. Lyssa Adkins had a ‚Äėtalk‚Äô about conflicts. Fun thing was that she actually facilitated the debate about scaled agile on stage. The session could have been better but the idea and potential of the subject is great.

Best session? Well probably the spotify guys. Still the greatest story out there of an agile company. The key take-out of that session for me is: agile is not an end-state, but a journey. And if you take it as serious as Spotify you might be able to make the working world a lot better. Looking at Xebia we might not even be considered to be trying agile compared to them. And that is meant in a humble way while looking up to these guys! - I know we are one of the frontiers of agile in the Netherlands. The greatest question in this session: ‚ÄėWhere is the PMO in your model‚Ķ.‚Äô

Well you clearly understand this …

Another inspiring session was the keynote session from the CFO of Statoil about beyond budgeting. This was a good story which should become bigger in the near future as this is one of the main questions I get when implementing agile in a company: ‚Äúhow do we plan / estimate and budget projects when we go and do agile?‚ÄĚ Beyond budgeting at least get‚Äôs us a little closer.
Long story short. I had a blast in Orlando. I learnt new things and met a lot of cool people.My main take out: Our community is growing which teaches us that we are not yet there by a long run. An awesome future is ahead! See you next year!

Little's Law in 3D

Sun, 08/17/2014 - 16:21

The much used relation between average cycle time, average total work and input rate (or throughput) is known as Little's Law. It is often used to argue that it is a good thing to work on less items at the same time (as a team or as an individual) and thus lowering the average cycle time. In this blog I will discuss the less known generalisation of Little's Law giving an almost unlimited number of additional relation. The only limit is your imagination.

I will show relations for the average 'Total Operational Cost in the system' and for the average 'Just-in-Timeness'.

First I will describe some rather straightforward generalisations and in the third part some more complex variations on Little's Law.

Little's Law Variations

As I showed in the previous blogs (Applying Little's Law in Agile Games and Why Little's Law Works...Always) Little's Law in fact states that measuring the total area from left-to-right equals summing it from top-to-bottom.

Once we realise this, it is easy to see some straightforward generalisations which are well-known. I'll mention them here briefly without ging into too much details.

Subsystem

new doc 8_1 Suppose a system that consists of 1 or more subsystems, e.g. in a kanban system consisting of 3 columns we can identify the subsystems corresponding to:

  1. first column (e.g. 'New') in 'red',
  2. second column (e.g. 'Doing') in 'yellow',
  3. third column (e.g. 'Done') in 'green'

See the figure on the right.

By colouring the subsystems different from each other we see immediately that Little's Law applies to the system as a whole as well as to every subsystem ('red' and 'yellow' area).

Note: for the average input rate consider only the rows that have the corresponding color, i.e. for the input rate of the column 'Doing' consider only the rows that have a yellow color; in this case the average input rate equals 8/3 items per round (entering the 'Doing' column). Likewise for the 'New' column.

Work Item Type

new doc 9_1Until now I assumed only 1 type of work items. In practise teams deal with more than one different work item types. Examples include class of service lanes, user stories, and production incidents. Again, by colouring the various work item type differently we see that Little's Law applies to each individual work item type.

In the example on the right, we have coloured user stories ('yellow') and production incidents ('red'). Again, Little's Law applies to both the red and yellow areas separately.

Doing the math we se that for 'user stories' (yellow area):

  • Average number in the system (N) = (6+5+4)/3 = 5 user stories,
  • Average input rate (\lambda\lambda = 6/3 = 2 user stories per round,
  • Average waiting time (W) = (3+3+3+3+2+1)/6 = 15/6 = 5/2 rounds.

As expected, the average number in the system equals the average input rate times the average waiting time.

The same calculation can be made for the production incidents which I leave as an exercise to the reader.

Expedite Items

new doc 10_1 Finally, consider items that enter and spend time in an 'expedite' lane. In Kanban an expedite lane is used for items that need special priority. Usually the policy for handling such items are that (a) there can be at most 1 such item in the system at any time, (b) the team stop working on anything but on this item so that it is completed as fast as possible, (c) they have priority over anything else, and (d) they may violate any WiP limits.

Colouring any work items blue that spend time in the expedite lane we can apply Little's Law to the expedite lane as well.

An example of the colouring is shown in the figure on the right. I leave the calculation to the reader.

3D


We can even further extend Little's Law. Until now I have considered only 'flat' areas.

The extension is that we can give each cell a certain height. See the figure to the right. A variation on Little's Law follows once we realise that measuring the volume from left-to-right is the same as calculating it from top-to-bottom. Instead of measuring areas we measure volumes instead.

The only catch here is that in order to write down Little's Law we need to give a sensible interpretation to the 'horizontal' sum of the numbers and a sensible interpretation to the 'vertical' sum of the numbers. In case of a height of '1' these are just 'Waiting Time' (W) and 'Number of items in the system' (N) respectively.

A more detailed, precise, and mathematical formulation can be found in the paper by Little himself: see section 3.2 in [Lit11].

Some Applications of 3D-Little's Law

Value

As a warming-up exercise consider as the height the (business) value of an item. Call this value 'V'. Every work item will have its own specific value.
new doc 12_1

 

 

 \overline{\mathrm{Value}} = \lambda \overline{V W} \overline{\mathrm{Value}} = \lambda \overline{V W}

The interpretation of this relation is that the 'average (business) value of unfinished work in the system at any time' is equal to the average input rate multiplied by the 'average of the product of cycle time and value'.

Teams may ant to minimise this while at the same time maximising the value output rate.

Total Operational Cost

As the next example let's take as the height for the cells a sequence of numbers 1, 2, 3, .... An example is shown in the figures below. What are the interpretations in this case?

Suppose we have a work item that has an operational cost of 1 per day. Then the sequence 1, 2, 3, ... gives the total cost to date. At day 3, the total cost is 3 times 1 which is the third number in the sequence.

new doc 12_2The 'vertical' sum is just the 'Total Cost of unfinished work in the system.

For the interpretation of the 'horizontal' sum we need to add the numbers. For a work item that is in the system for 'n' days, the total is 1+2+3+...+n1+2+3+...+n which equals 1/2 n (n+1)1/2 n (n+1). For 3 days this gives 1+2+3=1/2 * 3 * 4 = 61+2+3=1/2 * 3 * 4 = 6. Thus, the interpretation of the 'horizontal' sum is 1/2 W (W+1)1/2 W (W+1) in which 'W' represents the waiting time of the item.

Putting this together gives an additional Little's Law of the form:

 \overline{\mathrm{Cost}} = \frac{1}{2} \lambda C \overline{W(W + 1)} \overline{\mathrm{Cost}} = \frac{1}{2} \lambda C \overline{W(W + 1)}

where 'C' is the operational cost rate of a work item and \lambda\lambda is the (average) input rate. If instead of rounds in a game, the 'Total Cost in the system' is measured at a time interval 'T' the formula slightly changes into

 \overline{\mathrm{Cost}} = \frac{1}{2} \lambda C \overline{W\left(W + T\right)} \overline{\mathrm{Cost}} = \frac{1}{2} \lambda C \overline{W\left(W + T\right)}

Teams may want to minimise this which gives an interesting optimisation problem is different work item types have different associated operational cost rates. How should the capacity of the be divided over the work items? This is a topic for another blog.

Just-in-Time

For a slightly more odd relation consider items that have a deadline associated with them. Denote the date and time of the deadline by 'D'. As the height choose the number of time units before or after the deadline the item is completed. Further, call 'T' the time that the team has taken up to work on the item. Then the team finishes work on this item at time  T + W T + W , where 'W' represent the cycle time of the work item.

new doc 12_4

In the picture on the left a work item is shown that is finished 2 days before the deadline. Notice that the height decreases as the deadline is approached. Since it is finished 2 time units before the deadline, the just-in-timeness is 2 at the completion time.

new doc 12_3

The picture on the left shows a work item one time unit after the deadline and has an associated just-in-timeness of 1.

 

 \overline{\mathrm{Just-in-Time}} = \frac{1}{2} \lambda \overline{|T+W-D|(|T+W-D| + 1)} \overline{\mathrm{Just-in-Time}} = \frac{1}{2} \lambda \overline{|T+W-D|(|T+W-D| + 1)}

This example sounds like a very exotic one and not very useful. A team might want to look at what the best time is to start working on an item so as to minimise the above variable.

Conclusion

From our 'playing around' with the size of areas and volumes and realising that counting it in different ways (left-to-right and top-to-bottom) should give the same result I have been able to derive a new set of relations.

In this blog I have rederived well-known variations on Little's Law regarding subsystems and work items types. In addition I have derived new relations for the 'Average Total Operational Cost', 'Average Value', and 'Average Just-in-Timeness'.

Together with the familiar Little's Law these give rise to interesting optimisation problems and may lead to practical guidelines for teams to create even more value.

I'm curious to hear about the variations that you can come up with! Let me know by posting them here.

References

[Lit11] John D.C. Little, "Little’s Law as Viewed on Its 50th Anniversary", 2011, Operations Research, Vol. 59 , No 3, pp. 536-549, https://www.informs.org/content/download/255808/2414681/file/little_paper.pdf

 

 

The AngularJS Promise DSL

Mon, 08/11/2014 - 10:21

As promised in my previous post, I just pushed the first version of our "Angular Promise DSL" to Github. It extends AngularJS's $q promises with a number of helpful methods to create cleaner applications.

The project is a V1, it may be a bit rough around the edges in terms of practical applicability and documentation, but that's why it's open source now.

The repository is at https://github.com/fwielstra/ngPromiseDsl and licensed as MIT. It's the first OS project I've created, so bear with me. I am accepting pull requests and issues, of course.

Questions? Ask them on the issues page, ask me via Twitter (@frwielstra) or send me an e-mail. I'd offer you to come by my office too... if I had one.

Extending AngularJS services with the Decorate method

Fri, 08/08/2014 - 12:00

Many large Angular applications tend to see a lot of repetition - same API endpoint, same method of dealing with and transforming data, etcetera. One technique you can use to at least alleviate that is using AngularJS's decorate method, which allows you to extend, adjust or even fully replace any existing service.

As you'll see in this post, using this allows you to modify and extend the framework you build your app in, which will lead to a cleaner, more legible codebase, written in a more functional style (the what, not the how).

Update 11/8: The follow-up is now live, along with the GitHub repository.

A feature not often used when developing AngularJS applications is the $provide service, which is the primary service used to register components with the $injector. More commonly, a developer would use methods like $provide.service() or $provide.factory to do so, but those are merely utility methods defined in the $provide service and exposed via angular.module().

The main reasons to use $provide over the service() and factory() methods is to configure the service before it's instantiated, for example. While there may be more advanced use-cases for using $provide, I haven't yet encountered them while developing regular applications and I'm sure they won't occur often.

One of the methods listed at the very bottom of the $provide documentation is the decorate() method. It doesn't look like much (it's at the bottom, after all), but its documentation hints that it's very powerful:

"A service decorator intercepts the creation of a service, allowing it to override or modify the behaviour of the service. The object returned by the decorator may be the original service, or a new service object which replaces or wraps and delegates to the original service."

Nothing to add there. You can use decorate() to change, add to, or completely replace the behaviour of services without having to edit its code. This can be done on any code not your own - core AngularJS services, but also third-party libraries. It's the equivalent of overriding methods in OO languages or monkey-patching in the more dynamic languages.

‚ÄúIsn‚Äôt that evil?‚ÄĚ, I hear you ask. As with every programming-related question, the only correct answer is: it depends. I‚Äôm going to give a few practical examples of when I believe using decorate() is appropriate. In a future blog post, I'll expand on this example, showing how relatively simple code can positively influence your entire application architecture.

Here’s a practical example, a neat follow-up on my previous blog about angular promises: decorating $q to add methods to the promise object. The promise API itself defines only one method: then(). $q adds a few simple methods to that like catch() and finally(), but for your own application you can add a few more.

If you’ve been working with promises for a little while in your AngularJS application, you’ve probably noticed some operations are pretty common; assigning the promise result to the scope (or any object), logging the result in the console, or calling some other method. Using decorate(), we can add methods to the promise object to simplify those. Here's a bit of code from my previous post; we'll add a method to $q to remove the need for a callback:

CustomerService.getCustomer(currentCustomer)
 .then(CartService.getCart)
 .then(function(cart) {
   $scope.cart = cart;
 })
 .catch($log.error);

First, we’ll need to do some boilerplate: we create a function that adds our methods to the promise object, and then we replace all the default promise methods. Note that the decorating function will also apply itself to the given promise.then method again, so that our customisations aren’t lost further down a promise chain:

angular.module('ngPromiseDsl', [])
  .config(function ($provide) {
    $provide.decorator('$q', function ($delegate, $location) {

      // decorating method

      function decoratePromise(promise) {
        var then = promise.then;

        // Overwrite promise.then. Note that $q's custom methods (.catch and .finally) are implemented by using .then themselves, so they're covered too.

        promise.then = function (thenFn, errFn, notifyFn) {
          return decoratePromise(then(thenFn, errFn, notifyFn));
        };

        return promise;
      }

      // wrap and overwrite $q's deferred object methods

      var defer = $delegate.defer,
        when = $delegate.when,
        reject = $delegate.reject,
        all = $delegate.all;

      $delegate.defer = function () {
        var deferred = defer();
        decoratePromise(deferred.promise);
        return deferred;
      };

      $delegate.when = function () {
        return decoratePromise(when.apply(this, arguments));
      };

      $delegate.reject = function () {
        return decoratePromise(reject.apply(this, arguments));
      };

      $delegate.all = function () {
        return decoratePromise(all.apply(this, arguments));
      };

      return $delegate;

    });
  });

With that boilerplate in place, we can now start adding methods. As I mentioned earlier, one of the most common uses of a then() function is to set the result onto the scope (or some other object). This is a fairly trivial operation, and it’s pretty straightforward to add it to the promise object using our decorator, too:

function decoratePromise(promise) {
  var then = promise.then;

  promise.then = function (thenFn, errFn, notifyFn) {
    return decoratePromise(then(thenFn, errFn, notifyFn));
  };

  // assigns the value given to .then on promise resolution to the given object under the given varName
  promise.thenSet = function (obj, varName) {
    return promise.then(function (value) {
      obj[varName] = value;
      return value;
    });
  };

  return promise;
}

That’s all. Put this .config block in your application's module definition, or create a new module and add a dependency to it, and you can use it throughout your application. Here's the same piece of code, now with our new thenSet method:

CustomerService.getCustomer(currentCustomer)
  .then(CartService.getCart)
  .thenSet($scope, 'cart')
  .catch($log.error);

This particular example can be extended in a multitude ways to add useful utilities to promises. In my current project we’ve added a number of methods to the promise object, which allows us to reduce the number of callback definitions in our controllers and thus create cleaner, more legible code.

 

Replacing custom callbacks with named methods allows for a more functional programming style, and allows readers to read and write code as a list of ‚Äúwhats‚ÄĚ, instead of ‚Äúhows‚ÄĚ - and it's also fully asynchronous.

Extending $q is just the start though: Every angular service can be extended for various purposes - add performance monitoring and logging to $http, set common prefixes or fixed properties on $resource urls or template paths, you name it. Leave a remark in the comments about how you've used decorate() to create a better application.

Stay tuned for an upcoming post where I release a small open source project that extends angular’s promise objects with a number of helpful methods to perform common tasks.

Further reading, resources, helpful links:

A Valuable Sprint Review (a.k.a. Demo): How To

Thu, 08/07/2014 - 10:05

A valuable Sprint Review (from now on in this blog referred to as Demo) can be built in three steps. It starts during the Sprint planning session with agreeing on and understanding the user stories on the Sprint backlog. Then, during the Sprint, the team constantly exchanges ideas and results of the realisation of the Story. Finally, during the demo itself, the Product Owner and the rest of the team demo the stories to the stakeholders to display the value delivered and open up for feedback.

Planning for a good demo

During the planning session, it is imperative that the Product Owner and the rest of the team understand the stories that will be picked up. This sounds obvious, but it happens often that this is not the case. Stories might be too technical so the Product Owner is disconnected or stories are so high level that it is hard to determine what needs to be done.

Make sure stories are formulated from the perspective of an end-user of the functionality that will be delivered. This could be an actual user, a system that picks up whatever result is created or any other manifestation of who or what will use the result of the story.

Also take care of getting the acceptance criteria clear. This way it will be clear to developers what to build, to testers what to test for and designers what to design. It will help the Product Owner to have a better idea what is in and what might have to be defined in a new/other user story.

It is important that everyone understands the context in which the story ‚Äėlives‚Äô. What part of the system is touched (end-to-end is preferred but not always possible), which parties are affected by the change, what prerequisites are needed, etc.

Building for a great demo

When during the creation of the value of each story the whole team is in constant contact about intermediate results and decisions taken, everyone will be able to add to the value and be aware of what the result of the story will be. It is very important that the whole team is including the Product Owner. When the PO sees the intermediate results, she or he can already create an image of what the result will be like. Also, the PO can contact stakeholders that might have an opinion of what is created and, when needed, adjust the end result to match expectations.

Delivering an valuable demo

In the demo, the Product Owner should present to the stakeholders the value of each user story that has been delivered. So, per story, explain what has changed from the perspective of the end-user and have the rest of the team show this. Also, when stories are not done, explain which (sub-)functionality is not yet finished. Make sure to ask for feedback from the end-user or other stakeholders on what is demonstrated.

Conclusion

The value of the demo depends largely on the cooperation of the entire team. When the Product Owner and the rest of the team work together on understanding what will be delivered and help each other to get the most value from each story delivered the demo will be focused, valuable and fun.

Scrumban

Wed, 08/06/2014 - 19:19

Scrum has become THE revolution in the world of software development. The main philosophy behind scrum is accepting that a problem cannot be fully understood or defined at start; scrum has the focus on maximizing the team's ability to deliver quickly and respond to emerging requirements. It came as truly refreshing in the time when projects were ruled by procedure and MS-project planning. Because of scrum:

  1. Projects can deliver what the customer needs, not just what he thought he wanted.
  2. Teams are efficient. They work as a unit to reach a common goal.
  3. We have better project roles (like a product owner and scrum master), ceremonies (like daily stand-ups, grooming) and a scrumboard.

But the central question is: "are we there yet"? And the answer is: "No!". We can optimize scrum by mixing it with kanban, which leads to scrumban.

A kanban introduction for scrummers

Instead of scrum, which is a software development framework in the widest sense of the term, kanban is a method. It, or instance, does not define ceremonies and project roles. There are two main principles in kanban I would like to highlight:

  1. Each column on the kanban board represents a step in the workflow. So, instead of the lanes 'todo', 'inprogress' and 'done' like in scrum, you have 'defining', 'developing', 'testing' and 'deploying'. That is a more full-stack view; a task has a wider lifecycle. This concept is also called 'from concept to cash'; from user research and strategic planning to data center operations and product support.
  2. Another principle of Kanban is that it limits WIP (work in progress). An example of a WIP limit is limiting the number of cards allowed in each column. The advantage is that it reveals bottlenecks dynamically. Because of the WIP, Kanban is a pull mechanism. For instance, a tester can only pickup a next work item if there are items available in de done-column of development-lane and when the WIP limit of the test-lane isn't exceeded.

After all kanban is incredibly simple, but at the same time incredibly powerful.

What's wrong with scrum?

  1. The reason why we went to scrum is because we did not want the waterfall approach anymore. But, in fact each sprint in scrum has become a mini waterfall. In each sprint teams plan, try to design, develop and test. At the end the product owner reviews the completed work and decides which of the stories are shippable and ready for production. Those sprints can result in a staccato flow, which can be exhausted. With kanban we can make sprints more agile and the goal is to have a more continuous flow. In comparison with how to run a marathon? You don't make sprints of 200 meters, but rather with a constant rate.
  2. Scrum is a push mechanism and therefore 'pushes' the quality out of your product. When a sprint backlog is defined, the team is asked for a commitment. Whatever happened, a team must satisfy its commitment and at the end of the sprint the product owner must say 'decharge', else the team has failed. No team wants to publicly fail, so most of the time, at the end of the sprint, teams take shortcuts to satisfy the deadline. Those shortcuts are killing for quality! Asking for commitment is like not trusting the intrinsic motivation of the team. The correct commitment is visible during each standup. During a standup team members have to tell each other what they've done the day before. If they are working too long on a story, another team member will rebel. That is the real commitment.
  3. One of the reasons why we do scrum is that it is better to start immediately instead of doing an estimation and a feasibility study upfront, because almost always after the study is completed, the project will be executed. The estimation at the start is not reliable and the feasibility study is just a waste of time. Isn't that the same mistake we make with scrum with the grooming and ready sessions that causes a lot of overhead? The first overhead during grooming is that we give an estimation with relative precision. It is in a developer's nature to argue about the story points; is it 3, 5, 8 or maybe 1 points? And that is a waste. You should only talk about the story sizes large, medium or small. Making a more precise estimation is just a waste of time, because there are too many external factors. Second, with the grooming we do a mini feasibility study. With a team we will think about a direction of the solution, which is fine. But most of the time it takes two or three sprints before it is realized in the sprint. And with all the weekends of beer in between we've already forgot the solutions. So one smart guy says: 'yes, lets document it', but that is an inefficient way for the real problem: there is too much time between the grooming and the realization.

Scrumban: the board of kanban

Untitled

A scrumban board

The first column in a scrumban board is reserved for the backlog, where stories are ordered by their respective priority. There are several rules for the kanban backlog:

  1. It is the product owner's responsibility to fill this lane with stories, and keep it steadily supplied. The product owner must avoid creating or analyzing too many stories, because this is a waste and it corrupts with the Just-In-Time principle of scrumban. Therefore the scrumban board has a WIP-limit of 5 backlog stories.
  2. Assure the necessary level of analysis before starting development. Each story must be analyzed with a minimum effort. That should be done in the Weekly Time Box (WTB), which will be discussed later on.
  3. The backlog should be event-driven with an order point.
  4. Prioritization-on-demand. The ideal work planning process should always provide the team with best things to work on next, no more and no less.

Next to the backlog-column is the tasking-column, in which there should always be at least one story that is tasked (a minimum WIP-limit). If this isn't the case the team will task after the standup to satisfy this condition. A story is picked up from the backlog and is tasked by the team. Tip: put the tasked cards at the back of the story cards. The next columns are the realization columns. Each team is free to add, remove or change necessary columns so it suits the business. In the realization columns there should be a maximum number of stories that are worked on. If the maximum limit has not been reached, a story can be pulled from the tasking column and unfolded on the 'to implement' lane. Now the team can work on the tasks of the story. Each task that is implemented can be moved to the 'ready' lane. If all of the tasks are done for a story, the story can be moved to the next lane. When the story and tasks are ready, the cards can be moved to the right bottom of the board, so there is a new horizontal lane available for the next story.

Scrumban: the ceremonies of scrum

With scrumban we only have two types of meetings: the daily standup and the weekly timeblock. The Weeky Timeblock is a recurring meeting used for multiple purposes. It should be set up in the middle of the week. The big advantage of the weekly timeblock is that developers are distracted from their work only once a week (instead of the various of meetings with scrum).

The Weekly Timeblock contains three parts. First there is a demo of the work done. Second, there is a retrospective on the development process of the last week. Third, the team should have a preview of upcoming workitems. The team try to understand the intent of each item and provide feedback. The only size-indication a team has to make is if the story is small, medium or large. Avoid using poker cards/story points, which are too fine-grained and are to vague.

Conclusion

Scrumban is a mix of the scrum ceremonies and the kanban method. With scrumban we

  1. Introduce the weekly timeblock (WTB). The weekly timeblock should be around 4 hours and there are no more meetings
  2. Have a wider lifecycle of a story: 'from concept to cash'.
  3. Change the scrumboard to  company flows and avoid the push principle of a sprint but using a pull mechanism.

Kanban should be the default choice for DevOps teams

Wed, 08/06/2014 - 14:59

We had a successful workshop on DevOpsDays 2014. Our main point was that Kanban should be the default choice for DevOps teams. The presentation can be downloaded here.

DevOpsDays 2014 was a success

On the 19th, 20th & 21st of June 2014 the second edition of DevOpsDays Amsterdam was held in Pakhuis De Zwijger in Amsterdam. This year I was asked to teach a course there on Kanban for DevOps. At the 2013 edition I also gave a presentation about this subject and it was nice to be invited back to this great event.

With the Open Source mindset in mind I teamed up with Maarten Hoppen en Bas van Oudenaarde. Our message: Kanban should be the default choice for DevOps teams. 

The response to this workshop was very positive and because we received a lot of great feedback I thought I’d share the slide deck. The presentation assumes you are working in an environment where DevOps might work or is already being implemented. 

Main points of the presentation

DevOps is about Culture, Automation, Measurement and Sharing (CAMS). These four values require a way of working that looks past existing processes, handovers and role descriptions. 

The Kanban Method is about looking at your organization in a different way. From a point of:

  • Sustainability: by shaping demand and limiting Work in Progress
  • Service-Orientation: by creating an SLA based on past results and data
  • Survivability: create an improvement mindset in the organization to respond to rapidly changing environments

The three different ways the Kanban Method makes you look at your organization makes it an extremely powerful solution for DevOps.

If you are interested to learn more about the workshop, check out the slides here:

http://www.slideshare.net/jsonnevelt/kanban-bootcamp-devopsdays-2014

Playing around with Yo

Fri, 08/01/2014 - 13:09

Yo has been quite a bit in the news lately. Mainly because it got a lot of investment, which surprised and shocked some people because it all seems too simple. Yo only allows you to send a Yo notification to other users. However, it has a lot of potential to become big while staying that simple.

Screenshot 2014-08-01 14.23.48

After reading Why A Stupid App Like Yo May Have Billion-Dollar Platform Potential a few days ago I felt it was time to play around a bit with Yo and it's API.

I came up with 3 very simple use cases that should be simple to solve with Yo:

  • Get a Yo when it's time to have lunch when I'm at work
  • Get a Yo when I forgot to check out from my parking App
  • Get a Yo when a new blog post is published here on the Xebia Blog

Time to register a couple of Yo developer usernames. The Yo API is, just like Yo itself, very simple. You register a username from which you want to receive notifications at http://yoapi.justyo.co after which you'll receive an API token for that username. Once you have that, people can subscribe to that username with the Yo app. Then you can send a Yo to either all subscribers or to a single subscribe with a simple POST request and the token. All this is explained at https://medium.com/@YoAppStatus/e7f2f0ec5c3c in more detail.

Let's start with our lunch notifications. I created the TIME2LUNCH username for this. I want my notifications at 12:00pm, since that's the time I go to lunch. So all I need is some service that sends the POST request every day at 12:00pm (for now I don't care about getting one in the weekend as well and I only care about my own time zone, Central European Time). Using NodeJS, it's just a single line of code to do such a request:

require('request')
  .post('http://api.justyo.co/yoall/', 
    {
      form: {api_token: 'my_secret_api_token'}
    }
  );

Now we need to have a scheduler that executes it every day at 12:00pm. Luckily Heroku has a scheduler that can do exactly that:

Screenshot 2014-07-31 17.43.51

So after deploying our javascript file with a single line of code and creating our scheduled job we will receive our daily Yo from TIME2LUNCH. Not bad for a first attempt.

Usually my co-workers will remind me when it's time to go to lunch so let's do something that's actually a bit less useless.

To park my car near the office I always use the Parkmobile service. At the end of the day I have to check out to avoid paying too much. Unfortunately it's an easy thing to forget. Parkmobile knows this and can send sms alerts at a specific time or after parking a certain amount of hours. Unfortunately they charge € 0.25 per sms. They also send e-mails, but they're easier to miss. It would be nice to get a Yo instead, for free of course.

What we need is to send the Yo POST request each time we receive the Parkmobile e-mails. Sounds like we might be able to use IFTTT (if this then that) to accomplish this. When browsing for channels and recipes on IFTTT I saw that they already support Yo as a channel. I thought I was gonna be done fast. Unfortunately it's only possible to use Yo as a trigger (if Yo then that) and not as an action (if this then Yo). So we need another solution here. I couldn't find a way to send a cURL request directly from IFTTT, but when Googling for a solution I found a webhook project: https://github.com/captn3m0/ifttt-webhook. The ifttt-webhook works by acting as a WordPress site, which is something that can act as an action of IFTTT. It then allows us to send a POST request to a specific URL. Not exactly the POST requests that are accepted by the Yo API though. But we already made some NodeJS code to send a Yo request so I'm sure we can add some code to accept a request from the ifttt-webhook and then pass it on to something that Yo does understand.

If we follow the instructions on the Github page and set our username to our Yo username and use our API token as password, then the webhook will send a POST request with a JSON body that looks something like this:

{ user: 'MYUSERNAME', pass: 'ab1234-1234-abcd-1234-abcd1234abcd', title: '' }

We can handle that in NodeJS like this:

var express = require('express');
var bodyParser = require('body-parser')
var app = express();
var request = require('request');

app.use(bodyParser.json());
app.post('/api/yo', function (req, res) {
  var user = req.body.user;
  var apiToken = req.body.pass;
  request.post('http://api.justyo.co/yo/',
    {
      form: {
        api_token: apiToken,
        username: user
      }
    });
});

var port = Number(process.env.PORT || 5000);
app.listen(port, function() {
  console.log('Listening on ' + port);
});

This is just a simple express web server that listens for POST calls on /api/yo and then uses the user and pass fields from the body to send a POST request to the Yo API.

This is deployed at http://youser.herokuapp.com/ so everyone can use it as a IFTTT to Yo action.

We can now create our IFTTT recipe. Creating the this step is easy. I receive the e-mails from Parkmobile in my Gmail and their e-mail address is norepy@parkmobile.com. So the rule becomes to trigger each time when I receive an email from them. Then in the that step I activate the WordPress channel with the Yo username and api token and in the body I set the http://youser.herokuapp.com/api/yo URL.

Here is the recipe:

Screenshot 2014-07-31 18.32.12

The last use case I had is to send a Yo each time a new blog post was posted on this blog. For that I registered the XEBIABLOG username (so make sure to subscribe to that in your Yo app if you want to get Yo'd as well for each new blog post).

Since this blog has an RSS feed, I figured I could poll that once in a while to check for new posts. We also already used the Heroku scheduler, so we might as well use that again. I found a little node library called feed-read that makes reading RSS feeds easy. So here is our little app that runs every hour:

var feed = require("feed-read");
var request = require('request');
var ONE_HOUR = 60 * 60 * 1000;

feed("http://blog.xebia.com/feed/", function(err, articles) {
  if (err) throw err;

  var lastArticle = articles[0];
  if ((new Date() - lastArticle.published) < ONE_HOUR) {
    console.log('Sending Yo for ' + lastArticle.title);
    request.post('http://api.justyo.co/yoall/',
      {
        form: {
          api_token: 'my_secret_token'
        }
      });
  }
});

We now have completed our 3 little use cases. Not the most useful things but nice non nonetheless. When looking back on them, we can imagine a couple of improvements. For example for the TIME2LUNCH it would be possible to make a little service where people could register and set their timezone at which they want to receive their notification. We could create a little database that store Yo usernames and the zone. But at this moment it's not possible to verify that USERX is really USERX. Yo doesn't support third party authentication like Facebook and Twitter have with OAuth. Perhaps that's something they will add in the future to make platform more useable for user specific notifications.

Marketing scrum vs IT scrum - a report published and presented at agile 2014

Fri, 07/25/2014 - 17:49

As we know, Scrum is the perfect framework for IT / software development projects to learn, adapt to change and deliver great software of value, faster.

But is Scrum also usable outside of software development? Can we apply similar or maybe even the same principals in other departments in the enterprise?

Yes, we can! And yes there are differences but there are also a lot of similarities.

We (Remco en Me)  successfully implemented Scrum in the marketing departments of two large companies: The ANWB and ING Bank. Both companies are now using Scrum for the development of new campaigns, their full commercial expressions and even at the product development level. They wanted a faster time to market, more ownership, and greater innovation. How did we approach and realized a transition with those goals in the marketing environment? And what are the results?

So when we are not delivering software but other things, how does Scrum change? Well, a great deal actually. The people working in these other departments are, in general, quite different to those in Software Development (and yes more than you would expect). This means coaches or change agents need to take another approach.

Since the people are different, it is possible to go faster or ‚Äėdeeper‚Äô in certain areas. Entrepreneurial skills or ambitions are more present in marketing. This gives a sense of ‚Äėact first apologize later‚Äô, taking ownership, a higher drive to succeed, and upfront and willing behavior. Scrumming here means thinking more about business goals and KPIs (how to go from department to scrumteam goals for example). After that the fun begins‚Ķ

I will be speaking about this topic at agile 2014. A great honor offcourse to be standing there. I will also attende the conference and therefor try to post some updates here.

To read more about this topic you can read my publication about marketing scrum. It has the extensive research paper I publisched about this story. Please feel free to give me comments and questions either about agile 2014 or the paper.

 

Enjoy reading the paper:

Marketing scrum vs IT scrum ‚Äď two marketing case studies who now ‚Äėact first and apologize later'

 

WorldCup 2014 Retrospective: The Magic about Mindset and Leadership

Sun, 07/13/2014 - 13:56

This weekend preparing this bjohan-cruijff-hollande-1974logpost, I ran into a brilliant quote from Johan Cruijff. At a conference a few years ago for the Dutch local government, he told a great story about a talented blind golfer, Ronald Boef he played golf with.  Despite his handicap, Ronald Boef played his best golf in difficult mental circumstances like playing balls over a big pond or consistent putting. The conclusion of Johan Cruijff: "Ronald doesn’t “see" the problems, he is only focussing on the next target. He thinks from a positive mindset".   I couldn’t agree more.  In my opinion, this is one of the fundamentals behind eXtreme Manufacturing (XM) and the reason why the Dutch team didn’t made it through the WorldCup finals.

Like many consultants, topsport is an inspiring source for me.  Almost every day I show or tell stories from great sport coaches like Marc Lammers or Johan Cruijjff.   Like every major sports event, also this WorldCup in Brasil contained some interesting lessons for me I wanted to share with you.

The Big Question: You can have the best individual team members but still not be able to perform.  Why?

 3rd Place Playoff - 2014 FIFA World Cup Brazil

Top Team Ingredient #1: Mindset

The defeat of Spain against the Netherlands, the glorious win of Germany over Brazil showed having fun, faith and determination pay off and a lack of these ingredients will bring you in a lot of trouble.   Until the penalty series of the semi-finals, the right side of this recipe also worked for the Dutch squad. Now, penalty series are for no one a fun exercise, which only leaves faith and determination.   Unlike the previous penalty series against Costa Rica, the Dutch team had no faith in their keeper as a penalty-killer which directly effected the teams determination. They became more hesitant and aware of what could happen when missing a penalty.  Yes, Ronald Boef probably would have taken the penalties better than the Dutch team did against Argentina.. ;-)

Top Team Ingredient #2: Leadership

Like Johan Cruijjf stated in the same video, the leader on the pitch should be 100% concentrated on every detail and also (in my words) be the natural leader of the team, coaching them in keeping the spirit up and giving them enough room “to grow".  Despite his great qualities as a football-player, as a captain Robin van Persie was obviously not the natural leader of the team. Arjan Robben was. The natural leadership of Arjan Robben in combination with his determination was an important reason why The Netherlands were able to regain their motivation and pull off a highly respected 3rd place in this WorldCup.

In my opinion, a high performing team should always have a natural leader.  The options:

  1. A formal leader with natural leadership qualities is the perfect combination.
  2. A formal leader without natural leadership qualities but able to delegate this to another team member is also okay.
  3. A formal leader without natural leadership qualities and ignoring don’t having this competence, is bad news for the team, the team’s environment but above all, for the formal leader himself.

For the new coach van the Dutch team, Guus Hiddink, it will be a challenge convincing Robin van Persie to step back as the 1st captain after nominating Arjan Robben.  Robin van Persie should keep one thing in mind here:  no one is doubting his qualities as a top world class striker.  As a natural leader however, he is not that world class.  Trying to be one is effecting his performance as a world class striker and that would in the end be a disappointment for his supporters but above all, for Robin van Persie himself.

What does this imply for Leadership within organizations?

Leadership, especially natural leadership, is crucial for having highly motivated and productive teams.  The team stays motivated and focussed on their goal.

How ever, a lot of employees are still instrumentally ‚Äúnominated‚ÄĚ to become a coach or manager without having any leadership skills. ¬†In my opinion, natural leadership is something you can‚Äôt gain by nomination or just by learning it. ¬†You can improve it, but there should be some basis of natural leadership. ¬†Ignoring this can be even counter-productive: conflicts will arise, the spirit and productivity will go down.

Why Little's Law Works...Always

Fri, 07/11/2014 - 12:00

On the internet there is much information on Little's Law. It is described an explained in many places [Vac]. Recently, the conditions under which it is true got  attention [Ram11]. As will be explained in this blog the conditions under which the law is true are very mild. It will be shown that for teams working on backlog items virtually there are no conditions.

Why does it work? Under what conditions does it hold?

Airplane Folding

In the previous post (Applying Little's Law in Agile Games) I described how to calculate the quantities in Little's Law. As an example this was applied to the Agile game of folding airplanes consisting of 3 rounds of folding.

airplane-1

Let's look in more detail in which round an airplane was picked up and in which round it was completed. This is depicted in the following figure.

The horizontal axis shows the number of rounds. The vertical axis describes each airplane to fold. The picture is then interpreted as follows. Airplane no. 2 is picked up in round 1 and competed in the same round. It has a waiting time of 1 round. This is indicated at the right of the lowest shaded rectangle.
Airplane no. 8 was picked up in round 1 and finished in round 3. A waiting time of 3 rounds. Airplane no 12 (top most shaded area) was picked up in round 3 and unfinished. Up to round 3 a waiting time of 1 round.

The number 3, 5, and 10 denote the number of completed airplanes at the end of round 1, 2, and 3 respectively.

Waiting Times

The waiting times are determined by counting the number of 'cells' in a row.

The pictures show that we have 12 airplanes (12 'rows'), 3 completed in the first round, 2 more completed in the second round and 5 additionally  folded airplanes in the third and last round giving a total of 10 finished paper airplanes.

All twelve airplanes have waiting times of 1, 1, 1, 2, 2, 3, 3, 3, 3, 3, 3, and 1 respectively.

Work in Progress

In the figure below the number of airplanes picked up by the team in each round are indicated with red numbers above the round. new doc 6_2
In round 1 the team has taken up the folding of 11 airplanes (of which 3 are completed). In round 2 the team was folding 8 airplanes (of which 2 were competed) and in round 3 the team was folding 7 airplanes (of which it completed 5).

Work in progress is determined by counting the number of 'cells' in a column.

Little's Law.....Again

Now that we have determined the waiting times and amount of work in progress, let's calculate the average waiting time and average work in progress.

Average Waiting Time. This quantity we get by adding all waiting times and dividing by the number of items. This gives 26/12.

Average Work in Progress. This quantity is equal to (11+8+7)/3 = 26/3.

Average input rate. This is equal to 12 (the height of the third column) divided by 3 which gives 4.

Again we find that: Average Waiting Time = Average Work in Progress / Average input rate.

Why It Works

new doc 6_3 Little's Law works....always....because the average waiting times is got by adding the lengths of all the rows dividing by the number of rows, so it is proportional to the size of the shaded area in the picture to the right.

The average work in progress is got by adding the heights of the columns in the shaded area which is also proportional to the size of the shaded area.

Both the waiting time and work in progress relate to the size of the shaded area: one by adding the heights and the other by adding the rows. The proportionality corresponds to the average input rate.

Conditions

What assumptions did we make? None...well this is not exactly true. The only assumptions we make in this calculation:

  • We count discrete items
  • There are a finite number of rounds (or sprints)
  • Items enter and possibly leave the system.

That's it. It doesn't need to be stable, ageing (items having increasingly larger waiting times) is not a problem, prioritisation/scheduling of items (also known as queueing discipline), etc. Only the above assumptions need to be true.

Note: Especially the second condition is important, i.e. Little's Law is measured over a finite time interval. For infinite time interval additional conditions need to be fulfilled.

Note: When applying this to agile teams we always consider finite time intervals, e.g. 6 months, 1 year, 8 sprints, etc.

Conclusion

Little's Law is true because the average waiting time is proportional to the size of the shaded area (see figure) and the average work in progress is also proportional to the size of the same shaded area.

Only 3 basic conditions need to be met for Little's Law to be true.

References

[Vac] Little’s Law Part I, Dan Vacanti, http://corporatekanban.com/littles-law-part-i/

[Ram11] Little‚Äôs Law ‚Äď It‚Äôs not about the numbers,¬†Agile Ramblings, http://agileramblings.com/2012/12/11/littles-law-its-not-about-the-numbers/

Bootstrapping and monitoring multiple processes in Docker using monit

Thu, 07/10/2014 - 23:01

If you have every tried to start a docker container and keep it running, you must have encountered the problem that this is no easy task. Most stuff I like to start in container are things like http servers, application servers and various other middleware components which tend to have start scripts that daemonize the program. Starting a single process is a pain, starting multiple processes becomes nasty. My advise is to use monit to start all but the most simple Docker application containers! When I found monit while delving through the inner works Cloud Foundry, I was ecstatic about it! It was so elegant, small, fast, with a beautiful DSL that I thought it was the hottest thing since sliced bread! I was determined to blog it off the roof tops. Until.... I discovered that the first release dated from somewhere in 2002. So it was not hot and new; Clearly I had been lying under a UNIX rock for quite a while. This time, the time was right to write about it! Most of the middleware components I want to start in a docker container, have a habit to start the process, daemonize it and exit immediately, with the docker container on its tail. My first attempt to circumvent this while starting a tomcat server in Docker looked something like this:

/bin/bash -c "service tomcat7 start;while service tomcat7 status;do sleep 1;done

Quite horrific. Imaging the ugliness when you have to start multiple processes. A better solution is needed:  With the zabbix docker container  the problem was solved using simplevisor. As you can read in this post that was not a pleasant experience either. As I knew little about simplevisor and could not solve the problem, I put in an issue and  resorted to a plain installation. But a voice in my head started nagging: "Why don't you fix it and send a pull request?"  (actually, it was the voice of my colleague Arjan Molenaar). Then, I remembered from my earlier explorations to the inner workings of Cloud Foundry, a tool that would be really suitable for the job: monit. Why? It will:

  1. Give you a beautiful,readable specification file stating which processes to start
  2. Make sure that your processes will keep on running
  3. Deliver you a clean and crisp monitoring application
  4. Reduce all your Docker starts to a single command!

In the case of the Zabbix server there were seven processes to start: the zabbix server, agent, java agent, apache, mysql and sshd. In monit this looks as follows:

check process mysqld with pidfile /var/run/mysqld/mysqld.pid
        start program = "/sbin/service mysqld start"
        stop program = "/sbin/service mysqld stop"

check process zabbix-server with pidfile /var/run/zabbix/zabbix_server.pid
        start program = "/sbin/service zabbix-server start"
        stop program = "/sbin/service zabbix-server stop"
        depends on mysqld

check process zabbix-agent with pidfile /var/run/zabbix/zabbix_agentd.pid
        start program = "/sbin/service zabbix-agent start"
        stop program = "/sbin/service zabbix-agent stop"

check process zabbix-java-gateway with pidfile /var/run/zabbix/zabbix_java.pid
        start program = "/sbin/service zabbix-java-gateway start"
        stop program = "/sbin/service zabbix-java-gateway stop"

check process httpd with pidfile /var/run/httpd/httpd.pid
        start program = "/sbin/service httpd start"
        stop program = "/sbin/service httpd stop"
        depends on zabbix-server

check process sshd with pidfile /var/run/sshd.pid
        start program = "/sbin/service sshd start"
        stop program = "/sbin/service sshd stop"

Normally when you start monit it will start as a daemon. But fortunately, you can prevent this with the following configuration.

set init

Your Dockerfile CMD can now always look the same:

    monit -d 10 -Ic /etc/monitrc

Finally, by adding the following statement to the configuration you get an application to view the status of your container processes,

set httpd
     port 2812
     allow myuser:mypassword

After starting the container, surf to port 2812 and you will get a beautiful page showing the state of your processes and the ability to stop and restart them.

monit overview monit process control

Just delve into the documentation of monit and you will find much more features that will allow you to monitor network ports and files, start corrective actions and send out alerts.

Monit is true to its UNIX heritage: it is elegant and promotes an autonomous monitoring system. Monit is cool!