Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!


Neo4j: LOAD CSV – Handling conditionals

Mark Needham - Wed, 06/18/2014 - 00:41

While building up the Neo4j World Cup Graph I’ve been making use of the LOAD CSV function and I frequently found myself needing to do different things depending on the value in one of the columns.

For example I have one CSV file which contains the different events that can happen in a football match:

"1012","Antonin Panenka","174835",21,"penalty"
"1012","Faisal Al Dakhil","2204",57,"goal"
"102","Roger Milla","79318",106,"goal"
"102","Roger Milla","79318",108,"goal"
"102","Bernardo Redin","44555",115,"goal"
"102","Andre Kana-biyik","174649",44,"yellow"

If the type is ‘penalty’, ‘owngoal’ or ‘goal’ then I want to create a SCORED_GOAL relationship whereas if it’s ‘yellow’, ‘yellowred’ or ‘red’ then I want to create a RECEIVED_CARD relationship instead.

I learnt – from reading a cypher script written by Chris Leishman – that we can make FOREACH mimic a conditional by creating a collection with one item in to represent ‘true’ and an empty collection to represent ‘false’.

In this case we’d end up with something like this to handle the case where a row represents a goal:

// removed for conciseness
// goals
FOREACH(n IN (CASE WHEN csvLine.type IN ["penalty", "goal", "owngoal"] THEN [1] else [] END) |
  FOREACH(t IN CASE WHEN team = home THEN [home] ELSE [away] END |
    MERGE (stats)-[:SCORED_GOAL]->(penalty:Goal {time: csvLine.time, type: csvLine.type})

And equally when we want to process a row that represents a card we’d have this:

// cards
FOREACH(n IN (CASE WHEN csvLine.type IN ["yellow", "red", "yellowred"] THEN [1] else [] END) |
  FOREACH(t IN CASE WHEN team = home THEN [home] ELSE [away] END |
    MERGE (stats)-[:RECEIVED_CARD]->(card {time: csvLine.time, type: csvLine.type})

And if we put everything together we get this:

MATCH (home)<-[:HOME_TEAM]-(match:Match {id: csvLine.match_id})-[:AWAY_TEAM]->(away)
MATCH (player:Player {id: csvLine.player_id})-[:IN_SQUAD]->(squad)<-[:NAMED_SQUAD]-(team)
MATCH (player)-[:STARTED|:SUBSTITUTE]->(stats)-[:IN_MATCH]->(match)
// goals
FOREACH(n IN (CASE WHEN csvLine.type IN ["penalty", "goal", "owngoal"] THEN [1] else [] END) |
  FOREACH(t IN CASE WHEN team = home THEN [home] ELSE [away] END |
    MERGE (stats)-[:SCORED_GOAL]->(penalty:Goal {time: csvLine.time, type: csvLine.type})
// cards
FOREACH(n IN (CASE WHEN csvLine.type IN ["yellow", "red", "yellowred"] THEN [1] else [] END) |
  FOREACH(t IN CASE WHEN team = home THEN [home] ELSE [away] END |
    MERGE (stats)-[:RECEIVED_CARD]->(card {time: csvLine.time, type: csvLine.type})

You can have a look at the code on github or follow the instructions to get all the World Cup graph into your own local Neo4j.

Feedback welcome as always.

Categories: Programming

One Change at a Time

Xebia Blog - Tue, 06/17/2014 - 08:22

One of the twelve Agile principles states "At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behaviour accordingly" [1]. Many Agile teams have retrospectives every two weeks that result in concrete actions that are executed in the next time period (sprint). There are many types of tuning and adjustments that a team can do. Examples are actions that improve the flow of work, automation of tasks, team coorporation.

Is it good habit for retrospectives to focus on the same type of improvement or should the team alter the type of improvements that will be done? In this blog I will look into the effect of multiple consecutive actions that affect the flow of work.

The simulation is inspired by the GetKanban game [2].

An Experiment

Ideally one would have a set-up for an experiment in which two exactly equivalent teams are compared. One team that would perform consecutive actions to improve the flow of work. The other team would just make one adjustment to improve the flow, and focus subsequent improvements on other areas than flow. At the same time the flow is measured and verified after a certain period of time whether the first or the second team has achieved better results in terms of flow.

Such an experiment is in practise (very) difficult to perform. In this blog I will study the same experiment by making use of simulations.


For the purpose of simulation I consider a team consisting of three specialists: 1 designer, 1 developer, and 1 tester. The team uses a kanban process to achieve flow. See the picture below for the begin situation.


In the simulation, at the beginning of each working day it is determined how much work will be completed by the team. During the simulation the average cycle time is measured. The initial Work in Progress (WiP) limits are set to 3 for each column and is indicated by the red '3's in the picture.

The average amount of work done by the 'team' and the average effort of one work item are such that on average it takes one card about 5,5 days to complete.

At the end of each work day, cards are pulled into the next columns (if allowed by the WiP limits). The policy is to always pull in as much work as allowed so the columns are maximally filled. Furthermore, the backlog is assumed to always have enough user stories ready to be pulled into the 'design' column. This very much resembles developing a new product when the backlog is filled with more than enough stories.

The system starts with a clean board and all column empty. After letting the system run for 75 simulated work days, we will trigger a policy change. Particularly the WiP limit for the 'design' is increased from '3' to '5'. After this policy change the system runs for another 100 work days.

From the chart showing the average cycle time we will be able to study the effect of WiP limit changing adjustments.


The simulation assumes a simple uniform distribution for the amount of work done by the team and the effort assigned to a work item. I assume this is OK for the purpose of this blog. A consequence of this, is that the result probably can't be scaled. For instance, the situation in which a column in the picture above is a single scrum team is not applicable since a more complex probability distribution should be used instead of the uniform distribution.


The picture below shows the result of running the experiment.




After the start it takes the system little over 40 work days to reach the stable state of an average cycle time of about 24(*) days. This is the cycle time one would expect. Remember, the 'ready' column has a size of '3' and the other columns get work done. So, one would expect a cycle time of around 4 times 5,5 which equals 22 days which is close to 24.

At day 75 the WiP limit is changed. As can be inferred from the picture, the cycle time starts to rise only at day 100 (takes about 1 cycle time (24 days) to respond). The new stable state is reached at day 145 with an average cycle time of around 30(**) days. It takes 70 days(!) to reach the new equilibrium.

The chart shows the following interesting features:

  1. It takes roughly 2 times the (new) average cycle time to reach the equilibrium state,
  2. The response time (when one begins to notice an effect of the policy change) is about the length of the average cycle time.

(*) One can calculate (using transition matrices) the theoretical average cycle time for this system to be 24 days.

(**) Similar, the theoretical average cycle time of the system after policy change is 31 days.



In this blog we have seen that when a team makes adjustments that affect the flow, the system needs time to get to its new stable state. Until this state has been reached any new tuning of the flow is questionable. Simulations show that the time it takes to reach the new stable state is about 2 times the average cycle time.

For scrum teams that have 2-week sprints, the system may need about 2 months before new tuning of flow is effective. Meanwhile, the team can very well focus on other improvements, e.g. retrospectives that focus on the team aspect or collaboration with the team's environment.

Moreover, don't expect to see any changes in measurements of e.g. cycle time within the time period of the average cycle time after making a flow affecting change.

To summarise, after making flow affecting changes (e.g. increasing or decreasing WiP limits):

  • Let the system run for at least the duration of the average cycle time so it has time to respond to the change,
  • After it responds, notice the effect of the change,
  • If the effect is positive, let the system run for another duration of the average cycle time, to get to the new stable state,
  • If the effect is negative, do something else, e.g. go back to the old state, and remember that the system needs to respond to this as well!

[1] Agile manifesto,

[2] GetKanban,

Stop Taking Yourself So Seriously

Making the Complex Simple - John Sonmez - Mon, 06/16/2014 - 15:00

There is a pretty good chance I screwed up today. There is a pretty good chance that this blog article will flop. Maybe, you’ll come here to my blog, expecting to be enlightened, but instead find a bunch of worthless drivel that wasn’t worth the time it took you to read. But, guess what? I […]

The post Stop Taking Yourself So Seriously appeared first on Simple Programmer.

Categories: Programming

Combining Salt with Docker

Xebia Blog - Sat, 06/14/2014 - 10:59

You could use Salt to build and run Docker containers but that is not how I use it here. This blogpost is about Docker containers that run Salt minions, which is just an experiment. The use case? Suppose you have several containers that run a particular piece of middleware, and this piece of middleware needs a security update, i.e. an OpenSSL hotfix. It is necessary to perform the update immediately.


The Dockerfile

In order to build a container you have to write down the container description in a file called Dockerfile. Here is the Dockerfile:

# Standard heading stuff

FROM centos

# Do Salt install stuff and squeeze in a master.conf snippet that tells the minion
# to contact the master specified.

RUN rpm -Uvh
RUN yum install -y salt-minion --enablerepo=epel-testing
RUN [ ! -d /etc/salt/minion.d ] && mkdir /etc/salt/minion.d
ADD ./master.conf /etc/salt/minion.d/master.conf

# Run the Salt Minion and do not detach from the terminal.
# This is important because the Docker container will exit whenever
# the CMD process exits.

CMD /usr/bin/salt-minion


Build the image

Time to run the Dockerfile through docker. The command is:

$ docker build --rm=true -t salt-minion .

provided that you run this command in the directory where file Dockerfile and master.conf resides. Docker creates an image with tag ‘salt-minion’ and throws away all intermediate images after a successful build.


Run a container

The command is:

$ docker run -d salt-minion

and Docker returns:


The Salt minion on the container is started and searches for a Salt master to connect to, defined by the configuration setting “master” in file /etc/salt/minion.d/master.conf. You might want to run the Salt master in “auto_accept” mode so that minion keys are accepted automatically. Docker assigns a container id to the running container. That is the magic key that docker reports as a result of the run command.

The following command shows the running container:

$ docker ps
CONTAINER ID        IMAGE                COMMAND                CREATED             STATUS              NAMES
273a6b77a8fa        salt-minion:latest   /bin/sh -c /etc/rc.l   3 seconds ago       Up 3 seconds        distracted_lumiere


Apply the hot fix
There you are: the Salt minion is controlled by your Salt master. Provided that you have a state module that contains the OpenSSL hot fix, you can now easily update all docker nodes to include the hotfix:

salt \* state.sls openssl-hotfix

That is all there is to it.

Simplicity is the Ultimate Enabler

“Everything should be made as simple as possible, but not simpler.” – Albert Einstein

Simplicity is among the ultimate of pursuits.  It’s one of your most efficient and effective tools in your toolbox.  I used simplicity as the basis for my personal results system, Agile Results, and it’s served me well for more than a decade.

And yet, simplicity still isn’t treated as a first-class citizen.

It’s almost always considered as an afterthought.  And, by then, it’s too little, too late.

In the book, Simple Architectures for Complex Enterprises (Developer Best Practices), Roger Sessions shares his insights on how simplicity is the ultimate enabler to solving the myriad of problems that complexity creates.

Complex Problems Do Not Require Complex Solutions

Simplicity is the only thing that actually works.

Via Simple Architectures for Complex Enterprises (Developer Best Practices):

“So yes, the problems are complex.  But complex problems do not ipso facto require complex solutions.  Au contraire!  The basic premise of this book is that simple solutions are the only solutions to complex problems that work.  The complex solutions are simply too complex.”

Simplicity is the Antidote to Complexity

It sounds obvious but it’s true.  You can’t solve a problem with the same complexity that got you there in the first place.

Via Simple Architectures for Complex Enterprises (Developer Best Practices):

“The antidote to complexity is simplicity.  Replace complexity with simplicity and the battle is three-quarters over.  Of course, replacing complexity with simplicity is not necessarily simple.” 

Focus on Simplicity as a Core Value

If you want to achieve simplicity, you first have to explicitly focus on it as a core value.

Via Simple Architectures for Complex Enterprises (Developer Best Practices):

“The first thing you need to do to achieve simplicity is focus on simplicity as a core value.  We all discuss the importance of agility, security, performance, and reliability of IT systems as if they are the most important of all requirements.  We need to hold simplicity to as high a standard as we hold these other features.  We need to understand what makes architectures simple with as much critical reasoning as we use to understand what makes architectures secure, fast, or reliable.  In fact, I argue that simplicity is not merely the equal of these other characteristics; it is superior to all of them.  It is, in many ways, the ultimate enabler.”

A Security Example

Complex systems work against security.

Via Simple Architectures for Complex Enterprises (Developer Best Practices):

“Take security for example.  Simple systems that lack security can be made secure.  Complex systems that appear to be secure usually aren't.  And complex systems that aren't secure are virtually impossible to make either simple or secure.”

An Agility Example

Complexity works against agility, and agility is the key to lasting solutions.

Via Simple Architectures for Complex Enterprises (Developer Best Practices):

“Consider agility.  Simple systems, with their well-defined and minimal interactions, can be put together in new ways that were never considered when these systems were first created.  Complex systems can never used in an agile wayThey are simply too complex.  And, of course, retrospectively making them simple is almost impossible.”

Nobody Ever Considers Simplicity as a Critical Feature

And that’s the problem.

Via Simple Architectures for Complex Enterprises (Developer Best Practices):

“Yet, despite the importance of simplicity as a core system requirement, simplicity is almost never considered in architectural planning, development, or reviews.  I recently finished a number of speaking engagements.  I spoke to more than 100 enterprise architects, CIOs, and CTOs spanning many organizations and countries.  In each presentation, I asked if anybody in the audience had ever considered simplicity as a critical architectural feature for any projects on which they had participated. Not one person had. Ever.”

The Quest for Simplicity is Never Over

Simplicity is a quest.  And the quest is never over.  Simplicity is a ongoing pursuit and it’s a dynamic one.  It’s not a one time event, and it’s not static.

Via Simple Architectures for Complex Enterprises (Developer Best Practices):

“The quest for simplicity is never over.  Even systems that are designed from the beginning with simplicity in mind (rare systems, indeed!) will find themselves under a never-ending attack. A quick tweak for performance here, a quick tweak for interoperability there, and before you know it, a system that was beautifully simple two years ago has deteriorated into a mass of incomprehensibility.”

Simplicity is your ultimate sword for hacking your way through complexity … in work … in life … in systems … and ecosystems.

Wield it wisely.

You Might Also Like

10 Ways to Make Information More Useful

Reduce Complexity, Cost, and Time

Simple Enterprise Strategy

Categories: Architecture, Programming

Start exploring I/O 2014 with the new and improved Android mobile app

Google Code Blog - Thu, 06/12/2014 - 17:00

By Mónica Bagagem, Developer Marketing and Bruno Oliveira, Developer Relations

We’re pleased to announce the Google I/O 2014 app is now available on Google Play!

We designed the app to be your co-pilot to navigate the conference and make the best of I/O 2014, whether you’re attending in-person or remotely. This means you’ll be able to explore the full conference agenda, add events to a personalized schedule, find your next session on the map and enjoy facilitated pre-event WiFi configuration for those of you who will be joining us at Moscone.  blog-post-image-1.png blog-post-image-2.png  But the list goes on! Here is a rundown of the main features, with even more to come closer to I/O:

  • Find relevant content faster: Browse the full conference agenda and build your personalized schedule. With new filters for theme, topic and type of event, it's easy to find the content you want. Don't know what you are looking for? No problem! During the conference, the “Explore” screen always shows you what is happening right now and what's coming up, so you can decide where to go next. Plus, your schedule is kept in sync between your Android devices and the I/O website.
  • Relive past I/Os: Watch videos from previous years in the “Video Library”, organized by year and topic. Sit back and catch up on those great talks as you refresh your skills and prepare for I/O 2014!
  • Get reminded when and where it matters: The app will send you reminders when your sessions are about to begin, and once sessions end to provide feedback. You can also enable Google Calendar integration (in the app Settings) so you can see your I/O and personal events all in one place across your devices.
  • Enjoy I/O from anywhere: If you’re attending I/O remotely, you can enjoy live streamed content with closed captions directly from your device.
  • Take advantage of extra screen space: When designing the app, we dedicated a lot of time thinking through how to design for different screen sizes, from phone to tablet. The result is a beautiful and optimized experience across Android devices.
blog-post-image-3.png In addition to being useful to conference attendees, the I/O app serves as a real-life example of best practices for Android app development, and will be open-sourced shortly. Stay tuned!

We hope you are as excited as we are about this year’s conference! So have some fun exploring the things I/O 2014 has to offer, starting today - download the app here.
Mónica Bagagem is part of the Developer Marketing team, working on Google I/O and supporting Designer related efforts. She is a world traveler and a brunch lover.

Bruno Oliveira is a member of the Developer Relations team and is the Tech Lead for the Google I/O 2014 app. He is a geek who enjoys code, photography, sci-fi and old games with big pixels and square waves.
Categories: Programming

Switching My Videos to Once a Month

Making the Complex Simple - John Sonmez - Thu, 06/12/2014 - 15:00

I’ve decided to start cutting back on some of the work I am doing each week. I want to have more time to do other things and I don’t feel that I am able to offer as much value when I am only producing content and not doing other things. This video explains it all.

The post Switching My Videos to Once a Month appeared first on Simple Programmer.

Categories: Programming

Creating options by slicing features - #NoEstimates technique

Software Development Today - Vasco Duarte - Thu, 06/12/2014 - 04:00

Each feature (or story) in a product backlog contains many undiscovered options. By taking features as they are without slicing them into thin slices of functionality we implicitly commit to an implementation strategy. However, when we slice features we create options that allow us to pro-actively manage the scope of a project.

Let’s return to the IT Support Ticketing System project we discussed in a previous post. A feature like the one below will not allow us to manage the scope actively.

  • As an employee I want to be able to submit issues to IT so that I can fix a particular problem that prevents me from working.

The feature above is what I would call a “binary” feature. Either the employee is able to submit an issue to IT or not. This simple feature can have large implications in terms of the amount of work required to implement it. Taking the feature above and breaking it down into several smaller features or stories will allow us to make decisions regarding the implementation order, or delaying certain parts of the implementation. Let’s look at an example:

  • As an employee I want to be able to email an IT issue to the IT department so that I can have a fix for a problem that prevents me from working As an IT helpdesk employee I want to have a queue of issues to handle so that I know what items I should be working on at any given time.

By slicing the original feature in this particular way we unpacked the functionality under the term “submit issues” in the original feature into two different features: Email (replaces submit) and Queue of issues (replaces the receiving end of the submission process). We’ve potentially reduced the scope of the initial feature (no need to have a system to enter IT tickets, just send an email), and we’ve given ourselves the option to implement a solution based on standard tools. The two features we created allow for a solution based on email and a spreadsheet program with shared editing, like Google Docs.

These two stories could still be implemented with a full-fledged IT issue tracking system, but that is an option. Not a mandatory outcome of the initial feature. Slicing features into separate functional parts helps us actively manage the scope by creating different implementation options that are often implicit and non-negotiable when we have larger features in the backlog.

Picture credit: John Hammink, follow him on twitter

There is More Than One Path to Success

Making the Complex Simple - John Sonmez - Mon, 06/09/2014 - 15:00

For a long time I had a difficult time reconciling that both the cowboy coders and the meticulous software craftsmen are able to achieve extraordinary results. I’ve never wanted to believe the deploy-straight-to-production coder was effective. I always sought to find some way to prove that this kind of behavior or methodology always produced an […]

The post There is More Than One Path to Success appeared first on Simple Programmer.

Categories: Programming

Engineer Your Own Luck

“Chance favors the prepared mind.” - Louis Pasteur

Are you feeling lucky?

If you’re an engineer or a developer, you’ll appreciate the idea that you can design for luck, or stack the deck in your favor.

How do you do this?

As Harry Golden said, "The only thing that overcomes hard luck is hard work."

While I believe in hard work, I also believe in working smarter.

Luck is the same game.

It’s a game of skill.

And, success is a numbers game. 

You have to stay in long enough to get “lucky” over time.   That means finding a sustainable approach and using a sustainable system.  It means avoiding going all in without testing your assumptions and reducing the risk out of it.   It means taking emotion out of the equation, taking calculated risks, minimizing your exposure, and focusing on skills.

That’s why Agile methods and Lean approaches can help you outpace your failures.

Because they are test-driven and focus on continuous learning.

And because they focus on capacity and capability versus burnout or blowup.

So if you aren’t feeling the type of luck you’d like to see more of in your life, go back to the basics, and design for it.

They funny thing about luck is that the less you depend on it, the more of it you get.

BTW – Agile Results and Getting Results the Agile Way continue to help people “get lucky.”  Recently, I heard a story where a social worker shared Getting Results the Agile Way with two girls living off the streets.  They are off drugs now, have jobs, and are buying homes.   I’m not doing the story justice, but it’s great to hear about people turning their lives around and these kinds of life changing things that a simple system for meaningful results can help achieve.

It’s not luck. 

It’s desire, determination, and effective strategies applied in a sustainable way.

The Agile way.

Categories: Architecture, Programming

Ruby: Regex – Matching the Trademark ™ character

Mark Needham - Sun, 06/08/2014 - 02:34

I’ve been playing around with some World Cup data and while cleaning up the data I wanted to strip out the year and host country for a world cup.

I started with a string like this which I was reading from a file:

1930 FIFA World Cup Uruguay ™

And I wanted to be able to extract just the ‘Uruguay’ bit without getting the trademark or the space preceding it. I initially tried the following to match all parts of the line and extract my bit:

p text.match(/\d{4} FIFA World Cup (.*?) ™/)[1]

Unfortunately that doesn’t actually compile:

tm.rb:4: syntax error, unexpected $end, expecting ')'
p text.match(/\d{4} FIFA World Cup (.*?) ™/)[1]

I was initially able to work around the problem by matching the unicode code point instead:

p text.match(/\d{4} FIFA World Cup (.*?) \u2122/)[1]

While working on this blog post I also remembered that you can specify the character set of your Ruby file and by default it’s ASCII which would explain why it doesn’t like the ™ character.

If we add the following line at the top of the file then we can happily use the ™ character in our regex:

# encoding: utf-8
# ...
p text.match(/\d{4} FIFA World Cup (.*?) ™/)[1]
# returns "Uruguay"

This post therefore ends up being more of a reminder for future Mark when he comes across this problem again having forgotten about Ruby character sets!

Categories: Programming

Get Up And Code 057: Realistic Expectations

Making the Complex Simple - John Sonmez - Sat, 06/07/2014 - 15:00

How realistic are your expectations for your physique and fitness level? In this episode, I discuss some of the unrealistic expectations we can have after seeing bodybuilders and fitness models who utilize unnatural means to achieve their results. Full transcript below: show John:               Hey, welcome back to Get Up and CODE.  I’m John Sonmez and […]

The post Get Up And Code 057: Realistic Expectations appeared first on Simple Programmer.

Categories: Programming

Extreme Programming (XP) at a Glance (Visual)

“Each of the practices still has the same weaknesses as before, but what if those weaknesses were now made up for by the strengths of other practices? We might be able to get away with doing things simply." – Kent Beck

Extreme Programming (XP) has been around a while, but not everybody knows “what it looks like.”

What does it look like when you step back and take the balcony view and observe the flow of things?

It might look a bit like this …


I put this view together to help some folks get an idea of what the “system” sort of looks like.  It didn’t need to be perfect, but they needed at least a baseline idea or skeleton so they could better understand how the various practices fit together.

The beauty is that once you put a simple picture up on the whiteboard, then you can have real discussions with the team about where things can be improved.  Once again, a picture is worth 1,000 words.

For reference, here are the 12 Practices of Extreme Programming

  1. Coding Standards
  2. Collective Ownership
  3. Continuous Integration
  4. On-Site Customer
  5. Pair Programming
  6. Planning Game
  7. Refactoring
  8. Short Releases
  9. Simple Design
  10. Sustainable Pace (40 Hour Week)
  11. System Metaphor
  12. Test-Driven Development

The main idea here is to get simple visuals in your mind that you can easily draw on a whiteboard, and know the names of the various activities and artifacts. 

If you nail this down, it helps you build a simple vocabulary. 

This vocabulary will help you get others on board faster, and it will help you expand your own toolbox at a rapid rate, and you’ll soon find yourself composing new methods and creating interesting innovations in your process that will help you do things better, faster, and cheaper … on Cloud time.

You Might Also Like

What is Agile?

4 Circles of XP (Extreme Programming)

Extreme Programming at a Glance

Roles on Agile Teams

Scrum at a Glance (Visual)

Waterfall to Agile

Categories: Architecture, Programming

State of Web Development 2014

Xebia Blog - Fri, 06/06/2014 - 13:10

On the 5th of June, we (Lammert Westerhoff and Freek Wielstra) presented the State of Web Development 2014 at our yearly conference, XebiCon on the SS Rotterdam. It was a great success and we had some great feedback. We'd like to share the same presentation we gave there with everyone who couldn't make it to the conference in this blog post.

We started the presentation by showing an overwhelming tag cloud:




When you’re new to web development or when you haven’t done any web development for a long time, you’re likely to get lost in this jungle of web technologies. Don’t worry if you are overwhelmed with this, because that’s intentional. This tag cloud contains many languages and web frameworks that modern web applications consist of. But it also contains a number of development tools that are used during development. The goal of this post is to guide you through this jungle by explaining to you which technologies you can use to build a modern web application and which tools you need during the full development lifecycle.

The get a better understanding of which technologies and tools we should use, we should first look at bit at the evolution of a number of important technologies. Then we will understand why web development is quite complex these days and why we need tooling. Once that's clear, we can have a look at the available tooling.

The evolution of web applications

A traditional website consists of HTML, CSS and sometimes a little javascript. The HTML tells the browser the content that needs to be displayed. The CSS tells it how that content should be displayed and what it needs to look like. And JavaScript was mainly used to add small behaviour, like displaying a popup or showing an image carousel.


Of course, these static websites were hard to maintain. A change of content needed a change in the HTML file which then needed to be uploaded to the web server with FTP.

Luckily, there were a couple of technologies that solved this problem for us. Using a technology like PHP, JSP or ASP we could dynamically generate the websites at runtime, which were then sent to the browser. These websites usually were connected to a database. The database then stored the content instead of storing it directly in the HTML files. That way we could make changes to the website by changing the content stored in the database. This allowed us to already build more complex web applications that also stored user created content, like blogs or even entire web stores.

While we had the server side technologies that we needed to generate our websites, we needed new innovations on the client side as well. Especially because we entered the era of mobile devices, we needed new browser capabilities, like location services, browser storage and touch events. Therefore, new HTML and CSS standards were being developed that contain many features that make the web browsers much more powerful and richer, which allows us developers to build richer web applications for both mobile and desktop.

To make this a lot easier for everyone, Twitter released Bootstrap, which handles all the responsiveness automatically. It’s still being used by many responsive websites today.

With all the new powerful features of CSS3, the style sheets also became more complex and tedious. To overcome that problem, technologies like Sass and Less were developed. These are CSS preprocessors that offer new language features that make some of the more tedious styling easier to do. Both will produce CSS and both solve the same problems. Which one you should choose depends on your circumstances and personal flavour.


Users don’t only want web sites to work and look good on every device, they also want web sites to be more interactive. They want the web site to respond to their actions. For example, zoom into an image or drag an item into a shopping basket. Even though all this was already possible in JavaScript for a long time, jQuery made such things a lot easier. jQuery is a JavaScript library that makes Dom manipulation easier, especially for multiple browsers. With the popularity of jQuery, the web became overwhelmed with jQuery plugins that did all sort of things. Below we see a screenshot of the plugin registry. And of course a library for mobile had to be released.


One of the very important features of jQuery is that it makes it easy to make AJAX requests. AJAX stands for Asynchronous Javascript And XML. It allows the web browser to retrieve data from the backend without leaving the page or doing a page refresh. And only the data is transferred from the backend to the client, instead of entire generated pages. Once the data is retrieved, it is displayed dynamically using the DOM manipulation features of jQuery. And even though XML is in the name, nowadays JSON is often used instead of XML, which is a more lightweight and more flexible data format.

AJAX requests are also at the core of single web page applications (SPA).


These web applications load all the HTML, CSS and Javascript resources at once. Once the page is loaded, you won’t have any other pages loading or refreshing anymore. All the rendering and navigation is done at the client side without leaving the actual page. Of course the user might perceive that he’s navigating between different pages, but this is all handled on the client by Javascript.

The single page web applications are becoming more and more popular since they usually provide a better user experience. They rely heavily on the AJAX requests that we saw earlier to retrieve data from a server. The data that is retrieved by these request are one of the only types of communication between the client and server once the page is initially loaded.

Since with SPA’s, we move a lot of logic and complexity from the server to the client, it really comes in handy to use the framework that handles most of the complexity. Three of the most popular frameworks are Backbone.js, Ember.js and AngularJS. Which one you should choose depends a lot on your requirements, what you’re used to and personal preference. Though it is clear that AngularJS is the most popular one at this moment, has the most activity and the largest community. People seem to be moving from Backbone to Angular and Ember. We also use Angular ourselves and we’re happy with it. The Google Angular team is working on the next major version which should even be a big improvement. That’s why we think Angular is a good and safe choice.

From what we've seen in the previous paragraphs, it’s obvious that more and more functionality and complexity is moved from the server side to the client side. And by using frameworks and libraries such as Bootstrap, jQuery and Angular, we also increase our client side dependencies. This also means that our development focus and efforts are moving from the server to the client. It’s time to take front end development serious and to do it right. So let’s have a look at what this means for our development process.

Web Application Development

As we saw earlier, we start simple by creating a couple of HTML, CSS and Javascript files. Then we would like to take advantage of the power of Less or Sass for our stylesheets. And since we’re making a rich or single page web application, we’ll be making a lot of javascript files. And perhaps some CoffeeScript or Dart instead of just Javascript. For those who don’t know coffee script or Dart, these are alternative languages that both compile to Javascript. CoffeeScript is almost the same language as javascript but just with a slightly different syntax. Dart however is quite different is definitely worth checking out when you have a bit of spare time.

Of course we want to use many of the great third party libraries that we discussed earlier. And since those libraries often depend on other libraries, we’ll end up with a lot of external libraries.


With this picture we kind of get an idea which resources we have for our web application. But to just create these files, download the libraries and upload them to our web server is not enough. First of all, web browsers don’t understand Less, Sass, CoffeeScript or Dart. We need to produce something that the browser does understand. For Less and Sass that’s CSS and Coffee script and Dart need to be compiled to normal Javascript.


Now we’re at a stage that we could deploy these files to our web server, but still that’s not really the way we like to develop our apps. So we’re going to do a couple of extra steps. We will minify almost all of our files. This will grab all those files, do some smart tricks, and we’ll end up with much smaller files. This will safe the end user some download time, which is nice.


We also want to make sure that all of the files we made don’t have any syntax mistakes. That’s why we’ll run a syntax validator on all of those files.

And of course we need to test. So we write a lot of unit tests to cover all our JavaScript functionality. But we also write view tests to make sure that our views display the correct information to the user. With good decoupling we can run our unit tests and view tests in isolation, which is good since we can easily spot problems when one of our test fails and it makes it easy to maintain and refactor our code. But we also want to make sure everything works well together, which is why we also write a couple of end to end tests.


With all these steps in place we’re pretty confident that we have everything we need to make sure that we write quality code. But there is one more thing we need to worry about during development. And that’s dependency management. We want automatic downloads of the third party libraries that we use and their dependencies. And we want to specify their version as well, or specify a version range and get automatic updates within our range.

Now we’re completely happy with all the steps in our development process. But how do we do all this? Or better said, how do we automate all this? We need a build tool.

Let’s quickly recap the things we want our build tool to do. It needs to compile and minify our style sheets and scripts. After that it will take these files and together with the HTML build a distribution package that we can deploy to our production server.

And with every change in any of our files, we want to make sure that everything still works as we would expect. So we validate our files and run our tests. These are our quality assurance steps.

Like we just mentioned, we also want all of our dependencies to be managed automatically.

But there are even more things that we can do with modern build tools that we couldn’t even think of but are great for development. How about live watch and reload of our browser during development. We launch our app in a browser, change some logic or styling, and the web browser will refresh the page automatically to show our changes.

The web development community has been very innovative and came up with many more cool thing we can do with these tools. And you can even write your own build actions.


At this point we have a full understanding of what the web development process looks like. And by seeing this, we can say that front end development has become a first class citizen of software development. Now that we recognise the complexity of front end development and that we need tooling to manage this complexity, it time to have a deeper look at some of the most popular tools of this moment.

Modern tools

Most of the web development tools that are built these days are part of the Node.js ecosystem.


Node.js is a JavaScript engine to run JS outside a browser environment. It has powerful system libraries for basics like file and network access, all with a minimal footprint These characteristics make NodeJS a perfect platform to build server side applications and terminal applications.

Since it’s JavaScript, a language that web developers already know, it’s the perfect environment to build web development tools with.

The Node Package Manager makes it easy to install any registered Node.js tool or application with just a single command. It keeps an online registry of all available packages, and anyone can publish their own. The Node.js and NPM ecosystem have developed into an active and healthy developer community.

One of the most popular tools written in Node.js is Grunt. It is the standard build tool for front-end web projects. It’s a task based build tool, which means you can define a set of tasks that need to run for a production build, development build or a test run. Sounds pretty much what we need for our development process.


Since it’s plugin based, it has plugins for pretty much any task you want to run. That includes plugins for our compilation, minification, validation and testing steps, as well as dependency management. That makes it perfect for us.

Once you start grunt in development mode, it will launch its built-in webserver and host your web application. And with the right plugins, we get our live reload. The Grunt plugin will watch for any changes in our resources, perform necessary tasks and refresh the open browser page for us, so we can see the results of our work right away - drastically shortening the feedback loop compared to traditional web development.

Since Grunt is the standard and most popular build tool for web projects at this moment, it’s widely adopted and has a large and mature developer ecosystem.

But Grunt also has it’s drawbacks. Let’s have a look at Grunt’s configuration file, the Gruntfile.js. This file contains configuration for our tasks and the plugins that perform them.


With a lot of tasks, the Gruntfile really becomes a lot of configuration. The example below taken from our own project and it’s only about half of our configuration. No need to read what it says exactly, it’s just to give you an idea about the amount.


But the amount of configuration is not the only downside of Grunt. The other problem is that it works with temporary files and directories. For each task (depending on the task, of course), it will read a couple of files, transform or convert them into something else and write the output back to disk. The the next task will then use these new files as their input. This creates a chain of temporary files and directories. It’s not a huge problem, especially  but it certainly is not the fastest way of doing things.

That’s why Gulp.js, the streaming build system, was made. It’s a newer, alternative build tool that’s quickly gaining popularity. Gulp can do pretty much the same things as Grunt, so we don’t need to worry too much that we lose essential features of our development process. However, since Gulp is newer than Grunt, it doesn’t have as many plugins as Grunt yet. But since Gulp does the same as Grunt, but then better, it seems that many developers make the switch to Gulp and more and more plugins are made for Gulp as well.


So what’s the difference? Instead of requiring a lot of configuration for it’s tasks, Gulp instead uses “Code over configuration”. That means that you just tell the tool to do something in a certain way instead of configuring both what it should do and how it should do it. Much easier and much more direct. And resulting in far less code than Grunt’s configuration for doing the same.


It also solves Grunt’s problem of temporary files and directories since it’s stream based. Tasks will directly stream their output to the next task. This results in much faster build times.


So it seems that Gulp solves exactly the two problems we had with Grunt. So it’s to be expected that it will gain popularity and eventually pass Grunt. It’s just a matter of time until it reaches the same maturity that Grunt currently has.

We talked earlier about dependency management to manage our third party libraries. While Grunt and Gulp use the Node Package Manager directly to download it’s plugins and their dependencies, we still need to manage the dependencies for our app. To do that, we use Bower, a package manager for the web, by Twitter. Bower works similar as NPM in that you specify all your dependencies in a single configuration file. Bower will then download these libraries from the Internet. Since Grunt and Gulp have plugins for Bower, you don’t even need to worry about this, since it will make sure to download the files when you build your project or to download updates when needed - if your build system is properly configured.


So many tools and frameworks! Isn’t it too much work to setup and configure?
Yes; setting up a full web app development stack, configuring each plugin for your project, is a lot of development work which can take up days. Luckily there is a tool to solve that problem: Yeoman.

Yeoman is a project scaffolding tool that generates a basis for your project, using templates called Generators. Some of those generators will set up a web application project with a preconfigured build strait, containing everything we need for compilation, minification, validation, testing, dependency management and much more. It will also ask us if we want to use frameworks such as Bootstrap and AngularJS. This makes it an ideal solution to setup a new project and automated development process.

More recent generators will set up Gulp as the build tool, providing an alternative to Bower. This should help developers that aren’t familiar with Gulp yet to get up to speed quickly.


Let’s recap what we talked about in this post in a summary.



The full presentation slides will soon be available from the XebiCon website. Feel free to comment below with your feedback or reach out to us on Twitter.

Lammert Westerhoff @lwesterhoff

Freek Wielstra @frwielstra

What is Agile?

“Becoming limitless involves mental agility; the ability to quickly grasp and incorporate new ideas and concepts with confidence.” -- Lorii Myers

I was asked to give an Intro to Agile talk to a group in Microsoft, in addition to a talk on Getting Results the Agile Way.

It worked out well.

The idea was to do a level set and get everybody on the same page in terms of what Agile is.

I thought it was a great chance to build a shared vocabulary and establish some shared mental models.   I believe that when people have a shared vocabulary and common mental models, they can take a look from the balcony.  And, it makes it a lot easier to move up the chain and take things further, faster.

Anyway, here is how I summarized what Agile is:

  • Agile is a framework of values and principles to manage teams and projects.
  • Agile is an alternative approach to traditional project management. It embraces change while traditional project management fights change.
  • People across functional teams work together as one team, rather than different groups working in phases or stages.
  • More human communication, interaction, and face-to-face.
  • Continuous feedback from users and stakeholders.
  • Iterations, shorter development cycles, and more frequent releases.
  • Visibility of progress and transparency of process.

That said, I need to find something a bit more pithy and precise, yet insightful.

If I had to put it in a simple sentence, I’d say Agile is empowerment through flexibility.

One thing I’ve noticed over the years is that some people struggle when they try to go Agile.

They struggle because they can’t seem to “flip a switch.”  And if they don’t flip the switch, they don’t change their mindset.

And, if they don’t change their mindset, Agile remains just beyond their grasp.

Agile is like happiness, grow it right under your feet.

You Might Also Like

40 Hour Work Week at Microsoft

Why Agile

Waterfall to Agile

Categories: Architecture, Programming

Killing The Cynic

Making the Complex Simple - John Sonmez - Thu, 06/05/2014 - 16:00

It is easy to become a cynic. Most of us fall into the cynical cycle without even realizing it. When you go down that road it leads you right to pessimism and negativity. In this video I talk about why cynicism is so bad and how to break the cycle. Full transcript: John:               Hey, John […]

The post Killing The Cynic appeared first on Simple Programmer.

Categories: Programming

SQLitePCL.raw NuGet package, 0.2.0-alpha

Eric.Weblog() - Eric Sink - Wed, 06/04/2014 - 19:00

I have pushed up the second release of the NuGet package(s) for SQLitePCL.raw. There are no API changes, and all tests are passing on all my target platforms (Xamarin.iOS, Xamarin.Android, .NET 4.5, WinRT 8, WinRT 8.1, Windows Phone 8, Windows Phone 8.1 RT, and Windows Phone 8.1 Silverlight).

Changes in this release
  • Tweak the net45 platform assembly to [hopefully] make it compatible with Mono on non-Windows platforms

  • Better error messages for build configuration problems

  • Bundled builds of the sqlite3 library are now compiled with foreign keys defaulting to on

  • SQLitePCL.tests, a separate NuGet package containing my test suite in source code form

  • SQLitePCL.ugly, a separate NuGet package containing the "not-in-C#-style" API which I use for writing the test suite

Changes still coming [soon]
  • A release which is polished enough for real use

  • Release notes in the nuspec instead of here in a blog entry

  • Hopefully, automatic insertion of the Visual C++ Runtime SDK reference on RT-ish platforms

  • Support for sqlite-net in a NuGet package. I've been coordinating with @praeclarum and @timheuer on this. He/they accepted my pull request with the necessary changes (thanks!). And my build system is generating a csproj/nuspec for proof of concept purposes. Almost there.

  • SQLitePCL.kitchensink, a NuGet package for advanced users, containing all of the configurations

  • Documentation

  • Better support for SQLCipher


An Android Wear Design Story

Android Developers Blog - Tue, 06/03/2014 - 20:41
By Roman Nurik and Timothy Jordan, Design and Developer Advocates on Android Wear

A few weeks ago, Timothy and I were chatting about designing apps for wearables to validate some of the content we’re planning for Google I/O 20141. We talked a lot about how these devices require scrutiny to preserve user attention while exposing some unique new surface areas for developers. We also discussed user context and how the apps we make should be opportunistic, presenting themselves in contexts where they’re useful; it’s more important than ever to think of apps on wearable devices not as icons on a grid but rather as functional overlays on the operating system itself.

But while I’d designed a number of touch UIs for Android in the past and Timothy had a ton of experience with Glass, neither of us had really gone through the exercise of actually designing an app for Android Wear. So we set out to put our ideas in practice and see what designing for this new platform is like.

Before we got started, we needed an idea. Last year, I participated in an informal Glass design sprint in NYC run by Nadya Direkova, and my sprint team came up with a walking tour app. The idea was you’d choose from a set of nearby tours, walk between the stops, and at each stop on the tour, learn about the destination.

My rough mocks of a walking tour app from a Glass design sprint.

While the design sprint ended at rough mocks, the idea stuck around in my mind, and came up again during this exercise. It seemed like a perfect example of a contextually aware app that could enhance your Android Wear experience.

Designing a walking tour app for Android Wear

We started fleshing out the idea by thinking through the app’s entry points: how will users “launch” this app? While exposing a “start XYZ walking tours app” voice command is pretty standard, it’d be interesting to also suggest nearby walking tours as you go about your day by presenting notifications in the user’s context stream. These notifications would be “low priority,” so you’d only see them after addressing the more important stuff like text messages from friends. And with today’s geofencing and location functionality in Google Play services, this type of contextual awareness is possible in a battery-friendly way.

At this point we were pretty excited and decided to begin mocking up the UI. Rather than starting from scratch, we used Taylor Ling’s excellent Android Wear 0.1 design template as a baseline, which includes templates for both square and round devices. We started with square since we were most familiar with rectangle UI design:

Idea: You get a notification in the context stream when a walking tour is available nearby.

I’ve got to admit, it was pretty thrilling designing in such a constrained environment. 140x140 dp (280x280 px @ XHDPI) isn’t a lot of space to work with, so you need to make some tough choices about when and how to present information. But these are exactly the types of problems that make design really, really fun. You end up spending more time thinking and less time actually pushing pixels around in Photoshop or Sketch.

We pretty quickly fleshed out the rest of the app for square devices. They included just a handful of additional screens: a dynamic notification showing the distance to your next stop, and a 4-page detail screen when you arrive at the tour stop, where you can spend a few moments reading about where you’re standing.

A notification guiding you to your next stop, and a multi-page stop detail screen for learning about the stop when you get there. Seeing our design in real life

Here’s the thing—there’s only so much you can do in Photoshop. To truly understand a platform as a designer, you really need to use (and ideally live with) a real device, and see your work on that device. Only then can you fully evaluate the complexity of your flows, the size of your touch targets, or the legibility of your text.

Luckily, Timothy and I both had test devices—I sported an LG G Watch prototype and Timothy carried a Moto 360 prototype. We then needed a way to quickly send screens to our devices so we could iterate on the design. A few years ago I’d published the Android Design Preview tool that lets you mirror a part of your screen to a connected Android device. Much to our delight, the tool worked great with Android Wear! After seeing our mocks show up on my LG G Watch, we made a few small tweaks and felt much more confident that the overall idea “felt right” on the wrist.

Android Design Preview mirrors a part of your computer screen to an Android device. It’s especially awesome seeing your UI running on an LG G Watch prototype. Designing for round devices

We’d never designed round UIs before, so we weren’t sure what this new adventure would be like. Quite frankly, it ended up being unbelievably easy: tweaking all 8 of our screen mocks for round took under an hour. When you’re only showing the most important 2 or 3 pieces of information on screen at a time, that’s only 2 or 3 pieces of information you need to optimize for round devices. All in all, there were only a few types of minor tweaks we made:

  • Scaled up backgrounds to 160x160 dp (320x320 px @ XHDPI)
  • Bumped up content margins from 12dp on square to 26dp on round; this means content was 116x116 dp on square and only a little smaller at 108x108 dp on round
  • Pushed down circular actions like “Continue tour” to better vertically center with the watch frame
  • Center-aligned certain short snippets of text on round devices as opposed to left-aligning on square
  • Dropped the side padding for context stream cards (the platform automatically does this for notifications, so there isn’t any actual work to do here)
These weren’t completely different layouts—rather, the same layout with slightly tweaked metrics.

It’s hard to articulate the excitement we felt when we mirrored the mocks to Timothy’s Moto 360 prototype with Android Design Preview. To put it lightly, our minds were blown.

There’s something special and awe-inspiring about seeing one of your UIs running on a round screen..

And that was it—with round and square mocks complete, and mirrored on our devices, we’d gotten our first glimpse at designing apps for this exciting new platform. Below are our completed mocks for the tour discovery and engagement flows, not a grid of app icons in sight. You can download the full PSDs here.

An eye-opening experience

Designing for Android Wear is pretty different from designing for the desktop, phones or tablets. Just like with Glass, you really need to think carefully about the information and actions you present to the user, and even more so about the contexts in which your app will come to the surface.

As a designer, that’s the fun part—working with constraints involving scarce resources like device size and user attention means it’s more important than ever to think deeply about your ideas and iterate on them early and often. The actual pixel-pushing part of the process is far, far easier.

So there we were, putting our ideas into practice, on real actual device prototypes that we could’ve only dreamed about only a few years ago. It was the most fun I’ve had designing UIs in a long time. Remember that feeling when you first dreamed up an app, mocked or even coded it up, and ran it on your Android phone? It was that same feeling all over again, but amplified, because you were actually wearing your app. I can’t wait for you all to experience it!

1 Have we mentioned #io14 will have tons of great content around both design and wearable computing? Make sure to tune in June 25th and 26th!

Join the discussion on

+Android Developers
Categories: Programming

Why Agile?

I thought I had written about “Why Agile” before, but I don’t see anything crisp enough.

Anyway, here’s my latest rundown on Why Agile?

  1. Increase customer involvement which can build empathy and avoid rework
  2. Learn faster which means you can adapt to change
  3. Improve quality through focus
  4. Reduce risk through shorter feedback loops and customer interaction
  5. Simplify by getting rid of overhead and waste
  6. Reduce cycle time through timeboxing and parallel development
  7. Improve operational awareness through transparency
  8. Drive process improvement through continuous improvement
  9. Empower people through less mechanics and more interaction, continuous learning, and adaptation
  10. Flow more value through more frequent releases and less “big bang”

Remember that nature favors the flexible and agility is the key to success.

You Might Also Like

Agile vs. Waterfall

Agile Life-Cycle Frame

Methodologies at a Glance

Roles on Agile Teams

The Art of the Agile Retrospective

Categories: Architecture, Programming

New Demographic Stats in Google Play Games Services

Android Developers Blog - Mon, 06/02/2014 - 18:17

By Ben Frenkel, Google Play Games team

Hey game developers, back in March you may remember we added new game statistics in the Google Play Developer Console for those of you who had implemented Google Play Games: our cross-platform game services for Android, iOS and the web.

Starting today, we're providing more insights into how your games are being used by adding country, age, and gender dimensions to the existing set of reports available in the Developer console. You’ll see demographics integrated into Overview stats as well as the Players reports for New and Active users.

In the Overview stats you can now see highlights of activity by age group, most active countries, and gender.

With a better understanding of your users’ demographic composition, you'll be able to make more effective decisions to improve retention and monetization. Here a few ways you could imagine using these new stats:

  • You just launched your new game globally, and expected it do particularly well in Germany. Using country demographic data, you see that Germany is much less active than expected. After some digging, you realize that your tutorial was not properly translated to German. Based on this insight, you immediately roll out a fix to see if you can improve active users in Germany.

In the Players stats section the new metrics reveal trends in how your app is doing across age groups, countries, and gender.

  • After Looking at your new demographics report you realize that your game is really popular with women in their mid-20s. Your in-app purchase data corroborates this, showing that the one female hero character is the most popular purchase. Empowered by this data, you race to add female hero characters to your game.

Additionally, if you're already using Google Play game services, there's no extra integration needed! By logging in to the Google Play Developer Console you can start using demographics to better inform your decisions today.

Join the discussion on

+Android Developers
Categories: Programming