Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Xebia Blog
Syndicate content
Updated: 1 hour 56 min ago

A product manager's perfection....

Tue, 03/03/2015 - 15:59

is achieved not there are no more features to add, but when there are no more features to take away. -- Antoine de Saint Exupéry

Not only was Antoine a brilliant writer, philosopher and pilot (well arguably since he crashed in the Mediterranean) but most of all he had a sharp mind about engineering, and I frequent quote him when I train product owners, product managers or in general product companies, about what makes a good product great. I also tell them their most important word in their vocabulary is "no". But the question then becomes, what is the criteria to say "yes"?

Typically we will look at the value of a feature and use different ways to prioritise and rank different features, break them down to their minimal proposition and get the team going. But what if you already have a product? and it’s rate of development is slowing. Features have been stacked on each other for years or even decades, and it’s become more and more difficult for the teams to wade through the proverbial swamp the code has become?

Too many features

Too many features

Turns out there are a number of criteria that you can follow:

1.) Working software, means it’s actually being used.

Though it may sound obvious, it’s not that easy to figure out. I was once part of a team that had to rebuild a rather large piece (read huge) of software for an air traffic control system. The managers ensured us that every functionality was a must keep, but the cost would have been prohibitory high.

One of the functions of the system was a record and replay mode for legal purposes. It basically registers all events throughout the system to serve as evidence that picture compilers would be accountable, or at least verifiable. One of our engineers had the bright insight that we could catalogue this data anonymously to figure out which functions were used and which were not.

Turned out the Standish Group was pretty right in their claims that 80% of the software is never used. Carving that out was met with fierce resistance, but it was easier to convince management (and users) with data, than with gut.

Another upside? we also knew what functions they were using a lot, and figured out how to improve those substantially.

2.) The cost of platforms

Yippee we got it running on a gazillion platforms! and boy do we have a reach, the marketing guys are going frenzy. Even if is the right choice at the time, you need to revisit this assumption all the time, and be prepared to clean up! This is often looked upon as a disinvestment: “we spent so much money on finally getting Blackberry working” or “It’s so cost effective that we can also offer it on platform XYZ”.

In the web world it’s often the number of browsers we support, but for larger software systems it is more often operating systems, database versions or even hardware. For one customer we would refurbish hardware systems, simply because it was cheaper than moving on to a more modern machine.

Key take away: If the platform is deprecated, remove it entirely from the codebase, it will bog the team down and you need their speed to respond to an ever increasing pace of new platforms.

3.) Old strategies

Every market and every product company pivots at least every few years (or dies). Focus shifts from consumer groups, to type of clients, type of delivery, shift to service or something else which is novel, hip and most of all profitable. Code bases tend to have a certain inertia. The larger the product, the bigger the inertia, and before you know it there a are tons of features in their that are far less valuable in the new situation. Cutting away perfectly good features is always painful but at some point you end up with the toolbars of Microsoft Word. Nice features, but complete overkill for the average user.

4.) The cause and effect trap

When people are faced with an issue they tend to focus on fixing the issue as it manifests itself. It's hard for our brain to think in problems, it tries to think in solutions. There is an excellent blog post here that provides a powerful method to overcome this phenomena by asking five times "why".

  • "We need the system to automatically export account details at the end of the day."
  • "Why?"
  • "So we can enter the records into the finance system"
  • "So it sounds like the real problem is getting the data into the finance system, not exporting it. Exporting just complicates the issue. Let's implement a data feed that automatically feeds the data to the finance system"

The hard job is to continuously keep evaluating your features, and remove those who are no longer valuable. It may seem like your throwing away good code, but ultimately it is not the largest product that survives, but the one that is able to adapt fast enough to the changing market. (Freely after Darwin)

 

Tutum, first impressions

Mon, 03/02/2015 - 16:40

Tutum is a platform to build, run and manage your docker containers. After shortly playing with it some time ago, I decided to take a bit more serious look at it this time. This article describes first impressions of using this platform, more specifically looking at it from a continuous delivery perspective.

The web interface

First thing to notice is the clean and simple web interface. Basically there are two main sections, which are services and nodes. The services view lists the services or containers you have deployed with status information and two buttons, one to stop (or start) and one to terminate the container, which means to throw it away.

You can drill down to a specific service, which provides you with more detailed information per service. The detail page provides you information about the containers, a slider to scale up and down, endpoints, logging, some metrics for monitoring and more .

Screen Shot 2015-02-23 at 22.49.33

The second view is a list of nodes. The list contains the VM's on which containers can be deployed. Again with two simple buttons to start/stop and to terminate the node. For each node it displays useful information about the current status, where it runs, and how many containers are deployed on it.

The node page also allows you to drill down to get more information on a specific node.  The screenshot below shows some metrics in fancy graphs for a node, which can potentially be used to impress your boss.

Screen Shot 2015-02-23 at 23.07.30

 

Creating a new node

You’ll need a node to deploy containers on it. In the node view you see two big green buttons. One states: “Launch new node cluster”. This will bring up a form with currently four popular providers Amazon, Digital Ocean, Microsoft Azure and Softlayer. If you have linked your account(s) in the settings you can select that provider from a dropdown box. It only takes a few clicks to get a node up and running. In fact you create a node cluster, which allows you to easily scale up or down by adding or removing nodes from the cluster.

You also have an option to ‘Bring you own node’. This allows you to add your own Ubuntu Linux systems as nodes to Tutum. You need to install an agent onto your system and open up a firewall port to make your node available to Tutum. Again very easy and straight forward.

Creating a new service

Once you have created a node, you maybe want to do something with it. Tumtum provides jumpstart images with popular types of services for storage, cacheing, queueing and more, providing for example MongoDB, Elasticsearch or Tomcat. Using a wizard it takes only four steps to get a particular service up and running.

Besides the jumpstart images that Tutum provides, you can also search public repositories for your image of choice. Eventually you would like to have your own images running your homegrown software. You can upload your image to a Tutum private registry. You can either pull it from Docker Hub or upload your local images directly to Tutum.

Automating

We all know real (wo)men (and automated processes) don’t use GUI’s. Tutum provides a nice and extensive command line interface for both Linux and Mac. I installed it using brew on my MBP and seconds later I was logged in and doing all kinds of cool stuff with the command line.

Screen Shot 2015-02-24 at 22.23.30

The cli is actually doing rest calls, so you can skip the cli all together and talk HTTP directly to a REST API, or if it pleases you, you can use the python API to create scripts that are actually maintainable. You can pretty much automate all management of your nodes, containers, and services using the API, which is a must have in this era of continuous everything.

A simple deployment example

So let's say we've build a new version of our software on our build server. Now we want to get this software deployed to do some integration testing, or if you feeling lucky just drop it straight into production.

build the docker image

tutum build -t test/myimage .

upload the image to Tutum registry

tutum image push <image_id>

create the service

tutum service create <image_id>

run it on a node

tutum service run -p <port> -n <name> <image_id>

That's it. Of course there are lots of options to play with, for example deployment strategy, set memory, auto starting etc. But the above steps are enough to get your image build, deployed and run. Most time I had to spend was waiting while uploading my image using the flaky-but-expensive hotel wifi.

Conclusion for now

Tutum is clean, simple and just works. I’m impressed with ease and speed you can get your containers up and running. It takes only minutes to get from zero to running using the jumpstart services, or even your own containers. Although they still call it beta, everything I did just worked, and without the need to read through lots of complex documentation. The web interface is self explanatory and the REST API or cli provides everything you need to integrate Tutum in your build pipeline, so you can get your new features in production with automation speed.

I'm wondering how challenging managing would be at a scale of hundreds of nodes and even more containers, when using the web interface. You'd need a meta-overview or aggregate view or something. But then again, you have a very nice API to

Exploring container platforms: StackEngine

Fri, 02/20/2015 - 15:51

Docker has been around for more than a year already, and there are a lot of container platforms popping up. In this series of blogposts I will explore these platforms and share some insights. This blogpost is about StackEngine.

TL;DR: StackEngine is (for now) just a nice frontend to the Docker binary. Nothing...

Try, Option or Either?

Wed, 02/18/2015 - 09:45

Scala has a lot of different options for handling and reporting errors, which can make it hard to decide which one is best suited for your situation. In Scala and functional programming languages it is common to make the errors that can occur explicit in the functions signature (i.e. return type), in contrast with the common practice in other programming languages where either special values are used (-1 for a failed lookup anyone?) or an exception is thrown.

Let's go through the main options you have as a Scala developer and see when to use what!

Option
A special type of error that can occur is the absence of some value. For example when looking up a value in a database or a List you can use the find method. When implementing this in Java the common solution (at least until Java 7) would be to return null when a value cannot be found or to throw some version of the NotFound exception. In Scala you will typically use the Option[T] type, returning Some(value) when the value is found and None when the value is absent.

So instead of having to look at the Javadoc or Scaladoc you only need to look at the type of the function to know how a missing value is represented. Moreover you don't need to litter your code with null checks or try/catch blocks.

Another use case is in parsing input data: user input, JSON, XML etc.. Instead of throwing an exception for invalid input you simply return a None to indicate parsing failed. The disadvantage of using Option for this situation is that you hide the type of error from the user of your function which, depending on the use-case, may or may not be a problem. If that information is important keep on reading the next sections.

An example that ensures that a name is non-empty:

def validateName(name: String): Option[String] = {
  if (name.isEmpty) None
  else Some(name)
}

You can use the validateName method in several ways in your code:

// Use a default value

 validateName(inputName).getOrElse("Default name")

// Apply some other function to the result
 validateName(inputName).map(_.toUpperCase)

// Combine with other validations, short-circuiting on the first error
// returning a new Option[Person]
 for {
   name <- validateName(inputName)
   age <- validateAge(inputAge)
 } yield Person(name, age)

Either
Option is nice to indicate failure, but if you need to provide some more information about the failure Option is not powerful enough. In that case Either[L,R] can be used. It has 2 implementations, Left and Right. Both can wrap a custom type, respectively type L and type R. By convention Right is right, so it contains the successful result and Left contains the error. Rewriting the validateName method to return an error message would give:

def validateName(name: String): Either[String, String] = {
 if (name.isEmpty) Left("Name cannot be empty")
 else Right(name)
 }

Similar to Option Either can be used in several ways. It differs from option because you always have to specify the so-called projection you want to work with via the left or right method:

// Apply some function to the successful result
validateName(inputName).right.map(_.toUpperCase)

// Combine with other validations, short-circuiting on the first error
// returning a new Either[Person]
for {
 name <- validateName(inputName).right
 age <- validateAge(inputAge).right
} yield Person(name, age)

// Handle both the Left and Right case
validateName(inputName).fold {
  error => s"Validation failed: $error",
  result => s"Validation succeeded: $result"
}

// And of course pattern matching also works
validateName(inputName) match {
  case Left(error) => s"Validation failed: $error",
  case Right(result) => s"Validation succeeded: $result"
}

// Convert to an option:
validateName(inputName).right.toOption

This projection is kind of clumsy and can lead to several convoluted compiler error messages in for expressions. See for example the excellent and in detail discussion of the Either type in the The Neophyte's Guide to Scala Part 7. Due to these issues several alternative implementations for a kind of Either have been created, most well known are the \/  type in Scalaz and the Or type in Scalactic. Both avoid the projection issues of the Scala Either and, at the same time, add additional functionality for aggregating multiple validation errors into a single result type.

Try

Try[T] is similar to Either. It also has 2 cases, Success[T] for the successful case and Failure[Throwable] for the failure case. The main difference thus is that the failure can only be of type Throwable. You can use it instead of a try/catch block to postpone exception handling. Another way to look at it is to consider it as Scala's version of checked exceptions. Success[T] wraps the result value of type T, while the Failure case can only contain an exception.

Compare these 2 methods that parse an integer:

// Throws a NumberFormatException when the integer cannot be parsed
def parseIntException(value: String): Int = value.toInt

// Catches the NumberFormatException and returns a Failure containing that exception
// OR returns a Success with the parsed integer value
def parseInt(value: String): Try[Int] = Try(value.toInt)

The first function needs documentation describing that an exception can be thrown. The second function describes in its signature what can be expected and requires the user of the function to take the failure case into account. Try is typically used when exceptions need to be propagated, if the exception is not needed prefer any of the other options discussed.

Try offers similar combinators as Option[T] and Either[L,R]:

// Apply some function to the successful result
parseInt(input).map(_ * 2)

// Combine with other validations, short-circuiting on the first Failure
// returning a new Try[Stats]
for {
  age <- parseInt(inputAge)
  height <- parseDouble(inputHeight)
} yield Stats(age, height)

// Use a default value
parseAge(inputAge).getOrElse(0)

// Convert to an option
parseAge(inputAge).toOption

// And of course pattern matching also works
parseAge(inputAge) match {
  case Failure(exception) => s"Validation failed: ${exception.message}",
  case Success(result) => s"Validation succeeded: $result"
}

Note that Try is not needed when working with Futures! Futures combine asynchronous processing with the Exception handling capabilities of Try! See also Try is free in the Future.

Exceptions
Since Scala runs on the JVM all low-level error handling is still based on exceptions. In Scala you rarely see usage of exceptions and they are typically only used as a last resort. More common is to convert them to any of the types mentioned above. Also note that, contrary to Java, all exceptions in Scala are unchecked. Throwing an exception will break your functional composition and probably result in unexpected behaviour for the caller of your function. So it should be reserved as a method of last resort, for when the other options don’t make sense.
If you are on the receiving end of the exceptions you need to catch them. In Scala syntax:

try {
  dangerousCode()
} catch {
  case e: Exception => println("Oops")
} finally {
  cleanup
}

What is often done wrong in Scala is that all Throwables are caught, including the Java system errors. You should never catch Errors because they indicate a critical system error like the OutOfMemoryError. So never do this:

try {
  dangerousCode()
} catch {
  case _ => println("Oops. Also caught OutOfMemoryError here!")
}

But instead do this:

import scala.util.control.NonFatal

try {
  dangerousCode()
} catch {
  case NonFatal(_) => println("Ooops. Much better, only the non fatal exceptions end up here.")
}

To convert exceptions to Option or Either types you can use the methods provided in scala.util.control.Exception (scaladoc):

import scala.util.control.Exception._

val i = 0
val result: Option[Int] = catching(classOf[ArithmeticException]) opt { 1 / i }
val result: Either[Throwable, Int] = catching(classOf[ArithmeticException]) either { 1 / i }

Finally remember you can always convert an exception into a Try as discussed in the previous section.

TDLR;

  • Option[T], use it when a value can be absent or some validation can fail and you don't care about the exact cause. Typically in data retrieval and validation logic.
  • Either[L,R], similar use case as Option but when you do need to provide some information about the error.
  • Try[T], use when something Exceptional can happen that you cannot handle in the function. This, in general, excludes validation logic and data retrieval failures but can be used to report unexpected failures.
  • Exceptions, use only as a last resort. When catching exceptions use the facility methods Scala provides and never catch { _ => }, instead use catch { NonFatal(_) => }

One final advice is to read through the Scaladoc for all the types discussed here. There are plenty of useful combinators available that are worth using.

Cancelling $http requests for fun and profit

Tue, 02/17/2015 - 09:11

At my current client, we have a large AngularJS application that is configured to show a full-page error whenever one of the $http requests ends up in error. This is implemented with an error interceptor as you would expect it to be. However, we’re also using some calculation-intense resources that happen to timeout once in a while. This combination is tricky: a user triggers a resource request when navigating to a certain page, navigates to a second page and suddenly ends up with an error message, as the request from the first page triggered a timeout error. This is a particular unpleasant side effect that I’m going to address in a generic way in this post.

There are of course multiple solutions to this problem. We could create a more resilient implementation in the backend that will not time out, but accepts retries. We could change the full-page error in something less ‘in your face’ (but you still would get some out-of-place error notification). For this post I’m going to fix it using a different approach: cancel any running requests when a user switches to a different location (the route part of the URL). This makes sense; your browser does the same when navigating from one page to another, so why not mimic this behaviour in your Angular app?

I’ve created a pretty verbose implementation to explain how to do this. At the end of this post, you’ll find a link to the code as a packaged bower component that can be dropped in any Angular 1.2+ app.

To cancel a running request, Angular does not offer that many options. Under the hood, there are some places where you can hook into, but that won’t be necessary. If we look at the $http usage documentation, the timeout property is mentioned and it accepts a promise to abort the underlying call. Perfect! If we set a promise on all created requests, and abort these at once when the user navigates to another page, we’re (probably) all set.

Let’s write an interceptor to plug in the promise in each request:

angular.module('angularCancelOnNavigateModule')
  .factory('HttpRequestTimeoutInterceptor', function ($q, HttpPendingRequestsService) {
    return {
      request: function (config) {
        config = config || {};
        if (config.timeout === undefined && !config.noCancelOnRouteChange) {
          config.timeout = HttpPendingRequestsService.newTimeout();
        }
        return config;
      }
    };
  });

The interceptor will not overwrite the timeout property when it is explicitly set. Also, if the noCancelOnRouteChange option is set to true, the request won’t be cancelled. For better separation of concerns, I’ve created a new service (the HttpPendingRequestsService) that hands out new timeout promises and stores references to them.

Let’s have a look at that pending requests service:

angular.module('angularCancelOnNavigateModule')
  .service('HttpPendingRequestsService', function ($q) {
    var cancelPromises = [];

    function newTimeout() {
      var cancelPromise = $q.defer();
      cancelPromises.push(cancelPromise);
      return cancelPromise.promise;
    }

    function cancelAll() {
      angular.forEach(cancelPromises, function (cancelPromise) {
        cancelPromise.promise.isGloballyCancelled = true;
        cancelPromise.resolve();
      });
      cancelPromises.length = 0;
    }

    return {
      newTimeout: newTimeout,
      cancelAll: cancelAll
    };
  });

So, this service creates new timeout promises that are stored in an array. When the cancelAll function is called, all timeout promises are resolved (thus aborting all requests that were configured with the promise) and the array is cleared. By setting the isGloballyCancelled property on the promise object, a response promise method can check whether it was cancelled or another exception has occurred. I’ll come back to that one in a minute.

Now we hook up the interceptor and call the cancelAll function at a sensible moment. There are several events triggered on the root scope that are good hook candidates. Eventually I settled for $locationChangeSuccess. It is only fired when the location change is a success (hence the name) and not cancelled by any other event listener.

angular
  .module('angularCancelOnNavigateModule', [])
  .config(function($httpProvider) {
    $httpProvider.interceptors.push('HttpRequestTimeoutInterceptor');
  })
  .run(function ($rootScope, HttpPendingRequestsService) {
    $rootScope.$on('$locationChangeSuccess', function (event, newUrl, oldUrl) {
      if (newUrl !== oldUrl) {
        HttpPendingRequestsService.cancelAll();
      }
    })
  });

When writing tests for this setup, I found that the $locationChangeSuccess event is triggered at the start of each test, even though the location did not change yet. To circumvent this situation, the function does a simple difference check.

Another problem popped up during testing. When the request is cancelled, Angular creates an empty error response, which in our case still triggers the full-page error. We need to catch and handle those error responses. We can simply add a responseError function in our existing interceptor. And remember the special isGloballyCancelled property we set on the promise? That’s the way to distinguish between cancelled and other responses.

We add the following function to the interceptor:

      responseError: function (response) {
        if (response.config.timeout.isGloballyCancelled) {
          return $q.defer().promise;
        }
        return $q.reject(response);
      }

The responseError function must return a promise that normally re-throws the response as rejected. However, that’s not what we want: neither a success nor a failure callback should be called. We simply return a never-resolving promise for all cancelled requests to get the behaviour we want.

That’s all there is to it! To make it easy to reuse this functionality in your Angular application, I’ve packaged this module as a bower component that is fully tested. You can check the module out on this GitHub repo.

When development resembles the ageing of wine

Mon, 02/16/2015 - 20:29

Once upon a time I was asked to help out a software product company.  The management briefing went something like this: "We need you to increase productivity, the guys in development seem to be unable to ship anything! and if they do ship something it's only a fraction of what we expected".

And so the story begins. Now there are many ways how we can improve the teams outcome and its output (the first matters more), but it always starts with observing what they do today and trying to figure out why.

It turns out that requests from the business were treated like a good wine, and were allowed to "age", in the oak barrel that was called Jira. Not so much to add flavour in the form of details, requirements, designs, non functional requirements or acceptance criteria, but mainly to see if the priority of this request would remain stable over a period of time.

In the days that followed I participated in the "Change Control Board" and saw what he meant. Management would change priorities on the fly and make swift decisions on requirements that would take weeks to implement. To stay in vinotology terms, wine was poured in and out the barrels at such a rate that it bore more resemblance to a blender than to the art of wine making.

Though management was happy to learn I had unearthed to root cause to their problem, they were less pleased to learn that they themselves were responsible.  The Agile world created the Product Owner role for this, and it turned out that this is hat, that can only be worn by a single person.

Once we funnelled all the requests through a single person, both responsible for the success of the product and for the development, we saw a big change. Not only did the business got a reliable sparring partner, but the development team had a single voice when it came to setting the priorities. Once the team starting finishing what they started we started shipping at regular intervals, with features that we all had committed to.

Of course it did not take away the dynamics of the business, but it allowed us to deliver, and become reliable in how and when we responded to change. Perhaps not the most aged wine, but enough to delight our customers and learn what we should put in our barrel for the next round.

 

UC Berkeley paper unveils what business leaders should learn from the Agile WikiSpeed case.

Wed, 02/04/2015 - 23:26

WIKISPEED_my_next_car_gets_100mpg_1024Last fall, I was approached by Tom van Norden from the UC Berkeley School of Information. A team of professor Morten T. Hansen, famous from his bestseller with Jim Collins “Great by Choice”, was investigating the magic around Joe Justice’ WikiSpeed. His agile team outperformed companies like Tesla during the XPrize a couple of years ago. Berkeley did research on the Agile and Lean practices being applied by the WikiSpeed team and it’s current status.

Joe and I work closely together how we can apply powerful IT-practices like Scrum and TDD (Test Driven Development) in a non-IT environment.  In parallel with WikiSpeed, together with Xebia coaches Jeroen Molenaar and Jasper Sonnevelt we're currently coaching the hydrogen and solar raceteams of Delft University, Forze and Nuna. In our coaching we use many of WikiSpeeds XM-practices, especially Scrum. The Berkeley paper confirmed and unveiled a number of lessons I would like to share in this blogpost.

Unification

An inspiring recognizable goal gives the team extreme focus, spirit and energy. It actually provides them with an identity.  When the goal fades or is not unique and inspiring anymore, the focus, spirit and energy goes down significantly.  The Forze team was also struggling with this.  The 2014-team handed over the P6 Hydrogen Race Car to the 2015- team in september 2014.  This car was still buggy and last but not least, with a goal which was not realistic and committed by the 2015-team.  Not endless bug fixing on the previous car to complete a goal from last years team.  Their new goal is building a new car, competing at a spectacular race in 2016 with conventional race cars. Therefor, we recently advised the team to start a parallel track and preparations how we should develop this new P7-car the agile way. As a result, the spirit of the team went up significantly and also provided them with extra energy to fix the bugs in the P6 car.

slider_Forze_VI_Valkenburg

Nimble Networks

Like every team, the presence of an inspiring leader is important.  They're  like the glue keeping the team together and keeping them motivated. For example, when Joe was travelling, the attendance at agile ceremonies and the Wikispeed teams velocity went down.   The same phenomena I’ve observed coaching The Forze team.  Team members are recruited on their technical skills, not for their leadership qualities. As a result, we had a great bunch of technicians but no leadership in the team. TU-Delft students are are currently not educated to recognize develop natural and/or acquired leadership skills. As a result, the 2015-team is vulnerable. For example, there is only one member with electrotechnical knowledge rightnow.  A skill which is crucial for engineering a hydrogen car.   The recommendation of Berkeley is very useful here. The team should recruit new members with strong personalities combined with leadership qualities and develop a more distributed network which makes the team less vulnerable.

Finally, Berkeley stated the WikiSpeed solution is an exceptional example how disciplined collaboration could do magic but can not be blindly copied to another existing business to do the same magic there.  Agile methods should be tailored to fit it’s goals. I cannot more agree on this.  An agile process should be agile within itself, fit for purpose with a very clear goal.

Download the full paper here.

 

Microservices versus the common SOA implementation

Wed, 02/04/2015 - 15:02

When I was first reading about MSA architectures (MSA) I had a hard time figuring out what was so different from a Service Oriented Architecture (SOA). Main reason for this is that the SOA paradigm leaves quite a bit of room for interpretation and various people have different interpretations. When Martin Fowler wrote about MSA almost a year ago he also mentioned that some people see it as “SOA done right”, he himself considers MSA a subset of SOA. So, what are the differences, if any?
At first glance there seem to be more than a few similarities. Both talk about services as the cornerstone for the architecture, services need to loosely coupled, technology agnostic and there’s probably a few more. What sets the Microservices architectural style apart is that it’s more clear on where the focus needs to be when designing your services. The name suggests that the services need to very small and probably very fine grained but in reality no size restrictions apply. Ironically, it’s the size (and focus) of services that commonly cripple SOA implementations.
The main characteristics of MSA are: autonomy, high cohesion and low coupling. Out of these characteristics autonomy is the defining one. Here autonomy not only means that it can run or function independently (that is what low coupling is about), it means that a service should be able to provide business value on its own. This principle ties together the focus on low coupling and high cohesion. Lower coupling allows the service to operate independently and high cohesion increases it’s ability to add value on its own.
What you often see in SOA implementations is the focus on reuse. This means that a certain function should only exist in one place or a single service should handle a certain function. Where things go sour is in how these “functions” are determined. This is where cohesion comes into play. Cohesion is the degree in which functionality in services belongs together. The highest cohesion is functional cohesion where the services contribute to a well-defined task or business capability. MSA strives towards functional (high) cohesion. The cohesion found in common SOA implementations is logical cohesion (one step up from the worst kind, coincidental cohesion). With logical cohesion the services are grouped around similar tasks, like the one service for one function mentioned above. This approach leads to finer grained services that focus on atomic tasks. Take for instance a “data service” that handles all communication with a database. It accepts messages in a common format and then uses that information to execute the desired action on the database. Benefit is that applications that use this service don’t need to worry about the tech behind said service and only need to provide messages in a common format. If the database is shared by multiple services they all use this service when communicating with the database. Now imagine what would happen if this service goes down (intentionally or not). Or what the impact is when this service needs to be modified. And this is assuming that this service is well designed and built so it can handle all the different requests thrown at it.
With MSA the goal should be high cohesion, this means grouping things around a business capability (so it can add business value). It also provides all aspects of that capability end-to-end, from data storage to user interface functions. So instead of having a “data service” you create a “customer service” or “shipping service”.
Another aspect that is relevant here is that with MSA one should strive for low coupling (or better yet, no coupling). Low coupling boils down to the dependency between services. With no coupling the services can function completely independent of each other. If two services have no coupling downtime of one of the two will have zero impact on the other. The higher the coupling, the higher the impact.
With SOA implementations based on logical cohesion the coupling tends to be high. Because services depend on other services to function. High coupling increases the impact of changes.
For MSA the goal is no coupling. But lowering the coupling does not mean they can’t or shouldn’t interact. Services can still interact with other services but they don’t depend on them. Another distinction to take into consideration is that these are more technical dependencies, not functional ones. Take for instance an order service and a shipping service. Both can operate independently. The shipping service will process all the orders it has received, regardless of the availability of an order service. If the order service is down it simply means no new orders will be created for the shipping service. So when the shipping service is done handling its last known order it stops. Vice versa, if the shipping service is down the order service will still be able to process orders. They just won’t be processed by the shipping service. When the shipping service comes back up it will process the backlog created by the order service.
How much the Microservices architectural style differs from SOA depends on who you talk to. If nothing else, MSA offers a clearer and better defined approach on setting up an architecture built around services. The key differentiator being the focus on autonomously adding value. There is a lot more to say on the subject but hopefully this gives some insight into how MSA differentiates itself from what you typically see with SOA implementations.

Apache Spark

Tue, 02/03/2015 - 21:22

Spark is the new kid on the block when it comes to big data processing. Hadoop is also an open-source cluster computing framework, but when compared to the community contribution, Spark is much more popular. How come? What is so special and innovative about Spark? Is it that Spark makes big data processing easy and much more accessible to the developer? Or is it because the performance is outstanding, especially compared to Hadoop?

This article gives an introduction to the advantages of current systems and compares these two big data systems in depth in order to explain the power of Spark.

Parallel computing

First we have to understand the differences between Hadoop and the conventional approach of parallel computing before we can compare the differences between Hadoop and Spark.

Distributed computing

In the case of parallel computing, all tasks have access to shared data to exchange information and perform their calculations. With distributed computing, each task has its own data. Information is exchanged by passing data between the tasks.

One of the main concepts of distributed computing is data locality, which reduces network traffic. Because of data locality, data is processed faster and more efficiently. There is no separate storage network or processing network.

Apache Hadoop delivers an ecosystem for distributed computing. One of the biggest advantages of this approach is that it is easily scalable, and one can build a cluster with commodity hardware. Hadoop is designed in the way that it can handle server hardware failures.

Stack

To understand the main differences between Spark and Hadoop we have to look at their stacks. Both stacks consist of several layers.

Stacks of Spark and Hadoop

The storage layer is responsible for a distributed file system that stores data on commodity machines, providing very high aggregate bandwidth across the cluster. Spark uses the Hadoop layer. That means one can use HDFS (the file system of Hadoop) or other storage systems supported by the Hadoop API. The following storage systems are supported by Hadoop: your local file system, Amazon S3, Cassandra, Hive, and HBase.

The computing layer is the programming model for large scale data processing. Hadoop and Spark differ significantly in this area. Hadoop uses a disk-based solution provided by a map/reduce model. A disk-based solution persists its temporary data on disk. Spark uses a memory-based solution with its Spark Core. Therefore, Spark is much faster. The differences in their computing models will be discussed in the next chapter.

Cluster managers are a bit different from the other components. They are responsible for managing computing resources and using them for the scheduling of users' applications. Hadoop uses its own cluster manager (YARN). Spark can run over a variety of cluster managers, including YARN, Apache Mesos, and a simple cluster manager called the Standalone Scheduler.

A concept unique to Spark is its high-level packages. They provide lots of functionalities that aren't available in Hadoop. One can see this layer also as a sort of abstraction layer, whereby code becomes much easier to understand and produce. These packages are

  • Spark SQL is Spark’s package for working with structured data. It allows querying data via SQL.
  • Spark Streaming enables processing live streams of data, for example, log files or a twitter feed.
  • MLlib is a package for machine learning functionality. A practical example of machine learning is spam filtering.
  • GraphX is a library that provides an API for manipulating graphs (like social networks) and performing graph-parallel computations.

The Spark Core is also written in Scala and supports Scala natively, which is a far better language than Java for implementing the kinds of transformations it supports. This results in less code which is therefore more intuitive.

Screen Shot 2015-01-24 at 16.41.37

(Source: infoq)

Computational model

The main difference between Hadoop and Spark is the computational model. A computational model is the algorithm and the set of allowable operations to process the data.

Hadoop uses the map/reduce. A map/reduce involves several steps.

Hadoop computational model: Map/Reduce

  • This data will be processed and indexed on a key/value base. This processing is done by the map task.
  • Then the data will be shuffled and sorted among the nodes, based on the keys. So that each node contains all values for a particular key.
  • The reduce task will do computations for all the values of the keys (for instance count the total values of a key) and write these to disk.

With the Hadoop computational model, there are only two functions available: map and reduce.
Note that doing distributed computing with Hadoop results, most of the time, in several iterations of map/reduce.  For each iteration, all data is persisted on disk. That is the reason it is called disk-based computing.

Spark uses RDD, also called Resilient Distributed Datasets. Working and processing data with RDD is much easier:

Spark computational model: RDD

  • Reading input data and thereby creating an RDD.
  • Transforming data to new RDDs (by each iteration and in memory). Each transformation of data results in a new RDD. For transforming RDD's there are lots of functions one can use, like map, flatMap, filter, distinct, sample, union, intersection, subtract, etc. With map/reduce you only have the map-function. (..)
  • Calling operations to compute a result (output data). Again there are lots of actions available, like collect, count, take, top, reduce, fold, etc instead of only reduce with the map/reduce.

Performance Hadoop vs Spark

Behind the scenes, Spark does a lot, like distribute the data across your cluster and parallelizing the operations. Note that doing distributed computing is memory-based computing. Data between transformations are not saved to disk. That's why Spark is so much faster.

Conclusion

All in all Spark is the next step in the area of big data processing, and has several advantages compared to Hadoop. The innovation of Spark lies in its computational model. The biggest advantages of Spark against Hadoop are

  • Its in-memory computing capabilities that deliver speed
  • Packages like streaming and machine-learning
  • Ease of development - one can program natively in Scala

How to make the sprint review meeting worth your while

Mon, 02/02/2015 - 16:44

My work allows me to meet a lot of different people, who actively pursue Scrum. Some of them question the value of doing a sprint review meeting at the end of every sprint. Stakeholders presumably do not “use” nor “see” their work directly, or the iterated product is not yet releasable.

Looks like this Scrum ritual is not suited for all. If you are a person questioning the value of a demo, then focus on your stakeholders and start to demo the delta instead of just your product. Here is a 3-step plan to make your sprint reviews worth your while.

Step 1: Provide context

Stakeholders are not interested in individual user stories. They want to see releasable working software they can use and work with. That is were the value delta is for them; in practical services and products they can use to make their lives a little bit better than before.
So the thing you should do in the sprint review is to explain the completed stories in the context of the iteration and how this iteration will add value to the upcoming product-release. Providing the context will make it easier for stakeholders to give your team the feedback it is looking for.

Step 2: Demo the delta

Search for the abstraction level on which the user is impacted by the story you have completed, even if this means combining stories to demo as a whole. This is especially useful if you are working in component-based teams.

It will enable you to explicitly illustrate the added value from a stakeholder perspective after the change. It’s not just about adding an input screen or expansion of variables. It’s about being able to do something different as a stakeholder.

Think about the possibilities given the new additions to the system. Maybe people can be more effective or more efficient saving time, money and energy. If possible try to explicitly show the effect of the delta on the stakeholder, for instance by measuring key variables in the before and after situation. Seeing the explicit effect of the changes will get your stakeholders fired up to provide your team with additional ideas and feedback for your next release.

Step 3: Ask for feedback

The goal of the sprint review is to generate feedback. Often, this won’t happen automatically and means you have to explicitly ask for it. Doing it wrong is to do the review without interruption and to conclude by asking: “any questions?” This will certainly not evoke people to participate in a group discussion as posing questions is, unfortunately, still seen by many as a sign of weakness.

To counter this group dynamic, you should be the one asking the questions to your audience. Example stakeholder focused questions to generate feedback might be; “what are your thoughts on improving this feature further?” Or “How would this fit in your day-to-day routine?”

By doing so, you will lower the barrier for others to start asking questions themselves. Another tip to turn around the question dynamic is to specifically target your question to a single stakeholder, getting the group out of the conversation.

Up until now these steps helped me a lot in improving my sprint review meetings. These 3 simple steps will let your key-stakeholders be the subject of your review meeting, maximizing the chance that they will provide you with valuable feedback to improve your product.

Run your iOS app without overwriting the App Store version

Fri, 01/30/2015 - 13:59

Sometimes when you're developing a new version of your iOS app, you'd like to run it on your iPhone or iPad and still be able to run the current version that is released on the App Store. Normally when you run your app from Xcode on your device, it will overwrite any existing version. If you then want to switch back to the version from the App Store, you'll have to delete the development version and download it again from the App Store.

In this blog post I'll describe how you can run a test version of your app next to your production version. This method also works when you have embedded extensions (like the Today Widget or WatchKit app) in your app or when you want to beta test your app with Apple's TestFlight.

There are two different ways to accomplish this. The first method is to create a new target within your project that runs the same app but with a different name and identifier. With iOS 8 it is now possible to embed extensions in your app. Since these extensions are embedded in the app target, this approach doesn't work when you have extensions and therefore I'll only describe the second method which is based on User-Defined Settings in the build configuration.

Before going into detail, here is a quick explanation of how this works. Xcode already creates two build configurations for us: Release and Debug. By adding some User-Defined Settings to the build configurations we can run the Debug configuration with a different Bundle identifier than the Release configuration. This essentially creates a separate app on your device, keeping the one from the App Store (which used the Release configuration) intact.

To beta distribution of the app built with debug configuration easier we'll create multiple schemes.

The basics

Follow these steps exactly to run a test version on your device next to your App Store version. These steps are based on Xcode 6.1.1 with an Apple Developer Account with admin privileges.

Click on your project in the Project Navigator and make sure the main target is selected. Under both the General and Info tabs you will find the Bundle identifier. If you change this name and run your app, it will create a new app next to the old one. Add -test to your current Bundle identifier so you get something like com.example.myapp-test. Xcode will probably tell you that you don't have a provisioning profile. Let Xcode fix this issue for you and it will create a new Development profile for you named something like iOSTeam Provisioning Profile: com.example.myapp-test.

So we're already able to run a test version of our app next to the App Store version. However, manually changing the Bundle identifier all the time isn't very practical. The Bundle identifier is part of the Info.plist and it's not derived from a Build configuration property like for example the Bundle name (which uses ${PRODUCT_NAME} from the build configuration by default). Therefore we can't simply specify a different Bundle identifier for a different Build configuration using an existing property. But, we can use a User-Defined Setting for this.

Go to the Build Settings tab of your project and click the + button to add a User-Defined Setting. Name the new setting BUNDLE_ID and give it the value of the test Bundle identifier you created earlier (com.example.myapp-test). Then click the small triangle in front of the setting to expand the setting. Remove the -test part from the Release configuration. You should end up with something like this:

Screen Shot 2015-01-30 at 11.29.43

Now go to the Info tab and change the value of the Bundle identifier to ${BUNDLE_ID}. In the General tab you will still see the correct Bundle Identifier, but if you click on it you see the text is slightly grayed out, which means it's taken from a Build configuration setting.

Screen Shot 2015-01-30 at 11.35.52

To test if this works, you can change edit the current Scheme and change the Build Configuration of the Run action from Debug to Release. Close the Scheme window and go to the General tab again to see that the Bundle Identifier has changed to the Release BUNDLE_ID. (If you still had the General tab open and don't see the change then switch to another tab and back; the panel will reload the identifier). Make sure to change the Build configuration back to Debug in your scheme afterwards.

When you now Archive an app before you release it to the App Store, it will use the correct identifier from the Release configuration and when you run the app from Xcode on your device, it will use the identifier for testing. That way it no longer overwrites your App Store version on your device.

App name

Both our App Store app and test app still have the same name. This makes it hard to know which one is which. To solve this, find the Product Name in the Build Settings and change the name for the Debug configuration to something else, like MyApp Test. You can even use another app icon for your test build. Just change the Asset Catalog App Icon Set Name for the Debug configuration.

Beta distribution

What if you want to distribute a version of the app for Beta testing (through TestFlight or something similar) to other people that also shouldn't overwrite their Apple Store version? Our Archive action is using the Release build configuration. We could change that manually to Debug to have the test Bundle identifier but then we would be getting all of the Debug settings in our app, which is not something we want. We need to create another Build configuration.

Go to the project settings of your project (so not the target settings). Click the + button under Configurations and duplicate the Release configuration. Call the new configuration AdHoc. (You might already have such a Build configuration for other reasons, in that case you can skip this step and use that one for the next steps.)

Now go to the Build Settings of your main target and change the AdHoc value of the User-Defined Setting BUNDLE_ID to the same as the Debug value. Do the same for the Product name is you changed that in the previous step.

We could already make a Beta test Archive now by manually changing the configuration of the Archive action to Debug. But it makes it easier if we create a new scheme to do this. So go to the Manage Schemes and click to + button at the bottom left to create a new scheme. Make sure that your main target is selected as Target and add " Test" to the name so you end up with something like "MyApp Test". Also check the Shared checkbox if you are sharing your schemes on your version control system.

Double click the new scheme to edit it and change the build configuration of the Archive action to AdHoc. Now you can Archive with the MyApp Test scheme selected to create a test version of your app that you can distribute to your testers without it overwriting their App Store version.

To avoid confusion about which build configuration is used by which scheme action, you should also change the configuration of the Profile action to AdHoc. And in your normal non-test scheme, you can change the build configuration of all actions to Release. This allows you to run the app in both modes of your device, which sometimes might be necessary, for example when you need to test push notifications that only work for the normal Bundle identifier.

Extensions

As mentioned in the intro, the main reason to use multiple schemes with different build configurations and User-Defined settings as opposed to creating multiple targets with different Bundle identifiers is because of Extensions, like the Today extension or a WatchKit extension. An extension can only be part of a single target.

Extensions make things even more complex since their Bundle identifier needs to be prefixed with the parent app's bundle identifier. And since we just made that one dynamic, we need to make the Bundle identifier of our extensions dynamic as well.

If you don't already have an existing extension, create a new one now. I've tested the approach described below with Today extensions and WatchKit extensions but it should work with any other extension as well.

The steps for getting a dynamic Bundle identifier for the extensions is very similar as the once for the main target so I won't go into too much detail here.

First open the Info tab of the new target that was created for the extension, e.g. MyAppToday. You'll see here that the Bundle display name is not derived from the PRODUCT_NAME. This is probably because the product name (which is derived from the target name) is something not very user friendly like MyAppToday and it is assumed that you will change it. In case of the Today extension, running a test version of the app next to the App Store version will also create 2 Today extensions in the Today view of the phone. To be able to differentiate between the two we'll also make the Bundle display name dynamic.

Change the value of it to ${BUNDLE_DISPLAY_NAME} and then add a User-Defined Setting for BUNDLE_DISPLAY_NAME with different names for Debug/AdHoc and Release.

You might have noticed that the Bundle identifier of the extension is already party dynamic, something like com.example.myapp.$(PRODUCT_NAME:rfc1034identifier). Change this to ${PARENT_BUNDLE_ID}.$(PRODUCT_NAME:rfc1034identifier) and add a User-Defined Setting for PARENT_BUNDLE_ID to your Build Settings. The values of the PARENT_BUNDLE_ID should be exactly the same as the ones you used in your main target, e.g. com.example.myapp for Release and com.example.myapp-test for Debug and AdHoc.

That's it, you can now Run and Archive your app with extensions who's Bundle identifier are prefixed with the parent's Bundle identifier.

App Groups entitlements

You might have an extension that shares UserDefaults data or Core Data stores with the main app. In that case you need to have matching App Groups entitlements in both your main app and extensions. Since we have dynamic Bundle identifiers that use different provisioning profiles, we also have to make our App Groups dynamic.

If you don't have App Groups entitlements (or other entitlements) yet, go to the Capabilities tab of your main target and switch on App Groups. Add an app group in the form off group.[bundle identifier], e.g. group.com.example.myapp. This will generate an entitlements file for your project (MyApp.entitlements) and set the Code Signing Entitlements of your Build Settings to something like MyApp/MyApp.entitlements. Locate the entitlements file in Finder and duplicate it. Change the name of the copy by replacing " Copy" with "Test" (MyAppTest.entitlements). Drag the copy into your project. You should now have two entitlement files in your project. Open the Test entitlements file in Xcode's Property List editor and add "-test" to the value of Item 0 under com.apple.security.application-groups to match it with the Bundle identifier we used for testing, e.g. com.example.myapp-test. Now go back to the Build Settings and change the Debug and AdHoc values of Code Signing Entitlements to match with the file name of the Test entitlements.

Repeat all these steps for the Extension target. Xcode will also generate the entitlements file in the extension folder. You should end up with two entitlements files for your main target and two entitlements files for every extension.

The Application Groups for testing need to be added to the provisioning profiles which Xcode will handle automatically for you. It might warn/ask you for this while building.

Conclusion

It might be quite some work to follow all these steps and to get everything to work, but if you use your normal iPhone for development and still want to use or show a stable version of your app at the same time, it's definitely worth doing. And your Beta testers will thank your for it as well.

Continuous Delivery across multiple providers

Wed, 01/21/2015 - 13:04

Over the last year three of the four customers I worked with had a similar challenge with their environments. In different variations they all had their environments setup across separate domains. Ranging from physically separated on-premise networks to having environments running across different hosting providers managed by different parties.

Regardless of the reasoning behind having these kinds of setup it’s a situation where the continuous delivery concepts really add value. The stereotypical problems that exist with manual deployment and testing practices tend to get amplified when they occur in seperated domains. Things get even worse when you add more parties to the mix (like external application developers). Sticking to doing things manually is a recipe for disaster unless you enjoy going through expansive procedures every time you want to do anything in any of ‘your’ environments. And if you’ve outsourced your environments to an external party you probably don’t want to have to (re)hire a lot of people just so you can communicate with your supplier.

So how can continuous delivery help in this situation? By automating your provisioning and deployments you make deploying your applications, if nothing else, repeatable and predictable. Regardless of where they need to run.

Just automating your deployments isn’t enough however, a big question that remains is who does what. A question that is most likely backed by a lengthy contract. Agreements between all the parties are meant to provide an answer to that very question. A development partner develops, an outsourcing partners handles the hardware, etc. But nobody handles bringing everything together...

The process of automating your steps already provides some help with this problem. In order to automate you need some form of agreement on how to provide input for the tooling. This at least clarifies what the various parties need to produce. It also clarifies what the result of a step will be. This removes some of the fuzziness out of the process. Things like is the JVM part of the OS or part of the middleware should become clear. But not everything is that clearcut. It’s parts of the puzzle where pieces actually come together that things turn gray. A single tool may need input from various parties. Here you need to resists the common knee-jerk reaction to shield said tool from other people with procedures and red tape. Instead provide access to those tools to all relevant parties and handle your separation of concerns through a reliable access mechanism. Even then there might be some parts that can’t be used by just a single party and in that case, *gasp*, people will need to work together. 

What this results in is an automated pipeline that will keep your environments configured properly and allow applications to be deployed onto them when needed, within minutes, wherever they may run.

MultiProviderCD

The diagram above shows how we set this up for one of our clients. Using XL Deploy, XL Release and Puppet as the automation tooling of choice.

In the first domain we have a git repository to which developers commit their code. A Jenkins build is used to extract this code, build it and package it in such a way that the deployment automation tool (XL Deploy) understands. It’s also kind enough to make that package directly available in XL Deploy. From there, XL Deploy is used to deploy the application not only to the target machines but also to another instance of XL Deploy running in the next domain, thus enabling that same package to be deployed there. This same mechanism can then be applied to the next domain. In this instance we ensure that the machines we are deploying to are consistent by using Puppet to manage them.

To round things off we use a single instance of XL Release to orchestrate the entire pipeline. A single release process is able to trigger the build in Jenkins and then deploy the application to all environments spread across the various domains.

A setup like this lowers deployment errors that come with doing manual deployments and cuts out all the hassle that comes with following the required procedures. As an added bonus your deployment pipeline also speeds up significantly. And we haven’t even talked about adding automated testing to the mix


Try is free in the Future

Mon, 01/19/2015 - 09:40

Lately I have seen a few developers consistently use a Try inside of a Future in order to make error handling easier. Here I will investigate if this has any merits or whether a Future on it’s own offers enough error handling.

If you look at the following code there is nothing that a Future can’t supply but a Try can:

import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.{Await, Future, Awaitable}
import scala.concurrent.duration._
import scala.util.{Try, Success, Failure}

object Main extends App {

  // Happy Future
  val happyFuture = Future {
    42
  }

  // Bleak future
  val bleakFuture = Future {
    throw new Exception("Mass extinction!")
  }

  // We would want to wrap the result into a hypothetical http response
  case class Response(code: Int, body: String)

  // This is the handler we will use
  def handle[T](future: Future[T]): Future[Response] = {
    future.map {
      case answer: Int => Response(200, answer.toString)
    } recover {
      case t: Throwable => Response(500, "Uh oh!")
    }
  }

  {
    val result = Await.result(handle(happyFuture), 1 second)
    println(result)
  }

  {
    val result = Await.result(handle(bleakFuture), 1 second)
    println(result)
  }
}

After giving it some thought the only situation where I could imagine Try being useful in conjunction with Future is when awaiting a Future but not wanting to deal with error situations yet. The times I would be awaiting a future are very few in practice though. But when needed something like this migth do:

object TryAwait {
  def result[T](awaitable: Awaitable[T], atMost: Duration): Try[T] = {
    Try {
      Await.result(awaitable, atMost)
    }
  }
}

If you do feel that using Trys inside of Futures adds value to your codebase please let me know.

Meteor

Sun, 01/18/2015 - 12:11

Did you ever use AngularJS as a frontend framework? Then you should definitely give Meteor a try! Where AngularJS is powerful just as a client framework, meteor is great as a full stack framework. That means you just write your code in one language as if there is no back- and frontend at all. In fact, you get an Android and IOS client for free. Meteor is so incredibly simple that you are productive from the beginning.

Where meteor kicks angular

One of the killing features of meteor is that you'll have a shared code base for frontend and backend. In the next code snippet, you'll see a file shared by backend and frontend:

// Collection shared and synchronized accross client, server and database
Todos = new Mongo.Collection('todos');

// Shared validation logic
validateTodo = function (todo) {
  var errors = {};
  if (!todo.title)
    todo.title = "Please fill in a title";
  return errors;
}

Can you imagine how neat the code above is?

Scan 04 Jan 2015 18.48-page4

With one codebase, you get the full stack!

  1. As in the backend file and in the frontend file one can access and query over the Todos collection. Meteor is responsible for syncing the todos. Even when another user adds an item, it will be visible to your client directly. Meteor accomplishes this by a client-side Mongo implementation (MiniMongo).
  2. One can write validation rules once! And they are executed both on the front-end and on the back-end. So you can give my user quick feedback about invalid input, but you can also guarantee that no invalid data is processed by the backend (when someone bypasses the client). And this is all without duplicated code.

Another killer feature of meteor is that it works out of the box, and it's easy to understand. Angular can be a bit overwhelming; you have to learn concepts like directives, services, factories, filters, isolated scopes, transclusion. For some initial scaffolding, you need to know grunt, yeoman, etcetera. With meteor every developer can create, run and deploy a full-stack application within minutes. After installing meteor you can run your app within seconds.

$ curl https://install.meteor.com | /bin/sh
$ meteor create dummyapp
$ cd dummyapp
$ meteor
$ meteor deploy dummyapp.meteor.com
Screen Shot 2015-01-04 at 19.49.08

Meteor dummy application

Another nice aspect of meteor is that it uses DDP, the Distributed Data Protocol. The team invented the protocol and they are heavily promoting it as "REST for websockets". It is a simple, pragmatic approach allowing it to be used to deliver live updates as data changes in the backend. Remember that this works all out of the box. This talk walks you through the concepts of it. But the result is that if you change data on a client it will be updated immediately on the other client.

And there is so much more, like...

  1. Latency Compensation. On the client, Meteor prefetches data and simulates models to make it look like server method calls return instantly.
  2. Meteor is open source and integrates with existing open source tools and frameworks.
  3. Services (like an official package server and a build farm).
  4. Command line tools
  5. Hot deploys
Where meteor falls short

Yes, meteor is not the answer to all your problems. The reason, I'll still choose angular above meteor for my professional work, is because the view framework of angular rocks. It makes it easy to structure your client code into testable units and connect them via dependency injection. With angular you can separate your HTML from your javascript. With meteor your javascript contains HTML elements, (because their UI-library is based on handlebars. That makes testing harder and large projects will become unstructured very quickly.

Another flaw emerges if your project already has a backend. When you choose meteor, you choose their full stack. That means: Mongo as database and Node.js as backend. Despite you are able to create powerful applications, Meteor doesn't allow you (easily) to change this stack.

Under the hood

Meteor consists out of several subprojects. In fact, it is a library of libraries. In fact, it is a stack; a standard set of core packages that are designed to work well together:

Components used by meteor

  1. To make meteor reactive they've included the components blaze and tracker. The blaze component is heavily based on handlebars.
  2. The DDP component is a new protocol, described by meteor, for modern client-server communication.
  3. Livequery and full stack database take all the pain of data synchronization between the database, backend and frontend away! You don't have to think about in anymore.
  4. The Isobuild package is a unified build system for browser, server and mobile.
Conclusion

If you want to create a website or a mobile app with a backend in no time, with getting lots of functionality out of the box, meteor is a very interesting tool. If you want to have more control or connect to an existing backend, then meteor is probably less suitable.

You can watch this presentation I recently gave, to go along with the article.

Monitoring Akka with Kamon

Thu, 01/15/2015 - 13:49

Kamon is a framework for monitoring the health and performance of applications based on akka, the popular actor system framework often used with Scala. It provides good quick indicators, but also allows in-depth analysis.

Tracing

Beyond just collecting local metrics per actor (e.g. message processing times and mailbox size), Kamon is unique in that it also monitors message flow between actors.

Essentially, Kamon introduces a TraceContext that is maintained across asynchronous calls: it uses AOP to pass the context along with messages. None of your own code needs to change.

Because of convenient integration modules for Spray/Play, a TraceContext can be automatically started when an HTTP request comes in.

If nothing else, this can be easily combined with the Logback converter shipped with Kamon: simply logging the token is of great use right out of the gate.

Dashboarding

Kamon does not come with a dashboard by itself (though some work in this direction is underway).

Instead, it provides 3 'backends' to post the data to (4 if you count the 'LogReporter' backend that just dumps some statistics into Slf4j): 2 on-line services (NewRelic and DataDog), and statsd (from Etsy).

statsd might seem like a hassle to set up, as it needs additional components such as grafana/graphite to actually browse the statistics. Kamon fortunately provides a correctly set-up docker container to get you up and running quickly. We unfortunately ran into some issues with the image uploaded to the Docker Hub Registry, but building it ourselves from the definition on github resolved most of these.

Implementation

We found the source code to Kamon to be clear and to-the-point. While we're generally no great fan of AspectJ, for this purpose the technique seems to be quite well-suited.

'Monkey-patching' a core part of your stack like this can of course be dangerous, especially with respect to performance considerations. Unless you enable the heavier analyses (which are off by default and clearly marked), it seems this could be fairly light - but of course only real tests will tell.

Getting Started

Most Kamon modules are enabled by adding their respective akka extension. We found the quickest way to get started is to:

  • Add the Kamon dependencies to your project as described in the official getting started guide
  • Enable the Metrics and LogReporter extensions in your akka configuration
  • Start your application with AspectJ run-time weaving enabled. How to do this depends on how you start your application. We used the sbt-aspectj plugin.

Enabling AspectJ weaving can require a little bit of twiddling, but adding the LogReporter should give you quick feedback on whether you were successful: it should start periodically logging metrics information.

Next steps are:

  • Enabling Spray or Play plugins
  • Adding the trace token to your logging
  • Enabling other backends (e.g. statsd)
  • Adding custom application-specific metrics and trace points
Conclusion

Kamon looks like a healthy, useful tool that not only has great potential, but also provides some great quick wins.

The documentation that is available is of great quality, but there are some parts of the system that are not so well-covered. Luckily, the source code very approachable.

It is clear the Kamon project is not very popular yet, judging by some of the rough edges we encountered. These, however, seem to be mostly superficial: the core ideas and implementation seems solid. We highly recommend taking a look.

 

Remco Beckers

Arnout Engelen

Exploring Akka Stream's TCP Back Pressure

Wed, 01/14/2015 - 15:48

Some years ago, when Reactive Streams lived in utopia we got the assignment to build a high-volume message broker. A considerable amount of code of the solution we delivered back then was dedicated to prevent this broker being flooded with messages in case an endpoint became slow.

How would we have solved this problem today with the shiny new Akka Reactive Stream (experimental) implementation just within reach?

In this blog we explore Akka Streams in general and TCP Streams in particular. Moreover, we show how much easier we can solve the challenge we faced backed then using Streams.

A use-case for TCP Back Pressure

The high-volume message broker mentioned in the introduction basically did the following:

  • Read messages (from syslog) from a TCP socket
  • Parse the message
  • Forward the message to another system via a TCP connection

For optimal throughput multiple TCP connections were available, which allowed delivering messages to the endpoint system in parallel. The broker was supposed to handle about 4000 - 6000 messages per second. As follows a schema of the noteworthy components and message flow:

Waterhose2

Naturally we chose Akka as framework to implement this application. Our approach was to have an Actor for every TCP connection to the endpoint system. An incoming message was then forwarded to one of these connection Actors.

The biggest challenge was related to back pressure: how could we prevent our connection Actors from being flooded with messages in case the endpoint system slowed down or was not available? With 6000 messages per second an Actor's mailbox is flooded very quickly.

Another requirement was that message buffering had to be done by the client application, which was syslog. Syslog has excellent facilities for that. Durable mailboxes or something the like was out of the question. Therefore, we had to find a way to pull only as many messages in our broker as it could deliver to the endpoint. In other words: provide our own back pressure implementation.

A considerable amount of code of the solution we delivered back then was dedicated to back pressure. During one of our re-occurring innovation days we tried to figure out how much easier the back pressure challenge would have been if Akka Streams would have been available.

Akka Streams in a nutshell

In case you are new to Akka Streams as follows some basic information that help you understand the rest of the blog.

The core ingredients of a Reactive Stream consist of three building blocks:

  • A Source that produces some values
  • A Flow that performs some transformation of the elements produced by a Source
  • A Sink that consumes the transformed values of a Flow

Akka Streams provide a rich DSL through which transformation pipelines can be composed using the mentioned three building blocks.

A transformation pipeline executes asynchronously. For that to work it requires a so called FlowMaterializer, which will execute every step of the pipeline. A FlowMaterializer uses Actor's for the pipeline's execution even though from a usage perspective you are unaware of that.

A basic transformation pipeline looks as follows:


  import akka.stream.scaladsl._
  import akka.stream.FlowMaterializer
  import akka.actor.ActorSystem

  implicit val actorSystem = ActorSystem()
  implicit val materializer = FlowMaterializer()

  val numberReverserFlow: Flow[Int, String] = Flow[Int].map(_.toString.reverse)

  numberReverserFlow.runWith(Source(100 to 200), ForeachSink(println))

We first create a Flow that consumes Ints and transforms them into reversed Strings. For the Flow to run we call the runWith method with a Source and a Sink. After runWith is called, the pipeline starts executing asynchronously.

The exact same pipeline can be expressed in various ways, such as:


    //Use the via method on the Source that to pass in the Flow
    Source(100 to 200).via(numberReverserFlow).to(ForeachSink(println)).run()

    //Directly call map on the Source.
    //The disadvantage of this approach is that the transformation logic cannot be re-used.
    Source(100 to 200).map(_.toString.reverse).to(ForeachSink(println)).run()

For more information about Akka Streams you might want to have a look at this Typesafe presentation.

A simple reverse proxy with Akka Streams

Lets move back to our initial quest. The first task we tried to accomplish was to create a stream that accepts data from an incoming TCP connection, which is forwarded to a single outgoing TCP connection. In that sense this stream was supposed to act as a typical reverse-proxy that simply forwards traffic to another connection. The only remarkable quality compared to a traditional blocking/synchronous solution is that our stream operates asynchronously while preserving back-pressure.

import java.net.InetSocketAddress
import akka.actor.ActorSystem
import akka.stream.FlowMaterializer
import akka.stream.io.StreamTcp
import akka.stream.scaladsl.ForeachSink

implicit val system = ActorSystem("on-to-one-proxy")
implicit val materializer = FlowMaterializer()

val serverBinding = StreamTcp().bind(new InetSocketAddress("localhost", 6000))

val sink = ForeachSink[StreamTcp.IncomingConnection] { connection =>
      println(s"Client connected from: ${connection.remoteAddress}")
      connection.handleWith(StreamTcp().outgoingConnection(new InetSocketAddress("localhost", 7000)).flow)
}
val materializedServer = serverBinding.connections.to(sink).run()

serverBinding.localAddress(materializedServer)

First we create the mandatory instances every Akka reactive Stream requires, which is an ActorSystem and a FlowMaterializer. Then we create a server binding using the StreamTcp Extension that listens to incoming traffic on localhost:6000. With the ForeachSink[StreamTcp.IncomingConnection] we define how to handle the incoming data for every StreamTcp.IncomingConnection by passing a flow of type Flow[ByteString, ByteString]. This flow consumes ByteStrings of the IncomingConnection and produces a ByteString, which is the data that is sent back to the client.

In our case the flow of type Flow[ByteString, ByteString] is created by means of the StreamTcp().outgoingConnection(endpointAddress).flow. It forwards a ByteString to the given endpointAddress (here localhost:7000) and returns its response as a ByteString as well. This flow could also be used to perform some data transformations, like parsing a message.

Parallel reverse proxy with a Flow Graph

Forwarding a message from one connection to another will not meet our self defined requirements. We need to be able to forward messages from a single incoming connection to a configurable amount of outgoing connections.

Covering this use-case is slightly more complex. For it to work we make use of the flow graph DSL.


  import akka.util.ByteString
  import akka.stream.scaladsl._
  import akka.stream.scaladsl.FlowGraphImplicits._

  private def parallelFlow(numberOfConnections:Int): Flow[ByteString, ByteString] = {
    PartialFlowGraph { implicit builder =>
      val balance = Balance[ByteString]
      val merge = Merge[ByteString]
      UndefinedSource("in") ~> balance

      1 to numberOfConnections map { _ =>
        balance ~> StreamTcp().outgoingConnection(new InetSocketAddress("localhost", 7000)).flow ~> merge
      }

      merge ~> UndefinedSink("out")
    } toFlow (UndefinedSource("in"), UndefinedSink("out"))
  }

We construct a flow graph that makes use of the junction vertices Balance and Merge, which allow us to fan-out the stream to several other streams. For the amount of parallel connections we want to support, we create a fan-out flow starting with a Balance vertex, followed by a OutgoingConnection flow, which is then merged with a Merge vertex.

From an API perspective we faced the challenge of how to connect this flow to our IncomingConnection. Almost all flow graph examples take a concrete Source and Sink implementation as starting point, whereas the IncomingConnection does neither expose a Source nor a Sink. It only accepts a complete flow as input. Consequently, we needed a way to abstract the Source and Sink since our fan-out flow requires them.

The flow graph API offers the PartialFlowGraph class for that, which allows you to work with abstract Sources and Sinks (UndefinedSource and UndefinedSink). We needed quite some time to figure out how they work: simply declaring a UndefinedSource/Sink without a name won't work. It is essential that you give the UndefinedSource/Sink a name which must be identical to the one that is used in the UndefinedSource/Sink passed in the toFlow method. A bit more documentation on this topic would help.

Once the fan-out flow is created, it can be passed to the handleWith method of the IncomingConnection:

...
val sink = ForeachSink[StreamTcp.IncomingConnection] { connection =>
      println(s"Client connected from: ${connection.remoteAddress}")
      val parallelConnections = 20
      connection.handleWith(parallelFlow(parallelConnections))
    }
...

As a result, this implementation delivers all incoming messages to the endpoint system in parallel while still preserving back-pressure. Mission completed!

Testing the Application

To test our solution we wrote two helper applications:

  • A blocking client that pumps as many messages as possible into a socket connection to the parallel reverse proxy
  • A server that delays responses with a configurable latency in order to mimic a slow endpoint. The parallel reverse proxy forwards messages via one of its connections to this endpoint.

The following chart depicts the increase in throughput with the increase in amount of connections. Due to the nondeterministic concurrent behavior there are some spikes in the results but the trend shows a clear correlation between throughput and amount of connections:

Performance_Chart

End-to-end solution

The end-to-end solution can be found here.
By changing the numberOfConnections variable you can see the impact on performance yourself.

Check it out! ...and go with the flow ;-)

Information about TCP back pressure with Akka Streams

At the time of this writing there was not much information available about Akka Streams, due to the fact that it is one of the newest toys of the Typesafe factory. As follows some valuable resources that helped us getting started:

Peace of mind in a state of overload

Sun, 01/04/2015 - 21:23

This article is meant for knowledge workers that want to be more on top of things and feel secure that they haven’t forgotten about something, freeing their mind for the actual tasks at hand. It especially applies to those that are using or want to use SCRUM, a popular and formalized Agile methodology, in their day to day work.
I got hooked on Agile back in 2005, while working for Db4o, and never looked back since. Improving the process per iteration and having a manageable amount of work per sprint gave me peace of mind and focus, enabling me to deliver the right software solutions to the stakeholders. When I got asked to join a tech startup in 2011 as its CTO I suddenly had a lot more to deal with: hiring and managing staff, managing costs, reporting to the board, applying for subsidies and making sure the books were kept in order. On top of this I still operated as SCRUM Master and technical lead within a SCRUM team.
During this period one of my co-founders introduced me to Getting things done by David Allen. It took him only about 15 minutes to explain the basics and I got started with it straight away.

You can optionally watch this presentation to go along with the article:

Diverse responsibilities

As knowledge workers, and more specifically consultants, we have a diversity of responsibilities. You have personal responsibilities, like planning a doctor’s appointment or picking up your kids from daycare, and you have responsibilities from your employer like ordering a backup disk, co-creating a course or preparing a presentation. Lastly you also have responsibilities from your client, like sprint tasks, meetings and organizing innovation days. Truly staying on top of all these responsibilities is tough, real tough!

Agile

For those of you that are not familiar with Agile methodologies. Agile is an iterative process. In each iteration a team commits to a finite amount of work. A typical iteration has a length of two weeks, which is its deadline. All stories, the units of work in an iteration, can be made actionable by the Agile team.

GTD

Getting Things Done takes a slightly different approach. There is a single inbox, comparable to the part of a backlog in SCRUM, that hasn’t been groomed yet. By regularly reviewing the inbox, it can be emptied by turning items into actionable tasks, high-level projects, calendar items, reference material or just trash. Actionable tasks can be extracted from a project, that will move it forward. Actionable tasks should be kept together to be able to prioritize them upon review.

GTD Chart

GTD Chart

Please refer to the book Getting Things Done by David Allen in order to get the full explanation but below follows a quick overview.

Quick overview of GTD Inbox

The purpose of the GTD inbox is to collect anything new that might or might not require your attention. Whenever something pops up into your mind that you think requires your attention, either now or in the future, you collect it here. It doesn’t need to be organised. It doesn’t even need to be an attainable goal. As long as you get it off your mind and file it away for review, the inbox has done its job. Reviewing the inbox will empty it into one of the following categories.

Trash

You may find that a lot of the things you collect in your GTD inbox don’t really require you to take any action at all, so you can throw them away with impunity! Good riddance!

File

Many a time you get a piece of information that you don’t need immediately but would like to be able to reference in the future. These items go from your inbox into a file. This file can be physical, a folder structure on your computer system or something completely different.

Calendar

Though people have a tendency to put too many things in their calendar that really don’t need to be done at that exact time, inbox items that do have a specific time window move from there to your calendar.

Waiting for

If you delegate something you expect it to be done after a certain period of time. Though these dependencies are kept out of SCRUM for a good reason they are a part of everyday life. Move these from your inbox to your waiting for list so you can check in with whoever you delegated it to in order to be aware of their status.

Someday maybe

Colleague sent you an adorable youtube frolicking puppy compilation? Maybe you didn’t have time to watch it when it was brought to your attention but why not put it away to watch later? Items that you don’t need to do but would like to get around to eventually should be moved over here.

Projects

There are many things that people have on their to do list that they keep on staring at and get intimidated by for some reason. Often this pertains to items that are of a too high level to pick up straight away. This can range from Bring about world peace to Organise daughter’s birthday party. Both of these tasks really consist of multiple clearly actionable tasks even though one gets the impression the latter is easier to attain in one’s lifetime. Things that should be more clearly defined belong here. Review them and extract actionable tasks from them that move the projects closer to their goal.

Next actions

This is the work horse of the GTD system and is where you actually pick up tasks to do. You place well defined tasks here. If you need to call someone about a case, be sure to save this tasks along with the name of the person, their phone number and case reference to remove any impediments, like having to look up their number or the case reference. This way you make the barrier to get something done as low as possible.

Obstacles

Of course there are a few obstacles to overcome using these two methods side by side as a consultant.

Since GTD encourages you to work on the most important thing at that time, you could be required to make private phone calls during consultancy hours or respond to a mail from a co-worker while getting ready for work. This causes work-time and private time to overlap. The obvious solution would be to track work intervals for each task. However, this takes a little bit of time and discipline so should ideally be automated.

GTD requires you to review at least daily what should be done with stuff in your inbox and what the next most important actions are. This takes time. Is this personal time? Should it be billable? In my opinion working with GTD reduces stress and increases productivity and effectiveness. Thus working with GTD will make you a more valuable employee which is something your employer should invest in.

While both GTD as well as Agile work with definitions of work, the priority of stories in SCRUM is determined by the Product Owner. How can this be incorporated into GTD? Well, even though the Product Owner is in charge of the priorities inside the sprint, he does not have a full overview of everything that needs doing in your life. Therefore you are the one that determines the order in which your tasks need to be performed. Within these tasks only the ones coming from your sprint have a relative order pre-determined by the PO.

Improvements

In my day to day usage of GTD I found that there are a few identifiable improvements.

Due to work requirements I need to maintain multiple calendars. Since some GTD inbox items end up in your calendar this sometimes means having to create multiple items in your calendar, which causes needless overhead. It would be beneficial if this would be supported by software, so that a GTD inbox item can automatically be delegated to one or more calendars.

When tasks are coming from external applications like Jira they have to be kept in sync. It would save time if this could be managed automatically.

Lastly the question of ownership. Who owns a task? The assignee, the author or the company on who’s behalf it needs to be performed? I strongly believe that tasks belong to the author or organisation that the author wrote them for. If tasks have been delegated or synced from external systems they should be revocable by their owner. At the same hand an organisation should not have or control access to tasks a person authored without the author’s permission.

Conclusion

Unfortunately there is currently no software tool that would serve all my needs as outlined here. However the most essential properties of such a tool should be: multi-platform to ensure availability, tagging support to be able to categorise without having to split up the list and owned by you.

Swift self reference in inner closure

Fri, 12/19/2014 - 13:06

We all pretty much know how to safely use self within a Swift closure. But have you ever tried to use self inside a closure that's inside another closure? There is a big chance that the Swift compiler crashed (Xcode 6.1.1) without giving you an error in the editor and without any error message. So how can you solve this problem?

The basic working closure

Before we dive into the problem and solution, let's first have a look at a working code sample that only uses a single closure. We can create a simple Swift Playground to run it and validate that it works.

class Runner {
    var closures: [() -> ()] = []

    func doSomethingAsync(completion: () -> ()) {
        closures = [completion]
        completion()
    }
}

class Playground {

    let runner = Runner()

    func works() {
        runner.doSomethingAsync { [weak self] in
            self?.printMessage("This works") ?? ()
        }
    }

    func printMessage(message: String) {
        println(message)
    }

    deinit {
        println("Deinit")
    }

}

struct Tester {
    var playground: Playground? = Playground()
}

var tester: Tester? = Tester()
tester?.playground?.works()
tester?.playground = nil

The doSomethingAsync method takes a closure without arguments and has return type Void. This method doesn't really do anything, but imagine it would load data from a server and then call the completion closure once it's done loading. It does however create a strong reference to the closure by adding it to the closures array. That means we are only allowed to use a weak reference of self within our closure. Otherwise the Runner would keep a strong reference to the Playground instance and neither would ever be deallocated.

Luckily all is fine and the "This works" message is printed in our playground output. Also a "Deinit" message is printed. The Tester construction is used to make sure that the playground will actually deallocate it.

The failing situation

Let's make things slightly more complex. When our first async call is finished and calls our completion closure, we want to load something more and therefore need to create another closure within the outer closure. We add the method below to our Playground class. Keep in mind that the first closure doesn't have [weak self] since we only reference self in the inner closure.

func doesntWork() {
    weak var weakRunner = runner
    runner.doSomethingAsync {

        // do some stuff for which we don't need self

        weakRunner?.doSomethingAsync { [weak self] in
            self?.printMessage("This doesn't work") ?? ()
        } ?? ()
    }
}

Just adding it already makes the compiler crash, without giving us an error in the editor. We don't even need to run it.

Screen Shot 2014-12-19 at 10.30.59

It gives us the following message:

Communication with the playground service was interrupted unexpectedly.
The playground service "com.apple.dt.Xcode.Playground" may have generated a crash log.

And when you have such code in your normal project, the editor also doesn't give an error, but the build will fail with a Swift Compiler Error without clear message of what's wrong:
Command /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swiftc failed with exit code 1

The solution

So how can we work around this problem. Quite simple actually. We simply need to move the [weak self] to the most outer closure.

func doesWork() {
    weak var weakRunner = runner
    runner.doSomethingAsync { [weak self] in

        weakRunner?.doSomethingAsync {
            self?.printMessage("This now works again") ?? ()
        } ?? ()
    }
}

This does mean that it's possible that within the outer closure, self is not nil and in the inner closure it is nil. So don't write code like this:

    runner.doSomethingAsync { [weak self] in

        if self != nil {
            self!.printMessage("This is fine, self is not nil")

            weakRunner?.doSomethingAsync {
                self!.printMessage("This is not good, self could be nil now")
            } ?? ()
        }
    }

There is one more scenario you should be aware of. If you use an if let construction to safely unwrap self, you could create a strong reference again to self. The following sample illustrates this and will create a reference cycle since our runner will create a strong reference to the Playground instance because of the inner closure.

    runner.doSomethingAsync { [weak self] in

        if let this = self {

            weakRunner?.doSomethingAsync {
                this.printMessage("Captures a strong reference to self")
            } ?? ()
        }
    }

Also this is solved easily by including a weak reference to the instance again, now called this.

runner.doSomethingAsync { [weak self] in

    if let this = self {

        weakRunner?.doSomethingAsync { [weak this] in
            this?.printMessage("We're good again") ?? ()
        } ?? ()
    }
}
Conclusion

Most people working with Swift know that there are still quite a few bugs in it. In this case, Xcode should give us an error in the editor. If your editor doesn't complain, but your Swift compiler fails, look for closures like this and correct it. Always be safe and use [weak self] references within closures.

Agile: how hard can it be?!

Sun, 12/14/2014 - 13:48

Yesterday my colleagues and I ran an awesome workshop at the MIT conference in which we built a Rube Goldberg machine using Scrum and Extreme Engineering techniques. As agile coaches one would think that being an Agile team should come naturally to us, but I'd like to share our pitfalls and insights with you since "we learned a lot" about being an agile team and what an incredible powerful model a Rube Goldberg machine is for scaled agile product development.

If you're not the reading type, check out the video.

Rube ... what?

Goldberg. According to Wikipedia; A Rube Goldberg machine is a contraption, invention, device or apparatus that is deliberately over-engineered or overdone to perform a very simple task in a very complicated fashion, usually including a chain reaction. The expression is named after American cartoonist and inventor Rube Goldberg (1883–1970).

In our case we set out on a 6 by 4 meter stage divided in 5 sections. Each section had a theme like rolling, propulsion, swinging, lifting etc. In a fashion it resembled a large software product that requires to respond to some event in an (for outsiders) incredibly complex matter, by triggering a chain of sub-systems which end in some kind of end-result.

The workspace, scrum boards and build stuff

The workspace, scrum boards and build stuff

Extreme Scrum

During the day 5 teams worked in a total of 10 sprints to create the most incredible machine, experiencing everything one can find during "normal" product development. We had inexperienced team members, little to no documentation, legacy systems whom's engineering principles were shrouded in mystery, teams that forgot to retrospective, interfaces that were ignored because the problem "lies with the other team". The huge time pressure of the relative small sprint and the complexity of what we were trying to achieve created a pressure cooker that brought these problems to the surface faster than anything else and with Scrum we were forced to face and fix these problems.

Team scrumboard

Team scrumboard

Build, fail, improve, build

“Most people do not listen with the intent to understand; they listen with the intent to reply.” - Stephen R. Covey

Having 2 minutes to do your planning makes it very difficult to listen, especially when your head is buzzing with ideas, yet sometimes you have to slow down to speed up. Effective building requires us to really understand what your team mate is going to do, pairing proved a very effective way to slow down your own brain and benefit from both rubber ducking and the insight of your team mate. Once our teams reached 4 members we could pair and drastically improve the outcome.

Deadweight with pneumatic fuse

Deadweight with pneumatic fuse

Once the machine had reached a critical size integration tests started to fail. The teams responded by testing multiple times during the sprint and fix the broken build rather than adding new features. Especially in mechanical engineering that is not a simple as it sounds. Sometimes a part of the machine would be "refactored" and since we did not design for a simple end-to-end test to be applied continuously. It took a couple of sprints to get that right.

A MVP that made it to the final product

A MVP that made it to the final product

"Keep your code clean" we teach teams every day. "Don't accept technical or functional debt, you know it will slow you down in the end". Yet it is so tempting. Despite a Scrum Master and an "Über Scrum Master" we still had a hard time keeping our workspace clean, refactor broken stuff out, optimise and simplify...

Have an awesome goal

"A true big hairy audacious goal is clear and compelling, serves as unifying focal point of effort, and acts as a clear catalyst for team spirit. It has a clear finish line, so the organization can know when it has achieved the goal; people like to shoot for finish lines." - Collins and Porras, Built to Last: Successful Habits of Visionary Companies

Truth is: we got lucky with the venue. Building a machine like this is awesome and inspiring in itself, learning how Extreme Scrum can help teams to build machines better, faster, innovative and with a whole lot more fun is a fantastic goal in itself, but parallel to our build space was a true magnet, something that really focussed the teams and go that extra mile.

The ultimate goal of the machine

The ultimate goal of the machine

Biggest take away

Building things is hard, building under pressure is even harder. Even teams that are aware of the theory will be tempted to throw everything overboard and just start somewhere. Applying Extreme Engineering techniques can truly help you, it's a simple set of rules but it requires an unparalleled level of discipline. Having a Scrum coach can make all the difference between a successful and failed project.

Extreme Engineering - Building a Rube Goldberg machine with scrum

Fri, 12/12/2014 - 15:16

Is agile usable to do other things than software development? Well we knew that already; yes!
But to create a machine in 1 day with 5 teams and continuously changing members using scrum might be exciting!

See our report below (it's in Dutch for now)

 

Extreme engineering video