Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Xebia Blog
Syndicate content
Updated: 14 hours 7 min ago

Internet of things

Sat, 03/28/2015 - 11:26

Do you remember the time that you were not connected to the internet? When you had to call your friend instead of sending a message via Whatsapp to make an appointment. The times you had to apply for a job by writing a letter with  a pen instead of sending your Linkedin page? We all have gone through a major revolution in which we've all got a digital identity. The same revolution has started for things, right now!

Internet of Things

Potential

Things we use all day, like your watch or thermostat, get suddenly connected to the internet. These things are equipped with electronics, sensors and connectivity. Whether you like it or not, you will not escape this movement. According to Gartner there will be nearly 26 billion devices on the Internet of Things by 2020.
The potential of the Internet of Things isn’t just linking millions of devices. It is more like the potential of Google or Amazon had when building massive IT infrastructure. The IoT is about transforming business models and enabling companies to sell products in entirely new and better ways.

Definition

The definition of IoT is fairly simple. The internet of things involves `every thing that is connected to the internet`. You should exclude mobiles and computers, because they've already become mainstream.

In fact, the Internet of things is a name for a set of trends. First of all we have the trend that objects, we use all day,  are suddenly connected to the internet. For example bathroom scales that compares your weight with others.
Another trend is that devices that were already shipped with CPUs have become much more powerful when they’re connected to a network. Functionality of the hardware depends on local computing as well as powerful cloud computing. Fitness tracker, can carry out some functionality, like basic data cleaning and tabulation, on local hardware, but it depends on sophisticated machine-learning algorithms on an ever improving cloud service to offer prescription and insight.
Besides that we have the makers movement. Creating hardware isn’t for the big companies like Sony or Samsung anymore, just like creating software in the 80’s and 90’s was a privilege for IBM and Microsoft.

What’s new?

Devices that are connected to the internet aren’t new. For many years, we've connected devices to the internet, although not at that size. I think we have to compare it concepts like 'the cloud'. Before I heard the name 'the cloud' I was already using services that run on the internet, like Hotmail. But one swallow doesn't make a summer. When suddenly lots of services are running on the web, we speak of a trend and the market will give it a name to it. The same is happening with the internet of things. Suddenly all kinds of 'normal' devices are connected to the internet and we have a movement towards that direction that is unstoppable. And the possibilities it will bring are endless.

Splitting up in pieces

We can divide the internet of things into five smart categories. First of all we have the smart wearable. It is designed for a variety of purposes as well as for wear on a variety of parts of the body. Think of a bicycle helmet that detects a crash. Next to it we have the smart home. It’s goal is to make the experience of living at home more convenient and pleasant. A thermostat that learns what temperatures users like and builds a context-aware personalized schedule. A third and fourth category are the smart city and smart environment. It focuses on finding sustainable solutions to the growing problems. The last category is very interesting. It is the smart enterprise. One can think of real-time shipment tracking or smart metering solution that manages energy consumption at the individual appliance and machine level.

Changes of business models

There are plenty of new business models due to the internet of things. An example is a system that provides a sensor-embedded trash, so it’s capable of real-time context analysis and alerting the authorities when it is full and needs to be emptied. Another example is the insurance world, which will adopt the internet of things. There's already behaviour-based car insurance pricing available. Customers who agree to install a monitoring device on their car's diagnostic port, get a break on their insurance rates and pricing that varies according to usage and driving habits.

Big data

Other disciplines in software development also have interest in the new movement, like big data. Internet of things will not generate big data, but obesitas data. Did you know that within every flight, there are  3 terabytes of data generated with all the sensors in an airplane? Think about how much data will be generated, that has to be analyzed, when all buildings in a town are equipped with sensors to measure the quality of the air?

Conclusion

The intersection between software and the physical world becomes smaller every day and that is the result of the unstoppable movement of internet of things. It is still at the beginning. Business models are changing and there are lots of opportunities. You just have to find them...

A High Available Docker Container Platform using CoreOS and Consul

Tue, 03/24/2015 - 11:35

Docker containers are hot, but containers in themselves are not very interesting. It needs an eco-system to make it into  24x7 production deployments. Just handing your container names to operations, does not cut it.

In the blog post, we will show you how  CoreOS can be used to provide a High Available Docker Container Platform as a Service, with a box standard way to deploy Docker containers. Consul is added to the mix to create a lightweight HTTP Router to any docker application offering a HTTP service.

We will be killing a few processes and machine on the way to prove our point...

Architecture

The basic architecture for our Docker Container Platform as a Service, consists of the following components

coreos-caas

  • CoreOS cluster
    The CoreOS cluster will provide us with a cluster of Highly Available Docker Hosts. CoreOS is an open source lightweight operating system based on the Linux kernel and provides an infrastructure for clustered deployments of applications. The interesting part of CoreOS is that you cannot install applications or packages on CoreOS itself. Any custom application has to be packed and deployed as a Docker container. At the same time CoreOS provides only basic functionality for managing these applications.
  • Etcd
    etcd is the CoreOS distributed key value store and provides a reliable mechanism to distribute data through the cluster.
  • Fleet
    Fleet is the cluster wide init system of CoreOS which allows you to schedule applications to run inside the Cluster and provides the much needed nanny system for you apps.
  • Consul
    Consul from Hashicorp is a tool that eases service discovery and configuration. Consul allows services to be discovered via DNS and HTTP and provides us with the ability to respond to changes in the service registration.
  • Registrator
    The Registrator from Gliderlabs will automatically register and deregister any Docker container as a service in Consul. The registrator runs on each Docker Host.
  • HttpRouter
    Will dynamically route HTTP traffic to any application providing a HTTP services, running anywhere in the cluster.  It listens on port 80.
  • Load Balancer
    An external load balancer which will route the HTTP traffic to any of the CoreOS node listening on port 80.
  • Apps
    These are the actual applications that may advertise HTTP services to be discovered and accessed. These will be provided by you.

 

Getting Started

In order to get your own container platform as a service running, we have created a Amazon AWS CloudFormation file which installs the basic services: Consul, Registrator, HttpRouter and the load balancer.

In the infrastructure we create two autoscaling groups: one for the Consul Servers which is limited to 3 to 5 machines and one from the Consul clients which is basically unlimited and depends on your need.

The nice thing about the autoscaling group is that it will automatically launch a new machine if the number of machines drops below the minimum or desired number.  This adds robustness to the platform.

The Amazon Elastic Load Balancer balances incoming traffic to any port machine in either autoscaling group.

We created a little script that creates your CoreOS cluster. This has prerequisite that you are running MacOS and have installed:

In addition, the CloudFormation file assumes that you have a Route53 HostedZone in which we can add Records for your domain. It may work on other Linux platforms, but that I did not test.

 

git clone https://github.com/mvanholsteijn/coreos-container-platform-as-a-service
cd coreos-container-platform-as-a-service
./bin/create-stack.sh -d cargonauts.dutchdevops.net

...
{
"StackId": "arn:aws:cloudformation:us-west-2:233211978703:stack/cargonautsdutchdevopsnet/b4c802f0-d1ff-11e4-9c9c-5088484a585d"
}
INFO: create in progress. sleeping 15 seconds...
INFO: create in progress. sleeping 15 seconds...
INFO: create in progress. sleeping 15 seconds...
INFO: create in progress. sleeping 15 seconds...
INFO: create in progress. sleeping 15 seconds...
INFO: create in progress. sleeping 15 seconds...
INFO: create in progress. sleeping 15 seconds...
INFO: create in progress. sleeping 15 seconds...
INFO: create in progress. sleeping 15 seconds...
INFO: create in progress. sleeping 15 seconds...
INFO: create in progress. sleeping 15 seconds...
CoreOSServerAutoScale 54.185.55.139 10.230.14.39
CoreOSServerAutoScaleConsulServer 54.185.125.143 10.230.14.83
CoreOSServerAutoScaleConsulServer 54.203.141.124 10.221.12.109
CoreOSServerAutoScaleConsulServer 54.71.7.35 10.237.157.117

Now you are ready to look around. Use one of the external IP addresses to setup a tunnel for fleetctl.

export FLEETCTL_TUNNEL=54.203.141.124

fleetctl is the command line utility that allows you to manage the units that you deploy on CoreOS.


fleetctl list-machines
....
MACHINE		IP		METADATA
1cdadb87...	10.230.14.83	consul_role=server,region=us-west-2
2dde0d31...	10.221.12.109	consul_role=server,region=us-west-2
7f1f2982...	10.230.14.39	consul_role=client,region=us-west-2
f7257c36...	10.237.157.117	consul_role=server,region=us-west-2

will list all the machines in the platform with their private IP addresses and roles. As you can see we have tagged 3 machines for the consul server role and 1 machine for the consul client role. To see all the docker containers that have started on the individual machines, you can run the following script:

for machine in $(fleetctl list-machines -fields=machine -no-legend -full) ; do
   fleetctl ssh $machine docker ps
done
...
CONTAINER ID        IMAGE                                  COMMAND                CREATED             STATUS              PORTS                                                                                                                                                                                                                                      NAMES
ccd08e8b672f        cargonauts/consul-http-router:latest   "/consul-template -c   6 minutes ago       Up 6 minutes        10.221.12.109:80->80/tcp                                                                                                                                                                                                                   consul-http-router
c36a901902ca        progrium/registrator:latest            "/bin/registrator co   7 minutes ago       Up 7 minutes                                                                                                                                                                                                                                                   registrator
fd69ac671f2a        progrium/consul:latest                 "/bin/start -server    7 minutes ago       Up 7 minutes        172.17.42.1:53->53/udp, 10.221.12.109:8300->8300/tcp, 10.221.12.109:8301->8301/tcp, 10.221.12.109:8301->8301/udp, 10.221.12.109:8302->8302/udp, 10.221.12.109:8302->8302/tcp, 10.221.12.109:8400->8400/tcp, 10.221.12.109:8500->8500/tcp   consul
....

To inspect the Consul console, you need to first setup a tunnel to port 8500 on a server node in the cluster:

ssh-add stacks/cargonautsdutchdevopsnet/cargonauts.pem
ssh -A -L 8500:10.230.14.83:8500 core@54.185.125.143
open http://localhost:8500

Consul Console

You will now see that there are two services registered: consul and the consul-http-router. Consul registers itself and the http router was detected and registered by the Registrator on 4 machines.

Deploying an application

Now we can to deploy an application and we have a wonderful app to do so: the paas-monitor. It is a simple web application which continuously gets the status of a backend service and shows who is responding in a table.

In order to deploy this application we have to create a fleet unit file. which is basically a systemd unit file. It describes all the commands that it needs to execute for managing the life cycle of a unit. The paas-monitor unit file looks like this:


[Unit]
Description=paas-monitor

[Service]
Restart=always
RestartSec=15
ExecStartPre=-/usr/bin/docker kill paas-monitor-%i
ExecStartPre=-/usr/bin/docker rm paas-monitor-%i
ExecStart=/usr/bin/docker run --rm --name paas-monitor-%i --env SERVICE_NAME=paas-monitor --env SERVICE_TAGS=http -P --dns 172.17.42.1 --dns-search=service.consul mvanholsteijn/paas-monitor
ExecStop=/usr/bin/docker stop paas-monitor-%i

It states that this unit should always be restarted, with a 15 second interval. Before it starts, it stops and removes the previous container (ignoring any errors) and when it starts, it runs a docker container - non-detached. This allows systemd to detect that the process has stopped. Finally there is also a stop command.

The file also contains %i: This is a template file which means that more instances of the unit can be started.

In the environment settings of the Docker container, hints for the Registrator are set. The environment variable SERVICE_NAME indicates the name under which it would like to be registered in Consul and the SERVICE_TAGS indicates which tags should be attached to the service. These tags allow you to select the  'http' services in a domain or even from a single container.

If the container would expose more ports for instance 8080 en 8081 for  http and administrative traffic, you cloud specify environment variables.

SERVICE_8080_NAME=paas-monitor
SERVICE_8080_TAGS=http
SERVICE_8081_NAME=paas-monitor=admin
SERVICE_8081_TAGS=admin-http

Deploying the file goes in two stages: submitting the template file and starting an instance:

cd fleet-units/paas-monitor
fleetctl submit paas-monitor@.service
fleetctl start paas-monitor@1

Unit paas-monitor@1.service launched on 1cdadb87.../10.230.14.83

Now the fleet report that it is launched, but that does not mean it is running. In the background Docker has to pull the image which takes a while. You can monitor the progress using fleetctl status.

fleetctl status paas-monitor@1

paas-monitor@1.service - paas-monitor
   Loaded: loaded (/run/fleet/units/paas-monitor@1.service; linked-runtime; vendor preset: disabled)
   Active: active (running) since Tue 2015-03-24 09:01:10 UTC; 2min 48s ago
  Process: 3537 ExecStartPre=/usr/bin/docker rm paas-monitor-%i (code=exited, status=1/FAILURE)
  Process: 3529 ExecStartPre=/usr/bin/docker kill paas-monitor-%i (code=exited, status=1/FAILURE)
 Main PID: 3550 (docker)
   CGroup: /system.slice/system-paas\x2dmonitor.slice/paas-monitor@1.service
           └─3550 /usr/bin/docker run --rm --name paas-monitor-1 --env SERVICE_NAME=paas-monitor --env SERVICE_TAGS=http -P --dns 172.17.42.1 --dns-search=service.consul mvanholsteijn/paas-monitor

Mar 24 09:02:41 ip-10-230-14-83.us-west-2.compute.internal docker[3550]: 85071eb722b3: Pulling fs layer
Mar 24 09:02:43 ip-10-230-14-83.us-west-2.compute.internal docker[3550]: 85071eb722b3: Download complete
Mar 24 09:02:43 ip-10-230-14-83.us-west-2.compute.internal docker[3550]: 53a248434a87: Pulling metadata
Mar 24 09:02:44 ip-10-230-14-83.us-west-2.compute.internal docker[3550]: 53a248434a87: Pulling fs layer
Mar 24 09:02:46 ip-10-230-14-83.us-west-2.compute.internal docker[3550]: 53a248434a87: Download complete
Mar 24 09:02:46 ip-10-230-14-83.us-west-2.compute.internal docker[3550]: b0c42e8f4ac9: Pulling metadata
Mar 24 09:02:47 ip-10-230-14-83.us-west-2.compute.internal docker[3550]: b0c42e8f4ac9: Pulling fs layer
Mar 24 09:02:49 ip-10-230-14-83.us-west-2.compute.internal docker[3550]: b0c42e8f4ac9: Download complete
Mar 24 09:02:49 ip-10-230-14-83.us-west-2.compute.internal docker[3550]: b0c42e8f4ac9: Download complete
Mar 24 09:02:49 ip-10-230-14-83.us-west-2.compute.internal docker[3550]: Status: Downloaded newer image for mvanholsteijn/paas-monitor:latest

Once it is running you can navigate to http://paas-monitor.cargonauts.dutchdevops.net and click on start.

Screen Shot 2015-03-24 at 10.05.20

 

 

You can now add new instances and watch them appear in the paas-monitor! It definitively takes a while because the docker images have to be pulled from the registry before the can be started, but in the end it they will all appear!

fleetctl start paas-monitor@{2..10}
Unit paas-monitor@2.service launched on 2dde0d31.../10.221.12.109
Unit paas-monitor@4.service launched on f7257c36.../10.237.157.117
Unit paas-monitor@3.service launched on 7f1f2982.../10.230.14.39
Unit paas-monitor@6.service launched on 2dde0d31.../10.221.12.109
Unit paas-monitor@5.service launched on 1cdadb87.../10.230.14.83
Unit paas-monitor@8.service launched on f7257c36.../10.237.157.117
Unit paas-monitor@9.service launched on 1cdadb87.../10.230.14.83
Unit paas-monitor@7.service launched on 7f1f2982.../10.230.14.39
Unit paas-monitor@10.service launched on 2dde0d31.../10.221.12.109

to see all deployed units, use the list-units command

fleetctl list-units
...
UNIT MACHINE ACTIVE SUB
paas-monitor@1.service 94d16ece.../10.90.9.78 active running
paas-monitor@2.service f7257c36.../10.237.157.117 active running
paas-monitor@3.service 7f1f2982.../10.230.14.39 active running
paas-monitor@4.service 94d16ece.../10.90.9.78 active running
paas-monitor@5.service f7257c36.../10.237.157.117 active running
paas-monitor@6.service 7f1f2982.../10.230.14.39 active running
paas-monitor@7.service 7f1f2982.../10.230.14.39 active running
paas-monitor@8.service 94d16ece.../10.90.9.78 active running
paas-monitor@9.service f7257c36.../10.237.157.117 active running
How does it work?

Whenever there is a change to the consul service registry, the consul-http-router is notified, selects all http tagged services and generates a new nginx.conf. After the configuration is generated it is reloaded by nginx so that there is little impact on the current traffic.

The consul-http-router uses the Go template language to regenerate the config. It looks like this:

events {
    worker_connections 1024;
}

http {
{{range $index, $service := services}}{{range $tag, $services := service $service.Name | byTag}}{{if eq "http" $tag}}

    upstream {{$service.Name}} {
	least_conn;
	{{range $services}}server {{.Address}}:{{.Port}} max_fails=3 fail_timeout=60 weight=1;
	{{end}}
    }
{{end}}{{end}}{{end}}

{{range $index, $service := services}}{{range $tag, $services := service $service.Name | byTag}}{{if eq "http" $tag}}
    server {
	listen 		80;
	server_name 	{{$service.Name}}.*;

	location / {
	    proxy_pass 		http://{{$service.Name}};
	    proxy_set_header 	X-Forwarded-Host	$host;
	    proxy_set_header 	X-Forwarded-For 	$proxy_add_x_forwarded_for;
	    proxy_set_header 	Host 			$host;
	    proxy_set_header 	X-Real-IP 		$remote_addr;
	}
    }
{{end}}{{end}}{{end}}

    server {
	listen		80 default_server;

	location / {
	    root /www;
	    index index.html index.htm Default.htm;
	}
    }
}

It loops through all the services and selects all services tagged 'http' and creates a virtual host for servicename.* which sends all request to the registered upstream services. Using the following two commands you can see the current configuration file.

AMACHINE=$(fleetctl list-machines -fields=machine -no-legend -full | head -1)
fleetctl ssh $AMACHINE docker exec consul-http-router cat /etc/nginx/nginx.conf
...
events {
    worker_connections 1024;
}

http {

    upstream paas-monitor {
	least_conn;
	server 10.221.12.109:49154 max_fails=3 fail_timeout=60 weight=1;
	server 10.221.12.109:49153 max_fails=3 fail_timeout=60 weight=1;
	server 10.221.12.109:49155 max_fails=3 fail_timeout=60 weight=1;
	server 10.230.14.39:49153 max_fails=3 fail_timeout=60 weight=1;
	server 10.230.14.39:49154 max_fails=3 fail_timeout=60 weight=1;
	server 10.230.14.83:49153 max_fails=3 fail_timeout=60 weight=1;
	server 10.230.14.83:49154 max_fails=3 fail_timeout=60 weight=1;
	server 10.230.14.83:49155 max_fails=3 fail_timeout=60 weight=1;
	server 10.237.157.117:49153 max_fails=3 fail_timeout=60 weight=1;
	server 10.237.157.117:49154 max_fails=3 fail_timeout=60 weight=1;

    }

    server {
	listen 		80;
	server_name 	paas-monitor.*;

	location / {
	    proxy_pass 		http://paas-monitor;
	    proxy_set_header 	X-Forwarded-Host	$host;
	    proxy_set_header 	X-Forwarded-For 	$proxy_add_x_forwarded_for;
	    proxy_set_header 	Host 			$host;
	    proxy_set_header 	X-Real-IP 		$remote_addr;
	}
    }

    server {
	listen		80 default_server;

	location / {
	    root /www;
	    index index.html index.htm Default.htm;
	}
    }
}
[/code]

This also happens when you stop or kill and instance. Just stop an instance and watch your monitor respond.
1
fleetctl destroy paas-monitor@10
...
Destroyed paas-monitor@10.service
Killing a machine

Now let's be totally brave and stop and entire machine!

ssh core@$FLEETCTL_TUNNEL sudo shutdown -h now.
...
Connection to 54.203.141.124 closed by remote host.

Keep watching your paas-monitor. You will notice a slow down and also notice that a number of backend services are no longer responding. After a short while (1 or 2 minutes) you will see new instances appear in the list.
paas-monitor after restart
What happened is that Amazon AWS restarted a new instance into the cluster and all units that were running on the stopped node have been moved to running instances with only 6 HTTP errors!

Please note that CoreOS is not capable of automatically recovering from loss of a majority of the servers at the same time. In that case, manual recovery by operations is required.

Conclusion

CoreOS provides all the basic functionality to manage Docker containers and provided High Availability to your application with a minimum of fuss. Consul and Consul templates actually make it very easy to use custom components like NGiNX to implement dynamic service discovery.

Outlook

In the next blog we will be deploying an multi-tier application that uses Consul DNS to connect application parts to databases!

References

This blog is based on information, ideas and source code snippets from  https://coreos.com/docs/running-coreos/cloud-providers/ec2, http://cargonauts.io/mitchellh-auto-dc and https://github.com/justinclayton/coreos-and-consul-cluster-via-terraform

How to write an Amazon RDS service broker for Cloud Foundry

Mon, 03/23/2015 - 10:58

Cloud Foundry is a wonderful on-premise PaaS  that makes it very easy to build, deploy while providing scalability and high availability to your stateless applications. But Cloud Foundry is really a Application Platform Service and does not provide high availability and scalability for your data. Fortunately, there is Amazon RDS, which excels in providing this as a service.

In this blog I will show you how easy it is to build, install and use a Cloud Foundry Service Broker for Amazon RDS.  The broker was developed in Node.JS using the Restify framework and can be deployed as a normal Cloud Foundry application. Finally,  I will point you to a skeleton service broker which you can use as the basis for your own.

Cloud Foundry Service Broker Domain

Before I race of into the details of the implementation, I would like to introduce you into the Cloud Foundry lingo. If you are aware of the lingo, just skip to the paragraph 'AWS RDS Service Broker operations'.

Service - an external resource that can be used by an application. It can be a database, a messaging system or an external application.  Commonly provided services are mysql, postgres, redis and memcached.

Service Plan - a plan specify the quality of the service and governs the amount memory, disk space, nodes etc. provided with the service.

Service Catalog - a document containing all services and service plans of a service broker.

Service Broker - a program that is capable of creating services and providing the necessary information to applications to connect to the service.

Now a service broker can provide the following operations:

Describe Services - Show me all the services this broker can provide.

Create Service - Creating an instance of a service matching a specified plan. When the service is a database, it depends on the broker what this means: It may create an entire database server, or just a new database instance, or even just a database schema.   Cloud Foundry calls this 'provisioning a service instance'.

Binding a Service - providing a specific application with the necessary information to connect to an existing service.  When the service is a database, it provides the hostname, portname, database name, username and password. Depending on the service broker, the broker may even  create specific credentials for each  bind request/application. The Cloud Controller will store the returned credentials in a JSON document stored as an UNIX environment variable (VCAP_SERVICES).

Unbind service - depending on the service broker, undo what what done on the bind.

Destroy Service - Easy, just deleting what was created. Cloud Foundry calls this 'deprovisioning a service instance'.

In the next paragraph I will map these operations to Amazon AWS RDS services.

AWS RDS Service Broker operations

Any Service Broker has to implement a REST API of the Cloud Foundry specification.  To create the Amazon AWS RDS service broker, I had to implement four out of five methods:

  • describe services - returns available services and service plans
  • create service - call the createDBInstance operation and store generated credentials as tags in with the instance.
  • bind service - call the describeDBInstances operation and return the stored credentials.
  • delete service - just call the deleteDBInstance operation.

I implemented these REST calls using the Restify framework and the Amazon AWS RDS API for Javascript. the skeleton looks like this:

// get catalog
server.get('/v2/catalog', function(request, response, next) {
    response.send(config.catalog);
    next();
});

// create service
server.put('/v2/service_instances/:id', function(request, response, next) {
        response.send(501, { 'description' : 'create/provision service not implemented' });
        next();
    });

// delete service
server.del('/v2/service_instances/:id', function(req, response, next) {
        response.send(501, { 'description' : 'delete/unprovision service not implemented' });
        next();
    });

// bind service
server.put('/v2/service_instances/:instance_id/service_bindings/:id', function(req, response, next) {
        response.send(501, { 'description' : 'bind service not implemented' });
        next();
});

// unbind service
server.del('/v2/service_instances/:instance_id/service_bindings/:id', function(req, response, next) {
    response.send(501, { 'description' : 'unbind service not implemented' });
    next();
});

For the actual implementation of each operations on AWS RDS,  I would like to refer you to the source code of aws-rds-service-broker.js on github.com .

Design decisions

That does not look all too difficult does it?  Here are some of my design decisions:

Where do I store the credentials?

I store the credentials as tags on the  instance.  I wanted to create service broker that was completely stateless so that I could deploy it in Cloud Foundry itself. I did not want to be dependent on a complete database for a little bit of information. The tags seemed to fit the purpose.

Why does bind return the same credentials for every bind?

I wanted the bind service to be as simple as possible. I did not want to generate new user accounts and passwords, because if I did, I had even more state to maintain.  Even more, I found  that if I bind two applications to the same MySQL service, they could see each others data. So why bother creating users for binds? Finally, making the bind service simple, kept the unbind service even simpler because there is nothing to undo.

How to implement different service plans?

The createDBInstance operation of AWS RDS API operation, takes a JSON object as input parameter that is basically the equivalent of a plan. I just had to add an appropriate JSON record to the configuration file for each plan. See the description of the params parameter of the createDBInstance operation.

How do I create a AWS RDS service within 60 seconds?

Well, I don't.  The service broker API states that you have to create a service within the timeout of the cloud controller (which is 60 seconds), but RDS takes a whee bit more time. So the create request is initiated within seconds, but before you can bind an application to it may take a few minutes. Nothing I can do about that.

Why store the service broker credentials in environment variables?

I want the service broker to be configured upon deployment time. When the credentials are in the config file, you need to change the files of the application on each deployment.

Installation

In these instructions, I presume you have access to an AWS account and you have an installation of Cloud Foundry. I used  Stackato which is a Cloud Foundy implementation by ActiveState.  These instructions assume you are too!

  1. Create a AWS IAM user
    First create a AWS IAM user (cf-aws-service-broker) with at least the folllowing privileges
  2. Assign privileges to execute AWS RDS operations
    The newly created IAM user needs the privileges to create RDS databases. I used the following permissions:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [
             "rds:AddTagsToResource",
             "rds:CreateDBInstance",
             "rds:DeleteDBInstance",
             "rds:DescribeDBInstances",
             "rds:ListTagsForResource"
          ],
          "Resource": [
             "*"
          ]
        },
        {
          "Effect": "Allow",
          "Action": [
             "iam:GetUser"
          ],
          "Resource": [
              "*"
          ]
        }
      ]
    }
    
  3. Generate AWS access key and secret for the user 'cf-aws-service-broker'
  4. Create a Database Subnet
    Create a  database subnet 'stackato-db-subnet-group' in the AWS Region where you want to have the databases to be created.
  5. Check out the service broker
    git clone https://github.com/mvanholsteijn/aws-rds-service-broker
    cd aws-rds-service-broker
    
  6. Add all your parameters as environment variables to the manifest.yml
    applications:
       - name: aws-rds-service-broker
         mem: 256M
         disk: 1024M
         instances: 1
         env:
           AWS_ACCESS_KEY_ID: <fillin>
           AWS_SECRET_ACCESS_KEY: <fillin>
           AWS_REGION: <of db subnet group,eg eu-west-1>
           AWS_DB_SUBNET_GROUP: stackato-db-subnet-group
           SERVICE_BROKER_USERNAME: <fillin>
           SERVICE_BROKER_PASSWORD: <fillin>
         stackato:
           ignores:
             - .git
             - bin
             - node_modules
    
  7. Deploy the service broker
    stackato target <your-service-broker> --skip-ssl-validation
    stackato login
    push
    
  8. Install the service broker
    This script is a cunning implementation which create the service broker in Cloud Foundry and makes all the plans publicly available. In stackato we use the curl commands to achieve this. This script requires you to have installed jq, the wonderful JSON command line processor by Stephen Dolan.

    bin/install-service-broker.sh
    

Now you can use the service broker!

Using the Service Broker

Now we are ready to use the service broker.

  1. Deploy a sample application
    $ git clone https://github.com/mvanholsteijn/paas-monitor
    $ stackato push -n 
    
  2. Create a service for the mysql services
    $ stackato create-service
    1. filesystem 1.0, by core
    2. mysql
    3. mysql 5.5, by core
    4. postgres
    5. postgresql 9.1, by core
    6. redis 2.8, by core
    7. user-provided
    Which kind to provision:? 2
    1. 10gb: 10Gb HA MySQL database.
    2. default: Small 5Gb non-HA MySQL database
    Please select the service plan to enact:? 2
    Creating new service [mysql-844b1] ... OK
    
  3. Bind the service to the application
    stackato bind-service mysql-844b1 paas-monitor
      Binding mysql-844b1 to paas-monitor ... Error 10001: Service broker error: No endpoint set on the instance 'cfdb-3529e5764'. The instance is in state 'creating'. please retry a few minutes later (500)
    

    retry until the database is actually created (3-10 minutes on AWS)

    stackato bind-service mysql-844b1 paas-monitor
     Binding mysql-844d1 to paas-monitor ...
    Stopping Application [paas-monitor] ... OK
    Starting Application [paas-monitor] ...
    OK
    http://paas-monitor.<your-api-endpoint>/ deployed
    
  4. Check the environment of the application
    curl -s http://paas-monitor.<your-api-endpoint>/environment | jq .DATABASE_URL
    "mysql://root:e1zfMf7OXeq3@cfdb-3529e5764.c1ktcm2kjsfu.eu-central-1.rds.amazonaws.com:3306/mydb"
    

    As you can see the credentials for the newly created database has been inserted into the environment of the application.

Creating your own service broker

If you want to create your own service broker in Node.JS you may find the Skeleton Service Broker  a good starting point. It includes a number of utilities to test your broker in the bin directory.

  • catalog.sh - calls the catalog operation
  • provision.sh - calls the create operation
  • unprovision.sh - call the delete operation
  • bind.sh - calls the bind operation on a specified instance
  • unbind.sh - calls the unbind operation on a specified instance and bind id.
  • list.sh - calls the list all service instances operation
  • getenv.sh - gets the environment variables of an CF applications as sourceable output
  • install-service-broker.sh - installs the application and makes all plans public.
  • docurl.sh - calls the stackato CURL operation.

getenv.sh, install-service-broker.sh and provision.sh require jq to be installed.

Conclusion

As you can see, it is quite easy to create your own Cloud Foundry service broker!

Consul: the end of CNAMEs and PuppetDB?

Fri, 03/20/2015 - 16:24

Do you use CNAME records to identify services on your network? Do you feel life is impossible without PuppetDB and exported resources? In this blog I will explain how Consul can be used to replace both, and jump-start your transition towards container-based infrastructure in the process.

Microservices: coupling vs. autonomy

Wed, 03/18/2015 - 14:35

Microservices are the latest architectural style promising to resolve all issues we had we previous architectural styles. And just like other styles it has its own challenges. The challenge discussed in this blog is how to realise coupling between microservices while keeping the services as autonomous as possible. Four options will be described and a clear winner will be selected in the conclusion.

To me microservices are autonomous services that take full responsibility for one business capability. Full responsibility includes presentation, API, data storage and business logic. Autonomous is the keyword for me, by making the services autonomous the services can be changed with no or minimal impact on others. If services are autonomous, then operational issues in one service should have no impact on the functionality of other services. That all sounds like a good idea, but services will never be fully isolated islands. A service is virtually always dependent on data provided by another service. For example imagine a shopping cart microservice as part of a web shop, some other service must put items in the shopping cart and the shopping cart contents must be provided to yet other services to complete the order and get it shipped. The question now is how to realise these couplings while keeping maximum autonomy.  The goal of this blog post is to explain which pattern should be followed to couple microservices while retaining maximum autonomy.

rr-ps

I'm going to structure the patterns by 2 dimensions, the interaction pattern and the information exchanged using this pattern.

Interaction pattern: Request-Reply vs. Publish-Subscribe.

  • Request-Reply means that one service does a specific request for information (or to take some action). It then expects a response. The requesting service therefore needs to know what to aks and where to ask it. This could still be implemented asynchronously and of course your could put some abstraction in place such that the request service does not have to know the physical address of the other service, the point still remains that one service is explicitly asking a for specific information (or action to be taken) and functionally waiting for a response.
  • Publish-Subscribe: with this pattern a service registers itself as being interested in certain information, or being able to handle certain requests. The relevant information or requests will then be delivered to it and it can decide what to do with it. In this post we'll assume that there is some kind of middleware in place to take care of delivery of the published messages to the subscribed services.

Information exchanged: Events vs. Queries/Commands

  • Events are facts that cannot be argued about. For example, an order with number 123 is created. Events only state what has happened. They do not describe what should happen as a consequence of such an event.
  • Queries/Commands: Both convey what should happen. Queries are a specific request for information, commands are a specific request to the receiving service to take some action.

Putting these two dimensions in a matrix results into 4 options to realise couplings between microservices. So what are the advantages and disadvantages for each option? And which one is the best for reaching maximum autonomy?

In the description below we'll use 2 services to illustrate each pattern. The Order service which is responsible for managing orders and the Shipping service which is responsible for shipping stuff, for example the items included in an order. Services like these could be part of a webshop, which could then also contain services like a shopping cart, a product (search) service, etc.

1. Request-Reply with Events:rre

In this pattern one service asks a specific other service for events that took place (since the last time it asked). This implies strong dependency between these two services, the Shipping service must know which service to connect to for events related to orders. There is also a runtime dependency since the shipping service will only be able to ship new orders if the Order service is available.

Since the Shipping service only receives events it has to decide by itself when an order may be shipped based on information in these events. The Order service does not have to know anything about shipping, it simply provides events stating what happened to orders and leaves the responsibility to act on these events fully to the services requesting the events.

2. Request-Reply with Commands/Queries:

rrcIn this pattern the shipping Order service is going to request the Shipping service to ship an order. This implies strong coupling since the Order service is explicitly requesting a specific service to take care of the shipping and now the Order service must determine when an order is ready to be shipped. It is aware of the existence of a Shipping service and it even knows how to interact with it. If other factors not related to the order itself should be taken into account before shipping the order (e.g. credit status of the customer), then the order services should take this into account before requesting the shipping service to ship the order. Now the business process is baked into the architecture and therefore the architecture cannot be changed easily.

Again there is a runtime dependency since the Order service must ensure that the shipping request is successfully delivered to the Shipping service.

3. Publish-Subscribe with Events

pseIn Publish-Subscribe with Events the Shipping service registers itself as being interested in events related to Orders. After registering itself it will receive all events related to Orders without being aware what the source of the order events is. It is loosely coupled to the source of the Order events. The shipping service will need to retain a copy of the data received in the events such that is can conclude when an order is ready to be shipped. The Order service needs to have no knowledge about shipping. If multiple services provide order related events containing relevant data for the Shipping service then this is not recognisable by the Shipping service. If (one of) the service(s) providing order events is down, the Shipping service will not be aware, it just receives less events. The Shipping service will not be blocked by this.

4. Publish-Subscribe with Commands/Queries

pscIn Publish-Subscribe with Command/Queries the Shipping service registers itself as a service being able to ship stuff. It then receives all commands that want to get something shipped. The Shipping service does not have to be aware of the source of the Shipping commands and on the flip side the Order service is not aware of which service will take care of shipping. In that sense they are loosely coupled. However, the Order service is aware of the fact that orders must get shipped since it is sending out a ship command, this does make the coupling stronger.

Conclusion

Now that we have described the four options we go back to the original question, which pattern of the above 4 provides maximum autonomy?

Both Request-Reply patterns imply a runtime coupling between two services and that implies strong coupling. Both Command/Queries patterns imply that one service is aware of what another service should do (in the examples above the order service is aware that another service takes care of shipping) and that also implies strong coupling, but this time on functional level. That leaves one option: 3. Publish-Subscribe with Events. In this case both services are not aware of each others existence from both runtime and functional perspective. To me this is the clear winner for achieving maximum autonomy between services.

The next question pops up immediately, should you always couple services using Publish-Subscribe with events? If your only concern is maximum autonomy of services the answer would be yes, but, there are more factors that should be taken into the account. Always coupling using this pattern comes at a price, data is replicated, measures must be taken to deal with lost events, events driven architectures do add extra requirements on infrastructure, their might be extra latency, and more. In a next post I'll dive into these trade-offs and put things into perspective. For now remember that Publish-Subscribe with Events is a good bases for achieving autonomy of services.

Reducing the size of Docker Images

Wed, 03/18/2015 - 01:00

Using the basic Dockerfile syntax it is quite easy to create a fully functional Docker image. But if you just start adding commands to the Dockerfile the resulting image can become unnecessary big. This makes it harder to move the image around.

A few basic actions can reduce this significantly.

A product manager's perfection....

Tue, 03/03/2015 - 15:59

is achieved not there are no more features to add, but when there are no more features to take away. -- Antoine de Saint Exupéry

Not only was Antoine a brilliant writer, philosopher and pilot (well arguably since he crashed in the Mediterranean) but most of all he had a sharp mind about engineering, and I frequent quote him when I train product owners, product managers or in general product companies, about what makes a good product great. I also tell them their most important word in their vocabulary is "no". But the question then becomes, what is the criteria to say "yes"?

Typically we will look at the value of a feature and use different ways to prioritise and rank different features, break them down to their minimal proposition and get the team going. But what if you already have a product? and it’s rate of development is slowing. Features have been stacked on each other for years or even decades, and it’s become more and more difficult for the teams to wade through the proverbial swamp the code has become?

Too many features

Too many features

Turns out there are a number of criteria that you can follow:

1.) Working software, means it’s actually being used.

Though it may sound obvious, it’s not that easy to figure out. I was once part of a team that had to rebuild a rather large piece (read huge) of software for an air traffic control system. The managers ensured us that every functionality was a must keep, but the cost would have been prohibitory high.

One of the functions of the system was a record and replay mode for legal purposes. It basically registers all events throughout the system to serve as evidence that picture compilers would be accountable, or at least verifiable. One of our engineers had the bright insight that we could catalogue this data anonymously to figure out which functions were used and which were not.

Turned out the Standish Group was pretty right in their claims that 80% of the software is never used. Carving that out was met with fierce resistance, but it was easier to convince management (and users) with data, than with gut.

Another upside? we also knew what functions they were using a lot, and figured out how to improve those substantially.

2.) The cost of platforms

Yippee we got it running on a gazillion platforms! and boy do we have a reach, the marketing guys are going frenzy. Even if is the right choice at the time, you need to revisit this assumption all the time, and be prepared to clean up! This is often looked upon as a disinvestment: “we spent so much money on finally getting Blackberry working” or “It’s so cost effective that we can also offer it on platform XYZ”.

In the web world it’s often the number of browsers we support, but for larger software systems it is more often operating systems, database versions or even hardware. For one customer we would refurbish hardware systems, simply because it was cheaper than moving on to a more modern machine.

Key take away: If the platform is deprecated, remove it entirely from the codebase, it will bog the team down and you need their speed to respond to an ever increasing pace of new platforms.

3.) Old strategies

Every market and every product company pivots at least every few years (or dies). Focus shifts from consumer groups, to type of clients, type of delivery, shift to service or something else which is novel, hip and most of all profitable. Code bases tend to have a certain inertia. The larger the product, the bigger the inertia, and before you know it there a are tons of features in their that are far less valuable in the new situation. Cutting away perfectly good features is always painful but at some point you end up with the toolbars of Microsoft Word. Nice features, but complete overkill for the average user.

4.) The cause and effect trap

When people are faced with an issue they tend to focus on fixing the issue as it manifests itself. It's hard for our brain to think in problems, it tries to think in solutions. There is an excellent blog post here that provides a powerful method to overcome this phenomena by asking five times "why".

  • "We need the system to automatically export account details at the end of the day."
  • "Why?"
  • "So we can enter the records into the finance system"
  • "So it sounds like the real problem is getting the data into the finance system, not exporting it. Exporting just complicates the issue. Let's implement a data feed that automatically feeds the data to the finance system"

The hard job is to continuously keep evaluating your features, and remove those who are no longer valuable. It may seem like your throwing away good code, but ultimately it is not the largest product that survives, but the one that is able to adapt fast enough to the changing market. (Freely after Darwin)

 

Tutum, first impressions

Mon, 03/02/2015 - 16:40

Tutum is a platform to build, run and manage your docker containers. After shortly playing with it some time ago, I decided to take a bit more serious look at it this time. This article describes first impressions of using this platform, more specifically looking at it from a continuous delivery perspective.

The web interface

First thing to notice is the clean and simple web interface. Basically there are two main sections, which are services and nodes. The services view lists the services or containers you have deployed with status information and two buttons, one to stop (or start) and one to terminate the container, which means to throw it away.

You can drill down to a specific service, which provides you with more detailed information per service. The detail page provides you information about the containers, a slider to scale up and down, endpoints, logging, some metrics for monitoring and more .

Screen Shot 2015-02-23 at 22.49.33

The second view is a list of nodes. The list contains the VM's on which containers can be deployed. Again with two simple buttons to start/stop and to terminate the node. For each node it displays useful information about the current status, where it runs, and how many containers are deployed on it.

The node page also allows you to drill down to get more information on a specific node.  The screenshot below shows some metrics in fancy graphs for a node, which can potentially be used to impress your boss.

Screen Shot 2015-02-23 at 23.07.30

 

Creating a new node

You’ll need a node to deploy containers on it. In the node view you see two big green buttons. One states: “Launch new node cluster”. This will bring up a form with currently four popular providers Amazon, Digital Ocean, Microsoft Azure and Softlayer. If you have linked your account(s) in the settings you can select that provider from a dropdown box. It only takes a few clicks to get a node up and running. In fact you create a node cluster, which allows you to easily scale up or down by adding or removing nodes from the cluster.

You also have an option to ‘Bring you own node’. This allows you to add your own Ubuntu Linux systems as nodes to Tutum. You need to install an agent onto your system and open up a firewall port to make your node available to Tutum. Again very easy and straight forward.

Creating a new service

Once you have created a node, you maybe want to do something with it. Tumtum provides jumpstart images with popular types of services for storage, cacheing, queueing and more, providing for example MongoDB, Elasticsearch or Tomcat. Using a wizard it takes only four steps to get a particular service up and running.

Besides the jumpstart images that Tutum provides, you can also search public repositories for your image of choice. Eventually you would like to have your own images running your homegrown software. You can upload your image to a Tutum private registry. You can either pull it from Docker Hub or upload your local images directly to Tutum.

Automating

We all know real (wo)men (and automated processes) don’t use GUI’s. Tutum provides a nice and extensive command line interface for both Linux and Mac. I installed it using brew on my MBP and seconds later I was logged in and doing all kinds of cool stuff with the command line.

Screen Shot 2015-02-24 at 22.23.30

The cli is actually doing rest calls, so you can skip the cli all together and talk HTTP directly to a REST API, or if it pleases you, you can use the python API to create scripts that are actually maintainable. You can pretty much automate all management of your nodes, containers, and services using the API, which is a must have in this era of continuous everything.

A simple deployment example

So let's say we've build a new version of our software on our build server. Now we want to get this software deployed to do some integration testing, or if you feeling lucky just drop it straight into production.

build the docker image

tutum build -t test/myimage .

upload the image to Tutum registry

tutum image push <image_id>

create the service

tutum service create <image_id>

run it on a node

tutum service run -p <port> -n <name> <image_id>

That's it. Of course there are lots of options to play with, for example deployment strategy, set memory, auto starting etc. But the above steps are enough to get your image build, deployed and run. Most time I had to spend was waiting while uploading my image using the flaky-but-expensive hotel wifi.

Conclusion for now

Tutum is clean, simple and just works. I’m impressed with ease and speed you can get your containers up and running. It takes only minutes to get from zero to running using the jumpstart services, or even your own containers. Although they still call it beta, everything I did just worked, and without the need to read through lots of complex documentation. The web interface is self explanatory and the REST API or cli provides everything you need to integrate Tutum in your build pipeline, so you can get your new features in production with automation speed.

I'm wondering how challenging managing would be at a scale of hundreds of nodes and even more containers, when using the web interface. You'd need a meta-overview or aggregate view or something. But then again, you have a very nice API to

Exploring container platforms: StackEngine

Fri, 02/20/2015 - 15:51

Docker has been around for more than a year already, and there are a lot of container platforms popping up. In this series of blogposts I will explore these platforms and share some insights. This blogpost is about StackEngine.

TL;DR: StackEngine is (for now) just a nice frontend to the Docker binary. Nothing...

Try, Option or Either?

Wed, 02/18/2015 - 09:45

Scala has a lot of different options for handling and reporting errors, which can make it hard to decide which one is best suited for your situation. In Scala and functional programming languages it is common to make the errors that can occur explicit in the functions signature (i.e. return type), in contrast with the common practice in other programming languages where either special values are used (-1 for a failed lookup anyone?) or an exception is thrown.

Let's go through the main options you have as a Scala developer and see when to use what!

Option
A special type of error that can occur is the absence of some value. For example when looking up a value in a database or a List you can use the find method. When implementing this in Java the common solution (at least until Java 7) would be to return null when a value cannot be found or to throw some version of the NotFound exception. In Scala you will typically use the Option[T] type, returning Some(value) when the value is found and None when the value is absent.

So instead of having to look at the Javadoc or Scaladoc you only need to look at the type of the function to know how a missing value is represented. Moreover you don't need to litter your code with null checks or try/catch blocks.

Another use case is in parsing input data: user input, JSON, XML etc.. Instead of throwing an exception for invalid input you simply return a None to indicate parsing failed. The disadvantage of using Option for this situation is that you hide the type of error from the user of your function which, depending on the use-case, may or may not be a problem. If that information is important keep on reading the next sections.

An example that ensures that a name is non-empty:

def validateName(name: String): Option[String] = {
  if (name.isEmpty) None
  else Some(name)
}

You can use the validateName method in several ways in your code:

// Use a default value

 validateName(inputName).getOrElse("Default name")

// Apply some other function to the result
 validateName(inputName).map(_.toUpperCase)

// Combine with other validations, short-circuiting on the first error
// returning a new Option[Person]
 for {
   name <- validateName(inputName)
   age <- validateAge(inputAge)
 } yield Person(name, age)

Either
Option is nice to indicate failure, but if you need to provide some more information about the failure Option is not powerful enough. In that case Either[L,R] can be used. It has 2 implementations, Left and Right. Both can wrap a custom type, respectively type L and type R. By convention Right is right, so it contains the successful result and Left contains the error. Rewriting the validateName method to return an error message would give:

def validateName(name: String): Either[String, String] = {
 if (name.isEmpty) Left("Name cannot be empty")
 else Right(name)
 }

Similar to Option Either can be used in several ways. It differs from option because you always have to specify the so-called projection you want to work with via the left or right method:

// Apply some function to the successful result
validateName(inputName).right.map(_.toUpperCase)

// Combine with other validations, short-circuiting on the first error
// returning a new Either[Person]
for {
 name <- validateName(inputName).right
 age <- validateAge(inputAge).right
} yield Person(name, age)

// Handle both the Left and Right case
validateName(inputName).fold {
  error => s"Validation failed: $error",
  result => s"Validation succeeded: $result"
}

// And of course pattern matching also works
validateName(inputName) match {
  case Left(error) => s"Validation failed: $error",
  case Right(result) => s"Validation succeeded: $result"
}

// Convert to an option:
validateName(inputName).right.toOption

This projection is kind of clumsy and can lead to several convoluted compiler error messages in for expressions. See for example the excellent and in detail discussion of the Either type in the The Neophyte's Guide to Scala Part 7. Due to these issues several alternative implementations for a kind of Either have been created, most well known are the \/  type in Scalaz and the Or type in Scalactic. Both avoid the projection issues of the Scala Either and, at the same time, add additional functionality for aggregating multiple validation errors into a single result type.

Try

Try[T] is similar to Either. It also has 2 cases, Success[T] for the successful case and Failure[Throwable] for the failure case. The main difference thus is that the failure can only be of type Throwable. You can use it instead of a try/catch block to postpone exception handling. Another way to look at it is to consider it as Scala's version of checked exceptions. Success[T] wraps the result value of type T, while the Failure case can only contain an exception.

Compare these 2 methods that parse an integer:

// Throws a NumberFormatException when the integer cannot be parsed
def parseIntException(value: String): Int = value.toInt

// Catches the NumberFormatException and returns a Failure containing that exception
// OR returns a Success with the parsed integer value
def parseInt(value: String): Try[Int] = Try(value.toInt)

The first function needs documentation describing that an exception can be thrown. The second function describes in its signature what can be expected and requires the user of the function to take the failure case into account. Try is typically used when exceptions need to be propagated, if the exception is not needed prefer any of the other options discussed.

Try offers similar combinators as Option[T] and Either[L,R]:

// Apply some function to the successful result
parseInt(input).map(_ * 2)

// Combine with other validations, short-circuiting on the first Failure
// returning a new Try[Stats]
for {
  age <- parseInt(inputAge)
  height <- parseDouble(inputHeight)
} yield Stats(age, height)

// Use a default value
parseAge(inputAge).getOrElse(0)

// Convert to an option
parseAge(inputAge).toOption

// And of course pattern matching also works
parseAge(inputAge) match {
  case Failure(exception) => s"Validation failed: ${exception.message}",
  case Success(result) => s"Validation succeeded: $result"
}

Note that Try is not needed when working with Futures! Futures combine asynchronous processing with the Exception handling capabilities of Try! See also Try is free in the Future.

Exceptions
Since Scala runs on the JVM all low-level error handling is still based on exceptions. In Scala you rarely see usage of exceptions and they are typically only used as a last resort. More common is to convert them to any of the types mentioned above. Also note that, contrary to Java, all exceptions in Scala are unchecked. Throwing an exception will break your functional composition and probably result in unexpected behaviour for the caller of your function. So it should be reserved as a method of last resort, for when the other options don’t make sense.
If you are on the receiving end of the exceptions you need to catch them. In Scala syntax:

try {
  dangerousCode()
} catch {
  case e: Exception => println("Oops")
} finally {
  cleanup
}

What is often done wrong in Scala is that all Throwables are caught, including the Java system errors. You should never catch Errors because they indicate a critical system error like the OutOfMemoryError. So never do this:

try {
  dangerousCode()
} catch {
  case _ => println("Oops. Also caught OutOfMemoryError here!")
}

But instead do this:

import scala.util.control.NonFatal

try {
  dangerousCode()
} catch {
  case NonFatal(_) => println("Ooops. Much better, only the non fatal exceptions end up here.")
}

To convert exceptions to Option or Either types you can use the methods provided in scala.util.control.Exception (scaladoc):

import scala.util.control.Exception._

val i = 0
val result: Option[Int] = catching(classOf[ArithmeticException]) opt { 1 / i }
val result: Either[Throwable, Int] = catching(classOf[ArithmeticException]) either { 1 / i }

Finally remember you can always convert an exception into a Try as discussed in the previous section.

TDLR;

  • Option[T], use it when a value can be absent or some validation can fail and you don't care about the exact cause. Typically in data retrieval and validation logic.
  • Either[L,R], similar use case as Option but when you do need to provide some information about the error.
  • Try[T], use when something Exceptional can happen that you cannot handle in the function. This, in general, excludes validation logic and data retrieval failures but can be used to report unexpected failures.
  • Exceptions, use only as a last resort. When catching exceptions use the facility methods Scala provides and never catch { _ => }, instead use catch { NonFatal(_) => }

One final advice is to read through the Scaladoc for all the types discussed here. There are plenty of useful combinators available that are worth using.

Cancelling $http requests for fun and profit

Tue, 02/17/2015 - 09:11

At my current client, we have a large AngularJS application that is configured to show a full-page error whenever one of the $http requests ends up in error. This is implemented with an error interceptor as you would expect it to be. However, we’re also using some calculation-intense resources that happen to timeout once in a while. This combination is tricky: a user triggers a resource request when navigating to a certain page, navigates to a second page and suddenly ends up with an error message, as the request from the first page triggered a timeout error. This is a particular unpleasant side effect that I’m going to address in a generic way in this post.

There are of course multiple solutions to this problem. We could create a more resilient implementation in the backend that will not time out, but accepts retries. We could change the full-page error in something less ‘in your face’ (but you still would get some out-of-place error notification). For this post I’m going to fix it using a different approach: cancel any running requests when a user switches to a different location (the route part of the URL). This makes sense; your browser does the same when navigating from one page to another, so why not mimic this behaviour in your Angular app?

I’ve created a pretty verbose implementation to explain how to do this. At the end of this post, you’ll find a link to the code as a packaged bower component that can be dropped in any Angular 1.2+ app.

To cancel a running request, Angular does not offer that many options. Under the hood, there are some places where you can hook into, but that won’t be necessary. If we look at the $http usage documentation, the timeout property is mentioned and it accepts a promise to abort the underlying call. Perfect! If we set a promise on all created requests, and abort these at once when the user navigates to another page, we’re (probably) all set.

Let’s write an interceptor to plug in the promise in each request:

angular.module('angularCancelOnNavigateModule')
  .factory('HttpRequestTimeoutInterceptor', function ($q, HttpPendingRequestsService) {
    return {
      request: function (config) {
        config = config || {};
        if (config.timeout === undefined && !config.noCancelOnRouteChange) {
          config.timeout = HttpPendingRequestsService.newTimeout();
        }
        return config;
      }
    };
  });

The interceptor will not overwrite the timeout property when it is explicitly set. Also, if the noCancelOnRouteChange option is set to true, the request won’t be cancelled. For better separation of concerns, I’ve created a new service (the HttpPendingRequestsService) that hands out new timeout promises and stores references to them.

Let’s have a look at that pending requests service:

angular.module('angularCancelOnNavigateModule')
  .service('HttpPendingRequestsService', function ($q) {
    var cancelPromises = [];

    function newTimeout() {
      var cancelPromise = $q.defer();
      cancelPromises.push(cancelPromise);
      return cancelPromise.promise;
    }

    function cancelAll() {
      angular.forEach(cancelPromises, function (cancelPromise) {
        cancelPromise.promise.isGloballyCancelled = true;
        cancelPromise.resolve();
      });
      cancelPromises.length = 0;
    }

    return {
      newTimeout: newTimeout,
      cancelAll: cancelAll
    };
  });

So, this service creates new timeout promises that are stored in an array. When the cancelAll function is called, all timeout promises are resolved (thus aborting all requests that were configured with the promise) and the array is cleared. By setting the isGloballyCancelled property on the promise object, a response promise method can check whether it was cancelled or another exception has occurred. I’ll come back to that one in a minute.

Now we hook up the interceptor and call the cancelAll function at a sensible moment. There are several events triggered on the root scope that are good hook candidates. Eventually I settled for $locationChangeSuccess. It is only fired when the location change is a success (hence the name) and not cancelled by any other event listener.

angular
  .module('angularCancelOnNavigateModule', [])
  .config(function($httpProvider) {
    $httpProvider.interceptors.push('HttpRequestTimeoutInterceptor');
  })
  .run(function ($rootScope, HttpPendingRequestsService) {
    $rootScope.$on('$locationChangeSuccess', function (event, newUrl, oldUrl) {
      if (newUrl !== oldUrl) {
        HttpPendingRequestsService.cancelAll();
      }
    })
  });

When writing tests for this setup, I found that the $locationChangeSuccess event is triggered at the start of each test, even though the location did not change yet. To circumvent this situation, the function does a simple difference check.

Another problem popped up during testing. When the request is cancelled, Angular creates an empty error response, which in our case still triggers the full-page error. We need to catch and handle those error responses. We can simply add a responseError function in our existing interceptor. And remember the special isGloballyCancelled property we set on the promise? That’s the way to distinguish between cancelled and other responses.

We add the following function to the interceptor:

      responseError: function (response) {
        if (response.config.timeout.isGloballyCancelled) {
          return $q.defer().promise;
        }
        return $q.reject(response);
      }

The responseError function must return a promise that normally re-throws the response as rejected. However, that’s not what we want: neither a success nor a failure callback should be called. We simply return a never-resolving promise for all cancelled requests to get the behaviour we want.

That’s all there is to it! To make it easy to reuse this functionality in your Angular application, I’ve packaged this module as a bower component that is fully tested. You can check the module out on this GitHub repo.

When development resembles the ageing of wine

Mon, 02/16/2015 - 20:29

Once upon a time I was asked to help out a software product company.  The management briefing went something like this: "We need you to increase productivity, the guys in development seem to be unable to ship anything! and if they do ship something it's only a fraction of what we expected".

And so the story begins. Now there are many ways how we can improve the teams outcome and its output (the first matters more), but it always starts with observing what they do today and trying to figure out why.

It turns out that requests from the business were treated like a good wine, and were allowed to "age", in the oak barrel that was called Jira. Not so much to add flavour in the form of details, requirements, designs, non functional requirements or acceptance criteria, but mainly to see if the priority of this request would remain stable over a period of time.

In the days that followed I participated in the "Change Control Board" and saw what he meant. Management would change priorities on the fly and make swift decisions on requirements that would take weeks to implement. To stay in vinotology terms, wine was poured in and out the barrels at such a rate that it bore more resemblance to a blender than to the art of wine making.

Though management was happy to learn I had unearthed to root cause to their problem, they were less pleased to learn that they themselves were responsible.  The Agile world created the Product Owner role for this, and it turned out that this is hat, that can only be worn by a single person.

Once we funnelled all the requests through a single person, both responsible for the success of the product and for the development, we saw a big change. Not only did the business got a reliable sparring partner, but the development team had a single voice when it came to setting the priorities. Once the team starting finishing what they started we started shipping at regular intervals, with features that we all had committed to.

Of course it did not take away the dynamics of the business, but it allowed us to deliver, and become reliable in how and when we responded to change. Perhaps not the most aged wine, but enough to delight our customers and learn what we should put in our barrel for the next round.

 

UC Berkeley paper unveils what business leaders should learn from the Agile WikiSpeed case.

Wed, 02/04/2015 - 23:26

WIKISPEED_my_next_car_gets_100mpg_1024Last fall, I was approached by Tom van Norden from the UC Berkeley School of Information. A team of professor Morten T. Hansen, famous from his bestseller with Jim Collins “Great by Choice”, was investigating the magic around Joe Justice’ WikiSpeed. His agile team outperformed companies like Tesla during the XPrize a couple of years ago. Berkeley did research on the Agile and Lean practices being applied by the WikiSpeed team and it’s current status.

Joe and I work closely together how we can apply powerful IT-practices like Scrum and TDD (Test Driven Development) in a non-IT environment.  In parallel with WikiSpeed, together with Xebia coaches Jeroen Molenaar and Jasper Sonnevelt we're currently coaching the hydrogen and solar raceteams of Delft University, Forze and Nuna. In our coaching we use many of WikiSpeeds XM-practices, especially Scrum. The Berkeley paper confirmed and unveiled a number of lessons I would like to share in this blogpost.

Unification

An inspiring recognizable goal gives the team extreme focus, spirit and energy. It actually provides them with an identity.  When the goal fades or is not unique and inspiring anymore, the focus, spirit and energy goes down significantly.  The Forze team was also struggling with this.  The 2014-team handed over the P6 Hydrogen Race Car to the 2015- team in september 2014.  This car was still buggy and last but not least, with a goal which was not realistic and committed by the 2015-team.  Not endless bug fixing on the previous car to complete a goal from last years team.  Their new goal is building a new car, competing at a spectacular race in 2016 with conventional race cars. Therefor, we recently advised the team to start a parallel track and preparations how we should develop this new P7-car the agile way. As a result, the spirit of the team went up significantly and also provided them with extra energy to fix the bugs in the P6 car.

slider_Forze_VI_Valkenburg

Nimble Networks

Like every team, the presence of an inspiring leader is important.  They're  like the glue keeping the team together and keeping them motivated. For example, when Joe was travelling, the attendance at agile ceremonies and the Wikispeed teams velocity went down.   The same phenomena I’ve observed coaching The Forze team.  Team members are recruited on their technical skills, not for their leadership qualities. As a result, we had a great bunch of technicians but no leadership in the team. TU-Delft students are are currently not educated to recognize develop natural and/or acquired leadership skills. As a result, the 2015-team is vulnerable. For example, there is only one member with electrotechnical knowledge rightnow.  A skill which is crucial for engineering a hydrogen car.   The recommendation of Berkeley is very useful here. The team should recruit new members with strong personalities combined with leadership qualities and develop a more distributed network which makes the team less vulnerable.

Finally, Berkeley stated the WikiSpeed solution is an exceptional example how disciplined collaboration could do magic but can not be blindly copied to another existing business to do the same magic there.  Agile methods should be tailored to fit it’s goals. I cannot more agree on this.  An agile process should be agile within itself, fit for purpose with a very clear goal.

Download the full paper here.

 

Microservices versus the common SOA implementation

Wed, 02/04/2015 - 15:02

When I was first reading about MSA architectures (MSA) I had a hard time figuring out what was so different from a Service Oriented Architecture (SOA). Main reason for this is that the SOA paradigm leaves quite a bit of room for interpretation and various people have different interpretations. When Martin Fowler wrote about MSA almost a year ago he also mentioned that some people see it as “SOA done right”, he himself considers MSA a subset of SOA. So, what are the differences, if any?
At first glance there seem to be more than a few similarities. Both talk about services as the cornerstone for the architecture, services need to loosely coupled, technology agnostic and there’s probably a few more. What sets the Microservices architectural style apart is that it’s more clear on where the focus needs to be when designing your services. The name suggests that the services need to very small and probably very fine grained but in reality no size restrictions apply. Ironically, it’s the size (and focus) of services that commonly cripple SOA implementations.
The main characteristics of MSA are: autonomy, high cohesion and low coupling. Out of these characteristics autonomy is the defining one. Here autonomy not only means that it can run or function independently (that is what low coupling is about), it means that a service should be able to provide business value on its own. This principle ties together the focus on low coupling and high cohesion. Lower coupling allows the service to operate independently and high cohesion increases it’s ability to add value on its own.
What you often see in SOA implementations is the focus on reuse. This means that a certain function should only exist in one place or a single service should handle a certain function. Where things go sour is in how these “functions” are determined. This is where cohesion comes into play. Cohesion is the degree in which functionality in services belongs together. The highest cohesion is functional cohesion where the services contribute to a well-defined task or business capability. MSA strives towards functional (high) cohesion. The cohesion found in common SOA implementations is logical cohesion (one step up from the worst kind, coincidental cohesion). With logical cohesion the services are grouped around similar tasks, like the one service for one function mentioned above. This approach leads to finer grained services that focus on atomic tasks. Take for instance a “data service” that handles all communication with a database. It accepts messages in a common format and then uses that information to execute the desired action on the database. Benefit is that applications that use this service don’t need to worry about the tech behind said service and only need to provide messages in a common format. If the database is shared by multiple services they all use this service when communicating with the database. Now imagine what would happen if this service goes down (intentionally or not). Or what the impact is when this service needs to be modified. And this is assuming that this service is well designed and built so it can handle all the different requests thrown at it.
With MSA the goal should be high cohesion, this means grouping things around a business capability (so it can add business value). It also provides all aspects of that capability end-to-end, from data storage to user interface functions. So instead of having a “data service” you create a “customer service” or “shipping service”.
Another aspect that is relevant here is that with MSA one should strive for low coupling (or better yet, no coupling). Low coupling boils down to the dependency between services. With no coupling the services can function completely independent of each other. If two services have no coupling downtime of one of the two will have zero impact on the other. The higher the coupling, the higher the impact.
With SOA implementations based on logical cohesion the coupling tends to be high. Because services depend on other services to function. High coupling increases the impact of changes.
For MSA the goal is no coupling. But lowering the coupling does not mean they can’t or shouldn’t interact. Services can still interact with other services but they don’t depend on them. Another distinction to take into consideration is that these are more technical dependencies, not functional ones. Take for instance an order service and a shipping service. Both can operate independently. The shipping service will process all the orders it has received, regardless of the availability of an order service. If the order service is down it simply means no new orders will be created for the shipping service. So when the shipping service is done handling its last known order it stops. Vice versa, if the shipping service is down the order service will still be able to process orders. They just won’t be processed by the shipping service. When the shipping service comes back up it will process the backlog created by the order service.
How much the Microservices architectural style differs from SOA depends on who you talk to. If nothing else, MSA offers a clearer and better defined approach on setting up an architecture built around services. The key differentiator being the focus on autonomously adding value. There is a lot more to say on the subject but hopefully this gives some insight into how MSA differentiates itself from what you typically see with SOA implementations.

Apache Spark

Tue, 02/03/2015 - 21:22

Spark is the new kid on the block when it comes to big data processing. Hadoop is also an open-source cluster computing framework, but when compared to the community contribution, Spark is much more popular. How come? What is so special and innovative about Spark? Is it that Spark makes big data processing easy and much more accessible to the developer? Or is it because the performance is outstanding, especially compared to Hadoop?

This article gives an introduction to the advantages of current systems and compares these two big data systems in depth in order to explain the power of Spark.

Parallel computing

First we have to understand the differences between Hadoop and the conventional approach of parallel computing before we can compare the differences between Hadoop and Spark.

Distributed computing

In the case of parallel computing, all tasks have access to shared data to exchange information and perform their calculations. With distributed computing, each task has its own data. Information is exchanged by passing data between the tasks.

One of the main concepts of distributed computing is data locality, which reduces network traffic. Because of data locality, data is processed faster and more efficiently. There is no separate storage network or processing network.

Apache Hadoop delivers an ecosystem for distributed computing. One of the biggest advantages of this approach is that it is easily scalable, and one can build a cluster with commodity hardware. Hadoop is designed in the way that it can handle server hardware failures.

Stack

To understand the main differences between Spark and Hadoop we have to look at their stacks. Both stacks consist of several layers.

Stacks of Spark and Hadoop

The storage layer is responsible for a distributed file system that stores data on commodity machines, providing very high aggregate bandwidth across the cluster. Spark uses the Hadoop layer. That means one can use HDFS (the file system of Hadoop) or other storage systems supported by the Hadoop API. The following storage systems are supported by Hadoop: your local file system, Amazon S3, Cassandra, Hive, and HBase.

The computing layer is the programming model for large scale data processing. Hadoop and Spark differ significantly in this area. Hadoop uses a disk-based solution provided by a map/reduce model. A disk-based solution persists its temporary data on disk. Spark uses a memory-based solution with its Spark Core. Therefore, Spark is much faster. The differences in their computing models will be discussed in the next chapter.

Cluster managers are a bit different from the other components. They are responsible for managing computing resources and using them for the scheduling of users' applications. Hadoop uses its own cluster manager (YARN). Spark can run over a variety of cluster managers, including YARN, Apache Mesos, and a simple cluster manager called the Standalone Scheduler.

A concept unique to Spark is its high-level packages. They provide lots of functionalities that aren't available in Hadoop. One can see this layer also as a sort of abstraction layer, whereby code becomes much easier to understand and produce. These packages are

  • Spark SQL is Spark’s package for working with structured data. It allows querying data via SQL.
  • Spark Streaming enables processing live streams of data, for example, log files or a twitter feed.
  • MLlib is a package for machine learning functionality. A practical example of machine learning is spam filtering.
  • GraphX is a library that provides an API for manipulating graphs (like social networks) and performing graph-parallel computations.

The Spark Core is also written in Scala and supports Scala natively, which is a far better language than Java for implementing the kinds of transformations it supports. This results in less code which is therefore more intuitive.

Screen Shot 2015-01-24 at 16.41.37

(Source: infoq)

Computational model

The main difference between Hadoop and Spark is the computational model. A computational model is the algorithm and the set of allowable operations to process the data.

Hadoop uses the map/reduce. A map/reduce involves several steps.

Hadoop computational model: Map/Reduce

  • This data will be processed and indexed on a key/value base. This processing is done by the map task.
  • Then the data will be shuffled and sorted among the nodes, based on the keys. So that each node contains all values for a particular key.
  • The reduce task will do computations for all the values of the keys (for instance count the total values of a key) and write these to disk.

With the Hadoop computational model, there are only two functions available: map and reduce.
Note that doing distributed computing with Hadoop results, most of the time, in several iterations of map/reduce.  For each iteration, all data is persisted on disk. That is the reason it is called disk-based computing.

Spark uses RDD, also called Resilient Distributed Datasets. Working and processing data with RDD is much easier:

Spark computational model: RDD

  • Reading input data and thereby creating an RDD.
  • Transforming data to new RDDs (by each iteration and in memory). Each transformation of data results in a new RDD. For transforming RDD's there are lots of functions one can use, like map, flatMap, filter, distinct, sample, union, intersection, subtract, etc. With map/reduce you only have the map-function. (..)
  • Calling operations to compute a result (output data). Again there are lots of actions available, like collect, count, take, top, reduce, fold, etc instead of only reduce with the map/reduce.

Performance Hadoop vs Spark

Behind the scenes, Spark does a lot, like distribute the data across your cluster and parallelizing the operations. Note that doing distributed computing is memory-based computing. Data between transformations are not saved to disk. That's why Spark is so much faster.

Conclusion

All in all Spark is the next step in the area of big data processing, and has several advantages compared to Hadoop. The innovation of Spark lies in its computational model. The biggest advantages of Spark against Hadoop are

  • Its in-memory computing capabilities that deliver speed
  • Packages like streaming and machine-learning
  • Ease of development - one can program natively in Scala

How to make the sprint review meeting worth your while

Mon, 02/02/2015 - 16:44

My work allows me to meet a lot of different people, who actively pursue Scrum. Some of them question the value of doing a sprint review meeting at the end of every sprint. Stakeholders presumably do not “use” nor “see” their work directly, or the iterated product is not yet releasable.

Looks like this Scrum ritual is not suited for all. If you are a person questioning the value of a demo, then focus on your stakeholders and start to demo the delta instead of just your product. Here is a 3-step plan to make your sprint reviews worth your while.

Step 1: Provide context

Stakeholders are not interested in individual user stories. They want to see releasable working software they can use and work with. That is were the value delta is for them; in practical services and products they can use to make their lives a little bit better than before.
So the thing you should do in the sprint review is to explain the completed stories in the context of the iteration and how this iteration will add value to the upcoming product-release. Providing the context will make it easier for stakeholders to give your team the feedback it is looking for.

Step 2: Demo the delta

Search for the abstraction level on which the user is impacted by the story you have completed, even if this means combining stories to demo as a whole. This is especially useful if you are working in component-based teams.

It will enable you to explicitly illustrate the added value from a stakeholder perspective after the change. It’s not just about adding an input screen or expansion of variables. It’s about being able to do something different as a stakeholder.

Think about the possibilities given the new additions to the system. Maybe people can be more effective or more efficient saving time, money and energy. If possible try to explicitly show the effect of the delta on the stakeholder, for instance by measuring key variables in the before and after situation. Seeing the explicit effect of the changes will get your stakeholders fired up to provide your team with additional ideas and feedback for your next release.

Step 3: Ask for feedback

The goal of the sprint review is to generate feedback. Often, this won’t happen automatically and means you have to explicitly ask for it. Doing it wrong is to do the review without interruption and to conclude by asking: “any questions?” This will certainly not evoke people to participate in a group discussion as posing questions is, unfortunately, still seen by many as a sign of weakness.

To counter this group dynamic, you should be the one asking the questions to your audience. Example stakeholder focused questions to generate feedback might be; “what are your thoughts on improving this feature further?” Or “How would this fit in your day-to-day routine?”

By doing so, you will lower the barrier for others to start asking questions themselves. Another tip to turn around the question dynamic is to specifically target your question to a single stakeholder, getting the group out of the conversation.

Up until now these steps helped me a lot in improving my sprint review meetings. These 3 simple steps will let your key-stakeholders be the subject of your review meeting, maximizing the chance that they will provide you with valuable feedback to improve your product.

Run your iOS app without overwriting the App Store version

Fri, 01/30/2015 - 13:59

Sometimes when you're developing a new version of your iOS app, you'd like to run it on your iPhone or iPad and still be able to run the current version that is released on the App Store. Normally when you run your app from Xcode on your device, it will overwrite any existing version. If you then want to switch back to the version from the App Store, you'll have to delete the development version and download it again from the App Store.

In this blog post I'll describe how you can run a test version of your app next to your production version. This method also works when you have embedded extensions (like the Today Widget or WatchKit app) in your app or when you want to beta test your app with Apple's TestFlight.

There are two different ways to accomplish this. The first method is to create a new target within your project that runs the same app but with a different name and identifier. With iOS 8 it is now possible to embed extensions in your app. Since these extensions are embedded in the app target, this approach doesn't work when you have extensions and therefore I'll only describe the second method which is based on User-Defined Settings in the build configuration.

Before going into detail, here is a quick explanation of how this works. Xcode already creates two build configurations for us: Release and Debug. By adding some User-Defined Settings to the build configurations we can run the Debug configuration with a different Bundle identifier than the Release configuration. This essentially creates a separate app on your device, keeping the one from the App Store (which used the Release configuration) intact.

To beta distribution of the app built with debug configuration easier we'll create multiple schemes.

The basics

Follow these steps exactly to run a test version on your device next to your App Store version. These steps are based on Xcode 6.1.1 with an Apple Developer Account with admin privileges.

Click on your project in the Project Navigator and make sure the main target is selected. Under both the General and Info tabs you will find the Bundle identifier. If you change this name and run your app, it will create a new app next to the old one. Add -test to your current Bundle identifier so you get something like com.example.myapp-test. Xcode will probably tell you that you don't have a provisioning profile. Let Xcode fix this issue for you and it will create a new Development profile for you named something like iOSTeam Provisioning Profile: com.example.myapp-test.

So we're already able to run a test version of our app next to the App Store version. However, manually changing the Bundle identifier all the time isn't very practical. The Bundle identifier is part of the Info.plist and it's not derived from a Build configuration property like for example the Bundle name (which uses ${PRODUCT_NAME} from the build configuration by default). Therefore we can't simply specify a different Bundle identifier for a different Build configuration using an existing property. But, we can use a User-Defined Setting for this.

Go to the Build Settings tab of your project and click the + button to add a User-Defined Setting. Name the new setting BUNDLE_ID and give it the value of the test Bundle identifier you created earlier (com.example.myapp-test). Then click the small triangle in front of the setting to expand the setting. Remove the -test part from the Release configuration. You should end up with something like this:

Screen Shot 2015-01-30 at 11.29.43

Now go to the Info tab and change the value of the Bundle identifier to ${BUNDLE_ID}. In the General tab you will still see the correct Bundle Identifier, but if you click on it you see the text is slightly grayed out, which means it's taken from a Build configuration setting.

Screen Shot 2015-01-30 at 11.35.52

To test if this works, you can change edit the current Scheme and change the Build Configuration of the Run action from Debug to Release. Close the Scheme window and go to the General tab again to see that the Bundle Identifier has changed to the Release BUNDLE_ID. (If you still had the General tab open and don't see the change then switch to another tab and back; the panel will reload the identifier). Make sure to change the Build configuration back to Debug in your scheme afterwards.

When you now Archive an app before you release it to the App Store, it will use the correct identifier from the Release configuration and when you run the app from Xcode on your device, it will use the identifier for testing. That way it no longer overwrites your App Store version on your device.

App name

Both our App Store app and test app still have the same name. This makes it hard to know which one is which. To solve this, find the Product Name in the Build Settings and change the name for the Debug configuration to something else, like MyApp Test. You can even use another app icon for your test build. Just change the Asset Catalog App Icon Set Name for the Debug configuration.

Beta distribution

What if you want to distribute a version of the app for Beta testing (through TestFlight or something similar) to other people that also shouldn't overwrite their Apple Store version? Our Archive action is using the Release build configuration. We could change that manually to Debug to have the test Bundle identifier but then we would be getting all of the Debug settings in our app, which is not something we want. We need to create another Build configuration.

Go to the project settings of your project (so not the target settings). Click the + button under Configurations and duplicate the Release configuration. Call the new configuration AdHoc. (You might already have such a Build configuration for other reasons, in that case you can skip this step and use that one for the next steps.)

Now go to the Build Settings of your main target and change the AdHoc value of the User-Defined Setting BUNDLE_ID to the same as the Debug value. Do the same for the Product name is you changed that in the previous step.

We could already make a Beta test Archive now by manually changing the configuration of the Archive action to Debug. But it makes it easier if we create a new scheme to do this. So go to the Manage Schemes and click to + button at the bottom left to create a new scheme. Make sure that your main target is selected as Target and add " Test" to the name so you end up with something like "MyApp Test". Also check the Shared checkbox if you are sharing your schemes on your version control system.

Double click the new scheme to edit it and change the build configuration of the Archive action to AdHoc. Now you can Archive with the MyApp Test scheme selected to create a test version of your app that you can distribute to your testers without it overwriting their App Store version.

To avoid confusion about which build configuration is used by which scheme action, you should also change the configuration of the Profile action to AdHoc. And in your normal non-test scheme, you can change the build configuration of all actions to Release. This allows you to run the app in both modes of your device, which sometimes might be necessary, for example when you need to test push notifications that only work for the normal Bundle identifier.

Extensions

As mentioned in the intro, the main reason to use multiple schemes with different build configurations and User-Defined settings as opposed to creating multiple targets with different Bundle identifiers is because of Extensions, like the Today extension or a WatchKit extension. An extension can only be part of a single target.

Extensions make things even more complex since their Bundle identifier needs to be prefixed with the parent app's bundle identifier. And since we just made that one dynamic, we need to make the Bundle identifier of our extensions dynamic as well.

If you don't already have an existing extension, create a new one now. I've tested the approach described below with Today extensions and WatchKit extensions but it should work with any other extension as well.

The steps for getting a dynamic Bundle identifier for the extensions is very similar as the once for the main target so I won't go into too much detail here.

First open the Info tab of the new target that was created for the extension, e.g. MyAppToday. You'll see here that the Bundle display name is not derived from the PRODUCT_NAME. This is probably because the product name (which is derived from the target name) is something not very user friendly like MyAppToday and it is assumed that you will change it. In case of the Today extension, running a test version of the app next to the App Store version will also create 2 Today extensions in the Today view of the phone. To be able to differentiate between the two we'll also make the Bundle display name dynamic.

Change the value of it to ${BUNDLE_DISPLAY_NAME} and then add a User-Defined Setting for BUNDLE_DISPLAY_NAME with different names for Debug/AdHoc and Release.

You might have noticed that the Bundle identifier of the extension is already party dynamic, something like com.example.myapp.$(PRODUCT_NAME:rfc1034identifier). Change this to ${PARENT_BUNDLE_ID}.$(PRODUCT_NAME:rfc1034identifier) and add a User-Defined Setting for PARENT_BUNDLE_ID to your Build Settings. The values of the PARENT_BUNDLE_ID should be exactly the same as the ones you used in your main target, e.g. com.example.myapp for Release and com.example.myapp-test for Debug and AdHoc.

That's it, you can now Run and Archive your app with extensions who's Bundle identifier are prefixed with the parent's Bundle identifier.

App Groups entitlements

You might have an extension that shares UserDefaults data or Core Data stores with the main app. In that case you need to have matching App Groups entitlements in both your main app and extensions. Since we have dynamic Bundle identifiers that use different provisioning profiles, we also have to make our App Groups dynamic.

If you don't have App Groups entitlements (or other entitlements) yet, go to the Capabilities tab of your main target and switch on App Groups. Add an app group in the form off group.[bundle identifier], e.g. group.com.example.myapp. This will generate an entitlements file for your project (MyApp.entitlements) and set the Code Signing Entitlements of your Build Settings to something like MyApp/MyApp.entitlements. Locate the entitlements file in Finder and duplicate it. Change the name of the copy by replacing " Copy" with "Test" (MyAppTest.entitlements). Drag the copy into your project. You should now have two entitlement files in your project. Open the Test entitlements file in Xcode's Property List editor and add "-test" to the value of Item 0 under com.apple.security.application-groups to match it with the Bundle identifier we used for testing, e.g. com.example.myapp-test. Now go back to the Build Settings and change the Debug and AdHoc values of Code Signing Entitlements to match with the file name of the Test entitlements.

Repeat all these steps for the Extension target. Xcode will also generate the entitlements file in the extension folder. You should end up with two entitlements files for your main target and two entitlements files for every extension.

The Application Groups for testing need to be added to the provisioning profiles which Xcode will handle automatically for you. It might warn/ask you for this while building.

Conclusion

It might be quite some work to follow all these steps and to get everything to work, but if you use your normal iPhone for development and still want to use or show a stable version of your app at the same time, it's definitely worth doing. And your Beta testers will thank your for it as well.

Continuous Delivery across multiple providers

Wed, 01/21/2015 - 13:04

Over the last year three of the four customers I worked with had a similar challenge with their environments. In different variations they all had their environments setup across separate domains. Ranging from physically separated on-premise networks to having environments running across different hosting providers managed by different parties.

Regardless of the reasoning behind having these kinds of setup it’s a situation where the continuous delivery concepts really add value. The stereotypical problems that exist with manual deployment and testing practices tend to get amplified when they occur in seperated domains. Things get even worse when you add more parties to the mix (like external application developers). Sticking to doing things manually is a recipe for disaster unless you enjoy going through expansive procedures every time you want to do anything in any of ‘your’ environments. And if you’ve outsourced your environments to an external party you probably don’t want to have to (re)hire a lot of people just so you can communicate with your supplier.

So how can continuous delivery help in this situation? By automating your provisioning and deployments you make deploying your applications, if nothing else, repeatable and predictable. Regardless of where they need to run.

Just automating your deployments isn’t enough however, a big question that remains is who does what. A question that is most likely backed by a lengthy contract. Agreements between all the parties are meant to provide an answer to that very question. A development partner develops, an outsourcing partners handles the hardware, etc. But nobody handles bringing everything together...

The process of automating your steps already provides some help with this problem. In order to automate you need some form of agreement on how to provide input for the tooling. This at least clarifies what the various parties need to produce. It also clarifies what the result of a step will be. This removes some of the fuzziness out of the process. Things like is the JVM part of the OS or part of the middleware should become clear. But not everything is that clearcut. It’s parts of the puzzle where pieces actually come together that things turn gray. A single tool may need input from various parties. Here you need to resists the common knee-jerk reaction to shield said tool from other people with procedures and red tape. Instead provide access to those tools to all relevant parties and handle your separation of concerns through a reliable access mechanism. Even then there might be some parts that can’t be used by just a single party and in that case, *gasp*, people will need to work together. 

What this results in is an automated pipeline that will keep your environments configured properly and allow applications to be deployed onto them when needed, within minutes, wherever they may run.

MultiProviderCD

The diagram above shows how we set this up for one of our clients. Using XL Deploy, XL Release and Puppet as the automation tooling of choice.

In the first domain we have a git repository to which developers commit their code. A Jenkins build is used to extract this code, build it and package it in such a way that the deployment automation tool (XL Deploy) understands. It’s also kind enough to make that package directly available in XL Deploy. From there, XL Deploy is used to deploy the application not only to the target machines but also to another instance of XL Deploy running in the next domain, thus enabling that same package to be deployed there. This same mechanism can then be applied to the next domain. In this instance we ensure that the machines we are deploying to are consistent by using Puppet to manage them.

To round things off we use a single instance of XL Release to orchestrate the entire pipeline. A single release process is able to trigger the build in Jenkins and then deploy the application to all environments spread across the various domains.

A setup like this lowers deployment errors that come with doing manual deployments and cuts out all the hassle that comes with following the required procedures. As an added bonus your deployment pipeline also speeds up significantly. And we haven’t even talked about adding automated testing to the mix


Try is free in the Future

Mon, 01/19/2015 - 09:40

Lately I have seen a few developers consistently use a Try inside of a Future in order to make error handling easier. Here I will investigate if this has any merits or whether a Future on it’s own offers enough error handling.

If you look at the following code there is nothing that a Future can’t supply but a Try can:

import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.{Await, Future, Awaitable}
import scala.concurrent.duration._
import scala.util.{Try, Success, Failure}

object Main extends App {

  // Happy Future
  val happyFuture = Future {
    42
  }

  // Bleak future
  val bleakFuture = Future {
    throw new Exception("Mass extinction!")
  }

  // We would want to wrap the result into a hypothetical http response
  case class Response(code: Int, body: String)

  // This is the handler we will use
  def handle[T](future: Future[T]): Future[Response] = {
    future.map {
      case answer: Int => Response(200, answer.toString)
    } recover {
      case t: Throwable => Response(500, "Uh oh!")
    }
  }

  {
    val result = Await.result(handle(happyFuture), 1 second)
    println(result)
  }

  {
    val result = Await.result(handle(bleakFuture), 1 second)
    println(result)
  }
}

After giving it some thought the only situation where I could imagine Try being useful in conjunction with Future is when awaiting a Future but not wanting to deal with error situations yet. The times I would be awaiting a future are very few in practice though. But when needed something like this migth do:

object TryAwait {
  def result[T](awaitable: Awaitable[T], atMost: Duration): Try[T] = {
    Try {
      Await.result(awaitable, atMost)
    }
  }
}

If you do feel that using Trys inside of Futures adds value to your codebase please let me know.

Meteor

Sun, 01/18/2015 - 12:11

Did you ever use AngularJS as a frontend framework? Then you should definitely give Meteor a try! Where AngularJS is powerful just as a client framework, meteor is great as a full stack framework. That means you just write your code in one language as if there is no back- and frontend at all. In fact, you get an Android and IOS client for free. Meteor is so incredibly simple that you are productive from the beginning.

Where meteor kicks angular

One of the killing features of meteor is that you'll have a shared code base for frontend and backend. In the next code snippet, you'll see a file shared by backend and frontend:

// Collection shared and synchronized accross client, server and database
Todos = new Mongo.Collection('todos');

// Shared validation logic
validateTodo = function (todo) {
  var errors = {};
  if (!todo.title)
    todo.title = "Please fill in a title";
  return errors;
}

Can you imagine how neat the code above is?

Scan 04 Jan 2015 18.48-page4

With one codebase, you get the full stack!

  1. As in the backend file and in the frontend file one can access and query over the Todos collection. Meteor is responsible for syncing the todos. Even when another user adds an item, it will be visible to your client directly. Meteor accomplishes this by a client-side Mongo implementation (MiniMongo).
  2. One can write validation rules once! And they are executed both on the front-end and on the back-end. So you can give my user quick feedback about invalid input, but you can also guarantee that no invalid data is processed by the backend (when someone bypasses the client). And this is all without duplicated code.

Another killer feature of meteor is that it works out of the box, and it's easy to understand. Angular can be a bit overwhelming; you have to learn concepts like directives, services, factories, filters, isolated scopes, transclusion. For some initial scaffolding, you need to know grunt, yeoman, etcetera. With meteor every developer can create, run and deploy a full-stack application within minutes. After installing meteor you can run your app within seconds.

$ curl https://install.meteor.com | /bin/sh
$ meteor create dummyapp
$ cd dummyapp
$ meteor
$ meteor deploy dummyapp.meteor.com
Screen Shot 2015-01-04 at 19.49.08

Meteor dummy application

Another nice aspect of meteor is that it uses DDP, the Distributed Data Protocol. The team invented the protocol and they are heavily promoting it as "REST for websockets". It is a simple, pragmatic approach allowing it to be used to deliver live updates as data changes in the backend. Remember that this works all out of the box. This talk walks you through the concepts of it. But the result is that if you change data on a client it will be updated immediately on the other client.

And there is so much more, like...

  1. Latency Compensation. On the client, Meteor prefetches data and simulates models to make it look like server method calls return instantly.
  2. Meteor is open source and integrates with existing open source tools and frameworks.
  3. Services (like an official package server and a build farm).
  4. Command line tools
  5. Hot deploys
Where meteor falls short

Yes, meteor is not the answer to all your problems. The reason, I'll still choose angular above meteor for my professional work, is because the view framework of angular rocks. It makes it easy to structure your client code into testable units and connect them via dependency injection. With angular you can separate your HTML from your javascript. With meteor your javascript contains HTML elements, (because their UI-library is based on handlebars. That makes testing harder and large projects will become unstructured very quickly.

Another flaw emerges if your project already has a backend. When you choose meteor, you choose their full stack. That means: Mongo as database and Node.js as backend. Despite you are able to create powerful applications, Meteor doesn't allow you (easily) to change this stack.

Under the hood

Meteor consists out of several subprojects. In fact, it is a library of libraries. In fact, it is a stack; a standard set of core packages that are designed to work well together:

Components used by meteor

  1. To make meteor reactive they've included the components blaze and tracker. The blaze component is heavily based on handlebars.
  2. The DDP component is a new protocol, described by meteor, for modern client-server communication.
  3. Livequery and full stack database take all the pain of data synchronization between the database, backend and frontend away! You don't have to think about in anymore.
  4. The Isobuild package is a unified build system for browser, server and mobile.
Conclusion

If you want to create a website or a mobile app with a backend in no time, with getting lots of functionality out of the box, meteor is a very interesting tool. If you want to have more control or connect to an existing backend, then meteor is probably less suitable.

You can watch this presentation I recently gave, to go along with the article.