Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Architecture

Performance at Scale: SSDs, Silver Bullets, and Serialization

This is a guest post by Aaron Sullivan, Director & Principal Engineer at Rackspace.

We all love a silver bullet. Over the last few years, if I were to split the outcomes that I see with Rackspace customers who start using SSDs, the majority of the outcomes fall under two scenarios. The first scenario is a silver bullet—adding SSDs creates near-miraculous performance improvements. The second scenario (the most common) is typically a case of the bullet being fired at the wrong target—the results fall well short of expectations.

With the second scenario, the file system, data stores, and processes frequently become destabilized. These demoralizing results, however, usually occur when customers are trying to speed up the wrong thing.

A common phenomena at the heart of the disappointing SSD outcomes is serialization. Despite the fact that most servers have parallel processors (e.g. multicore, multi-socket), parallel memory systems (e.g. NUMA, multi-channel memory controllers), parallel storage systems (e.g. disk striping, NAND), and multithreaded software, transactions still must happen in a certain order. For some parts of your software and system design, processing goes step by step. Step 1. Then step 2. Then step 3. That’s serialization.

And just because some parts of your software or systems are inherently parallel doesn’t mean that those parts aren’t serialized behind other parts. Some systems may be capable of receiving and processing thousands of discrete requests simultaneously in one part, only to wait behind some other, serialized part. Software developers and systems architects have dealt with this in a variety of ways. Multi-tier web architecture was conceived, in part, to deal with this problem. More recently, database sharding also helps to address this problem. But making some parts of a system parallel doesn’t mean all parts are parallel. And some things, even after being explicitly enhanced (and marketed) for parallelism, still contain some elements of serialization.

How far back does this problem go? It has been with us in computing since the inception of parallel computing, going back at least as far as the 1960s(1). Over the last ten years, exceptional improvements have been made in parallel memory systems, distributed database and storage systems, multicore CPUs, GPUs, and so on. The improvements often follow after the introduction of a new innovation in hardware. So, with SSDs, we’re peering at the same basic problem through a new lens. And improvements haven’t just focused on improving the SSD, itself. Our whole conception of storage software stacks is changing, along with it. But, as you’ll see later, even if we made the whole storage stack thousands of times faster than it is today, serialization will still be a problem. We’re always finding ways to deal with the issue, but rarely can we make it go away.

Parallelization and Serialization
Categories: Architecture

How to verify Web Service State in a Protractor Test

Xebia Blog - Sat, 06/21/2014 - 08:24

Sometimes it can be useful to verify the state of a web service in an end-to-end test. In my case, I was testing a web application that was using a third-party Javascript plugin that logged page views to a Rest service. I wanted to have some tests to verify that all our web pages did include the plugin, and that it was communicating with the Rest service properly when a new page was opened.
Because the webpages were written with AngularJS, Protractor was our framework of choice for our end-to-end test suite. But how to verify web service state in Protractor?

My first draft of a Protractor test looked like this:

var businessMonitoring = require('../util/businessMonitoring.js');
var wizard = require('./../pageobjects/wizard.js');

describe('Business Monitoring', function() {
  it('should log the page name of every page view in the wizard', function() {
    wizard.open();
    expect(wizard.activeStepNumber.getText()).toBe('1');

    // We opened the first page of the wizard and we expect it to have been logged
    expect(businessMonitoring.getMonitoredPageName()).toBe('/wizard/introduction');

    wizard.nextButton.click();
    expect(wizard.completeStep.getAttribute('class')).toContain('active');
    // We have clicked the ‘next’ button so the ‘completed’ page has opened, this should have // been logged as well
    expect(businessMonitoring.getMonitoredPageName()).toBe('/wizard/completed');
  });
});

The next thing I had to write was the businessMonitoring.js script, which should somehow make contact with the Rest service to verify that the correct page name was logged.
First I needed a simple plugin to make http requests. I found the 'request' npm package , which provides a simple API to make a http request like this:

var request = require('request');

var executeRequest = function(method, url) {
  var defer = protractor.promise.defer();
  
  // method can be ‘GET’, ‘POST’ or ‘PUT’
  request({uri: url, method: method, json: true}, function(error, response, body) {

    if (error || response.statusCode >= 400) {
      defer.reject({
        error : error,
        message : response
      });
    } else {
      defer.fulfill(body);
    }
  });

  // Return a promise so the caller can wait on it for the request to complete
  return defer.promise;
};

Then I completed the businessmonitoring.js script with a method that gets the last request from the Rest service, using the request plugin.
It looked like this:

var businessMonitoring = exports; 

< .. The request wrapper with the executeRequest method is included here, left out here for brevity ..>

businessMonitoring.getMonitoredPageName = function () {

    var defer = protractor.promise.defer();

    executeRequest('GET', 'lastRequest')  // Calls the method which was defined above
      .then(function success(data) {
        defer.fulfill(data,.url);
      }, function error(e) {
        defer.reject('Error when calling BusinessMonitoring web service: ' + e);
      });

    return defer.promise;
 };

It just fires a GET request to the Rest service to see which page was logged. It is an Ajax call so the result is not immediately available, so a promise is returned instead.
But when I plugged the script into my Protractor test, it didn't work.
I could see that the requests to the Rest service were done, but they were done immediately before any of my end-to-end tests were executed.
How come?

The reason is that Protractor uses the WebdriverJS framework to handle its control flow. Statements like expect(), which we use in our Protractor tests, don't execute their assertions immediately, but instead they put their assertions on a queue. WebdriverJS first fills the queue with all assertions and other statements from the test, and then it executes the commands on the queue. Click here for a more extensive explanation of the WebdriverJs control flow.

That means that all statements in Protractor tests need to return promises, otherwise they will execute immediately when Protractor is only building its test queue. And that's what happened with my first implementation of the businessMonitoring mock.
The solution is to let the getMonitoredPageName return its promise within another promise, like this:

var businessMonitoring = exports; 

businessMonitoring.getMonitoredPageName = function () {
  // Return a promise that will execute the rest call,
  // so that the call is only done when the controlflow queue is executed.
  var deferredExecutor = protractor.promise.defer();

  deferredExecutor.then(function() {
    var defer = protractor.promise.defer();

    executeRequest('GET', 'lastRequest')
      .then(function success(data) {
        defer.fulfill(data.url);
      }, function error(e) {
        defer.reject('Error when calling BusinessMonitoring mock: ' + e);
      });

    return defer.promise;
  });

  return deferredExecutor;
};

Protractor takes care of resolving all the promises, so the code in my Protractor test did not have to be changed.

Concordion without the JUnit Code

Xebia Blog - Fri, 06/20/2014 - 20:58

Concordion is a framework to support Behaviour Driven Design. It is based on JUnit to run tests and HTML enriched with a little Concordion syntax to call fixture methods and make assertions on test outcome. I won't describe Concordion because it is well documented here: http://concordion.org/.
Instead I'll describe a small utility class I've created to avoid code duplication. Concordion requires a JUnit class for each test. The utility I'll describe below allows you to run all Concordion tests without having a utility class for each test.

In Concordion you specify test cases and expected outcomes in a HTML file. Each HTML file is accompanied by a Java class of the same name that has the @RunWith(ConcordionRunner.class annotation. This Java class is comparable with Fitnesse's fixture classes. Here you can create methods that are used in the HTML file to connect the test case to the business code.

In my particular use case the team ended up writing lots of mostly empty Java files. The system under test was processing XML message files, so all the test needed to do was call a single method to hand the XML to the business code and then validate results. Each Java class was basically the same, except for its name.

To avoid this duplication I created a class that uses the JavaAssist library to generate a Java class on the fly and run it as a JUnit test. You can find my code on Github:

git clone git@github.com:jvermeir/concordionDemo

ConcordionRunner generates a class using a template. The template can be really simple like in my example where FixtureTemplate extends MyFixture. MyFixture holds all fixture code to connect the test to the application under test. This is where we would put all fixture code necessary to call a service using a XML message. In the example there's just the single getGreeting() method.
HelloWorldAgain.html is the actual Concordion test. It shows the call to getGreeting(), which is a method of MyFixture.
The dependencies are like this:
FixtureTemplate extends MyFixture
YourTest.html uses Concordion
Example uses ConcordionRunner uses JUnitCore
The example in Example.java shows how to use ConcordionRunner to execute a test. This could easily be extended to recursively go through a directory and execute all tests found. Note that Example writes the generated class to a file. This may help in troubleshooting but isn't really necessary.
Now it would be nice to adapt the Eclipse plugin to allow you to right click the HTML file and run it as a Concordion test without adding a unit test.

Time is The Great Equalizer

Time really is the great equalizer.

I was reading an article by Dr. Donald E. Wemore, a time management specialist, and here’s what he had to say:

"Time is the great equalizer for all of us. We all have 24 hours in a day, 7 days a week, yielding 168 hours per week. Take out 56 hours for sleep (we do spend about a third of our week dead) and we are down to 112 hours to achieve all the results we desire. We cannot save time (ever have any time left over on a Sunday night that you could lop over to the next week?); it can only be spent. And there are only two ways to spend our time: we can spend it wisely, or, not so wisely."

Well put.

And what’s his recommendation to manage time better?

Work smarter, not harder.

In my experience, that’s the only approach that works.

If you find yourself struggling too much, there’s a good chance your time management strategies are off.

Don’t keep throwing time and energy at things if it’s not working.

Change your approach.

The fastest thing you can change in any situation is you.

You Might Also Like

7 Days of Agile Results: A Time Management Boot Camp for Productivity on Fire

10 Big Ideas from Getting Results the Agile Way

Productivity on Fire

Categories: Architecture, Programming

How I Explained My Job to My Grandmother

Well, she wasn’t my grandmother, but you get the idea.

I was trying to explain to somebody that’s in a very different job, what my job is all about.

Here’s what I said …

As far as my day job, I do complex, complicated things. 

I'm in the business of business transformation

I help large Enterprises get ahead in the world through technology and innovation.

I help Enterprises change their capabilities -- their business capabilities, technology capabilities, and people capabilities. 

It’s all about capabilities.

This involves figuring out their current state, their desired future state, the gaps between, the ROI of addressing the gaps, and then a Roadmap for making it happen.  

The interesting thing I've learned though is how much business transformation applies to personal transformation

It's all about figuring out your unique service and contribution to the world -- your unique value -- and then optimizing your strengths to realize your potential and do what you do best in a way that's valued -- where you can both generate value, as well as capture the value -- and lift the world to a better place.

Interestingly, she said she got it, it made sense, and it sounds inspiring.

What a relief.

Categories: Architecture, Programming

Introduction to Agile Presentation

I gave an Introduction to Agile talk recently:

Introduction to Agile Presentation (Slideshow)

I kept it focused on three simple things:

  1. What is Agile and the Agile Mindset (the Values and Principles)
  2. A rapid tour of the big 3 (Extreme Programming, Scrum, and Lean)
  3. Build a shared vocabulary and simple mental models so teams could hit the ground running and work more effectively with each other.

The big take away that I wanted the audience to have was that it’s a journey, but a very powerful one.

It’s a very healthy way to create an organization that embraces agility, empowers people, and ship stuff that customers care about.

In fact, the most powerful aspect of going Agile is that you create a learning organization.

The system and ecosystem you are in can quickly improve if you simply embrace change and focus on learning as a way of driving both continues improvement as well as growing capability.

So many things get a lot better over time, if they get a little better every day.

This was actually my first real talk on Agile and Agile development.  I’ve done lots of talks on Getting Results the Agile Way, and lots of other topics from security to performance to application architecture to team development and the Cloud.  But this was the first time a group asked me to share what I learned from Agile development in patterns & practices.

It was actually fun.

As part of the talk, I shared some of my favorite take aways and insights from the Agile World.

I’ll be sure to share some of these insights in future posts.

For now, if there is one thing to take away, it’s a reminder from David Anderson (Agile Management):

“Don’t do Agile.  Embrace agility.”

Way to be.

I shared my slides on SlideShare at Introduction to Agile Presentation (Slides) to help you learn the language, draw the visuals, and spread the word.

I’ll try to share more of my slides in the future, now that SlideShare seems to be a bit more robust.

You Might Also Like

Don’t Push Agile, Pull It

Extreme Programing at a Glance (Visual)

Scrum at a Glance (Visual)

Waterfall to Agile

What is Agile?

Categories: Architecture, Programming

Deploying a Node.js app to Docker on CoreOS using Deis

Xebia Blog - Wed, 06/18/2014 - 17:00

The world of on-premise private PaaSes is changing rapidly. A few years ago, we were building on on-premise private PaaSes based upon the existing infrastructure and using Puppet as an automation tool to quickly provision new application servers.  We provided a self-service portal where development teams could get any type of server in any type of environment running within minutes.  We created a virtual server for each application to keep it manageable, which of course is quite resource intensive.

Since June 9th, Docker has been declared production ready, so this opens  the option of provisioning light weight containers to the teams instead of full virtual machine. This will increase the speed of provisioning even further while reducing the cost of creating a platform and minimising resource consumption.

To illustrate how easy life is becoming, we are going to deploy an original CloudFoundry node.js application to Docker on a CoreOS cluster. This hands-on experiment is based on MacOS, Vagrant and VirtualBox.

Step 1. Installing  etcdctl and fleetctl

Before you start, you need to install etcdctl and fleetctl on your host.  etcd is a nifty distributed key-value store while fleet manages the deployment of  (Docker) services to a CoreOS cluster.
$ brew install go etcdctl
$ git clone https://github.com/coreos/fleet.git
$ cd fleet && ./build && mv bin/fleetctl /usr/local/bin

 

Step 2. Install the Deis Command Line Client

To control the PaaS you need to install the Deis command line client:

$ brew install python
$ sudo pip install deis

Step 3. Build the platform

Deis  provides all the facilities for building, deployment and managing applications.

$ git clone https://github.com/deis/deis.git
$ cd deis
$ vagrant up

$ ssh-add ~/.vagrant.d/insecure_private_key
$ export DOCKER_HOST=tcp://172.17.8.100:4243
$ export FLEETCTL_TUNNEL=172.17.8.100
$ make pull

Step 4. Start the platform

Now all is set to start the platform:

$ make run

After this run has completed, you can see that the 7 components in the Deis Architecture have been started using the list-units command: The builder, the cache, the controller, the database, the logger, the registry and the router. This architecture looks quite similar to the architecture of CloudFoundry.

$ fleetctl list-units

UNIT STATE LOAD ACTIVE SUB DESC MACHINE deis-builder.service launched loaded active running deis-builder 79874bde.../172.17.8.100 deis-cache.service launched loaded active running deis-cache 79874bde.../172.17.8.100 deis-controller.service launched loaded active running deis-controller 79874bde.../172.17.8.100 deis-database.service launched loaded active running deis-database 79874bde.../172.17.8.100 deis-logger.service launched loaded active running deis-logger 79874bde.../172.17.8.100 deis-registry.service launched loaded active running deis-registry 79874bde.../172.17.8.100 deis-router.1.service launched loaded active running deis-router 79874bde.../172.17.8.100

Alternatively, you can inspect the result by looking inside the virtual machine:

$ vagrant ssh -c "docker ps"

Now we have our platform running, we can start using it!

Step 5. Register a new user to Deis and add the public ssh key

$ deis register 172.17.8.100:8000 \
     --username=mark \
     --password=goddesses \
     --email=mark.van.holsteijn@..com
$ deis keys:add ~/.ssh/id_rsa.pub

Step 6. Create a Cluster

Create a application cluster under the domain 'dev.172.17.8.100.xip.io'.  The --hosts specifies all hosts in the cluster: the only available host  at this moment in the cluster is 172.17.8.100.

$ deis clusters:create dev  dev.172.17.8.100.xip.io \
        --hosts=172.17.8.100 \
        --auth=~/.vagrant.d/insecure_private_key

Step 7. Get the app

We created a simple but effective  node.js application that show you what happens when you scale or push a new version of the application.

$ git clone git@github.com:mvanholsteijn/sample_nodejs_cf.git
$ deis apps:create appmon --cluster=dev
$ deis config:set RELEASE=deis-v1
$ git push deis master

Step 8. open your application

Voila! Your application is running. Now click on start monitoring.

$ deis apps:open 

you should see something like this:

app-mon-1

Step 9. scaling your application

To see scaling in action,  type the following command:

$ deis ps:scale web=4

It will start 3 new containers which will show up in the list.

app-mon-4

 

Step 10. upgrading your application

Now make a change to the application, for instance change the message to 'Greetings from Deis release' and push your change:

$ git commit -a -m "Message changed"
$ git  push deis master

After a while you will see the following on your monitor!

app-mon-8

 

Step 11. Looking on CoreOS

You can use  fleetctl again to look at the new services that have been added to the platform!

 

$ fleetctl list-units

UNIT STATE LOAD ACTIVE SUB DESC MACHINE app-mon_v7.web.1-announce.service launched loaded active running app-mon_v7.web.1 announce 79874bde.../172.17.8.100 app-mon_v7.web.1-log.service launched loaded active running app-mon_v7.web.1 log 79874bde.../172.17.8.100 app-mon_v7.web.1.service launched loaded active running app-mon_v7.web.1 79874bde.../172.17.8.100 app-mon_v7.web.2-announce.service launched loaded active running app-mon_v7.web.2 announce 79874bde.../172.17.8.100 app-mon_v7.web.2-log.service launched loaded active running app-mon_v7.web.2 log 79874bde.../172.17.8.100 app-mon_v7.web.2.service launched loaded active running app-mon_v7.web.2 79874bde.../172.17.8.100 app-mon_v7.web.3-announce.service launched loaded active running app-mon_v7.web.3 announce 79874bde.../172.17.8.100 app-mon_v7.web.3-log.service launched loaded active running app-mon_v7.web.3 log 79874bde.../172.17.8.100 app-mon_v7.web.3.service launched loaded active running app-mon_v7.web.3 79874bde.../172.17.8.100 app-mon_v7.web.4-announce.service launched loaded active running app-mon_v7.web.4 announce 79874bde.../172.17.8.100 app-mon_v7.web.4-log.service launched loaded active running app-mon_v7.web.4 log 79874bde.../172.17.8.100 app-mon_v7.web.4.service launched loaded active running app-mon_v7.web.4 79874bde.../172.17.8.100 deis-builder.service launched loaded active running deis-builder 79874bde.../172.17.8.100 deis-cache.service launched loaded active running deis-cache 79874bde.../172.17.8.100 deis-controller.service launched loaded active running deis-controller 79874bde.../172.17.8.100 deis-database.service launched loaded active running deis-database 79874bde.../172.17.8.100 deis-logger.service launched loaded active running deis-logger 79874bde.../172.17.8.100 deis-registry.service launched loaded active running deis-registry 79874bde.../172.17.8.100 deis-router.1.service launched loaded active running deis-router 79874bde.../172.17.8.100

 

Conclusion

Deis is a very simple and easy to use way to create a PaaS using Docker and CoreOS. The node.js application we created, could be deployed using Deis without a single modification. We will be diving into Deis and CoreOS in more details in following posts!

One Change at a Time

Xebia Blog - Tue, 06/17/2014 - 08:22

One of the twelve Agile principles states "At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behaviour accordingly" [1]. Many Agile teams have retrospectives every two weeks that result in concrete actions that are executed in the next time period (sprint). There are many types of tuning and adjustments that a team can do. Examples are actions that improve the flow of work, automation of tasks, team coorporation.

Is it good habit for retrospectives to focus on the same type of improvement or should the team alter the type of improvements that will be done? In this blog I will look into the effect of multiple consecutive actions that affect the flow of work.

The simulation is inspired by the GetKanban game [2].

An Experiment

Ideally one would have a set-up for an experiment in which two exactly equivalent teams are compared. One team that would perform consecutive actions to improve the flow of work. The other team would just make one adjustment to improve the flow, and focus subsequent improvements on other areas than flow. At the same time the flow is measured and verified after a certain period of time whether the first or the second team has achieved better results in terms of flow.

Such an experiment is in practise (very) difficult to perform. In this blog I will study the same experiment by making use of simulations.

Simulation

For the purpose of simulation I consider a team consisting of three specialists: 1 designer, 1 developer, and 1 tester. The team uses a kanban process to achieve flow. See the picture below for the begin situation.


foto

In the simulation, at the beginning of each working day it is determined how much work will be completed by the team. During the simulation the average cycle time is measured. The initial Work in Progress (WiP) limits are set to 3 for each column and is indicated by the red '3's in the picture.

The average amount of work done by the 'team' and the average effort of one work item are such that on average it takes one card about 5,5 days to complete.

At the end of each work day, cards are pulled into the next columns (if allowed by the WiP limits). The policy is to always pull in as much work as allowed so the columns are maximally filled. Furthermore, the backlog is assumed to always have enough user stories ready to be pulled into the 'design' column. This very much resembles developing a new product when the backlog is filled with more than enough stories.

The system starts with a clean board and all column empty. After letting the system run for 75 simulated work days, we will trigger a policy change. Particularly the WiP limit for the 'design' is increased from '3' to '5'. After this policy change the system runs for another 100 work days.

From the chart showing the average cycle time we will be able to study the effect of WiP limit changing adjustments.

Note:

The simulation assumes a simple uniform distribution for the amount of work done by the team and the effort assigned to a work item. I assume this is OK for the purpose of this blog. A consequence of this, is that the result probably can't be scaled. For instance, the situation in which a column in the picture above is a single scrum team is not applicable since a more complex probability distribution should be used instead of the uniform distribution.

Results

The picture below shows the result of running the experiment.

 

retro_length

 

After the start it takes the system little over 40 work days to reach the stable state of an average cycle time of about 24(*) days. This is the cycle time one would expect. Remember, the 'ready' column has a size of '3' and the other columns get work done. So, one would expect a cycle time of around 4 times 5,5 which equals 22 days which is close to 24.

At day 75 the WiP limit is changed. As can be inferred from the picture, the cycle time starts to rise only at day 100 (takes about 1 cycle time (24 days) to respond). The new stable state is reached at day 145 with an average cycle time of around 30(**) days. It takes 70 days(!) to reach the new equilibrium.

The chart shows the following interesting features:

  1. It takes roughly 2 times the (new) average cycle time to reach the equilibrium state,
  2. The response time (when one begins to notice an effect of the policy change) is about the length of the average cycle time.

(*) One can calculate (using transition matrices) the theoretical average cycle time for this system to be 24 days.

(**) Similar, the theoretical average cycle time of the system after policy change is 31 days.

 

Conclusion

In this blog we have seen that when a team makes adjustments that affect the flow, the system needs time to get to its new stable state. Until this state has been reached any new tuning of the flow is questionable. Simulations show that the time it takes to reach the new stable state is about 2 times the average cycle time.

For scrum teams that have 2-week sprints, the system may need about 2 months before new tuning of flow is effective. Meanwhile, the team can very well focus on other improvements, e.g. retrospectives that focus on the team aspect or collaboration with the team's environment.

Moreover, don't expect to see any changes in measurements of e.g. cycle time within the time period of the average cycle time after making a flow affecting change.

To summarise, after making flow affecting changes (e.g. increasing or decreasing WiP limits):

  • Let the system run for at least the duration of the average cycle time so it has time to respond to the change,
  • After it responds, notice the effect of the change,
  • If the effect is positive, let the system run for another duration of the average cycle time, to get to the new stable state,
  • If the effect is negative, do something else, e.g. go back to the old state, and remember that the system needs to respond to this as well!
References

[1] Agile manifesto, http://agilemanifesto.org/principles.html

[2] GetKanban, http://getkanban.com

Migrating to XtraDB Cluster in EC2 at PagerDuty

This is a guest post by Doug Barth, a software generalist who has currently found himself doing operations work at PagerDuty. Prior to joining PagerDuty, he worked for Signal in Chicago and Orbitz, an online travel company.

A little over six months ago, PagerDuty switched its production MySQL database to XtraDB Cluster running inside EC2. Here's the story of why and how we made the change.

How the Database Looked Before

PagerDuty's MySQL database was a fairly typical deployment. We had:

  • A pair of Percona Servers writing data to a DRBD-backed volume.

  • EBS disks for both the primary and secondary servers backing the DRBD volume.

  • Two synchronous copies of the production database. (In the event of a failure of the primary, we wanted to be able to quickly switch to the secondary server without losing any data.)

  • A number of asynchronous replication slaves for disaster recovery, backups and accidental modification recovery.

Problems With the Old Setup
Categories: Architecture

Combining Salt with Docker

Xebia Blog - Sat, 06/14/2014 - 10:59

You could use Salt to build and run Docker containers but that is not how I use it here. This blogpost is about Docker containers that run Salt minions, which is just an experiment. The use case? Suppose you have several containers that run a particular piece of middleware, and this piece of middleware needs a security update, i.e. an OpenSSL hotfix. It is necessary to perform the update immediately.

 

The Dockerfile

In order to build a container you have to write down the container description in a file called Dockerfile. Here is the Dockerfile:

#-------
# Standard heading stuff

FROM centos
MAINTAINER No Reply noreply@xebia.com

# Do Salt install stuff and squeeze in a master.conf snippet that tells the minion
# to contact the master specified.

RUN rpm -Uvh http://ftp.linux.ncsu.edu/pub/epel/6/i386/epel-release-6-8.noarch.rpm
RUN yum install -y salt-minion --enablerepo=epel-testing
RUN [ ! -d /etc/salt/minion.d ] && mkdir /etc/salt/minion.d
ADD ./master.conf /etc/salt/minion.d/master.conf

# Run the Salt Minion and do not detach from the terminal.
# This is important because the Docker container will exit whenever
# the CMD process exits.

CMD /usr/bin/salt-minion
#-------

 

Build the image

Time to run the Dockerfile through docker. The command is:

$ docker build --rm=true -t salt-minion .

provided that you run this command in the directory where file Dockerfile and master.conf resides. Docker creates an image with tag ‘salt-minion’ and throws away all intermediate images after a successful build.

 

Run a container

The command is:

$ docker run -d salt-minion

and Docker returns:

aab154310ba6452ba2c686d15b1e3ca5fd85124d38c7935f1200d33b3a3e7ced

The Salt minion on the container is started and searches for a Salt master to connect to, defined by the configuration setting “master” in file /etc/salt/minion.d/master.conf. You might want to run the Salt master in “auto_accept” mode so that minion keys are accepted automatically. Docker assigns a container id to the running container. That is the magic key that docker reports as a result of the run command.

The following command shows the running container:

$ docker ps
CONTAINER ID        IMAGE                COMMAND                CREATED             STATUS              NAMES
273a6b77a8fa        salt-minion:latest   /bin/sh -c /etc/rc.l   3 seconds ago       Up 3 seconds        distracted_lumiere

 

Apply the hot fix
There you are: the Salt minion is controlled by your Salt master. Provided that you have a state module that contains the OpenSSL hot fix, you can now easily update all docker nodes to include the hotfix:

salt \* state.sls openssl-hotfix

That is all there is to it.

Setting up the hostname in Ubuntu

Agile Testing - Grig Gheorghiu - Fri, 06/13/2014 - 23:04
Most people recommend setting up the hostname on a Linux box so that:

1) running 'hostname' returns the short name (i.e. myhost)
2) running 'hostname -f' returns the FQDN (i.e. myhost.prod.example.com)
3) running 'hostname -d' returns the domain name (i.e prod.example.com)

After experimenting a bit and also finding this helpful Server Fault post, here's what we did to achieve this (we did it via Chef recipes, but it amounts to the same thing):

  • make sure we have the short name in /etc/hostname:
myhost

(also run 'hostname myhost' at the command line)
  • make sure we have the FQDN as the first entry associated with the IP of the server in /etc/hosts:
10.0.1.10 myhost.prod.example.com myhost myhost.prod
  • make sure we have the domain name set up as the search domain in /etc/resolv.conf:
search prod.example.com

Reboot the box when you're done to make sure all of this survives reboots.


Simplicity is the Ultimate Enabler

“Everything should be made as simple as possible, but not simpler.” – Albert Einstein

Simplicity is among the ultimate of pursuits.  It’s one of your most efficient and effective tools in your toolbox.  I used simplicity as the basis for my personal results system, Agile Results, and it’s served me well for more than a decade.

And yet, simplicity still isn’t treated as a first-class citizen.

It’s almost always considered as an afterthought.  And, by then, it’s too little, too late.

In the book, Simple Architectures for Complex Enterprises (Developer Best Practices), Roger Sessions shares his insights on how simplicity is the ultimate enabler to solving the myriad of problems that complexity creates.

Complex Problems Do Not Require Complex Solutions

Simplicity is the only thing that actually works.

Via Simple Architectures for Complex Enterprises (Developer Best Practices):

“So yes, the problems are complex.  But complex problems do not ipso facto require complex solutions.  Au contraire!  The basic premise of this book is that simple solutions are the only solutions to complex problems that work.  The complex solutions are simply too complex.”

Simplicity is the Antidote to Complexity

It sounds obvious but it’s true.  You can’t solve a problem with the same complexity that got you there in the first place.

Via Simple Architectures for Complex Enterprises (Developer Best Practices):

“The antidote to complexity is simplicity.  Replace complexity with simplicity and the battle is three-quarters over.  Of course, replacing complexity with simplicity is not necessarily simple.” 

Focus on Simplicity as a Core Value

If you want to achieve simplicity, you first have to explicitly focus on it as a core value.

Via Simple Architectures for Complex Enterprises (Developer Best Practices):

“The first thing you need to do to achieve simplicity is focus on simplicity as a core value.  We all discuss the importance of agility, security, performance, and reliability of IT systems as if they are the most important of all requirements.  We need to hold simplicity to as high a standard as we hold these other features.  We need to understand what makes architectures simple with as much critical reasoning as we use to understand what makes architectures secure, fast, or reliable.  In fact, I argue that simplicity is not merely the equal of these other characteristics; it is superior to all of them.  It is, in many ways, the ultimate enabler.”

A Security Example

Complex systems work against security.

Via Simple Architectures for Complex Enterprises (Developer Best Practices):

“Take security for example.  Simple systems that lack security can be made secure.  Complex systems that appear to be secure usually aren't.  And complex systems that aren't secure are virtually impossible to make either simple or secure.”

An Agility Example

Complexity works against agility, and agility is the key to lasting solutions.

Via Simple Architectures for Complex Enterprises (Developer Best Practices):

“Consider agility.  Simple systems, with their well-defined and minimal interactions, can be put together in new ways that were never considered when these systems were first created.  Complex systems can never used in an agile wayThey are simply too complex.  And, of course, retrospectively making them simple is almost impossible.”

Nobody Ever Considers Simplicity as a Critical Feature

And that’s the problem.

Via Simple Architectures for Complex Enterprises (Developer Best Practices):

“Yet, despite the importance of simplicity as a core system requirement, simplicity is almost never considered in architectural planning, development, or reviews.  I recently finished a number of speaking engagements.  I spoke to more than 100 enterprise architects, CIOs, and CTOs spanning many organizations and countries.  In each presentation, I asked if anybody in the audience had ever considered simplicity as a critical architectural feature for any projects on which they had participated. Not one person had. Ever.”

The Quest for Simplicity is Never Over

Simplicity is a quest.  And the quest is never over.  Simplicity is a ongoing pursuit and it’s a dynamic one.  It’s not a one time event, and it’s not static.

Via Simple Architectures for Complex Enterprises (Developer Best Practices):

“The quest for simplicity is never over.  Even systems that are designed from the beginning with simplicity in mind (rare systems, indeed!) will find themselves under a never-ending attack. A quick tweak for performance here, a quick tweak for interoperability there, and before you know it, a system that was beautifully simple two years ago has deteriorated into a mass of incomprehensibility.”

Simplicity is your ultimate sword for hacking your way through complexity … in work … in life … in systems … and ecosystems.

Wield it wisely.

You Might Also Like

10 Ways to Make Information More Useful

Reduce Complexity, Cost, and Time

Simple Enterprise Strategy

Categories: Architecture, Programming

Sponsored Post: Apple, Netflix, Salesforce, Blizzard Entertainment, Cloudant, CopperEgg, Logentries, Wargaming.net, PagerDuty, HelloSign, CrowdStrike, Gengo, ScaleOut Software, Couchbase, MongoDB, BlueStripe, AiScaler, Aerospike, LogicMonitor, AppDynamics

Who's Hiring?

  • Apple has multiple openings. Changing the world is all in a day's work at Apple. Imagine what you could do here.
    • Enterprise Software Engineer. Apple's Emerging Technology Services group provides a Java based SOA platform for various applications to interact with each other. The platform is designed to handle millions of messages a day with very low latency. We have an immediate opening for a talented Software Engineer in a highly visible team who is passionate about exploring emerging technologies to create elegant scalable solutions. Please apply here
    • Mobile Services Software Engineer. The Emerging Technologies/Mobile Services team is looking for a proactive and hardworking software engineer to join our team. The team is responsible for a variety of high quality and high performing mobile services and applications for internal use. Please apply here
    • Sr. Software Engineer-iOS Systems. Do you love building highly scalable, distributed web applications? Does the idea of performance tuning Java applications make your heart leap? If so, iOS Systems is looking for a highly motivated, detail-oriented, energetic individual with excellent written and oral skills who is not afraid to think outside the box and question assumptions. Please apply here
    • Senior Software Engineering Manager. As a Senior Software Engineering Manager on our team, you will be managing teams of very dedicated and talented engineering team. You will be responsible for managing the development of mobile point of sale system on iPod touch hardware. Please apply here.
    • Sr Software Engineer - Messaging Services. An exciting opportunity for a Software Engineer to join Apple's Messaging Services team. We build the cloud systems that power some of the busiest applications in the world, including iMessage, FaceTime and Apple Push Notifications. We handle hundreds of millions of active users using some of the most desirable devices on the planet and several Billion iMesssages/day, 40 billion push notifications/day, 16+ trillion push notifications sent to date. Please apply here.
    • Senior Software Engineer. Join Apple's Internet Applications Team, within the Information Systems and Technology group, as a Senior Software Engineer. Be involved in challenging and fast paced projects supporting Apple's business by delivering Java based IS Systems. Please apply here.
    • Sr Software Engineer. Join Apple's Internet Applications Team, within the Information Systems and Technology group, as a Senior Software Engineer. Be involved in challenging and fast paced projects supporting Apple's business by delivering Java based IS Systems. Please apply here.
    • Senior Payment Engineer. As a Software Engineer on our team, you will be responsible for working with cross-functional teams and developing Java server-based solutions to address business and technological needs. You will be helping design and build next generation retail solutions. You will be reviewing design and code developed by others on the team.You will build services and integrate with both internal as well as external services in a SOA environment. You will design and develop frameworks to be used by a large community of developers within the organization. Please apply here.

  • The Salesforce.com Core Application Performance team is seeking talented and experienced software engineers to focus on system reliability and performance, developing solutions for our multi-tenant, on-demand cloud computing system. Ideal candidate is an experienced Java developer, likes solving real-world performance and scalability challenges and building new monitoring and analysis solutions to make our site more reliable, scalable and responsive. Please apply here.

  • Sr. Software Engineer - Distributed Systems. Membership platform is at the heart of Netflix product, supporting functions like customer identity, personalized profiles, experimentation, and more. Are you someone who loves to dig into data structure optimization, parallel execution, smart throttling and graceful degradation, SYN and accept queue configuration, and the like? Is the availability vs consistency tradeoff in a distributed system too obvious to you? Do you have an opinion about asynchronous execution and distributed co-ordination? Come join us

  • Java Software Engineers of all levels, your time is now. Blizzard Entertainment is leveling up its Battle.net team, and we want to hear from experienced and enthusiastic engineers who want to join them on their quest to produce the most epic customer-facing site experiences possible. As a Battle.net engineer, you'll be responsible for creating new (and improving existing) applications in a high-load, high-availability environment. Please apply here.

  • Engine Programmer - C/C++. Wargaming|BigWorld is seeking Engine Programmers to join our team in Sydney, Australia. We offer a relocation package, Australian working visa & great salary + bonus. Your primary responsibility will be to work on our PC engine. Please apply here

  • Senior Engineer wanted for large scale, security oriented distributed systems application that offers career growth and independent work environment. Use your talents for good instead of getting people to click ads at CrowdStrike. Please apply here.

  • Ops Engineer - Are you passionate about scaling and automating cloud-based websites? Love Puppet and deployment scripts? Want to take advantage of both your sys-admin and DevOps skills? Join HelloSign as our second Ops Engineer and help us scale as we grow! Apply at http://www.hellosign.com/info/jobs

  • Human Translation Platform Gengo Seeks Sr. DevOps Engineer. Build an infrastructure capable of handling billions of translation jobs, worked on by tens of thousands of qualified translators. If you love playing with Amazon’s AWS, understand the challenges behind release-engineering, and get a kick out of analyzing log data for performance bottlenecks, please apply here.

  • UI EngineerAppDynamics, founded in 2008 and lead by proven innovators, is looking for a passionate UI Engineer to design, architect, and develop our their user interface using the latest web and mobile technologies. Make the impossible possible and the hard easy. Apply here.

  • Software Engineer - Infrastructure & Big DataAppDynamics, leader in next generation solutions for managing modern, distributed, and extremely complex applications residing in both the cloud and the data center, is looking for a Software Engineers (All-Levels) to design and develop scalable software written in Java and MySQL for backend component of software that manages application architectures. Apply here.
Fun and Informative Events

  • The Biggest MongoDB Event Ever Is On. Will You Be There? Join us in New York City June 23-25 for MongoDB World! The conference lineup includes Amazon CTO Werner Vogels and Cloudera Co-Founder Mike Olson for keynote addresses.  You’ll walk away with everything you need to know to build and manage modern applications. Register before April 4 to take advantage of super early bird pricing.

  • Upcoming Webinar: Practical Guide to SQL - NoSQL Migration. Avoid common pitfalls of NoSQL deployment with the best practices in this May 8 webinar with Anton Yazovskiy of Thumbtack Technology. He will review key questions to ask before migration, and differences in data modeling and architectural approaches. Finally, he will walk you through a typical application based on RDBMS and will migrate it to NoSQL step by step. Register for the webinar.
Cool Products and Services
  • The NoSQL "Family Tree" from Cloudant explains the NoSQL product landscape using an infographic. The highlights: NoSQL arose from "Big Data" (before it was called "Big Data"); NoSQL is not "One Size Fits All"; Vendor-driven versus Community-driven NoSQL.  Create a free Cloudant account and start the NoSQL goodness

  • Finally, log management and analytics can be easy, accessible across your team, and provide deep insights into data that matters across the business - from development, to operations, to business analytics. Create your free Logentries account here.

  • CopperEgg. Simple, Affordable Cloud Monitoring. CopperEgg gives you instant visibility into all of your cloud-hosted servers and applications. Cloud monitoring has never been so easy: lightweight, elastic monitoring; root cause analysis; data visualization; smart alerts. Get Started Now.

  • PagerDuty helps operations and DevOps engineers resolve problems as quickly as possible. By aggregating errors from all your IT monitoring tools, and allowing easy on-call scheduling that ensures the right alerts reach the right people, PagerDuty increases uptime and reduces on-call burnout—so that you only wake up when you have to. Thousands of companies rely on PagerDuty, including Netflix, Etsy, Heroku, and Github.

  • Aerospike Releases Client SDK for Node.js 0.10.x. This client makes it easy to build applications in Node.js that need to store and retrieve data from a high-performance Aerospike cluster. This version exposes Key-Value Store functionality - which is the core of Aerospike's In-Memory NoSQL Database. Platforms supported: CentOS 6, RHEL 6, Debian 6, Debian7, Mac OS X, Ubuntu 12.04. Write your first app: https://github.com/aerospike/aerospike-client-nodejs.

  • consistent: to be, or not to be. That’s the question. Is data in MongoDB consistent? It depends. It’s a trade-off between consistency and performance. However, does performance have to be sacrificed to maintain consistency? more.

  • Do Continuous MapReduce on Live Data? ScaleOut Software's hServer was built to let you hold your daily business data in-memory, update it as it changes, and concurrently run continuous MapReduce tasks on it to analyze it in real-time. We call this "stateful" analysis. To learn more check out hServer.

  • LogicMonitor is the cloud-based IT performance monitoring solution that enables companies to easily and cost-effectively monitor their entire IT infrastructure stack – storage, servers, networks, applications, virtualization, and websites – from the cloud. No firewall changes needed - start monitoring in only 15 minutes utilizing customized dashboards, trending graphs & alerting.

  • BlueStripe FactFinder Express is the ultimate tool for server monitoring and solving performance problems. Monitor URL response times and see if the problem is the application, a back-end call, a disk, or OS resources.

  • aiScaler, aiProtect, aiMobile Application Delivery Controller with integrated Dynamic Site Acceleration, Denial of Service Protection and Mobile Content Management. Cloud deployable. Free instant trial, no sign-up required.  http://aiscaler.com/

  • ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.

  • www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below. Please click to read more...

Categories: Architecture

Engineer Your Own Luck

“Chance favors the prepared mind.” - Louis Pasteur

Are you feeling lucky?

If you’re an engineer or a developer, you’ll appreciate the idea that you can design for luck, or stack the deck in your favor.

How do you do this?

As Harry Golden said, "The only thing that overcomes hard luck is hard work."

While I believe in hard work, I also believe in working smarter.

Luck is the same game.

It’s a game of skill.

And, success is a numbers game. 

You have to stay in long enough to get “lucky” over time.   That means finding a sustainable approach and using a sustainable system.  It means avoiding going all in without testing your assumptions and reducing the risk out of it.   It means taking emotion out of the equation, taking calculated risks, minimizing your exposure, and focusing on skills.

That’s why Agile methods and Lean approaches can help you outpace your failures.

Because they are test-driven and focus on continuous learning.

And because they focus on capacity and capability versus burnout or blowup.

So if you aren’t feeling the type of luck you’d like to see more of in your life, go back to the basics, and design for it.

They funny thing about luck is that the less you depend on it, the more of it you get.

BTW – Agile Results and Getting Results the Agile Way continue to help people “get lucky.”  Recently, I heard a story where a social worker shared Getting Results the Agile Way with two girls living off the streets.  They are off drugs now, have jobs, and are buying homes.   I’m not doing the story justice, but it’s great to hear about people turning their lives around and these kinds of life changing things that a simple system for meaningful results can help achieve.

It’s not luck. 

It’s desire, determination, and effective strategies applied in a sustainable way.

The Agile way.

Categories: Architecture, Programming

Extreme Programming (XP) at a Glance (Visual)

“Each of the practices still has the same weaknesses as before, but what if those weaknesses were now made up for by the strengths of other practices? We might be able to get away with doing things simply." – Kent Beck

Extreme Programming (XP) has been around a while, but not everybody knows “what it looks like.”

What does it look like when you step back and take the balcony view and observe the flow of things?

It might look a bit like this …

image

I put this view together to help some folks get an idea of what the “system” sort of looks like.  It didn’t need to be perfect, but they needed at least a baseline idea or skeleton so they could better understand how the various practices fit together.

The beauty is that once you put a simple picture up on the whiteboard, then you can have real discussions with the team about where things can be improved.  Once again, a picture is worth 1,000 words.

For reference, here are the 12 Practices of Extreme Programming

  1. Coding Standards
  2. Collective Ownership
  3. Continuous Integration
  4. On-Site Customer
  5. Pair Programming
  6. Planning Game
  7. Refactoring
  8. Short Releases
  9. Simple Design
  10. Sustainable Pace (40 Hour Week)
  11. System Metaphor
  12. Test-Driven Development

The main idea here is to get simple visuals in your mind that you can easily draw on a whiteboard, and know the names of the various activities and artifacts. 

If you nail this down, it helps you build a simple vocabulary. 

This vocabulary will help you get others on board faster, and it will help you expand your own toolbox at a rapid rate, and you’ll soon find yourself composing new methods and creating interesting innovations in your process that will help you do things better, faster, and cheaper … on Cloud time.

You Might Also Like

What is Agile?

4 Circles of XP (Extreme Programming)

Extreme Programming at a Glance

Roles on Agile Teams

Scrum at a Glance (Visual)

Waterfall to Agile

Categories: Architecture, Programming

State of Web Development 2014

Xebia Blog - Fri, 06/06/2014 - 13:10

On the 5th of June, we (Lammert Westerhoff and Freek Wielstra) presented the State of Web Development 2014 at our yearly conference, XebiCon on the SS Rotterdam. It was a great success and we had some great feedback. We'd like to share the same presentation we gave there with everyone who couldn't make it to the conference in this blog post.

We started the presentation by showing an overwhelming tag cloud:

WebXebiCon.006

 

 

When you’re new to web development or when you haven’t done any web development for a long time, you’re likely to get lost in this jungle of web technologies. Don’t worry if you are overwhelmed with this, because that’s intentional. This tag cloud contains many languages and web frameworks that modern web applications consist of. But it also contains a number of development tools that are used during development. The goal of this post is to guide you through this jungle by explaining to you which technologies you can use to build a modern web application and which tools you need during the full development lifecycle.

The get a better understanding of which technologies and tools we should use, we should first look at bit at the evolution of a number of important technologies. Then we will understand why web development is quite complex these days and why we need tooling. Once that's clear, we can have a look at the available tooling.

The evolution of web applications

A traditional website consists of HTML, CSS and sometimes a little javascript. The HTML tells the browser the content that needs to be displayed. The CSS tells it how that content should be displayed and what it needs to look like. And JavaScript was mainly used to add small behaviour, like displaying a popup or showing an image carousel.

WebXebiCon.016

Of course, these static websites were hard to maintain. A change of content needed a change in the HTML file which then needed to be uploaded to the web server with FTP.

Luckily, there were a couple of technologies that solved this problem for us. Using a technology like PHP, JSP or ASP we could dynamically generate the websites at runtime, which were then sent to the browser. These websites usually were connected to a database. The database then stored the content instead of storing it directly in the HTML files. That way we could make changes to the website by changing the content stored in the database. This allowed us to already build more complex web applications that also stored user created content, like blogs or even entire web stores.

While we had the server side technologies that we needed to generate our websites, we needed new innovations on the client side as well. Especially because we entered the era of mobile devices, we needed new browser capabilities, like location services, browser storage and touch events. Therefore, new HTML and CSS standards were being developed that contain many features that make the web browsers much more powerful and richer, which allows us developers to build richer web applications for both mobile and desktop.

To make this a lot easier for everyone, Twitter released Bootstrap, which handles all the responsiveness automatically. It’s still being used by many responsive websites today.

With all the new powerful features of CSS3, the style sheets also became more complex and tedious. To overcome that problem, technologies like Sass and Less were developed. These are CSS preprocessors that offer new language features that make some of the more tedious styling easier to do. Both will produce CSS and both solve the same problems. Which one you should choose depends on your circumstances and personal flavour.

WebXebiCon.021

Users don’t only want web sites to work and look good on every device, they also want web sites to be more interactive. They want the web site to respond to their actions. For example, zoom into an image or drag an item into a shopping basket. Even though all this was already possible in JavaScript for a long time, jQuery made such things a lot easier. jQuery is a JavaScript library that makes Dom manipulation easier, especially for multiple browsers. With the popularity of jQuery, the web became overwhelmed with jQuery plugins that did all sort of things. Below we see a screenshot of the plugin registry. And of course a library for mobile had to be released.

WebXebiCon.026

One of the very important features of jQuery is that it makes it easy to make AJAX requests. AJAX stands for Asynchronous Javascript And XML. It allows the web browser to retrieve data from the backend without leaving the page or doing a page refresh. And only the data is transferred from the backend to the client, instead of entire generated pages. Once the data is retrieved, it is displayed dynamically using the DOM manipulation features of jQuery. And even though XML is in the name, nowadays JSON is often used instead of XML, which is a more lightweight and more flexible data format.

AJAX requests are also at the core of single web page applications (SPA).

WebXebiCon.027

These web applications load all the HTML, CSS and Javascript resources at once. Once the page is loaded, you won’t have any other pages loading or refreshing anymore. All the rendering and navigation is done at the client side without leaving the actual page. Of course the user might perceive that he’s navigating between different pages, but this is all handled on the client by Javascript.

The single page web applications are becoming more and more popular since they usually provide a better user experience. They rely heavily on the AJAX requests that we saw earlier to retrieve data from a server. The data that is retrieved by these request are one of the only types of communication between the client and server once the page is initially loaded.

Since with SPA’s, we move a lot of logic and complexity from the server to the client, it really comes in handy to use the framework that handles most of the complexity. Three of the most popular frameworks are Backbone.js, Ember.js and AngularJS. Which one you should choose depends a lot on your requirements, what you’re used to and personal preference. Though it is clear that AngularJS is the most popular one at this moment, has the most activity and the largest community. People seem to be moving from Backbone to Angular and Ember. We also use Angular ourselves and we’re happy with it. The Google Angular team is working on the next major version which should even be a big improvement. That’s why we think Angular is a good and safe choice.

From what we've seen in the previous paragraphs, it’s obvious that more and more functionality and complexity is moved from the server side to the client side. And by using frameworks and libraries such as Bootstrap, jQuery and Angular, we also increase our client side dependencies. This also means that our development focus and efforts are moving from the server to the client. It’s time to take front end development serious and to do it right. So let’s have a look at what this means for our development process.

Web Application Development

As we saw earlier, we start simple by creating a couple of HTML, CSS and Javascript files. Then we would like to take advantage of the power of Less or Sass for our stylesheets. And since we’re making a rich or single page web application, we’ll be making a lot of javascript files. And perhaps some CoffeeScript or Dart instead of just Javascript. For those who don’t know coffee script or Dart, these are alternative languages that both compile to Javascript. CoffeeScript is almost the same language as javascript but just with a slightly different syntax. Dart however is quite different is definitely worth checking out when you have a bit of spare time.

Of course we want to use many of the great third party libraries that we discussed earlier. And since those libraries often depend on other libraries, we’ll end up with a lot of external libraries.

WebXebiCon.033

With this picture we kind of get an idea which resources we have for our web application. But to just create these files, download the libraries and upload them to our web server is not enough. First of all, web browsers don’t understand Less, Sass, CoffeeScript or Dart. We need to produce something that the browser does understand. For Less and Sass that’s CSS and Coffee script and Dart need to be compiled to normal Javascript.

WebXebiCon.034

Now we’re at a stage that we could deploy these files to our web server, but still that’s not really the way we like to develop our apps. So we’re going to do a couple of extra steps. We will minify almost all of our files. This will grab all those files, do some smart tricks, and we’ll end up with much smaller files. This will safe the end user some download time, which is nice.

WebXebiCon.036

We also want to make sure that all of the files we made don’t have any syntax mistakes. That’s why we’ll run a syntax validator on all of those files.

And of course we need to test. So we write a lot of unit tests to cover all our JavaScript functionality. But we also write view tests to make sure that our views display the correct information to the user. With good decoupling we can run our unit tests and view tests in isolation, which is good since we can easily spot problems when one of our test fails and it makes it easy to maintain and refactor our code. But we also want to make sure everything works well together, which is why we also write a couple of end to end tests.

WebXebiCon.042

With all these steps in place we’re pretty confident that we have everything we need to make sure that we write quality code. But there is one more thing we need to worry about during development. And that’s dependency management. We want automatic downloads of the third party libraries that we use and their dependencies. And we want to specify their version as well, or specify a version range and get automatic updates within our range.

Now we’re completely happy with all the steps in our development process. But how do we do all this? Or better said, how do we automate all this? We need a build tool.

Let’s quickly recap the things we want our build tool to do. It needs to compile and minify our style sheets and scripts. After that it will take these files and together with the HTML build a distribution package that we can deploy to our production server.

And with every change in any of our files, we want to make sure that everything still works as we would expect. So we validate our files and run our tests. These are our quality assurance steps.

Like we just mentioned, we also want all of our dependencies to be managed automatically.

But there are even more things that we can do with modern build tools that we couldn’t even think of but are great for development. How about live watch and reload of our browser during development. We launch our app in a browser, change some logic or styling, and the web browser will refresh the page automatically to show our changes.

The web development community has been very innovative and came up with many more cool thing we can do with these tools. And you can even write your own build actions.

WebXebiCon.050

At this point we have a full understanding of what the web development process looks like. And by seeing this, we can say that front end development has become a first class citizen of software development. Now that we recognise the complexity of front end development and that we need tooling to manage this complexity, it time to have a deeper look at some of the most popular tools of this moment.

Modern tools

Most of the web development tools that are built these days are part of the Node.js ecosystem.

WebXebiCon.054

Node.js is a JavaScript engine to run JS outside a browser environment. It has powerful system libraries for basics like file and network access, all with a minimal footprint These characteristics make NodeJS a perfect platform to build server side applications and terminal applications.

Since it’s JavaScript, a language that web developers already know, it’s the perfect environment to build web development tools with.

The Node Package Manager makes it easy to install any registered Node.js tool or application with just a single command. It keeps an online registry of all available packages, and anyone can publish their own. The Node.js and NPM ecosystem have developed into an active and healthy developer community.

One of the most popular tools written in Node.js is Grunt. It is the standard build tool for front-end web projects. It’s a task based build tool, which means you can define a set of tasks that need to run for a production build, development build or a test run. Sounds pretty much what we need for our development process.

WebXebiCon.060

Since it’s plugin based, it has plugins for pretty much any task you want to run. That includes plugins for our compilation, minification, validation and testing steps, as well as dependency management. That makes it perfect for us.

Once you start grunt in development mode, it will launch its built-in webserver and host your web application. And with the right plugins, we get our live reload. The Grunt plugin will watch for any changes in our resources, perform necessary tasks and refresh the open browser page for us, so we can see the results of our work right away - drastically shortening the feedback loop compared to traditional web development.

Since Grunt is the standard and most popular build tool for web projects at this moment, it’s widely adopted and has a large and mature developer ecosystem.

But Grunt also has it’s drawbacks. Let’s have a look at Grunt’s configuration file, the Gruntfile.js. This file contains configuration for our tasks and the plugins that perform them.

WebXebiCon.062

With a lot of tasks, the Gruntfile really becomes a lot of configuration. The example below taken from our own project and it’s only about half of our configuration. No need to read what it says exactly, it’s just to give you an idea about the amount.

WebXebiCon.064

But the amount of configuration is not the only downside of Grunt. The other problem is that it works with temporary files and directories. For each task (depending on the task, of course), it will read a couple of files, transform or convert them into something else and write the output back to disk. The the next task will then use these new files as their input. This creates a chain of temporary files and directories. It’s not a huge problem, especially  but it certainly is not the fastest way of doing things.

That’s why Gulp.js, the streaming build system, was made. It’s a newer, alternative build tool that’s quickly gaining popularity. Gulp can do pretty much the same things as Grunt, so we don’t need to worry too much that we lose essential features of our development process. However, since Gulp is newer than Grunt, it doesn’t have as many plugins as Grunt yet. But since Gulp does the same as Grunt, but then better, it seems that many developers make the switch to Gulp and more and more plugins are made for Gulp as well.

WebXebiCon.065

So what’s the difference? Instead of requiring a lot of configuration for it’s tasks, Gulp instead uses “Code over configuration”. That means that you just tell the tool to do something in a certain way instead of configuring both what it should do and how it should do it. Much easier and much more direct. And resulting in far less code than Grunt’s configuration for doing the same.

WebXebiCon.066

It also solves Grunt’s problem of temporary files and directories since it’s stream based. Tasks will directly stream their output to the next task. This results in much faster build times.

WebXebiCon.067

So it seems that Gulp solves exactly the two problems we had with Grunt. So it’s to be expected that it will gain popularity and eventually pass Grunt. It’s just a matter of time until it reaches the same maturity that Grunt currently has.

We talked earlier about dependency management to manage our third party libraries. While Grunt and Gulp use the Node Package Manager directly to download it’s plugins and their dependencies, we still need to manage the dependencies for our app. To do that, we use Bower, a package manager for the web, by Twitter. Bower works similar as NPM in that you specify all your dependencies in a single configuration file. Bower will then download these libraries from the Internet. Since Grunt and Gulp have plugins for Bower, you don’t even need to worry about this, since it will make sure to download the files when you build your project or to download updates when needed - if your build system is properly configured.

WebXebiCon.068

So many tools and frameworks! Isn’t it too much work to setup and configure?
Yes; setting up a full web app development stack, configuring each plugin for your project, is a lot of development work which can take up days. Luckily there is a tool to solve that problem: Yeoman.

Yeoman is a project scaffolding tool that generates a basis for your project, using templates called Generators. Some of those generators will set up a web application project with a preconfigured build strait, containing everything we need for compilation, minification, validation, testing, dependency management and much more. It will also ask us if we want to use frameworks such as Bootstrap and AngularJS. This makes it an ideal solution to setup a new project and automated development process.

More recent generators will set up Gulp as the build tool, providing an alternative to Bower. This should help developers that aren’t familiar with Gulp yet to get up to speed quickly.

WebXebiCon.072

Let’s recap what we talked about in this post in a summary.

WebXebiCon.079

More

The full presentation slides will soon be available from the XebiCon website. Feel free to comment below with your feedback or reach out to us on Twitter.

Lammert Westerhoff @lwesterhoff

Freek Wielstra @frwielstra

What is Agile?

“Becoming limitless involves mental agility; the ability to quickly grasp and incorporate new ideas and concepts with confidence.” -- Lorii Myers

I was asked to give an Intro to Agile talk to a group in Microsoft, in addition to a talk on Getting Results the Agile Way.

It worked out well.

The idea was to do a level set and get everybody on the same page in terms of what Agile is.

I thought it was a great chance to build a shared vocabulary and establish some shared mental models.   I believe that when people have a shared vocabulary and common mental models, they can take a look from the balcony.  And, it makes it a lot easier to move up the chain and take things further, faster.

Anyway, here is how I summarized what Agile is:

  • Agile is a framework of values and principles to manage teams and projects.
  • Agile is an alternative approach to traditional project management. It embraces change while traditional project management fights change.
  • People across functional teams work together as one team, rather than different groups working in phases or stages.
  • More human communication, interaction, and face-to-face.
  • Continuous feedback from users and stakeholders.
  • Iterations, shorter development cycles, and more frequent releases.
  • Visibility of progress and transparency of process.

That said, I need to find something a bit more pithy and precise, yet insightful.

If I had to put it in a simple sentence, I’d say Agile is empowerment through flexibility.

One thing I’ve noticed over the years is that some people struggle when they try to go Agile.

They struggle because they can’t seem to “flip a switch.”  And if they don’t flip the switch, they don’t change their mindset.

And, if they don’t change their mindset, Agile remains just beyond their grasp.

Agile is like happiness, grow it right under your feet.

You Might Also Like

40 Hour Work Week at Microsoft

Why Agile

Waterfall to Agile

Categories: Architecture, Programming

Cloud Architecture Revolution

The introduction of cloud technologies is not a simple evolution of existing ones, but a real revolution.  Like all revolutions, it changes the points of views and redefines all the meanings. Nothing is as before.  This post wants to analyze some key words and concepts, usually used in traditional architectures, redefining them according the standpoint of the cloud.  Understanding the meaning of new words is crucial to grasp the essence of a pure cloud architecture.

<<There is no greater impediment to the advancement of knowledge than the ambiguity of words.>> THOMAS REID, Essays on the Intellectual Powers of Man

Nowadays, it is required to challenge the limits of traditional architectures that go beyond the normal concepts of scalability and support millions of users (What's Up 500 Million) billions of transactions per day (Salesforce 1.3 billion), five 9s of availability (99.999 AOL).  I wish all of you the success of the examples cited above, but do not think that it is completely impossible to reach mind-boggling numbers. Using cloud technology, everyone can create a service with a little investment and immediately have a world stage.  If successful, the architecture must be able to scale appropriately.

Using the same design criteria or move the current configuration to the cloud simply does not work and it could reveal unpleasant surprises.

Infrastructure - commodity HW instead of high-end HW

Categories: Architecture

Why Agile?

I thought I had written about “Why Agile” before, but I don’t see anything crisp enough.

Anyway, here’s my latest rundown on Why Agile?

  1. Increase customer involvement which can build empathy and avoid rework
  2. Learn faster which means you can adapt to change
  3. Improve quality through focus
  4. Reduce risk through shorter feedback loops and customer interaction
  5. Simplify by getting rid of overhead and waste
  6. Reduce cycle time through timeboxing and parallel development
  7. Improve operational awareness through transparency
  8. Drive process improvement through continuous improvement
  9. Empower people through less mechanics and more interaction, continuous learning, and adaptation
  10. Flow more value through more frequent releases and less “big bang”

Remember that nature favors the flexible and agility is the key to success.

You Might Also Like

Agile vs. Waterfall

Agile Life-Cycle Frame

Methodologies at a Glance

Roles on Agile Teams

The Art of the Agile Retrospective

Categories: Architecture, Programming

Is there a future for Map/Reduce?

8w9jj

Google’s Jeffrey Dean and Sanjay Ghemawat filed the patent request and published the map/reduce paper  10 year ago (2004). According to WikiPedia Doug Cutting and Mike Cafarella created Hadoop, with its own implementation of Map/Reduce,  one year later at Yahoo – both these implementations were done for the same purpose – batch indexing of the web.

Back than, the web began its “web 2.0″ transition, pages became more dynamic , people began to create more content – so an efficient way to reprocess and build the web index was needed and map/reduce was it. Web Indexing was a great fit for map/reduce since the initial processing of each source (web page) is completely independent from any other – i.e.  a very convenient map phase and you need  to combine the results to build the reverse index. That said, even the core google algorithm –  the famous pagerank is iterative (so less appropriate for map/reduce), not to mention that  as the internet got bigger and the updates became more and more frequent map/reduce wasn’t enough. Again Google (who seem to be consistently few years ahead of the industry) began coming up with alternatives like Google Percolator  or  Google Dremel (both papers were published in 2010, Percolator was introduced at that year, and dremel has been used in Google since 2006).

So now, it is 2014, and it is time for the rest of us to catch up with Google and get over Map/Reduce and  for multiple reasons:

  • end-users’ expectations (who hear “big data” but interpret that as  “fast data”)
  • iterative problems like graph algorithms which are inefficient as you need to load and reload the data each iteration
  • continuous ingestion of data (increments coming on as small batches or streams of events) – where joining to existing data can be expensive
  • real-time problems – both queries and processing

In my opinion, Map/Reduce is an idea whose time has come and gone – it won’t die in a day or a year, there is still a lot of working systems that use it and the alternatives are still maturing. I do think, however, that if you need to write or implement something new that would build on map/reduce – you should use other option or at the very least carefully consider them.

So how is this change going to happen ?  Luckily, Hadoop has recently adopted YARN (you can see my presentation on it here), which opens up the possibilities to go beyond map/reduce without changing everything … even though in effect,  a lot  will change. Note that some of the new options do have migration paths and also we still retain the  access to all that “big data” we have in Hadoopm as well as the extended reuse of some of the ecosystem.

The first type of effort to replace map/reduce is to actually subsume it by offering more  flexible batch. After all saying Map/reduce is not relevant, deosn’t mean that batch processing is not relevant. It does mean that there’s a need to more complex processes. There are two main candidates here  Tez and Spark where Tez offers a nice migration path as it is replacing map/reduce as the execution engine for both Pig and Hive and Spark has a compelling offer by combining Batch and Stream processing (more on this later) in a single engine.

The second type of effort or processing capability that will help kill map/reduce is MPP databases on Hadoop. Like the “flexible batch” approach mentioned above, this is replacing a functionality that map/reduce was used for – unleashing the data already processed and stored in Hadoop.  The idea here is twofold

  • To provide fast query capabilities* – by using specialized columnar data format and database engines deployed as daemons on the cluster
  • To provide rich query capabilities – by supporting more and more of the SQL standard and enriching it with analytics capabilities (e.g. via MADlib)

Efforts in this arena include Impala from Cloudera, Hawq from Pivotal (which is essentially greenplum over HDFS), startups like Hadapt or even Actian trying to leverage their ParAccel acquisition with the recently announced Actian Vector . Hive is somewhere in the middle relying on Tez on one hand and using vectorization and columnar format (Orc)  on the other

The Third type of processing that will help dethrone Map/Reduce is Stream processing. Unlike the two previous types of effort this is covering a ground the map/reduce can’t cover, even inefficiently. Stream processing is about  handling continuous flow of new data (e.g. events) and processing them  (enriching, aggregating, etc.)  them in seconds or less.  The two major contenders in the Hadoop arena seem to be Spark Streaming and Storm though, of course, there are several other commercial and open source platforms that handle this type of processing as well.

In summary – Map/Reduce is great. It has served us (as an industry) for a decade but it is now time to move on and bring the richer processing capabilities we have elsewhere to solve our big data problems as well.

Last note  - I focused on Hadoop in this post even thought there are several other platforms and tools around. I think that regardless if Hadoop is the best platform it is the one becoming the de-facto standard for big data (remember betamax vs VHS?)

One really, really last note – if you read up to here, and you are a developer living in Israel, and you happen to be looking for a job –  I am looking for another developer to join my Technology Research team @ Amdocs. If you’re interested drop me a note: arnon.rotemgaloz at amdocs dot com or via my twitter/linkedin profiles

*esp. in regard to analytical queries – operational SQL on hadoop with efforts like  Phoenix ,IBM’s BigSQL or Splice Machine are also happening but that’s another story

illustration idea found in  James Mickens’s talk in Monitorama 2014 –  (which is, by the way, a really funny presentation – go watch it) -ohh yeah… and pulp fiction :)

Categories: Architecture