Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Programming

R: String to Date or NA

Mark Needham - Sun, 12/07/2014 - 20:29

I’ve been trying to clean up a CSV file which contains some rows with dates and some not – I only want to keep the cells which do have dates so I’ve been trying to work out how to do that.

My first thought was that I’d try and find a function which would convert the contents of the cell into a date if it was in date format and NA if not. I could then filter out the NA values using the is.na function.

I started out with the as.Date function…

> as.Date("2014-01-01")
[1] "2014-01-01"
 
> as.Date("foo")
Error in charToDate(x) : 
  character string is not in a standard unambiguous format

…but that throws an error if we have a non date value so it’s not so useful in this case.

Instead we can make use of the strptime function which does exactly what we want:

> strptime("2014-01-01", "%Y-%m-%d")
[1] "2014-01-01 GMT"
 
> strptime("foo", "%Y-%m-%d")
[1] NA

We can then feed those values into is.na..

> strptime("2014-01-01", "%Y-%m-%d") %>% is.na()
[1] FALSE
 
> strptime("foo", "%Y-%m-%d") %>% is.na()
[1] TRUE

…and we have exactly the behaviour we were looking for.

Categories: Programming

How Do I Learn C++?

Making the Complex Simple - John Sonmez - Thu, 12/04/2014 - 16:00

In this video, I give some tips on how to learn C++.

The post How Do I Learn C++? appeared first on Simple Programmer.

Categories: Programming

Sky Force 2014 Reimagined for Android TV

Android Developers Blog - Thu, 12/04/2014 - 09:33

By Jamil Moledina, Games Strategic Partnerships Lead, Google Play

In the coming months, we’ll be seeing more media players, like the recently released Nexus Player, and TVs from partners with Android TV built-in hit the market. While there’s plenty of information available about the technical aspects of adapting your app or game to Android TV, it’s also useful to consider design changes to optimize for the living room. That way you can provide lasting engagement for existing fans as well as new players discovering your game in this new setting. Here are three things one developer did, and how you can do them too.

Infinite Dreams is an indie studio out of Poland, co-founded by hardcore game fans Tomasz Kostrzewski and Marek Wyszyński. With Sky Force 2014 TV, they brought their hit arcade style game to Android TV in a particularly clever way. The mobile-based version of Sky Force 2014 reimaged the 2004 classic by introducing stunning 3D visuals, and a free-to-download business model using in-app purchasing and competitive tournaments to increase engagement. In bringing Sky Force 2014 to TV, they found ways to factor in the play style, play sessions, and real-world social context of the living room, while paying homage to the title’s classic arcade heritage. As Wyszyński puts it, “We decided not to take any shortcuts, we wanted to make the game feel like it was designed to be played on TV.”

Orientation

For starters, Sky Force 2014 is played vertically on a smartphone or tablet, also known as portrait mode. In the game, you’re piloting a powerful fighter plane flying up the screen over a scrolling landscape, targeting waves of steampunk enemies coming down at you. You can see far enough up the screen, enabling you to plan your attacks and dodge enemies in advance.

Vertical play on the mobile version

When bringing the game to TV, the quickest approach would have been to preserve that vertical orientation of the gameplay, by pillarboxing the field of play.

With Sky Force 2014, Infinite Dreams considered their options, and decided to scale the gameplay horizontally, in landscape mode, and recompose the view and combat elements. You’re still aiming up the screen, but the world below and the enemies coming at you are filling out a much wider field of view. They also completely reworked the UI to be comfortably operated with a gamepad or simple remote. From Wyszyński’s point of view, “We really didn't want to just add support for remote and gamepad on top of what we had because we felt it would not work very well.” This approach gives the play experience a much more immersive field of view, putting you right there in the middle of the action. More information on designing for landscape orientation can be found here.

Multiplayer

Like all mobile game developers building for the TV, Infinite Dreams had to figure out how to adapt touch input onto a controller. Sky Force 2014 TV accepts both remote control and gamepad controller input. Both are well-tuned, and fighter handling is natural and responsive, but Infinite Dreams didn’t stop there. They took the opportunity to add cooperative multiplayer functionality to take advantage of the wider field of view from a TV. In this way, they not only scaled the visuals of the game to the living room, but also factored in that it’s a living room where people play together. Given the extended lateral patterns of advancing enemies, multiplayer strategies emerge, like “divide and conquer,” or “I got your back” for players of different skill levels. More information about adding controller support to your Android game can be found here, handling controller actions here, and mapping each player’s paired controllers here.

Players battle side by side in the Android TV version

Business Model

Infinite Dreams is also experimenting with monetization and extending play session length. The TV version replaces several $1.99 in-app purchases and timers with a try-before-you-buy model which charges $4.99 after playing the first 2 levels for free. We’ve seen this single purchase model prove successful with other arcade action games like Mediocre’s Smash Hit for smartphones and tablets, in which the purchase unlocks checkpoint saves. We’re also seeing strong arcade action games like Vector Unit’s Beach Buggy Racing and Ubisoft’s Hungry Shark Evolution retain their existing in-app purchase models for Android TV. More information on setting up your games for these varied business models can be found here. We’ll be tracking and sharing these variations in business models on Android TV, including variations in premium, as the Android TV platform grows.

Reflecting on the work involved in making these changes, Wyszyński says, “From a technical point of view the process was not really so difficult – it took us about a month of work to incorporate all of the features and we are very happy with the results.” Take a moment to check out Sky Force 2014 TV on a Nexus Player and the other games in the Android TV collection on Google Play, most of which made no design changes and still play well on a TV. Consider your own starting point, take a look at the Android TV section on our developer blog, and build the version of your game that would be most satisfying to players on the couch.

Join the discussion on

+Android Developers
Categories: Programming

R: Applying a function to every row of a data frame

Mark Needham - Thu, 12/04/2014 - 07:31

In my continued exploration of London’s meetups I wanted to calculate the distance from meetup venues to a centre point in London.

I’ve created a gist containing the coordinates of some of the venues that host NoSQL meetups in London town if you want to follow along:

library(dplyr)
 
# https://gist.github.com/mneedham/7e926a213bf76febf5ed
venues = read.csv("/tmp/venues.csv")
 
venues %>% head()
##                        venue      lat       lon
## 1              Skills Matter 51.52482 -0.099109
## 2                   Skinkers 51.50492 -0.083870
## 3          Theodore Bullfrog 51.50878 -0.123749
## 4 The Skills Matter eXchange 51.52452 -0.099231
## 5               The Guardian 51.53373 -0.122340
## 6            White Bear Yard 51.52227 -0.109804

Now to do the calculation. I’ve chosen the Centre Point building in Tottenham Court Road as our centre point. We can use the distHaversine function in the geosphere library allows us to do the calculation:

options("scipen"=100, "digits"=4)
library(geosphere)
 
centre = c(-0.129581, 51.516578)
aVenue = venues %>% slice(1)
aVenue
##           venue   lat      lon
## 1 Skills Matter 51.52 -0.09911

Now we can calculate the distance from Skillsmatter to our centre point:

distHaversine(c(aVenue$lon, aVenue$lat), centre)
## [1] 2302

That works pretty well so now we want to apply it to every row in the venues data frame and add an extra column containing that value.

This was my first attempt…

venues %>% mutate(distHaversine(c(lon,lat),centre))
## Error in .pointsToMatrix(p1): Wrong length for a vector, should be 2

…which didn’t work quite as I’d imagined!

I eventually found my way to the by function which allows you to ‘apply a function to a data frame split by factors’. In this case I wouldn’t be grouping rows by a factor – I’d apply the function to each row separately.

I wired everything up like so:

distanceFromCentre = by(venues, 1:nrow(venues), function(row) { distHaversine(c(row$lon, row$lat), centre)  })
 
distanceFromCentre %>% head()
## 1:nrow(venues)
##      1      2      3      4      5      6 
## 2301.6 3422.6  957.5 2280.6 1974.1 1509.5

We can now add the distances to our venues data frame:

venuesWithCentre = venues %>% 
  mutate(distanceFromCentre = by(venues, 1:nrow(venues), function(row) { distHaversine(c(row$lon, row$lat), centre)  }))
 
venuesWithCentre %>% head()
##                        venue   lat      lon distanceFromCentre
## 1              Skills Matter 51.52 -0.09911             2301.6
## 2                   Skinkers 51.50 -0.08387             3422.6
## 3          Theodore Bullfrog 51.51 -0.12375              957.5
## 4 The Skills Matter eXchange 51.52 -0.09923             2280.6
## 5               The Guardian 51.53 -0.12234             1974.1
## 6            White Bear Yard 51.52 -0.10980             1509.5

Et voila!

Categories: Programming

Quota for Calendar API v3 now 10 times higher

Google Code Blog - Wed, 12/03/2014 - 19:52

Posted by Lucia Fedorova, Tech Lead, Google Calendar API

At Google we like to make 10x rather than 10% improvements. In this spirit, we are increasing the default quota for the Calendar API v3 by a factor of ten, to 1 million requests per day. That means your application can support ten times as many users without any need to apply for more quota.

And if you need even more free quota, you can apply for it in the developer console under APIs -> Calendar API -> Quota -> “Apply for higher quota.” We have also streamlined the process of quota handling to make sure you receive your quota as quickly as possible.

A few tips to work efficiently with your quota:

  • Use push notifications instead of polling.
  • If you cannot avoid polling, make sure you only poll when necessary (for example poll very seldomly at night).
  • Use incremental synchronization with sync tokens for all collections instead of repeatedly retrieving all the entries.
  • Increase page size to retrieve more data at once by using the maxResults parameter.
  • Update events when they change, avoid re-creating all the events on every sync.
  • Use exponential backoff for error retries.
Categories: Programming

Zephyr vs. qTest

Making the Complex Simple - John Sonmez - Wed, 12/03/2014 - 15:00

I often get asked about test management software. Specifically, which solution is best and which ones I recommend. So, I thought I would do a little post comparing Zephyr to qTest, since they have fairly similar offerings. In all fairness, I want to provide a full disclosure and let you know that Zephyr sponsored this post. But… as you know, ... Read More

The post Zephyr vs. qTest appeared first on Simple Programmer.

Categories: Programming

Bandwidth challenged? Amazing Google Developer content is now available offline

Google Code Blog - Tue, 12/02/2014 - 19:56
We currently provide a robust array of content and resources online, including documentation, SDKs, and videos, that empower developers to create awesome apps and services on a variety of Google platforms. However, due to the challenge of limited Internet access in many parts of the world, many developers are not able to easily search through our vast library of content.

Today, we’re going to make a select number of resources available for download through a kit, while offline, as part of a pilot to solve the broadband challenge. As an example, you will no longer have to worry about needing to download and install the SDK over a slow or non-existent internet connection when running a code lab, or spending 30 minutes to watch a 10 minute video tutorial on the latest API due to constant buffering.

So far, we distributed more than 2,000 kits through Google Developer Groups in Sub-Saharan Africa, India and Bangladesh. Developers now have offline access to many Google SDKs, developer videos, and documentation when they need them.
devkitcds
The current pilot kit contains:
Check out this document to find out more about the offline developer kit and the many ways you can put it to use, If you are interested in piloting one of these developer kits, please get in touch with a Google Developer Group near you or follow these instructions if you are an organisation like a school, tech hub or incubator.

Posted by Chukwuemeka Afigbo, Program Manager, Developer Ecosystems
Categories: Programming

Testing Feature Branches Remotely with Grunt

Xebia Blog - Tue, 12/02/2014 - 17:48

At my current job we are working on multiple features simultaneously, using git feature branches. We have a Jenkins build server which we use for integration testing of the master branch, which runs about 20 jobs simultaneously for Protractor and Fitnesse tests. An individual job typically takes around 10 minutes to complete.

Our policy is to keep the master branch production ready at all times. Therefore we have a review process in place that should assure that feature branches are only pushed to master when they can't break the application.
This all works very well as long as the feature which you are working on requires only one or two integration test suites to test its functionality. But every once in a while you're working on something that could have effects all over the application, and you would like to run a larger number of integration test suites. And of course before you merge your feature branch to master.
Running all the integration suites on your local machine would take too way much time. And Jenkins is configured to run all its suites against the master branch. So what to do?

The Solution

In this article I'm going to show a solution that we developed for this problem, which lets us start multiple remote Jenkins jobs on the branch that we are working on. This way we can continue working on our local machine while Jenkins is running integration tests on the build server.
Most of our integration suites run against our frontend modules, and for those modules we use grunt as our build tool.
Therefore the most practical step was extend grunt with a task for starting the integration tests on Jenkins: we'd like to type 'grunt jenkins' and then grunt should figure out which branch we have checked out, send that information to Jenkins, and start all the integration suites.
To accomplish that we need to take the following steps:

  • Have some Jenkins integration tests suites which can take a git branch as a parameter
  • Create a custom grunt task called 'jenkins'
  • Let the grunt jenkins task figure out which branch we have checked out
  • Let the grunt jenkins task start a bunch of remote jenkins jobs with the branch as a parameter
The Parameterized Jenkins job

Jenkins offers the feature to configure your build with a parameter. Here is how we do it:
In the configuration of a Jenkins job there's a little checkbox saying 'the build is parameterized'. Upon checking it, you can enter a parameter name, which will be available in your Jenkins build script.
We'll add a parameter called BRANCH, like in the screenshot below:

Jenkins job Parameter

Then in our Jenkins build script, we can check if the parameter is set, and if this is the case, check out the branch. It will look something like this:

git fetch
if [[ -n "$BRANCH" ]]; then
  git checkout -f $BRANCH
  git pull
else
  git checkout -f ${PROMOTED_GIT_COMMIT-"origin/master"}
fi

What's nice about our parameterized build job is that we can invoke it via a Rest call and include our parameter as a query parameter. I'll show that later on.

Our custom 'jenkins' Grunt task

In our grunt.js configuration file, we can load custom tasks. The following snippet loads all files in the conf/grunt/tasks folder.

 grunt.loadTasks('conf/grunt/tasks');

In our tasks folder we create a jenkins.js file, containing our custom jenkins task.
The next thing to do is to retrieve the name of the branch which we have checked out on our machine. There's a grunt plugin called 'gitinfo' which will help us with that.
When the gitinfo plugin is invoked it will add a section to the grunt configuration which contains, amongst others, the name of our current local branch:

module.exports = function (grunt) {
  grunt.registerTask('jenkins', ['gitinfo', 'build-branch']);
  
  grunt.registerTask('build-branch', function () {
    var git = grunt.config().gitinfo;
    grunt.log.ok('Building branch: ' + git.local.branch.current.name);


And now we can start our parameterized job with the correct value for the branch parameter, like this:

    var request = require('request');

    var jenkinsUser = 'your username';
    var jenkinsPassword = 'your password';
    var jenkinsHost = 'your jenkins host';
    var job = 'my-parameterized-integration-suite'; 

    var url = 'http://' + jenkinsUser + ':' + jenkinsHost + '@' + jenkinsHost + ':8080/job/' + job + '/buildWithParameters?BRANCH=' + git.local.branch.current.name + '&delay=0sec';

      request({
        url: url,
        method: 'POST'
      },
      jobFinished(job));
    });

First we acquire a reference to the 'request' package. This is a simple Node package that lets you perform http requests.
We then build the Rest url; to connect to jenkins we need to supply our Jenkins username and password.
And finally we post a request to the Rest endpoint of Jenkins, which will start our job. We supply a callback called 'jobFinished'.

Putting it all together: starting multiple jobs

With these steps in place, we have a new grunt task which we can invoke with 'grunt jenkins' from the commandline, and which will start a Jenkins job on the feature branch that we have checked out locally.
But this will only be useful if our grunt jenkins task is able to start not just one job, but a bunch of them.
Here is the full source code of the jenkins.js file. It has a (hardcoded) array of jobs, starts all of them and keeps track of how many of them have finished:

module.exports = function (grunt) {

  grunt.registerTask('jenkins', ['gitinfo', 'build-branch']);

  grunt.registerTask('build-branch', function () {
    var request = require('request');

    var jenkinsUser = 'your username';
    var jenkinsPassword = 'your password';
    var jenkinsHost = 'your jenkins host';
  
    var jobs = [
      'my-parameterized-integration-suite-1',
      'my-parameterized-integration-suite-2',
      'my-parameterized-integration-suite-3',
      'my-parameterized-integration-suite-4',
      'my-parameterized-integration-suite-5'
    ];
    var git = grunt.config().gitinfo;
    var done = this.async();
    var jobCounter = 0;

    grunt.log.writeln();
    grunt.log.ok('Building branch: ' + git.local.branch.current.name);
    grunt.log.writeln();

    function jobFinished (job) {
      return function (error, response, body) {
        jobCounter++;
        grunt.log.ok('[' + jobCounter + '/' + jobs.length + '] Started: ' + job);

        if (error) {
          grunt.log.error('Error: ' + error + (response ? ', status: ' + response.statusCode : ''));
        } else if (response.statusCode === 301) {
          grunt.log.writeln('See: ' + response.headers.location);
        }

        if (body) {
          grunt.log.writeln(body);
        }

        if (jobCounter === jobs.length) {
          grunt.log.ok();
          done();
        }
      };
    }

    jobs.forEach(function (job, i) {
      var url = 'http://' + jenkinsUser + ':' + jenkinsHost + '@' + jenkinsHost + ':8080/job/' + job + '/buildWithParameters?BRANCH=' + git.local.branch.current.name + '&delay=0sec';
      grunt.log.ok('[' + (i + 1) + '/' + jobs.length + '] Starting: ' + job);

      request({
        url: url,
        method: 'POST'
      },
      jobFinished(job));
    });

    grunt.log.ok();

  });
};

And here's the console output:

$ grunt jenkins
Running "gitinfo" task

Running "build-branch" task

>> Building branch: my-feature-branch

>> [1/5] Starting: my-parameterized-integration-suite-1
>> [2/5] Starting: my-parameterized-integration-suite-2
>> [3/5] Starting: my-parameterized-integration-suite-3
>> [4/5] Starting: my-parameterized-integration-suite-4
>> [5/5] Starting: my-parameterized-integration-suite-5
OK
>> [1/5] Started: my-parameterized-integration-suite-1
>> [2/5] Started: my-parameterized-integration-suite-2
>> [3/5] Started: my-parameterized-integration-suite-3
>> [4/5] Started: my-parameterized-integration-suite-4
>> [5/5] Started: my-parameterized-integration-suite-5
OK

Done, without errors.

If You Want to Thrive at Microsoft

I was reading back through Satya Nadella’s email on Bold Ambition and Our Core, and a few things caught my eye.

One of them was the idea that if you want to thrive at Microsoft, you need to drive change.

Satya writes:

“And if you want to thrive at Microsoft and make a world impact, you and your team must add numerous more changes to this list that you will be enthusiastic about driving.

Nothing is off the table in how we think about shifting our culture to deliver on this core strategy. Organizations will change. Mergers and acquisitions will occur. Job responsibilities will evolve. New partnerships will be formed. Tired traditions will be questioned. Our priorities will be adjusted. New skills will be built. New ideas will be heard. New hires will be made. Processes will be simplified. And if you want to thrive at Microsoft and make a world impact, you and your team must add numerous more changes to this list that you will be enthusiastic about driving.”

Change is in the air, and Satya has given everyone a license to thrive by re-imagining how to change the world, or at least their part of it.

For me, I’m focused on how to accelerate business transformation with Cloud, Mobile, Social, Big Data and the Internet of Things.

Together, these technology trends are enabling new end-to-end customer experiences, workforce transformation, and operations transformation.

It’s all about unleashing what individuals and businesses are capable of.

Categories: Architecture, Programming

New blogpost on kibana 4 beta

Gridshore - Tue, 12/02/2014 - 12:55

If you are like me interested in elasticsearch and kibana, than you might be interested in a blog post I wrote on my employers blog about the new Kibana 4 beta. If so, head over to my employers blog:

http://amsterdam.luminis.eu/2014/12/01/experiment-with-the-kibana-4-beta/

The post New blogpost on kibana 4 beta appeared first on Gridshore.

Categories: Architecture, Programming

Snowflakes

Phil Trelford's Array - Tue, 12/02/2014 - 08:45

Welcome to day 2 of the F# Advent Calendar in English, and don’t miss Scott Wlaschin’s introduction to property-based testing from yesterday.

In A Christmas Carol Charles Dickens wrote of cold winters with snow as a matter of course. White Christmases were common during the Little Ice Age that lasted from the 1550s to the 1850s. Nowadays the chances of a snowfall on Christmas day are much lower, but the imagery of a white Christmas persists.

In this post we’ll generate our snowflakes instead.

Koch Snowflake

The Koch snowflake is a mathematical curve constructed from an equilateral triangle where each line segment is recursively altered:

Koch

The picture above was generated in the F# REPL, using WinForms to display a bitmap:

let snowflake (graphics:Graphics) length =
   use pen = new Pen(Color.White)
   let angle = ref 0.0
   let x = ref ((float width/2.0) - length/2.0)
   let y = ref ((float height/2.0) - length/3.0)
   let rec segment n depth =
      if depth = 0 then
         line n
      else
         segment (n/3.0) (depth-1)
         rotate -60.0
         segment (n/3.0) (depth-1)
         rotate 120.0
         segment (n/3.0) (depth-1)
         rotate -60.0
         segment (n/3.0) (depth-1)
   and line n =
      let r = !angle * Math.PI / 180.0
      let x2 = !x + cos(r) * n
      let y2 = !y + sin(r) * n
      graphics.DrawLine(pen, float32 !x,float32 !y, float32 x2, float32 y2)
      x := x2
      y := y2
   and rotate a =
      angle := !angle + a
   let depth = 5
   segment length depth
   rotate 120.0  
   segment length depth  
   rotate 120.0
   segment length depth

The full snippet is available on F# Snippets: http://fssnip.net/oA

Paper and Scissors

Snowflakes can be created by folding paper and cutting holes with scissors. We can get a similar effect using transparent polygons and rotational symmetry:

Paper

Here the polygons are selected randomly and like snowflakes each one is different:

let paperSnowflake () =   
   let image = new Bitmap(int width, int height)
   use graphics = Graphics.FromImage(image)  
   use brush = new SolidBrush(Color.Black)
   graphics.FillRectangle(brush, 0, 0, int width, int height)
   graphics.TranslateTransform(float32 (width/2.0),float32 (height/2.0))
   let color = Color.FromArgb(128,0,128,255)
   use brush = new SolidBrush(color)
   let rand = Random()
   let polys =
      [for i in 1..12 ->
         let w = rand.Next(20)+1 // width
         let h = rand.Next(20)+1 // height
         let m = rand.Next(h)    // midpoint
         let s = rand.Next(30)   // start
         [|0,s; -w,s+m; 0,s+h; w,s+m|]
      ]
   for i in 0.0..60.0..300.0 do
      graphics.RotateTransform(float32 60.0)
      let poly points =
         let points = [|for (x,y) in points -> Point(x*5,y*5)|]
         graphics.FillPolygon(brush,points)
      polys |> List.iter poly 
   image

 

The full snippet is on F# Snippets: http://fssnip.net/oB

Another interesting method of generating snowflakes is cellular automata, but I’ll leave that as an exercise for the reader.

Happy holidays!
Categories: Programming

3, 2, 1 Code-in: Inviting teens to contribute to open source

Google Code Blog - Mon, 12/01/2014 - 21:00

Code-in 2014 logo

We believe that the key to getting students excited about computer science is to give them opportunities at ever younger ages to explore their creativity with computer science. That’s why we’re running the Google Code-in contest again this year, and today’s the day students can go to the contest site, register and start looking for tasks that interest them.

Ignacio Rodriguez was just 10 years old when he became curious about Sugar, the open source learning platform introduced nationally to students in Uruguay when he was in elementary school. With the encouragement of his teacher, Ignacio started asking questions of the developers writing and maintaining the code and he started playing around with things, a great way to learn to code. When he turned 14 he entered the Google Code-in contest completing tasks that included writing two new web services for Sugar and he created four new Sugar activities. He even continued to mentor other students throughout the contest period.  His enthusiasm for coding and making the software even better for future users earned him a spot as a 2013 Grand Prize Winner.

Ignacio is one of the 1,575 students from 78 countries that have participated in Google Code-in since 2010. We are encouraging 13-17 year old students to explore the many varied ways they can contribute to open source software development through the Google Code-in contest. Because open source is a collaborative effort, the contest is designed as a streamlined entry point for students into software development by having mentors assigned to each task that a student works on during the contest. Students don’t have to be coders to participate; as with any software project, there are many ways to contribute to the project.  Students will be able to choose from documentation, outreach, research, training, user interface and quality assurance tasks in addition to coding tasks.

This year, students can choose tasks created by 12 open source organizations working ondisaster relief, content management, desktop environments, gaming, medical record systems for developing countries, 3D solid modeling computer-aided design and operating systems to name a few.  

For more information on the contest, please visit the contest site where you can find the timeline, Frequently Asked Questions and information on each of the open source projects students can work with during the seven week contest.

Good luck students!
By Stephanie Taylor, Open Source Programs
Categories: Programming

What It Actually Means to Market Yourself as a Software Developer

Making the Complex Simple - John Sonmez - Mon, 12/01/2014 - 17:00

Today is your lucky day! No, really it is. I am going to tell you exactly what it means to market yourself as a software developer and why it just might not be such a bad thing. Believe me, I know what you are thinking. I get a lot of flak about the idea of marketing yourself or doing any ... Read More

The post What It Actually Means to Market Yourself as a Software Developer appeared first on Simple Programmer.

Categories: Programming

About snowmen and mathematical proof why agile works

Xebia Blog - Mon, 12/01/2014 - 16:05

Last week I had an interesting course by Roger Sessions on Snowman Architecture. The perishable nature of Snowmen under any serious form of pressure fortunately does not apply to his architecture principles, but being an agile fundamentalist I noticed some interesting patterns in the math underlying the Snowmen Architecture that are well rooted in agile practices. Understanding these principles may give facts to feed your gut feeling about these philosophies and give mathematical proof as to why Agile works.

Complexity

“What has philosophy got to do with measuring anything? It's the mathematicians you have to trust, and they measure the skies like we measure a field. “ - Galileo Galilei, Concerning the New Star (1606).

In his book “Facts and Fallacies of Software Engineering” Robert Glass implied that when the functionality of a system increases by 25% the complexity of it effectively doubles. So in formula form:

                      

This hypothesis is supported by empirical evidence, and also explains why planning poker that focuses on the complexity of the implementation rather than functionality delivered is a more accurate estimator of what a team can deliver in sprint.

Basically the smaller you can make the functionality the better, and that is better to the power 3 for you! Once you start making functionality smaller, you will find that your awesome small functionality needs to talk to other functionalities in order to be useful for an end user. These dependencies are penalized by Roger’s model.

“An outside dependency contributes as much complexity as any other function, but does so independently of the functions.”

In other words, splitting a functionality of say 4 points (74 complexity points) in two equal separate functions reduces the overall complexity to 17 complexity points. This benefit however vanishes when each module has more than 3 connections.

An interesting observation that one can derive from this is a mathematical model that helps you to find which functions “belong” together. It stands to reason that when those functions suffer from technical interfacing, they will equally suffer from human interfaces. But how do we find which functions “belong” together, and does it matter if we get it approximately right? 

Endless possibilities

“Almost right doesn’t count” – Dr. Taylor; on landing a spacecraft after a 300 million miles journey 50 meter from a spot with adequate sunlight for the solar panels. 

Partitioning math is incredibly complex, and the main problem with the separation of functions and interfaces is that it has massive implications if you get it “just about right”. This is neatly covered by “the Bell number” (http://en.wikipedia.org/wiki/Bell_number).

These numbers grow quite quickly e.g. a set of 2 functions can be split 2 ways, but a set of 3 already has 5 options, at 6 it is 203 and if your application covers a mere 16 business functions, we already have more than 10 billion ways to create sets, and only a handful will give that desired low complexity number.

So how can math help us to find the optimal set division? the one with the lowest complexity factor?

Equivalence Relations

In order to find business functions that belong together or at lease have so much in common that the number of interfaces will outweigh the functional complexity, we can resort to the set equivalence relation (http://en.wikipedia.org/wiki/Equivalence_relation). It is both the strong and the weak point in the Snowmen architecture. It provides a genius algorithm for separating a set in the most optimal subsets (and doing so in O(n + k log k) time). The equivalence relation that Session proposes is as follows:

            Two business functions {a, b} have synergy if, and only if, from a business perspective {a} is not useful without {b} and visa versa.

The weak point is the subjective measurement in the equation. When played at a too high level it will all be required, and on a too low level not return any valuable business results.

In my last project we split a large eCommerce platform in the customer facing part and the order handling part. This worked so well that the teams started complaining that the separation had lowered their knowledge of each other’s codebase, since very little functionality required coding on both subsystems.

We effectively had reduced complexity considerable, but could have taken it one step further. The order handling system was talking to a lot of other systems in order to get the order fulfilled. From a business perspective we could have separated further, reducing complexity even further. In fact, armed with Glass’s Law, we’ll refactor the application to make it even better than it is today.

Why bother?

Well, polynomial growing problems can’t be solved with linear solutions.

Polynomial problems vs linear solutions plotted against time

Polynomial problems vs linear solutions plotted against time

As long as the complexity is below the solution curve, things will be going fine. Then there is a point in time where the complexity surpasses our ability to solve it. Sure we can add a team, or a new technology, but unless we change nature of our problem, we are only postponing the inevitable.

This is the root cause why your user stories should not exceed the sprint boundaries. Scrum forces you to chop the functionality into smaller pieces that move the team in a phase where linear development power supersedes the complexity of the problem. In practice, in almost every case where we saw a team breaking this rule, they would end up at the “uh-oh moment” at some point in the future, at that stage where there are no neat solutions any more.

So believe in the math and divide your complexity curve in smaller chunks, where your solution capacity exceeds the problems complexity. (As a bonus you get a happy and thriving team.)

Spark: Write to CSV file with header using saveAsFile

Mark Needham - Sun, 11/30/2014 - 09:21

In my last blog post I showed how to write to a single CSV file using Spark and Hadoop and the next thing I wanted to do was add a header row to the resulting row.

Hadoop’s FileUtil#copyMerge function does take a String parameter but it adds this text to the end of each partition file which isn’t quite what we want.

However, if we copy that function into our own FileUtil class we can restructure it to do what we want:

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.io.IOUtils;
import java.io.IOException;
 
public class MyFileUtil {
    public static boolean copyMergeWithHeader(FileSystem srcFS, Path srcDir, FileSystem dstFS, Path dstFile, boolean deleteSource, Configuration conf, String header) throws IOException {
        dstFile = checkDest(srcDir.getName(), dstFS, dstFile, false);
        if(!srcFS.getFileStatus(srcDir).isDir()) {
            return false;
        } else {
            FSDataOutputStream out = dstFS.create(dstFile);
            if(header != null) {
                out.write((header + "\n").getBytes("UTF-8"));
            }
 
            try {
                FileStatus[] contents = srcFS.listStatus(srcDir);
 
                for(int i = 0; i < contents.length; ++i) {
                    if(!contents[i].isDir()) {
                        FSDataInputStream in = srcFS.open(contents[i].getPath());
 
                        try {
                            IOUtils.copyBytes(in, out, conf, false);
 
                        } finally {
                            in.close();
                        }
                    }
                }
            } finally {
                out.close();
            }
 
            return deleteSource?srcFS.delete(srcDir, true):true;
        }
    }
 
    private static Path checkDest(String srcName, FileSystem dstFS, Path dst, boolean overwrite) throws IOException {
        if(dstFS.exists(dst)) {
            FileStatus sdst = dstFS.getFileStatus(dst);
            if(sdst.isDir()) {
                if(null == srcName) {
                    throw new IOException("Target " + dst + " is a directory");
                }
 
                return checkDest((String)null, dstFS, new Path(dst, srcName), overwrite);
            }
 
            if(!overwrite) {
                throw new IOException("Target " + dst + " already exists");
            }
        }
        return dst;
    }
}

We can then update our merge function to call this instead:

def merge(srcPath: String, dstPath: String, header:String): Unit =  {
  val hadoopConfig = new Configuration()
  val hdfs = FileSystem.get(hadoopConfig)
  MyFileUtil.copyMergeWithHeader(hdfs, new Path(srcPath), hdfs, new Path(dstPath), false, hadoopConfig, header)
}

We call merge from our code like this:

merge(file, destinationFile, "type,count")

I wasn’t sure how to import my Java based class into the Spark shell so I compiled the code into a JAR and submitted it as a job instead:

$ sbt package
[info] Loading global plugins from /Users/markneedham/.sbt/0.13/plugins
[info] Loading project definition from /Users/markneedham/projects/spark-play/playground/project
[info] Set current project to playground (in build file:/Users/markneedham/projects/spark-play/playground/)
[info] Compiling 3 Scala sources to /Users/markneedham/projects/spark-play/playground/target/scala-2.10/classes...
[info] Packaging /Users/markneedham/projects/spark-play/playground/target/scala-2.10/playground_2.10-1.0.jar ...
[info] Done packaging.
[success] Total time: 8 s, completed 30-Nov-2014 08:12:26
 
$ time ./bin/spark-submit --class "WriteToCsvWithHeader" --master local[4] /path/to/playground/target/scala-2.10/playground_2.10-1.0.jar
Spark assembly has been built with Hive, including Datanucleus jars on classpath
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.propertie
...
14/11/30 08:16:15 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool
14/11/30 08:16:15 INFO SparkContext: Job finished: saveAsTextFile at WriteToCsvWithHeader.scala:49, took 0.589036 s
 
real	0m13.061s
user	0m38.977s
sys	0m3.393s

And if we look at our destination file:

$ cat /tmp/singlePrimaryTypes.csv
type,count
THEFT,859197
BATTERY,757530
NARCOTICS,489528
CRIMINAL DAMAGE,488209
BURGLARY,257310
OTHER OFFENSE,253964
ASSAULT,247386
MOTOR VEHICLE THEFT,197404
ROBBERY,157706
DECEPTIVE PRACTICE,137538
CRIMINAL TRESPASS,124974
PROSTITUTION,47245
WEAPONS VIOLATION,40361
PUBLIC PEACE VIOLATION,31585
OFFENSE INVOLVING CHILDREN,26524
CRIM SEXUAL ASSAULT,14788
SEX OFFENSE,14283
GAMBLING,10632
LIQUOR LAW VIOLATION,8847
ARSON,6443
INTERFERE WITH PUBLIC OFFICER,5178
HOMICIDE,4846
KIDNAPPING,3585
INTERFERENCE WITH PUBLIC OFFICER,3147
INTIMIDATION,2471
STALKING,1985
OFFENSES INVOLVING CHILDREN,355
OBSCENITY,219
PUBLIC INDECENCY,86
OTHER NARCOTIC VIOLATION,80
RITUALISM,12
NON-CRIMINAL,12
OTHER OFFENSE ,6
NON - CRIMINAL,2
NON-CRIMINAL (SUBJECT SPECIFIED),2

Happy days!

The code is available as a gist if you want to see all the details.

Categories: Programming

Spark: Write to CSV file

Mark Needham - Sun, 11/30/2014 - 08:40

A couple of weeks ago I wrote how I’d been using Spark to explore a City of Chicago Crime data set and having worked out how many of each crime had been committed I wanted to write that to a CSV file.

Spark provides a saveAsTextFile function which allows us to save RDD’s so I refactored my code into the following format to allow me to use that:

import au.com.bytecode.opencsv.CSVParser
import org.apache.spark.rdd.RDD
import org.apache.spark.SparkContext._
 
def dropHeader(data: RDD[String]): RDD[String] = {
  data.mapPartitionsWithIndex((idx, lines) => {
    if (idx == 0) {
      lines.drop(1)
    }
    lines
  })
}
 
// https://data.cityofchicago.org/Public-Safety/Crimes-2001-to-present/ijzp-q8t2
val crimeFile = "/Users/markneedham/Downloads/Crimes_-_2001_to_present.csv"
 
val crimeData = sc.textFile(crimeFile).cache()
val withoutHeader: RDD[String] = dropHeader(crimeData)
 
val file = "/tmp/primaryTypes.csv"
FileUtil.fullyDelete(new File(file))
 
val partitions: RDD[(String, Int)] = withoutHeader.mapPartitions(lines => {
  val parser = new CSVParser(',')
  lines.map(line => {
    val columns = parser.parseLine(line)
    (columns(5), 1)
  })
})
 
val counts = partitions.
  reduceByKey {case (x,y) => x + y}.
  sortBy {case (key, value) => -value}.
  map { case (key, value) => Array(key, value).mkString(",") }
 
counts.saveAsTextFile(file)

If we run that code from the Spark shell we end up with a folder called /tmp/primaryTypes.csv containing multiple part files:

$ ls -lah /tmp/primaryTypes.csv/
total 496
drwxr-xr-x  66 markneedham  wheel   2.2K 30 Nov 07:17 .
drwxrwxrwt  80 root         wheel   2.7K 30 Nov 07:16 ..
-rw-r--r--   1 markneedham  wheel     8B 30 Nov 07:16 ._SUCCESS.crc
-rw-r--r--   1 markneedham  wheel    12B 30 Nov 07:16 .part-00000.crc
-rw-r--r--   1 markneedham  wheel    12B 30 Nov 07:16 .part-00001.crc
-rw-r--r--   1 markneedham  wheel    12B 30 Nov 07:16 .part-00002.crc
-rw-r--r--   1 markneedham  wheel    12B 30 Nov 07:16 .part-00003.crc
...
-rwxrwxrwx   1 markneedham  wheel     0B 30 Nov 07:16 _SUCCESS
-rwxrwxrwx   1 markneedham  wheel    28B 30 Nov 07:16 part-00000
-rwxrwxrwx   1 markneedham  wheel    17B 30 Nov 07:16 part-00001
-rwxrwxrwx   1 markneedham  wheel    23B 30 Nov 07:16 part-00002
-rwxrwxrwx   1 markneedham  wheel    16B 30 Nov 07:16 part-00003
...

If we look at some of those part files we can see that it’s written the crime types and counts as expected:

$ cat /tmp/primaryTypes.csv/part-00000
THEFT,859197
BATTERY,757530
 
$ cat /tmp/primaryTypes.csv/part-00003
BURGLARY,257310

This is fine if we’re going to pass those CSV files into another Hadoop based job but I actually want a single CSV file so it’s not quite what I want.

One way to achieve this is to force everything to be calculated on one partition which will mean we only get one part file generated:

val counts = partitions.repartition(1).
  reduceByKey {case (x,y) => x + y}.
  sortBy {case (key, value) => -value}.
  map { case (key, value) => Array(key, value).mkString(",") }
 
 
counts.saveAsTextFile(file)

part-00000 now looks like this:

$ cat !$
cat /tmp/primaryTypes.csv/part-00000
THEFT,859197
BATTERY,757530
NARCOTICS,489528
CRIMINAL DAMAGE,488209
BURGLARY,257310
OTHER OFFENSE,253964
ASSAULT,247386
MOTOR VEHICLE THEFT,197404
ROBBERY,157706
DECEPTIVE PRACTICE,137538
CRIMINAL TRESPASS,124974
PROSTITUTION,47245
WEAPONS VIOLATION,40361
PUBLIC PEACE VIOLATION,31585
OFFENSE INVOLVING CHILDREN,26524
CRIM SEXUAL ASSAULT,14788
SEX OFFENSE,14283
GAMBLING,10632
LIQUOR LAW VIOLATION,8847
ARSON,6443
INTERFERE WITH PUBLIC OFFICER,5178
HOMICIDE,4846
KIDNAPPING,3585
INTERFERENCE WITH PUBLIC OFFICER,3147
INTIMIDATION,2471
STALKING,1985
OFFENSES INVOLVING CHILDREN,355
OBSCENITY,219
PUBLIC INDECENCY,86
OTHER NARCOTIC VIOLATION,80
NON-CRIMINAL,12
RITUALISM,12
OTHER OFFENSE ,6
NON - CRIMINAL,2
NON-CRIMINAL (SUBJECT SPECIFIED),2

This works but it’s quite a bit slower than when we were doing the aggregation across partitions so it’s not ideal.

Instead, what we can do is make use of one of Hadoop’s merge functions which squashes part files together into a single file.

First we import Hadoop into our SBT file:

libraryDependencies += "org.apache.hadoop" % "hadoop-hdfs" % "2.5.2"

Now let’s bring our merge function into the Spark shell:

import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs._
 
def merge(srcPath: String, dstPath: String): Unit =  {
  val hadoopConfig = new Configuration()
  val hdfs = FileSystem.get(hadoopConfig)
  FileUtil.copyMerge(hdfs, new Path(srcPath), hdfs, new Path(dstPath), false, hadoopConfig, null)
}

And now let’s make use of it:

val file = "/tmp/primaryTypes.csv"
FileUtil.fullyDelete(new File(file))
 
val destinationFile= "/tmp/singlePrimaryTypes.csv"
FileUtil.fullyDelete(new File(destinationFile))
 
val counts = partitions.
reduceByKey {case (x,y) => x + y}.
sortBy {case (key, value) => -value}.
map { case (key, value) => Array(key, value).mkString(",") }
 
counts.saveAsTextFile(file)
 
merge(file, destinationFile)

And now we’ve got the best of both worlds:

$ cat /tmp/singlePrimaryTypes.csv
THEFT,859197
BATTERY,757530
NARCOTICS,489528
CRIMINAL DAMAGE,488209
BURGLARY,257310
OTHER OFFENSE,253964
ASSAULT,247386
MOTOR VEHICLE THEFT,197404
ROBBERY,157706
DECEPTIVE PRACTICE,137538
CRIMINAL TRESPASS,124974
PROSTITUTION,47245
WEAPONS VIOLATION,40361
PUBLIC PEACE VIOLATION,31585
OFFENSE INVOLVING CHILDREN,26524
CRIM SEXUAL ASSAULT,14788
SEX OFFENSE,14283
GAMBLING,10632
LIQUOR LAW VIOLATION,8847
ARSON,6443
INTERFERE WITH PUBLIC OFFICER,5178
HOMICIDE,4846
KIDNAPPING,3585
INTERFERENCE WITH PUBLIC OFFICER,3147
INTIMIDATION,2471
STALKING,1985
OFFENSES INVOLVING CHILDREN,355
OBSCENITY,219
PUBLIC INDECENCY,86
OTHER NARCOTIC VIOLATION,80
RITUALISM,12
NON-CRIMINAL,12
OTHER OFFENSE ,6
NON - CRIMINAL,2
NON-CRIMINAL (SUBJECT SPECIFIED),2

The full code is available as a gist if you want to play around with it.

Categories: Programming

(Edu) Scrum at XP Days Benelux: beware of the next generation

Xebia Blog - Sat, 11/29/2014 - 09:21

Xp Days Benelux 2014 is over, and it was excellent.
Good sessions, interesting mix of topics and presenters, and a wonderful atmosphere of knowledge sharing, respect and passion for Agile.

After 12 years, XP Days Benelux continues to be inspiring and surprising.

The greatest surprise for me was the participation of 12 high school students from the Valuas College in Venlo, who arrived on the second day. These youngsters did not only attend the conference, but they actually hosted a 120-minute session on Scrum at school, called EduScrum.

eduscrum

 

Eduscrum

EduScrum uses the ceremonies, roles and artifacts of Scrum to help young people learn in a better way. Students work together in small teams, and thus take ownership of their own learning process. At the Valuas College, two enthusiastic Chemistry teachers introduced EduScrum in their department two years ago, and have made the switch to teaching Chemistry in this new way.

In an interactive session, we, the adults, learned from the youngsters how they work and what EduScrum brought them. They showed their (foldable!) Scrum boards, explained how their teams are formed, and what the impact was on their study results. Forcing themselves to speak English, they were open, honest, courageous and admirable.

eduscrum2

 

Learnings

Doing Scrum in school has many similarities with doing Scrum at work. However, there is also a lot we can learn from the youngsters. These are my main takeaways:

- Transition is hard
It took the students some time to get used to working in the new way. At first they thought it was awkward. The transition took about… 4 lessons. That means that these youngsters were up and running with Scrum in 2 weeks (!).

- Inform your stakeholders
When the teachers introduced Scrum, they did not inform their main stakeholders, the parents. Some parents, therefore, were quite worried about this strange thing happening at school. However,  after some explanations, the parents recognised that eduScrum actually helps to prepare their children for today’s society and were happy with the process.

- Results count
In schools more than anywhere else, your results (grades) count. EduScrum students are graded as a team as well as individually. When they transitioned to Scrum the students experienced a drop in their grades at first, maybe due to the greater freedom and responsibility they had to get used to. Soon after, theirs grades got better.

- Compliancy is important
Schools and teachers have to comply with many rules and regulations. The knowledge that needs to get acquired each year is quite fixed. However, with EduScrum the students decide how they will acquire that knowledge.

- Scrum teaches you to cooperate
Not surprisingly, all students said that, next to Chemistry, they now learned to cooperate and communicate better. Because of this teamwork, most students like to work this way. However, this is also the reason a few classmates would like to return to the old, individual, style of learning. Teamwork does not suit everyone.

- Having fun helps you to work better
School (and work) should not be boring, and we work better together when we have some fun too. Therefore, next to a Definition of Done, the student teams also have a Definition of Fun.  :-)

Next generation Scrum

At the conference, the youngsters were surprised to see that so many companies that they know personally (like Bol.com) are actually doing Scrum. ‘I thought this was just something I learned to do in school ‘, one girl said. ‘But now I see that it is being used in so many companies and I will actually be able to use it after school, too.’

Beware of these youngsters. When this generation enters the work force, they will embrace Scrum as the natural way of working. In fact, this generation is going to take Scrum to the next level.

 

 

 

 

Take the Harder Road

Making the Complex Simple - John Sonmez - Thu, 11/27/2014 - 16:00

In this video, I talk about why it is usually a good idea to take the harder road, when faced with two choices and also how important it is to specialize.

The post Take the Harder Road appeared first on Simple Programmer.

Categories: Programming

Docker/Neo4j: Port forwarding on Mac OS X not working

Mark Needham - Thu, 11/27/2014 - 13:28

Prompted by Ognjen Bubalo’s excellent blog post I thought it was about time I tried running Neo4j on a docker container on my Mac Book Pro to make it easier to play around with different data sets.

I got the container up and running by following Ognien’s instructions and had the following ports forwarded to my host machine:

$ docker ps
CONTAINER ID        IMAGE                 COMMAND                CREATED             STATUS              PORTS                                              NAMES
c62f8601e557        tpires/neo4j:latest   "/bin/bash -c /launc   About an hour ago   Up About an hour    0.0.0.0:49153->1337/tcp, 0.0.0.0:49154->7474/tcp   neo4j

This should allow me to access Neo4j on port 49154 but when I tried to access that host:port pair I got a connection refused message:

$ curl -v http://localhost:49154
* Adding handle: conn: 0x7ff369803a00
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x7ff369803a00) send_pipe: 1, recv_pipe: 0
* About to connect() to localhost port 49154 (#0)
*   Trying ::1...
*   Trying 127.0.0.1...
*   Trying fe80::1...
* Failed connect to localhost:49154; Connection refused
* Closing connection 0
curl: (7) Failed connect to localhost:49154; Connection refused

My first thought was the maybe Neo4j hadn’t started up correctly inside the container so I checked the logs:

$ docker logs --tail=10 c62f8601e557
10:59:12.994 [main] INFO  o.e.j.server.handler.ContextHandler - Started o.e.j.w.WebAppContext@2edfbe28{/webadmin,jar:file:/usr/share/neo4j/system/lib/neo4j-server-2.1.5-static-web.jar!/webadmin-html,AVAILABLE}
10:59:13.449 [main] INFO  o.e.j.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@192efb4e{/db/manage,null,AVAILABLE}
10:59:13.699 [main] INFO  o.e.j.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@7e94c035{/db/data,null,AVAILABLE}
10:59:13.714 [main] INFO  o.e.j.w.StandardDescriptorProcessor - NO JSP Support for /browser, did not find org.apache.jasper.servlet.JspServlet
10:59:13.715 [main] INFO  o.e.j.server.handler.ContextHandler - Started o.e.j.w.WebAppContext@3e84ae71{/browser,jar:file:/usr/share/neo4j/system/lib/neo4j-browser-2.1.5.jar!/browser,AVAILABLE}
10:59:13.807 [main] INFO  o.e.j.server.handler.ContextHandler - Started o.e.j.s.ServletContextHandler@4b6690b1{/,null,AVAILABLE}
10:59:13.819 [main] INFO  o.e.jetty.server.ServerConnector - Started ServerConnector@495350f0{HTTP/1.1}{c62f8601e557:7474}
10:59:13.900 [main] INFO  o.e.jetty.server.ServerConnector - Started ServerConnector@23ad0c5a{SSL-HTTP/1.1}{c62f8601e557:7473}
2014-11-27 10:59:13.901+0000 INFO  [API] Server started on: http://c62f8601e557:7474/
2014-11-27 10:59:13.902+0000 INFO  [API] Remote interface ready and available at [http://c62f8601e557:7474/]

Nope! It’s up and running perfectly fine which suggested the problemw was with port forwarding.

I eventually found my way to Chris Jones’ ‘How to use Docker on OS X: The Missing Guide‘ which explained the problem:

The Problem: Docker forwards ports from the container to the host, which is boot2docker, not OS X.

The Solution: Use the VM’s IP address.

So to access Neo4j on my machine I need to use the VM’s IP address rather than localhost. We can get the VM’s IP address like so:

$ boot2docker ip
 
The VM's Host only interface IP address is: 192.168.59.103

Let’s strip out that surrounding text though:

$ boot2docker ip 2> /dev/null
192.168.59.103

Now if we cURL using that IP instead:

$ curl -v http://192.168.59.103:49154
* About to connect() to 192.168.59.103 port 49154 (#0)
*   Trying 192.168.59.103...
* Adding handle: conn: 0x7fd794003a00
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x7fd794003a00) send_pipe: 1, recv_pipe: 0
* Connected to 192.168.59.103 (192.168.59.103) port 49154 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.30.0
> Host: 192.168.59.103:49154
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json; charset=UTF-8
< Access-Control-Allow-Origin: *
< Content-Length: 112
* Server Jetty(9.0.5.v20130815) is not blacklisted
< Server: Jetty(9.0.5.v20130815)
<
{
  "management" : "http://192.168.59.103:49154/db/manage/",
  "data" : "http://192.168.59.103:49154/db/data/"
* Connection #0 to host 192.168.59.103 left intact

Happy days!

Chris has solutions to lots of other common problems people come across when using Docker with Mac OS X so it’s worth having a flick through his post.

Categories: Programming

How to implement validation callbacks in AngularJS 1.3

Xebia Blog - Wed, 11/26/2014 - 09:21

In my current project we've recently switched from AngularJS 1.2 to 1.3. Except for a few breaking changes the upgrade was quite trivial. However, after diving into the changelog we noticed that the way AngularJS handles form validation changed drastically. Since we're working on a greenfield application we decided it was worth the effort to rewrite the validation logic. The main argument for this was that the validation we had could be drastically simplified by using the new validation pipeline.

This article is aimed at AngularJS developers interested in the new validation pipeline offered by AngularJS 1.3. Except for a small introduction, this article will not be about all the different aspects related to validating forms. I will showcase 2 different cases in which we had to come up with custom solutions:

  • Displaying additional information after successful validation
  • Validating equality of multiple password fields

What has changed?

Whereas in AngularJS 1.2 we could use $parsers and $formatters for form validation, AngularJS 1.3 introduces the concept of $validators and $asyncValidators. As we can deduce from the names, the latter is for server-side validations using HTTP calls and the former is for validations on the client-side.

All validators are directives that are registered on a specific ngModel by either adding it to the ngModel.$validators or ngModel.$asyncValidators. When validating, all $validators run before executing the $asyncValidators.

AngularJS 1.3 also utilises the HTML5 validation API wherever possible. For each of the HTML5 validation attributes, AngularJS offers a directive. For example, for minlength, add ng-minlength to your input field to incorporate the minimum length check.

When it comes down to showing error messages we can rely on the $error property on a ngModel. Whenever one of the validators fail it will add the name of the validator to the $error property. Using the brand new ngMessages module we can then easily display specific error messaging depending on the type of validator.

Displaying additional information after successful validation

Implementing the new validation pipeline came with a few challenges. The biggest being that we had quite a few use cases in which, after successfully validating a field, we wanted to display some data returned by the web service. Below I will discuss how we've solved this.

The directive itself is very simple and merely does the following:

  1. Clear the data displayed next to the field. If the user has already entered text and the validation succeeds the data from validation call will be shown next to the input field. If the user were to change the input's value and it would not validate correctly, the data displayed next to the field would be stale. To prevent this we first clear the data displayed next to the field at the start of the validation.
  2. Validate the content against the web service using the HelloResource. Besides returning the promise the resource gives us we invoke the callback() method when the promise is successfully resolved.
  3. Display data returned by the HTTP call using a callback method
'use strict';

angular.module('angularValidators')
  .directive('validatorWithCallback', function (HelloResource) {
    return {
      require: 'ngModel',
      link: function (scope, element, attrs, ngModel) {
        function callback(response) {}

        ngModel.$asyncValidators.validateWithCallback = function (modelValue, viewValue) {
          callback('');

          var value = modelValue || viewValue;

          return HelloResource.get({name: value}).$promise.then(function (response) {
            callback(response);
          });
        };
      }
    };
});

We can add the validator to our input by adding the validator-with-callback attribute to the input which we would like to validate.

<form name="form">
    <input type="text" name="name" ng-model="name" required validator-with-callback />
</form>
Implementing the clear and callback

Because this directive should be independent from any specific ngModel we have to find a way to pass the ngModel to the directive. To accomplish this we add a value to the validator-with-callback attribute. We also change the ng-model attribute value to name.value. Why this is required will be explained later on. To finish we also add a div that will only display when the form element is valid and we will set it to display the value of name.detail.

<form name="form">
    <input type="text" name="name" ng-model="name.value" required validator-with-callback="name" />
    <div ng-if="form.name.$valid">{{name.detail}}</div>
</form>

The $eval method from scope can be used to resolve the object using the attribute's value. Displaying the data won't work if we simply supply and overwrite any scoped object (f.e. $scope.data). We have to add a scoped object name which contains 2 properties: value and detail. Note: the naming is not important.

We will add a controller to our HTML file which will be responsible for setting the default value for our scoped object name. As shown in our HTML view above, the value property will be used for storing the value of the field. The detail property will be used to store the response from the web service call and display it to the user.

'use strict';

angular.module('angularValidators')
  .controller('ValidationController', function ($scope) {
    $scope.name = {
      value: undefined,
      detail: ''
    };
});

The last thing is changing the directive implementation to retrieve the target object and implement the clear and callback methods. We do this by retrieving the value from the validator-with-callback attribute by calling scope.$eval(attrs.validatorWithCallback). When we have the target object we can implement the callback method.

'use strict';

angular.module('angularValidators')
  .directive('validatorWithCallback', function (HelloResource) {
    return {
      require: 'ngModel',
      link: function (scope, element, attrs, ngModel) {
        var target = scope.$eval(attrs.validatorWithCallback);

        function callback(response) {
          if (_.isUndefined(target)) {
            return;
          }

          target.detail = response.msg;
        }
        ngModel.$asyncValidators.validateWithCallback = function (modelValue, viewValue) {
          ...omitted...
        };
      }
    };
});

This is all that's needed to create a directive with a callback method. This callback method uses data returned by the web service to populate the property value but it can of course be adjusted to do anything we desire.

Validating equality of multiple password fields

The second example we would like to show is a synchronous validator that validates the values of 2 different fields. The use case we had for this were 2 password fields which required to be equal.

Requirements
  • (In)validate both fields when the user changes the value of either one of them and they are (not) equal
  • The validator successfully validates the field if the second input has not been touched or the value of the second input is empty.
  • Only display 1 error message and only when no other validators (required & min-length) are invalid
Implementation

We start of by creating the 2 different form elements with both a required and a ng-minlength validator. We also add a button to the form to show how enabling/disabling the button depending on the validity of the form works.

Both password fields also have the validate-must-equal-to="other_field_name" attribute. This indicates that we wish to validate the value of this field against the field defined by the attribute. We also add a form-name="form" attribute to pass in the name of the form to our directive. This is needed for invalidating the second input on our form without hardcoding the form name inside the directive and thus making this directive fully independent from form and field names.

To conclude we also conditionally show or hide the error displaying containers. For the errors related to the first input field we also specify that it should not display if the notEqualTo error has been set by our directive. This ensures that no empty div will be displayed if our validator invalidates the first field.

<form name="form">
    <input type="password" name="password" ng-model="password" required ng-minlength="8" validate-must-equal-to="password2" form-name="form" />
    <div ng-messages="form.password.$error" ng-if="form.password.$touched && form.password.$invalid && !form.password.$error.notEqualTo">
        <div ng-message="required">This field is required</div>
        <div ng-message="minlength">Your password must be at least 8 characters long</div>
    </div>
    <br />
    <input type="password" name="password2" ng-model="password2" required ng-minlength="8" validate-must-equal-to="password" form-name="form" />
    <div ng-messages="form.password2.$error" ng-if="form.password2.$touched && form.password2.$invalid">
        <div ng-message="required">This field is required</div>
        <div ng-message="minlength">Your password must be at least 8 characters long</div>
    </div>

    <p>The submit button will only be enabled when the entire form is valid</p>
    <button ng-disabled="form.$invalid">Submit</button>
</form>

The validator itself is again very compact. Basically all we want is to retrieve the value from the input and pass it to a isEqualToOther method which returns a boolean. At the beginning of the link method we also do a check to see if the form-name attribute is provided. If not, we throw an error. We do this to communicate to any developer reusing this directive that this directive requires the form name to function correctly. Unfortunately at this moment there is no other way to communicate the additional mandatory attribute.

'use strict';

angular.module('angularValidators')
  .directive('validateMustEqualTo', function () {
    return {
      require: 'ngModel',
      link: function (scope, element, attrs, ngModel) {
        if (_.isUndefined(attrs.formName)) {
          throw 'For this directive to function correctly you need to supply the form-name attribute';
        }

        function isEqualToOther(value) {
          ...omitted...
        }

        ngModel.$validators.notEqualTo = function (modelValue, viewValue) {
          var value = modelValue || viewValue;

          return isEqualToOther(value);
        };
      }
    };
  });

The isEqualToOther method itself does the following:

  • Retrieve the other input form element
  • Throw an error if it cannot be found which again means this directive won't function as intended
  • Retrieve the value from the other input and validate the field if the input has not been touched or the value is empty
  • Compare both values
  • Set the validity of the other field depending on the comparison
  • Return the comparison to (in)validate the field this directive is linked to
'use strict';

angular.module('angularValidators')
  .directive('validateMustEqualTo', function () {
    return {
      require: 'ngModel',
      link: function (scope, element, attrs, ngModel) {
        ...omitted...

        function isEqualToOther(value) {
          var otherInput = scope[attrs.formName][attrs.validateMustEqualTo];
          if (_.isUndefined(otherInput)) {
            throw 'Cannot retrieve the second field to compare with from the scope';
          }

          var otherValue = otherInput.$modelValue || otherInput.$viewValue;
          if (otherInput.$untouched || _.isEmpty(otherValue)) {
            return true;
          }

          var isEqual = (value === otherValue);

          otherInput.$setValidity('notEqualTo', isEqual);

          return isEqual;
        }

        ngModel.$validators.notEqualTo = function (modelValue, viewValue) {
          ...omitted...
        };
      }
    };
  });
Alternative solution

An alternative solution to the validate-must-equal-to directive could be to implement a directive that encapsulates both password fields and has a scoped function that would handle validation using ng-blur on the fields or a $watch on both properties. However, this approach does not use the out-of-the-box validation pipeline and we would thus have to extend the logic in the form button's ng-disabled to allow the user to submit the form.

Conclusion

AngularJS 1.3 introduces a new validation pipeline which is incredibly easy to use. However, when faced with more advanced validation rules it becomes clear that certain features (like the callback mechanism) are lacking for which we had to find custom solutions. In this article we've shown you 2 different validation cases which extend the standard pipeline.

Demo application

I've set up a stand-alone demo application which can be cloned from GitHub. This demo includes both validators and karma tests that cover all different scenario's. Please feel free to use and modify this code as you feel appropriate.