Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Feed aggregator

Why is Statistical Thinking Hard?

Herding Cats - Glen Alleman - Sat, 06/28/2014 - 21:26

I'll admit up front I'm hugely biased toward statistical thinking. As one trained in physics  and the mathematics that goes with physics, and Systems Engineering and the math that goes with that  thinking about statistics is what we do in our firm. We work programs with cost and schedule development, do triage on programs for cost and schedule, guide the development of technology solutions using probabilistic models, assess risk to cost, schedule, and technical performance using probability and statistics, and build business cases, performance models, Estimates To Complete, Estimates At Completion, the probability of program success, the probability  of a proposal win, and  the probability that the Go-Live date will occur on or before the need date, and be at or below the planned cost, and the probability the mandatory needed capabilities will be there as well.

We use probability and statistics not because we want to, but because we have to. Many intelligent, trained, and educated people in our domain - software intensive systems and the management of projects - find themselves frozen in fear when confronted by any mathematical problem beyond the level of basic arithmetic - especially in the software development domain. The algorithm writers on flight control systems we work  with are not actually software developers in the common sense, but are control system engineers who implement their algorithm in Handel-C - so they don't count - at leats not in this sense.

We have to deal with probability and statistics for a simple reason - ever variable on a project is a random variable. Only accountants deal with Point Numbers. The balance in your checking account is not subject to a statistical estimate. The price of General Electric stock in your 401(K) is a random variable. All the world is a non-stationary, stochastic process, and many times a non-linear, non-stationary, stochastic process.

Stochastic processes are everywhere. They are time series subject to random fluctuations. Your heart beat, the stock market, the productivity of your software team, the stability of technical requirements, the performance of the database server, the number of defects in the code you write.

In our software development domain there is an overwhelming need to predict the future. Not for the reason you may think. Because at the same time there is a movement underway to Not Estimate the future. But it turns out this is a need not necessarily a desire. The need to predict - to have some sense of what is going to happen - is based on a few very simple principles of microeconomics

  • It's not our money. If it were we could do with it as we please.
  • Those providing the money have a  finite amount of money. They alos have a finite amount of time in which to exchange that money for value produced by us, the development team.
  • If there were a non-finite amount of money and time, we won't have to talk about things like estimates of when we'll be done, or how much it will cost, or the probability that the produced outcomes will meet the needs of the users.

Our natural tendencies are to focus on observation - empirical data - rather than the statistical data that drives the probabilistic aspects of our work.

Probability and Statistics

This approach - the statistical processes and probabilistic outcomes requires we know something about our underlying processes. Our capacity for work, the generated defect rate, the defect fix rate. Without that knowledge the probabilistic answers aren't forth coming and if th ey are forced out in the Dilbert Style management, they'll be bogus at best, and down right lies at worst.

Let's stop here for some critically important points:

  • If we don't have some sense of the underlying processes driving our project, we're in much bigger trouble than we think - we don't know what done looks like in units of measure meaningful to the decision makers
    • When will we be done? Approximately? - I don't know -  then we're late before we start.
    • How much will this cost? Approximately? - I don't know - we're over budget before we start.
    • What's the probability we'll be able to deliver all the needed features - minimal features or mandatory features, for a cost and schedule goal? I don't know - this project is going to be a Death March project before we run out of time and money.
  • If we don't know our capacity for work, which should be developed from empirical data - we can't make duration and cost estimates.
    • Knowing this once the project is going if fine. But it's likely too late to make business decisions needed to start the project.
    • The very naive assumption that all the work can be broken down into same sized chunks has not broad evidence, and is likely to be highly domain dependent. 

So let's look at the core problem of estimating

As humans we are poor at estimating. Fine, does then mean we should not estimate. Hardly. We need to become aware of our built in problem and deal with them. The need for estimating in business is not going away, it is at the heart of business itself and core to all decision making.

Here's an example from Daniel Kahneman's "Thinking. Fast and Slow",

Linda is 31 years old, single, outspoken, and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations.

Which of these is more probable?

1. Linda is a bank clerk.
2. Linda is a bank clerk and an active member of a feminist movement.

In Kahneman's studies, 90% of all respondents selected the latter option. Why? Because the description of Linda and the word feminism intuitively raise an idea of compatibility. Being outspoken and an activist are not usually associated with the job of a bank clerk. Quickly thinking, the bank clerk is a slightly more probable option if she is at least a feminist. However, the second option is horribly wrong because the probability is lower when there are more variables. Statistical probability is always lower when the target group grows – there are more bank clerks who are not active feminists.

So Now What, We've Confirmed We're Bad At Making Estimates

How do we make it better? First, come to realize the good - meaning credible - estimates are part of good business. Knowing the cost of the value delivered is at the core of all business success. Second, is look for the root causes of poor estimating outcomes. These comes in many sizes. But the Dilbert excuse is not an excuse. It's a carton of bad management. So let's drop that right away. If you work for a Dilbert boss or have a Dilbert boss for a customer, not much good estimating is going to do for you. So let's dispense with the charde of trying too.

So what are some root causes of poor estimates:

  1. Poor understanding of what done looks like - what capabilities do we need and when do we need them. Without this understanding building a list of requirements has no home and the project becomes an endless series of emerging work efforts in an attempt to discover the needed capabilities.
  2. Inability to create a model of past performance with statistical behaviour
    • All project variables are random variables 
    • Discover these variables, use the capabilities delivery
  3. Lack of experience and knowledge in the basic processes of software estimating
  4. The mistaken belief that the future cannot be discovered before it arrives 


Related articles Statistical Process Control The Basis of All Modern Control Systems Four Critical Elements of Project Success All Project Numbers are Random Numbers - Act Accordingly Everything is a Random Variable Critical Thinking Insight Probability Theory - A Primer How to "Lie" with Statistics How To Fix Martin Fowler's Estimating Problem in 3 Easy Steps
Categories: Project Management

Neo4j: Cypher – Finding movies by decade

Mark Needham - Sat, 06/28/2014 - 12:12

I was recently asked how to find the number of movies produced per decade in the movie data set that comes with the Neo4j browser and can be imported with the following command:

:play movies

We want to get one row per decade and have a count alongside so the easiest way is to start with one decade and build from there.

MATCH (movie:Movie)
WHERE movie.released >= 1990 and movie.released <= 1999
RETURN 1990 + "-" + 1999 as years, count(movie) AS movies
ORDER BY years

Note that we’re doing a label scan of all nodes of type Movie as there are no indexes for range queries. In this case it’s fine as we have few movies but If we had 100s of thousands of movies then we’d want to optimise the WHERE clause to make use of an IN which would then use any indexes.

If we run the query we get the following result:

==> +----------------------+
==> | years       | movies |
==> +----------------------+
==> | "1990-1999" | 21     |
==> +----------------------+
==> 1 row

Let’s pull out the start and end years so they’re explicitly named:

WITH 1990 AS startDecade, 1999 AS endDecade
MATCH (movie:Movie)
WHERE movie.released >= startDecade and movie.released <= endDecade
RETURN startDecade + "-" + endDecade as years, count(movie)
ORDER BY years

Now we need to create a collection of start and end years so we can return more than one. We can use the UNWIND function to take a collection of decades and run them through the rest of the query:

UNWIND [{start: 1970, end: 1979}, {start: 1980, end: 1989}, {start: 1980, end: 1989}, {start: 1990, end: 1999}, {start: 2000, end: 2009}, {start: 2010, end: 2019}] AS row
WITH row.start AS startDecade, row.end AS endDecade
MATCH (movie:Movie)
WHERE movie.released >= startDecade and movie.released <= endDecade
RETURN startDecade + "-" + endDecade as years, count(movie)
ORDER BY years
==> +----------------------------+
==> | years       | count(movie) |
==> +----------------------------+
==> | "1970-1979" | 2            |
==> | "1980-1989" | 2            |
==> | "1990-1999" | 21           |
==> | "2000-2009" | 13           |
==> | "2010-2019" | 1            |
==> +----------------------------+
==> 5 rows

Alistair pointed out that we can simplify this even further by using the RANGE function:

UNWIND range(1970,2010,10) as startDecade
WITH startDecade, startDecade + 9 as endDecade
MATCH (movie:Movie)
WHERE movie.released >= startDecade and movie.released <= endDecade
RETURN startDecade + "-" + endDecade as years, count(movie)
ORDER BY years

And here’s a graph gist for you to play with.

Categories: Programming

Fixing The Top Five Issues in Project Estimation

Sometimes you need a seeing eye dog to see the solution.

Sometimes you need a seeing eye dog to see the solution.

In the entry, The Top Five Issues In Project Estimation, we identified the five macro categories of estimation problems generated when I asked a group of people the question “What are the two largest issues in project estimation?”  Knowing what the issues are is important, however equally important is having a set of solutions.

  1. Requirements. Techniques that reduce the impact of unclear and changing requirements on budgeting and estimation include release plans, identifying a clear minimum viable product and changing how requirements changes are viewed when judging project success. See Requirements: The Chronic Problem with Project Estimation.
  2. Estimate Reliability. Recognize that budgets, estimates and plans are subject to the cone of uncertainty.  The cone of uncertainty is a reflection of the fact earlier in a project the less you know about the project.  Predictions of the future will be more variable the less you know about the project.  Budgets, estimates and plans are predictions of cost, effort, duration or size.
  3. Project History.  Collect predicted and actual project size, effort, duration and other project demographics for each project.  Project history can be used both as the basis for analogous estimates and/or to train parametric estimation tools.  The act of collecting the quantitative history and the qualitative story about how projects performed is a useful form of introspection that can drive change.
  4. Labor Hours Are Not The Same As Size.  Implement functional (e.g. IFPUG Function Points) or relative sizing (Story Points) as a step in the estimation process. The act of focusing on size separately allows estimators to gain greater focus on the other parts of the estimation process like team capabilities, processes, risks or changes that will affect velocity.  Greater focus leads to greater understanding, which leads to a better estimate.
  5. No One Dedicated to Estimation.  Estimating is a skill that that not only requires but practice to develop consistency.  While everyone should understand the concepts of estimation, consistency will be gained faster if someone is dedicated to learn and to execute the estimation process.

Solving the five macro estimation problems requires organizational change.  Many of the changes required are difficult because they are less about “how” to estimate and more about what we think estimates are, which leads into a discussion of why we estimate.  Organization’s budget and estimate to provide direction at a high level.   At this level budgets and estimates affect planning for tax accruals and for communicating portfolio level decisions to organizational stakeholders.  Investing in improving how organizations estimate will improve communication between CIOs, CFOs and business stakeholders.

Categories: Process Management

Episode 205: Martin Lippert on Eclipse Flux

Eberhard Wolff talks with Martin Lippert of Pivotal about the Eclipse Flux project. This projects is in its early stages — and has a very interesting goal: It aims to put software development tools into the cloud. It is a lot more than just an IDE (integrated development environment) in a browser. Instead the IDE […]
Categories: Programming

Data Science is the Art of Asking Better Questions

I heard a colleague make a great comment today …

“Data science is the art of asking better questions.

It’s not the art of finding a solution … the data keeps evolving.”

Categories: Architecture, Programming

Billions without Buzz

Eric.Weblog() - Eric Sink - Fri, 06/27/2014 - 19:00

I've been thinking a lot lately about the distorted perspective I get when I extrapolate from my daily sources of content.

Using Twitter buzz to sip from a firehose

I currently follow 304 people on Twitter. These people are a primary means for me to hear about stuff that is (1) happening, and (2) important to me.

I rely more on echoes than voices. For example, I don't follow Satya Nadella (@satyanadella). But whenever he tweets something I would find important, I hear about it anyway, because a dozen people I do follow are talking about it.

Twitter for me is all about conversations. It's about buzz. I'm interested in what people are talking about. And I like having the option of participating in the chatter.

And the people I choose to follow pretty much cover everything important. I don't miss anything that I need to know.

Or do I?

What am I missing?

Obviously, my method is designed to exclude information. Sometimes my buzz filter works exactly as I want. I rarely see any tweets about Miley Cyrus. That is "by design".

I also like the fact that buzz is naturally slanted toward things that are new even though lots of very important things are old. (As long as I don't get confused. If I extrapolate from buzz with a black-and-white mentality, I might believe that COBOL, SQL, and Subversion are dead because everybody has switched to Java, Mongo, and Git. Older technologies have little or no buzz, but tons of people are still using them everyday.)

The problem is the stuff that is new and important (to me) but has very little buzz.

The boring wave of B2B apps

I am absolutely convinced that the B2B wave of mobile apps has barely started.

As Benedict Evans (@BenedictEvans) says, mobile is eating the world. But so far, most of the action is with consumers, not businesses.

And this whole wave has been very high-buzz, mostly because everything had to get bigger, and the stuff that could not was, er, "disrupted".

Add two billion mobile devices to the world and lots of people are forced to think about scale in a whole new way:

  • Manufacturing. Foxconn is enormous. What was the biggest manufacturing facility on earth before the iPhone?

  • Venture Capital. I remember when $100M was an exit. Now it's a series A.

  • Servers. Mobile is accelerating the growth of the cloud, served by data centers that make Soldier Field look small.

  • Databases. When you have 500 million users, you can't afford unique constraints and foreign keys in the db layer anymore.

All this big-ness is causing a lot of disruption. And buzz.

Meanwhile, the majority of the corporate world is still trying to figure out what to do about mobile.

Consider this in terms of the classic marketing bell curve:

  • In the consumer world, mobile is in the conservatives, and starting to sell to the laggards.

  • In the corporate world, mobile is in the early adopters. It's pre-chasm.

Companies who still need high growth to justify their stock price (read: Apple) are distressed about the notion that everybody on the planet who can afford a smart phone already has one. Marketing reseach firms are still doing surveys asking corporate IT about when they are going to dive into mobile even as my Mom has two tablets.

Has there ever been a technology wave where the consumers got so far ahead of business?

I was one of those people in the early 80s who had a computer before they were prevalent in the business world. But I was a hobbyist and a nerd. In terms of volume and sheer revenue, adoption of PCs was driven by companies, not consumers. They wanted to run Lotus 1-2-3, so they led the way. It was a long time before computers solved real problems for consumers in the way that they solved real problems for business. In fact, I'd argue this didn't happen until around 1995 when the Web came along.

In mobile devices, consumers have gotten so far ahead that they have provided the client side of the infrastructure that Corporate IT will use. Going forward, it's going to be BYOD (Bring Your Own Device). Big companies are not going to buy 10,000 BlackBerries for their workforce when everybody already has a smart phone. Instead, they have to figure out how they're going to securely integrate all these different devices into their corporate systems (which is why cross-platform is getting even more important in the mobile space, which is contributing to Xamarin's pursuit of world domination). This is not the kind of policy to which Corporate IT is accustomed, and that is slowing them down even more.

But this is going to happen. A lot of enterprise apps are going to get written.

And this wave will be very low-buzz compared to the "Angry Birds and Candy Crush" wave. People won't be talking about it (and even when they do, the sound will get drowned out by the buzz over the Internet of Things wave).

Quite simply, the B2B apps wave is not interesting enough to get serious buzz:

  • It won't be as disruptive.

  • It won't re-challenge our notions of scale. (Walmart's 2.2 million employees would be a small user base by today's standards.)

  • It's going to be built on, and integrated with, technologies that are old. (Corporate IT will prefer to add mobile incrementally, with as little change to existing systems as possible.)

Compared to my first computer(s), an iPhone is a supercomputer. We are moving into a wave where companies like Procter & Gamble are going to gain operational efficiency because all their employees carry supercomputers in their pocket. And this will be considered boring.

But despite all the yawns, this wave is going to involve a lot of money. Many billions of dollars are going to move around as all these enterprise apps get written.

In other words, this is going to be boring in the same way that IBM has been boring for the last decade. This week, my Twitter feed is dominated by people talking about the Google I/O conference. Nobody is talking about IBM, even though they have almost twice as much revenue as Google. That's how buzz works.

If I want to closely follow something that doesn't have much buzz, I'm gonna have to work harder.

Let's talk about Alex Bratton

I am currently reading a book called Billion Dollar Apps, by Alex Bratton (@alexbratton), CEO of Lextech, a custom mobile app dev shop in Chicago.

My buzz filter certainly didn't find this book for me:

  • @alexbratton has even fewer Twitter followers than I do.

  • @LextechApps has fewer followers than some high school students I know.

  • Almost all of Bratton's followers are people I don't know.

  • Google searches reveal no apparent connection between Lextech and Xamarin.

  • Twitter searches suggest that Bratton and Lextech have seldom or never been mentioned by Scott Hansleman (@shanselman).

There just aren't many connections between Lextech's world and mine. In fact, it looks like maybe I am the only occupant of the overlapping portion of the Venn diagram.

Maybe Lextech just doesn't have much buzz. Or maybe Lextech has the kind of buzz that happens mostly off Twitter. Maybe the CIOs of Fortune 500 companies share rides on their private jets where they sip Macallan 25 and talk about how great Lextech is.

Whatever. I've been following Alex Bratton on Twitter largely because he and I were students at UIUC around the same time, and I sort-of vaguely remember him. Were it not for this very thin college connection, I might never have heard of him or his company or his book.

And that would be sad, because Lextech's activities land squarely in an area of interest for me. Bratton's company may be low-buzz, but they're building B2B mobile apps for banner-name companies including GE, Fidelity, John Deere, Blue Cross, and H&R Block. They're riding the front of the wave. I think that's pretty darn cool.

So I am halfway through the book, and two things are already quite clear:

  • The book is really good.

  • I find it boring.

Basically, I'm not enjoying this book because it was clearly not written for people like me. I enjoy fiction books about crime investigation and books with lots of curly braces and semicolons. Billion Dollar Apps is neither of these. It's written for CxOs at big companies who have to make big decisions about mobile technology.

But I'm forcing myself to read this book for the same reason I force myself to eat certain unappealing vegetables. I don't like the taste, but I think it's good for me.

Thinking about Lextech and its customers is forcing me to widen my perspective. It is reminding me that the boredom goes both ways:

  • Most developers don't care about the details of how John Deere is using mobile to make its business run better. They tend to focus more on the technology than on solutions to problems.

  • Probably nobody at John Deere is spending their time critiquing the design of Xamarin.Forms or figuring out best practices for PCLs. They see mobile technology as a solution to their problems.

This book is good because it speaks to its intended audience, and that group of people doesn't care about the details of why certain .NET things are incompatible with the AOT compiler in Xamarin.iOS. Billion Dollar Apps is a book that talks about sensibly applying mobile technology to make regular non-geek businesses work better.

Like most other geeks, I am prone to getting lost in technology for its own sake. But I'm not just a developer. I'm an entrepreneur. I need to keep a sense of balance.

So, I don't particularly want to read this book, but I need to read it.

Bratton's book is about the beginning of a big boring bazaar where beaucoup billions bounce bereft of buzz.

(Sorry, the forced alliteration was shameful, but I couldn't resist.)

Bottom line

I've wandered around a bit, so let me close with a summary of key points you might take away from this blog entry:

  • Consider reading Alex Bratton's book. If you are a CxO of a Fortune 500 company (and you probably are not, because you're reading my blog), you'll probably like it. If you are a developer (and you probably are, because you're reading my blog), you probably won't like it any more than I do. Eat your brussel sprouts.

  • Widening your perspective is always good advice. If you're like me, there's a really good chance that your daily information flow has boundaries. Those boundaries make you efficient, but they also constantly protect you from seeing the perspective of people who are not like you. Look outside your boundaries.

  • If you are interested in the boring B2B wave of mobile apps, so am I. Maybe we should talk. Maybe I should be following you on Twitter. Maybe we should see if we can create a little buzz.


Don't be an Accidental Project Manager

Herding Cats - Glen Alleman - Fri, 06/27/2014 - 18:44

A common problem in our development of the Program Management Office is getting so caught up in putting out fires. This is Covey's “addiction of the urgent.” In this process we lose the big-picture perspective. This note is about the big-picture view of the project management process as it pertains to our collection of projects. These are very rudimentary principles, but they are important to keep in mind.

5 Basic Principles

1. Be conscious of what you're doing, don’t be an accidental manager. Learn PM theory and practice. Realize you don't often have direct control. Focus on being a professional and the PM's mantra:

"I am a project professional. I work on projects. Projects are undertakings that are goal-oriented, complex, finite, and unique. They pass through a life cycle, which begins with project selection and ends with project termination."

2. Invest in front-end work; get it right the first time. We often leap before we look due to an over–focus on results-oriented processes, simple and many times simple-minded platitudes about project management and the technical processes and ignore basic steps. Trailblazers often achieve breakthroughs, but projects need forethought. Projects are complex, and the planning, structure, and time spent with stakeholders are required for success. Doing things right takes time and effort, but this time and effort is much cheaper than rework.

3. Anticipate the problems that will inevitably arise. Most problems are predictable. Well-known examples are:

  • Little direct control over staff, little staff commitment to the project.
  • Staff workers are not precisely what we want or need.
  • Functional managers have different goals, and these will suboptimize the project.
  • Variances to schedule and budget will occur, and customer needs will shift.
  • Project requirements will be misinterpreted.
  • Overplanning and overcontrol are as bad as underplanning and weak control.
  • There are hidden agendas, and these are probably more important than the stated one.

4. Go beneath surface illusions; dig deep to find the real situation. Don't accept things at face value. Don't treat the symptom, treat the root cause, and the symptoms will be corrected. Our customers usually understands their own needs, but further probing will bring out new needs. Robert Block suggests a series of steps: 

  • Identify all the players, in particular those who can impact project outcome.
  • Determine the goals of each player and organization, focusing on hidden goals.
  • Assess your own situation and start to define the problems.

5. Be as flexible as possible; don’t get sucked into unnecessary rigidity and formality. Project Management is the reverse of Fermi's 2nd law: we're trying to create order out of chaos. But in this effort:

  • More formal structure & bureaucracy doesn't necessarily reduce chaos.
  • We need flexibility to bend but not break to deal with surprises, especially with intangibles our information-technology projects.
  • The goal is to have both order and flexibility at the same time.
  • Heavy formality is appropriate on large budget or low-risk projects with lots of communication expense and few surprises. Information-age projects have a low need for this because they deal more with information and intangibles, and have a high degree of uncertainly.

[1] The Politics of Projects, Robert Block, Yourdon Press, 1983.

Related articles Elements of Project Success How to Deal With Complexity In Software Projects?
Categories: Project Management


Herding Cats - Glen Alleman - Fri, 06/27/2014 - 16:11

When we hear about a process, a technique, or a tool, ask in what unit of measure are you assessing the beneficial outcome of applying those?

This idea started with P. W. Bridgman's principle that the meaning of any concept is in its measurement or other test. This was put forth in the 1930's in which Bridgman made a famous, useful, and very operational statement, usually remembered as:

The scientific method is doing your damnedest, no holds barred. †

Developing software is not a scientific process, even though Computer Science is a discipline at the university level, where probability and statistics are taught, IEEE/ACM Computer Science Education Curricula

When we want to make choices about a future outcome, we can apply statistical thinking using the mathematics  used in scientific discussions - cost, schedule, and performance (C,S,P which are random variables).

These decisions are based on the probabilistic and statistical behavior of the underlying processes that create the alternatives for our decisions. Should we spend $X on a system that will return $Y value? Since both X and Y are random variables - they are in the future - our decision making processes needs to estimate the behaviour of these random variables and determine the impact on our outcomes.

Probability and Statistics

When we hear there are alternatives to making decisions about the future impacted by cost, schedule and technical performance without estimating the impact of that decision, we need to ask what are those alternatives, what are their units of measure, and when can we find them described?

For those interested in further reading on the topic of Decision Making in the Presence of Uncertainty

† Reflections of a Physicist, P. W. Bridgman, pp. 535. The passage reads, "The scientific method, as far as it is a method, is nothing more than doing one's damnedest with one's mind, no holds barred."

Categories: Project Management

Neo4j: Cypher – Separation of concerns

Mark Needham - Fri, 06/27/2014 - 11:51

While preparing my talk on building Neo4j backed applications with Clojure I realised that some of the queries I’d written were incredibly complicated and went against anything I’d learnt about separating different concerns.

One example of this was the query I used to generate the data for the following page of the meetup application I’ve been working on:

2014 06 27 08 19 34 2014 06 27 08 31 13

Depending on the selected tab you can choose to see the people signed up for the meetup and the date that they signed up or the topics that those people are interested in.

For reference, this is an outline of the schema of the graph behind the application:

2014 06 27 11 51 00

This was my initial query to get the data:

MATCH (event:Event {id: {eventId}})-[:HELD_AT]->(venue)
OPTIONAL MATCH (event)<-[:TO]-(rsvp)<-[:RSVPD]-(person)
OPTIONAL MATCH (person)-[:INTERESTED_IN]->(topic) WHERE ()-[:HAS_TOPIC]->(topic)
WITH event, venue, rsvp, person, COLLECT(topic) as topics ORDER BY rsvp.time
OPTIONAL MATCH (rsvp)<-[:NEXT]-(initial)
WITH event, venue, COLLECT({rsvp: rsvp, initial: initial, person: person, topics: topics}) AS responses
WITH event, venue,
    [response in responses WHERE response.initial is null AND = "yes"] as attendees,
    [response in responses WHERE NOT response.initial is null] as dropouts, responses
UNWIND([response in attendees | response.topics]) AS topics
UNWIND(topics) AS topic
WITH event, venue, attendees, dropouts, {id:,, freq:COUNT(*)} AS t
RETURN event, venue, attendees, dropouts, COLLECT(t) AS topics

The first two lines of the query works out which people have RSVP’d to a particular event, the 3rd line captures the topics they’re interested in as long as the topic is linked to at least one of the NoSQL London groups.

We then optionally capture their initial RSVP in case they’ve changed it before doing a bit of data manipulation to group everything together.

If we run a slight variation of that which only shows a few of the topics, attendees and dropouts this is the type of result we get:

|                               |      | [a IN attendees[0..5] |]                                 | [d in dropouts[0..5] |]                              | topics[0..5]                                                                                                                                                                                                                                                    |
| "Building Neo4j backed web applications" | "Skills Matter" | ["Mark Needham","Alistair Jones","Jim Webber","Axel Morgner","Ramesh"] | ["Frank Gibson","Keith Hinde","Richard Mason","Ollie Glass","Tom"] | [{id -> 10538, name -> "Business Intelligence", freq -> 3},{id -> 61680, name -> "HBase", freq -> 3},{id -> 61679, name -> "Hive", freq -> 2},{id -> 193021, name -> "Graph Databases", freq -> 12},{id -> 85951, name -> "JavaScript Frameworks", freq -> 10}] |

The problem is we’ve mixed together two different concerns – the attendees to a meetup and the topics they’re interested in – which made the query quite hard to understand when I came back to it a couple of months later.

Instead what we can do is split the query in two and make two different calls to the server. We then end up with the following:

// Get the event + attendees + dropouts
MATCH (event:Event {id: {eventId}})-[:HELD_AT]->(venue)
OPTIONAL MATCH (event)<-[:TO]-(rsvp)<-[:RSVPD]-(person)
WITH event, venue, rsvp, person ORDER BY rsvp.time
OPTIONAL MATCH (rsvp)<-[:NEXT]-(initial)
WITH event, venue, COLLECT({rsvp: rsvp, initial: initial, person: person}) AS responses
WITH event, venue,
    [response in responses WHERE response.initial is null 
                           AND = "yes"] as attendees,
    [response in responses WHERE NOT response.initial is null] as dropouts
RETURN event, venue, attendees, dropouts
// Get the topics the attendees are interested in
MATCH (event:Event {id: {eventId}})
MATCH (event)<-[:TO]-(rsvp {response: "yes"})<-[:RSVPD]-(person)-[:INTERESTED_IN]->(topic)
WHERE ()-[:HAS_TOPIC]->(topic)
RETURN AS id, AS name, COUNT(*) AS freq

The first query is still a bit complex but that’s because there’s a bit of tricky logic to distinguish people who signed up and dropped out. However, the second query is now quite easy to read and expresses it’s intent very clearly.

Categories: Programming

How do I split these 40 stories?

Good Requirements - Jeffrey Davidson - Fri, 06/27/2014 - 04:15

A student from last week’s Agile Bootcamp Class taught at Harvard University (yes, that Harvard) asks,


Question: I have a project that involves 3 different users who want approximately 25 new fields on an application. Most of these fields are view only with the exception of 4-5 fields which allows input changes. However, user #1 wants to see all 25 fields, user #2 wants to see only 10 of the same fields, and user #3 wants to see 5 of the same fields. How should I approach writing these stories? Would I write a story for each user and each field – which could potentially be 40 stories? Or should I just combine the fields that all 3 users would want to view? For example, one field might be “name”. My user story might say“As a user #1, user #2, user #3, I want to see all names of employees eligible for an annual salary increase so that I can view all eligible employees.”


Answer: This sounds like an interesting set-up. My answers, of course, will be a bit general because I don’t know all the specifics. Also, I will make more stories if the developers are unfamiliar with the systems / tables / data, and probably fewer stories if the team knows quite a bit about the different systems. With caveats out of the way, let’s dig in! First, 40 stories? Yuck. Too many. Second, because you have 3 different users, I would start with stories for satisfying all of them (unless there is a reason to focus on just one). My presumption is doing this adds value the quickest, which is my guiding answer for how to break up stories.

  1. As user #3, I want to view < information from 1 field> so I can do my job.
  2. As user #3, I want to see < more information > so I can . . . .
  3. As user #2, I want to view < even more information > so I can . . . .
  4. As user #1, I want to see < all the information > so I can . . . .

The point of #1 is to prove we can display information, while the other stories add more details for the same and additional users. None of these is about editing the data. Depending on what keeps the users and developers happiest, there are a couple of options. I can insert story 1A; Edit the first field. After this I would insert more edit stories, probably grouped like the last 3 stories above, as appropriate. A different approach might be to insert the edit stories after the last story. Again, where is the value?

A couple side notes:
  • It doesn’t make much difference if you use “see” or “view,” the goal is understanding, not the worlds best grammar.
  • I would be remiss if I didn’t mention the value statement in your user story is a bit weak. What is the specific reason to view eligible employees; the actual awarding of salary increases, a validation check, a fascination with other people’s salary, something else entirely?
Categories: Requirements

Coming out of the closet - the life and adventure of a traditional project manager turned Agilist

Software Development Today - Vasco Duarte - Fri, 06/27/2014 - 04:00

I’m coming out of the closet today. No, not that closet. Another closet, the tabu closet in the Agile community. Yes, I was (and to a point still am) a control freak, traditional, command and control project manager. Yes, that’s right you read it correctly. Here’s why this is important: in 2003 when I first started to consider Agile in any shape or form I was a strong believer of the Church of Order. I did all the rites of passage, I did my Gantt charts, my PERT charts, my EVM-charts and, of course, my certification.

I was certified Project Manager by IPMA, the European cousin of PMI.

I too was a control freak, order junkie, command and control project manager. And I've been clean for 9 years and 154 days.

Why did I turn to Agile? No, it wasn’t because I was a failed project manager, just ask anyone who worked with me then. It was the opposite reason. I was a very successful project manager, and that success made me believe I was right. That I had the recipe. After all, I had been successful for many years already at that point.

I was so convinced I was right, that I decided to run our first Agile project. A pilot project that was designed to test Agile - to show how Agile fails miserably (I thought, at that time). So I decided to do the project by the book. I read the book and went to work.

I was so convinced I was right that I wanted to prove Agile was wrong. Turned out, I was wrong.

The project was a success... I swear, I did not see that coming! After that project I could never look back. I found - NO! - I experienced a better way to develop software that spoiled me forever. I could no longer look back to my past as a traditional project manager and continue to believe the things I believed then. I saw a new land, and I knew I was meant to continue my journey in that land. Agile was my new land.

Many of you have probably experienced a similar journey. Maybe it was with Test-Driven Development, or maybe it was with Acceptance Testing, or even Lean Startup. All these methods have one thing in common: they represent a change in context for software development. This means: they fundamentally change the assumptions on which the previous methods were based. They were, in our little software development world a paradigm shift.

Test-driven development, acceptance testing, lean startup are methods that fundamentally change the assumptions on which the previous software development methods were based.

NoEstimates is just another approach that challenges basic assumptions of how we work in software development. It wasn’t the first, it will not be the last, but it is a paradigm shift. I know this because I’ve used traditional, Agile with estimation, and Agile with #NoEstimates approaches to project management and software delivery.

A world premier?

That’s why me and Woody Zuill will be hosting the first ever (unless someone jumps the gun ;) #NoEstimates public workshop in the world. It will happen in Finland, of course, because that’s the country most likely to change the world of software development. A country of only five million people yet with a huge track record of innovation: The first ever mobile phone throwing world championship was created in Finland. The first ever wife-carrying world championship was created in Finland. The first ever swamp football championship was created in Finland. And my favourite: the Air Guitar World Championship is hosted in Finland.

#NoEstimates being such an exotic approach to software development it must, of course, have its first world-premier workshop in Finland as well! Me and Woody Zuill (his blog) will host a workshop on #NoEstimates on the week of October 20th in Helsinki. So whether you love it, or hate it you can meet us both in Helsinki!

In this workshop will cover topics such as:

  • Decision making frameworks for projects that do not require estimates.
  • Investment models for software projects that do not require estimates.
  • Project management (risk management, scope management, progress reporting, etc.) approaches that do not require estimates.
  • We will give you the tools and arguments you need to prove the value of #NoEstimates to your boss, and how to get started applying it right away.
  • We will discuss where we see #NoEstimates going and what are the likely changes to software development that will come next. This is the future delivered to you!

Which of these topics interest you the most? What topics would you like us to cover in the workshop. Tell us now and you have a chance to affect the topics we will cover.

Contact us at and tell us. We will reply to all emails, even flame bombs! :)

You can receive exclusive content (not available on the blog) on the topic of #NoEstimates, just subscribe to the #NoEstimates mailing list below. As a bonus you will get my #NoEstimates whitepaper, where I review the background and reasons for using #NoEstimates #mc_embed_signup{background:#fff; clear:left; font:14px Helvetica,Arial,sans-serif; } /* Add your own MailChimp form style overrides in your site stylesheet or in this style block. We recommend moving this block and the preceding CSS link to the HEAD of your HTML file. */ Subscribe to our mailing list* indicates required Email Address * First Name Last Name

Picture credit: John Hammink, follow him on twitter

The Top Five Issues In Project Estimation


Sometimes estimation leaves you in a fog!

Sometimes estimation leaves you in a fog!

When I recently asked a group of people the question “What are the two largest issues in project estimation?”, I received a wide range of answers. The range of answers is probably a reflection of the range of individuals answering.  Five macro categories emerged from the answers. They are:

  1. Requirements. The impact of unclear and changing requirements on budgeting and estimation was discussed in detail in the entry, Requirements: The Chronic Problem with Project Estimation.  Bottom line, change is required to embrace dynamic development methods and that change will require changes in how the organization evaluates projects.
  2. Estimate Reliability. The perceived lack of reliability of an estimate can be generated by many factors including differences in between development and estimation processes. One of the respondents noted, “most of the time the project does not believe the estimate and thus comes up with their own, which is primarily based on what they feel the customer wants to hear.”
  3. Project History. Both analogous and parametric estimation processes use the past as an input in determining the future.  Collection of consistent historical data is critical to learning and not repeating the same mistakes over and over.  According to Joe Schofield, “few groups retain enough relevant data from their experiences to avoid relearning the same lesson.”
  4. Labor Hours Are Not The Same As Size.  Many estimators either estimate the effort needed to perform the project or individual tasks.  By jumping immediately to effort, estimators miss all of the nuances that effect the level of effort required to deliver value.  According to Ian Brown, “then the discussion basically boils down to opinions of the number of hours, rather that assessing other attributes that drive the number of hours that something will take.”
  5. No One Dedicated to Estimation.  Estimating is a skill built on a wide range of techniques that need to be learned and practiced.  When no one is dedicated to developing and maintaining estimates it is rare that anyone can learn to estimate consistently, which affects reliability.  To quote one of the respondents, “consistency of estimation from team to team, and within a team over time, is non-existent.”


Each of the top five issues are solvable without throwing out the concept of estimation that are critical for planning at the organization, portfolio and product levels.  Every organization will have to wrestle with their own solution to the estimation conundrum. However the first step is to recognize the issues you face and your goals from the estimation process.

Categories: Process Management

Cast Away with Android TV and Google Cast

Android Developers Blog - Thu, 06/26/2014 - 22:02

By Dave Burke and Majd Bakar, Engineering Directors and TV Junkies

Last summer, we launched Chromecast, a small, affordable device that lets you cast online video, music and anything from the web to your TV. Today at Google I/O, we announced Android TV, the newest form factor to the Android platform, and a way to extend the reach of Google Cast to more devices, like televisions, set-top boxes and consoles.

Check out Coming to a Screen Near You for some details on everything we’re doing to make your TV the place to be.

For developers though--sorry, you don’t get to unwind in front of the TV. We need you to get to work and help us create the best possible TV experience, with all of the new features announced at I/O today.

Get started with Android TV

In addition to Google Cast apps that send content to the TV, you can now build immersive native apps and console-style games on Android TV devices. These native apps work with TV remotes and gamepads, even if you don’t have your phone handy. The Android L Developer Preview SDK includes the new Leanback support library that allows you to design smoother, simpler, living room apps.

And this is just the beginning. In the fall, new APIs will allow you to cast directly to these apps, so users can control the app with the phone, the remote, or even their Android Wear watch. You’ll also start seeing Android TV set-top boxes, consoles and televisions from Sony, TP Vision, Sharp, Asus, Razer and more.

Help more users find your Google Cast app

We want to help users more easily find your content, so we’ve improved the Google Cast SDK developer console to let you upload your app icon, app name, and app category for Android, iOS and Chrome. These changes will help your app get discovered on and on Google Play.

Additional capabilities have also been added to the Google Cast SDK. These include: Media Player Library enhancements, bringing easier integration with MPEG-DASH Smooth Streaming, and HLS. We’ve also added WebAudio & WebGL support, made the Cast Companion Library available, and added enhanced Closed Caption support. And coming soon, we will add support for queuing and ID delegation.

Ready to get started? Visit and for the SDKs, style guides, tutorials, sample code, and the API references. You can also request an ADT-1 devkit to bootstrap your Android TV development.

.footer__link, .footer__text{ color: #8b8b8b !important; display: block; font-size: 12px; line-height: 1.33333; margin-bottom: 10px; text-decoration: none; } .hoverable .footer__link:hover { text-decoration: underline; }

Google I/O 2014 I/O Livestreams I/O Bytes Videos +Android Developers

L Developer Preview Material Design Android Wear Android TV Android Auto

Get it on Google Play
Categories: Programming

Games at Google I/O '14: Everyone's Playing Games

Android Developers Blog - Thu, 06/26/2014 - 22:01

By Greg Hartrell, Product Manager, Google Play games

With Google I/O ‘14 here, we see Android and Google Play as a huge opportunity for game developers: 3 in 4 Android users are playing games, and with over one billion active Android users around the world, games are reaching and delighting almost everyone.

At Google, we see a great future where mobile and cloud services bring games to all the screens in your life and connect you with others. Today we announced a number of games related launches and upcoming technologies across Google Play Games, the Android platform and its new form factors.

Google Play Games

At last year’s Google I/O, we announced Google Play Games -- Google’s online game platform, with services and user experiences designed to bring players together and take Android and mobile games to the next level.

Google Play Games has grown at tremendous speed, activating 100 million users in the past 6 months. It’s the fastest growing mobile game network, and with such an incredible response, we announced more awesome enhancements to Google Play Games today. Game Profile

The Play Games app now gives players a Game Profile, where they earn points and vanity titles from unlocking achievements. Players can also compare their profile with friends. Developers can benefit from this meta-game by continuing to design great achievements that reward players for exploring all the content and depth of their game.

Quests and Saved Games

Two new game services will launch with the next update for Google Play Services on Android, and through the Play Games iOS SDK:

  • Quests is a service that enables developers to create online, time-based goals in their games without having to launch an update each time. Now developers can easily run weekend or daily challenges for their player community, and reward them in unique ways.
  • Saved Games is a service that stores a player’s game progress across many screens, along with a cover image, description and total time played. Players never have to play level 1 again by having their progress stored with Google, and cover images and descriptions are used in Play Games experiences to indicate where they left off and attract them to launch their favorite game again.

We have many great partners who have started integrating Quests and Saved Games, here are just a few current or upcoming games.

More tools for game developers

Other developer tools are now available for Play Games, including:

  • Play Games Statistics — Play Games adopters get easy effort game analytics through the Google Play Developer console today, including visualization of Player & Engagement statistics. What’s new is aggregated player demographic information for signed-in users, so you can understand the distribution of your player’s ages, genders and countries.
  • Play Games C++ SDK is updated with more cross-platform support for the new services and experiences we announced. Cocos2D-x, a popular game engine, is an early adopter of the Play Games C++ SDK bringing the power of Play Games to their developers.
Game enhancements for the Android Platform

With the announcement of the developer preview of the Android L-release, there are some new platform capabilities that will make Android an even more compelling platform for game development.

  • Support for OpenGL ES 3.1 in the L Developer Preview — Introducing powerful features like compute shaders, stencil textures, and texture gather, enables more interesting physics or pixel effects on mobile devices. Additional API and shading language improvements improve usability and reduce overhead.
  • Android Extension Pack (AEP) in the L Developer Preview — a new set of extensions to OpenGL ES that bring desktop class graphics to Android. Games will be able to take advantage of tessellation and geometry shaders, and use ASTC texture compression.

    We're pleased to be working with different GPU vendors to adopt AEP including Nvidia, ARM, Qualcomm, and Imagination Technologies.

  • Google Gamepad standards — We recently published a standard for gamepad input for OEMs and partners who create and enable these awesome input devices on Android. The standard makes this input mechanism compatible across Google platforms on Android, Chrome and Chromebooks. You can learn more here: Supporting Game Controllers.
Play Games on Android TV

And Google's game network is a part of the Android TV announcement — so think of Android on a TV, with a rich interface on a large screen, and fun games in your living room! Players will be able to earn achievements, climb leaderboards and play online with friends from an Android TV. This is only available through the developer preview, so game developers seeking a hardware development kit (the ADT-1) can make a request at

Updates rolling out soon

That’s a lot of games announcements! Our Play Games changes will roll out over the next few weeks with the update of Google Play Services and the Play Games App, and Android L-release changes are part of the announced developer preview. This gets us a big step closer to a world where Android and our cloud services enable games to reach all the screens in your life and connect you with others.

Greg Hartrell is the lead product manager for Google Play Games: Google's game platform that helps developers reach and unite millions of players. Before joining Google, he was VP of Product Development at Capcom/Beeline, and prior to that, led product development for 8 years at Microsoft for Xbox Live/360 and other consumer and enterprise product lines. In his spare time, he enjoys flying birds through plumbing structures, boss battles and pulling rare objects out of mystery boxes.

.footer__link, .footer__text{ color: #8b8b8b !important; display: block; font-size: 12px; line-height: 1.33333; margin-bottom: 10px; text-decoration: none; } .hoverable .footer__link:hover { text-decoration: underline; }

Google I/O 2014 I/O Livestreams I/O Bytes Videos +Android Developers

L Developer Preview Material Design Android Wear Android TV Android Auto

Get it on Google Play
Categories: Programming

New in Android: L Developer Preview and Google Play Services 5.0

Android Developers Blog - Thu, 06/26/2014 - 22:00

By Jamal Eason, Product Manager, Android

Earlier today, at Google I/O, we showed a number of projects we’ve been working on to the thousands of developers in the audience and the millions more tuning in on the livestream. These projects extend Android to the TV (Android TV), to the car (Android Auto) and to wearables (Android Wear), among others.

At Google, our focus is providing a seamless experience for users across all of the screens in their lives. An important component to that is making sure that you as developers have all of the tools necessary to easily deploy your apps across to those screens. Increasingly, Android is becoming the fabric that weaves these experiences together, which is why you’ll be excited about a number of things we unveiled today.

Android L Developer Preview

For the first time since we launched Android, we’re giving you early access to a development version of an upcoming release. The L Developer Preview, available starting tomorrow, lets you explore many of the new features and capabilities of the next version of Android, and offers everything you need to get started developing and testing on the new platform. This is important because the platform is evolving in a significant way — not only for mobile but also moving beyond phones and tablets. Here are a few of the highlights for developers:

  • Material design for the multiscreen world — We’ve been working on a new design language at Google that takes a comprehensive approach to visual, motion, and interaction design across a number of platforms and form factors. Material design is a new aesthetic for designing apps in today’s multi-device world. The L Developer Preview brings material design to Android, with a full set of tools for your apps. The system is incredibly flexible, allowing your app to express its individual character and brand with bold colors and a variety of responsive UI patterns and themeable elements.
  • Enhanced notifications — New lockscreen notifications let you surface content, updates, and actions to users at a glance, without unlocking. Visibility controls let you manage the types of information shown on the lockscreen. Heads-up notifications display content and actions in a small floating window that’s managed by the system, no matter which app is in the foreground. Notifications are material themed and you can express your brand through accent colors and more.
  • Document-centric Recents — Now you can organize your app by tasks and present these concurrently as individual “documents” in the Recents screen. Users can flip through Recents to find the specific task they want and then jump deep into your app with a single tap.
  • Project Volta — New tools and APIs help your app run efficiently and conserve power. Battery Historian is a new tool that lets you visualize power events over time and understand how your app is using battery. A job scheduler API lets you set the conditions under which your background tasks and other jobs should run, such as when the device is idle or connected to an unmetered to a charger, to minimize battery impact.
  • BLE Peripheral Mode — Android devices can now function in Bluetooth Low Energy (BLE) peripheral mode. Apps can use this capability to broadcast their presence to nearby devices — for example, you can now build apps that let a device to function as a pedometer or health monitor and transmit data to another BLE device.
  • Multi-networking — Apps can work with the system to dynamically scan for available networks with specific capabilities and then automatically connect. This is useful when you want to manage handoffs or connect to a specialized network, such as a carrier-billing network.
  • Advanced camera capabilities — A new camera API gives you new capabilities for image capture and processing. On supported devices, your app can capture uncompressed YUV capture at full 8 megapixel resolution at 30 FPS. The API also lets you capture raw sensor data and control parameters such as exposure time, ISO sensitivity, and frame duration, on a per-frame basis.
  • New features for game developers — Support for OpenGL ES 3.1, gives you capabilities such as compute shaders, stencil textures, and texture gather for your games. Android Extension Pack (AEP) is a new set of extensions to OpenGL ES that bring desktop-class graphics to Android. Games will be able to take advantage of tessellation and geometry shaders, and use ASTC texture compression across multiple GPU techonolgies.
  • Android Runtime (ART) — The L Developer Preview introduces the Android Runtime (ART) as the system default. ART offers ahead-of-time (AOT) compilation, more efficient garbage collection, and improved development and debugging features. In many cases it improves performance of the device with no action required by the developer.
  • 64-bit support — The L Developer Preview adds support for 64-bit ABIs, for additional address space and improved performance with certain compute workloads. Apps written in the Java language can run immediately on 64-bit architectures with no modifications required. To support apps using native code, we’re also releasing an updated NDK that includes 64-bit support.

Watch for more details coming out tomorrow (26 June) on what’s in the L Developer Preview and how to get it.

Google Play Services 5.0

Along with the L Developer Preview, we also announced a new version of Google Play services that brings new capabilities and the latest optimizations to devices across the Android ecosystem. Google Play services ensures that you can build on the latest features from Google for your users, with the confidence that those services will work properly everywhere. The latest version has begun rolling out and here are some of the highlights:

  • Services for Android wearables — Your apps can more easily communicate and sync with code running on Android wearables through an automatically synchronized, persistent data store and a reliable messaging interface.
  • Play Games services — Build a great gaming experience with Quests, which allow event-based challenges for players to complete for rewards, Saved Games (a snapshot API allow synchronization of game data along with a cover-image and description), and Game Profile (providing experience points for players).
  • App Indexing API — Surface deep content in your native mobile applications on Google search and drive additional user engagement.
  • Google Cast — Use media tracks to enable closed-caption support for Chromecast.
  • Drive — Sort query results, create offline folders, and select any mime type in the file picker by default.
  • Wallet — Build a "Save to Wallet" button for offers directly into your app; use geo-fenced in-store notifications to prompt the user to show and scan digital cards. Split tender allows payment to be split between Wallet Balance and a credit/debit card in Google Wallet.
  • Analytics — Get insights into the full user journey and understand how different user acquisition campaigns are performing with Enhanced Ecommerce, letting you measure product impressions, product clicks, and more.
  • Mobile Ads — Use improved in-app purchase ads and integrations for the Play store in-app purchase API client.
  • Dynamic Security Provider — Offers an alternative to the platform's secure networking APIs that can be updated more frequently, for faster delivery of security patches.

We expect the rollout of Google Play services 5.0 to take several days, after which time you’ll be able to get started developing with these new APIs.

Join us at the Google I/O sessions

If you’d like to learn more, join us for sessions on Android development, material design, game development, and more. You’ll find the full session list on the Google I/O 2014 site, and you can filter the schedule to find livestreamed sessions of interest.

.footer__link, .footer__text{ color: #8b8b8b !important; display: block; font-size: 12px; line-height: 1.33333; margin-bottom: 10px; text-decoration: none; } .hoverable .footer__link:hover { text-decoration: underline; }

Google I/O 2014 I/O Livestreams I/O Bytes Videos +Android Developers

L Developer Preview Material Design Android Wear Android TV Android Auto

Get it on Google Play
Categories: Programming

Google I/O: Design, Develop, Distribute

Android Developers Blog - Thu, 06/26/2014 - 21:59

By Monica Tran, Head of Developer Marketing

Today at Moscone, we kicked off our 7th annual Google I/O. This year, we’re focusing on three key themes: design, develop, distribute, helping you build your app from start to finish.

It’s been amazing to see how far you’ve come: in fact, since the last Google I/O, we’ve paid developers more than $5 billion, a testament to the experiences you’re creating. In the keynote, we had a number of announcements geared towards meeting the user wherever they go: on the TV, in the car and on your wrist. Below is a taste of some of the goodies we unveiled to help you along the way.

  • Material design — we introduced material design, which uses tactile surfaces, bold graphic design, and fluid motion to create beautiful, intuitive experiences.
  • L-Release of Android, with material design — Bringing material design to Android is a big part of the L-Release of Android: we’ve added the new Material theme (which you can apply to your apps for a new style) and the ability to specify a view’s elevation, allowing you to cast dynamic, real-time shadows in your apps.
  • Bringing material design to Polymer — As a developer, you’ll now have access to all the capabilities of material design via Polymer, bringing tangibility, bold graphics, and animations to your applications on the web, all at 60fps.
  • Android L Developer Preview — Get extra lead time to make great apps for the next version of Android, with lots of new APIs to make Android simpler and more consistent on screens everywhere
  • Google Play services 5.0 is rolling out worldwide with great new features for developers.
  • Android TV SDK — Explore, learn and build apps and games for the biggest screen in the home. Your hard work will pay off in the fall when Asus, Razer and other partners launch their first Android TV devices.
  • Google Cast SDK — Help users find your content more easily with the improved Google Cast SDK developer console, which lets your app get discovered on and on Google Play.
  • Android Auto SDK coming — Bring your app experience to the car by extending your existing app with Android Auto APIs. Be in millions of cars — with just one app.
  • Google Fit — An open fitness platform giving users control of their fitness data so that developers can focus on building smarter apps and manufacturers can focus on creating amazing devices.
  • Gaming — Learn what's new about Google Play Games and the Android platform to take games to the next level.
  • Google Cloud Platform — Get help with debugging, tracing, and monitoring applications in with new developer productivity tooling. Also, try Cloud Dataflow, a new fully managed service that simplifies the process of creating data pipelines.
  • The new Gmail API — Add Gmail features to your app with RESTful access to threads, messages, labels, drafts and history.
  • Android features for Enterprise — Secure apps and data without complicating the user experience. Build for the enterprise with no changes to the apps you're already developing. Learn more here.
DISTRIBUTE .footer__link, .footer__text{ color: #8b8b8b !important; display: block; font-size: 12px; line-height: 1.33333; margin-bottom: 10px; text-decoration: none; } .hoverable .footer__link:hover { text-decoration: underline; }

Google I/O 2014 I/O Livestreams I/O Bytes Videos +Android Developers

L Developer Preview Material Design Android Wear Android TV Android Auto

Get it on Google Play
Categories: Programming

Android L Developer Preview and Android Studio Beta

Android Developers Blog - Thu, 06/26/2014 - 21:57

By Jamal Eason, Product Manager, Android

At the Google I/O keynote yesterday we announced the L Developer Preview — a development version of an upcoming Android release. The Developer Preview lets you explore features and capabilities of the L release and get started developing and testing on the new platform. You can take a look at the developer features and APIs in the API Overview page.

Starting today, the L Developer Preview is available for download from the L Developer Preview site. We're also announcing that Android Studio is now in beta, and making great progress toward a full release.

Let’s take a deeper dive into what’s included in the preview and what it means for you as a developer as you prepare your apps for the next Android release.

What’s in the L Developer Preview

The L Developer Preview includes updated SDK tools, system images for testing on an emulator, and system images for testing on a Nexus 5 or Nexus 7 device.

You can download these components through the Android SDK Manager:

  • L Developer Preview SDK Tools
  • L Developer Preview Emulator System Image - 32-bit (64-bit experimental emulator image coming soon)
  • L Developer Preview Emulator System Image for Android TV (32-bit)

(Note: the full release of Android Wear is a part of Android KitKat, API Level 20. Read more about Android Wear development here.)

Today, we are also providing system image downloads for these Nexus devices to help with your testing as well:

  • Nexus 5 (GSM/LTE) “hammerhead” Device System Image
  • Nexus 7 [2013] - (Wifi) “razor” Device System Image

You can download both of these system images from the L Developer Preview site.

With the SDK Tools, and Nexus device images, you can get a head start on testing out your app on the latest Android platform months before the official launch. You can use the extra lead time to take advantage of all the new app features and APIs in your apps. The Nexus device images can help you with testing, but keep in mind that they are meant for development purposes only and should not be used on a production device.

Notes on APIs and publishing

The L Developer Preview is a development release and does not have a standard API level. The APIs are not final, and you can expect minor API changes over time.

To ensure a great user experience and broad compatibility, you can not publish versions of your app to Google Play that are compiled against L Developer Preview. Apps built for L Developer Preview will have to wait until the full official launch to publish on Google Play.

Android Studio Beta

To help you develop your apps for the upcoming Android version and for new Android device types, we’re also happy to announce Android Studio Beta. Android Studio Beta helps you develop apps by enabling you to:

  • Incorporate the new material design and interaction elements of the L Developer Preview SDK
  • Quickly create and build apps with a new app wizard and layout editor support for Android Wear and Android TV

Building on top of the build variants and flavors features we introduced last year, the Android Studio build system now supports creating multiple apks, such as for devices like Android Wear. You can try out all the new features with the L Developer Preview by downloading the Android Studio Beta today.

How to get started

To get started with the L Developer Preview and prepare your apps for the full release, just follow these steps:

  1. Try out Android Studio Beta
  2. Visit the L Developer Preview site
  3. Explore the new APIs
  4. Enable the material theme and try out material design on your apps
  5. Get the emulator system images through the SDK Manager or download the Nexus device system images.
  6. Test your app on the new Android Runtime (ART) with your device or emulator
  7. Give us feedback

As you use the new developer features and APIs in the L Developer Preview, we encourage you to give us your feedback using the L Developer Preview Issue Tracker. During the developer preview period, we aim to incorporate your feedback into our new APIs and adjust features as best as we can.

You can get all the latest downloads, documentation, and tools information from the L Developer Preview site on You can also check our Android Developer Preview Google+ page for updates and information.

We hope you try the L Developer Preview as you start building the next generation of amazing Android user experiences.

.footer__link, .footer__text{ color: #8b8b8b !important; display: block; font-size: 12px; line-height: 1.33333; margin-bottom: 10px; text-decoration: none; } .hoverable .footer__link:hover { text-decoration: underline; }

Google I/O 2014 I/O Livestreams I/O Bytes Videos +Android Developers

L Developer Preview Material Design Android Wear Android TV Android Auto

Get it on Google Play
Categories: Programming

Cloud Enabling your Mobile App

Google Code Blog - Thu, 06/26/2014 - 20:00
By Jason Polites, Cloud Platform Team

Many mobile apps today suffer from “app-nesia” — the affliction that causes an app to forget who you are. Have you ever re-installed an app only to discover you have to re-create all your carefully crafted preferences? This is typically because the user’s app data lives only on the device.

By connecting your apps to a backend platform, you can solve this issue, but it can be challenging. Whether it’s building basic plumbing, or just trying to load and save data in a network & battery-efficient way, spending time dealing with the backend can take precious time away from building an awesome app. So, we’re introducing two new features to help make your life easier.

Google Cloud Save
Google Cloud Save allows you to easily load and save user data to the cloud without needing to code up the backend. This is handy for situations where you want to save user state and have that state synchronized to multiple devices, or survive an app reinstall.

We handle all the backend logic as well as the synchronization services on the client. The synchronization services work in the background, providing offline support for the data, and minimizing impact on the battery. All you need to do is tell us when and what to save, and you do this with just 4 simple methods:
  • .save(client, List<Entity>)
  • .delete(client, Query)
  • .query(client, Query)
  • .requestSync(client)
All data is written locally first, then automatically synchronized in the background. The save, delete and query methods provide your basic CRUD operations while the requestSync method allows you to force a synchronization at any time.On the backend the data is stored in Google Cloud Datastore which means you can access the raw data directly from a Google App Engine or Google Compute Engine instance using the existing Datastore API. Changes on the server will even be automatically synced back to client devices. Importantly, this per-user data belongs to you, the developer, and stored in your own Google Cloud Datastore database. Cloud Save (3).pngGoogle Cloud Save is currently in private beta and will be available for general use soon. If you’re interested in participating in the private beta, you can sign up here!

Cloud Tools for Android Studio
To simplify the process of adding an App Engine backend to your app, Android Studio now provides three App Engine backend module templates which you can add to your app:
  • App Engine Java Servlet Module - Minimal Backend
  • App Engine Java Endpoints Module - Basic Endpoint scaffolding 
  • App Engine with Google Cloud Messaging - Push notification wireup 
When you choose one of these template types your project is updated with a new Gradle module containing your new App Engine backend. All of the required dependencies/permissions will be automatically set up for you. Built-in rich editing support for Google Cloud Endpoints
Once you have added the backend module to your Android application, you can use Google Cloud Endpoints to streamline the communication between your backend and your Android app. Cloud Endpoints automatically generates strongly-typed, mobile optimized client libraries from simple Java server-side API annotations, automates Java object marshalling to and from JSON, and provides built-in OAuth 2.0 support.On deployment, this annotated Endpoints API definition class generates a RESTful API. You can explore this generated API (and even make calls to it) by navigating to Endpoints API explorer as shown in the image below: api-explorer.png

To simplify calling this generated API from your Android app, Android Studio will automatically set up your project to include all compile dependencies and permissions required to consume Cloud Endpoints, and will re-generate strongly-typed client libraries if your backend changes. This means that you can start calling the client libraries from your Android app immediately after defining the server-side Endpoints API. The underlying work-horses: Gradle, and Gradle plug-in for App EngineUnder the hood, Gradle is used to build both your app and your App Engine backend. In fact, when you add an App Engine backend to your Android app, the open-source App Engine plug-in for Gradle is automatically downloaded by Android Studio, and common App Engine tasks become available as Gradle targets. This allows you to use the same build system across your IDE, command-line or continuous integration environments.Checkout more details on the new Cloud Endpoints features in Android Studio on the Android Developer Blog.

Posted by Louis Gray, Googler
Categories: Programming

Mocking a REST backend for your AngularJS / Grunt web application

Xebia Blog - Thu, 06/26/2014 - 17:15

Anyone who ever developed a web application will know that a lot of time is spend in a browser to check if everything works as well and looks good. And you want to make sure it looks good in all possible situations. For a single-page application, build with a framework such as AngularJS, that gets all it's data from a REST backend this means you should verify your front-end against different responses from your backend. For a small application with primarily GET requests to display data, you might get away with testing against your real (development) backend. But for large and complex applications, you need to mock your backend.

In this post I'll go in to detail how you can solve this by mocking GET requests for an AngularJS web application that's built using Grunt.

In our current project, we're building a new mobile front-end for an existing web application. Very convenient since the backend already exists with all the REST services that we need. An even bigger convenience is that the team that built the existing web application also built an entire mock implementation of the backend. This mock implementation will give standard responses for every possible request. Great for our Protractor end-to-end tests! (Perhaps another post about that another day.) But this mock implementation is not so great for the non standard scenario's. Think of error messages, incomplete data, large numbers or a strange valuta. How can we make sure our UI displays these kind of cases correct? We usually cover all these cases in our unit tests, but sometimes you just want to see it right in front of you as well. So we started building a simple solution right inside our Grunt configuration.

To make this solution work, we need to make sure that all our REST requests go through the Grunt web server layer. Our web application is served by Grunt on localhost port 9000. This is the standard configuration that Yeoman generates (you really should use Yeoman to scaffold your project). Our development backend is also running on localhost, but on port 5000. In our web application we want to make all REST calls using the `/api` path so we need to rewrite all requests to http://localhost:9000/api to our backend: http://localhost:5000/api. We can do this by adding middleware in the connect:livereload configuration of our Gruntfile.

livereload: {
  options: {
    open: true,
    middleware: function (connect, options) {
      return [
        require('connect-modrewrite')(['^/api http://localhost:5000/api [P L]']),

        /* The lines below are generated by Yeoman */

Do the same for the connect:test section as well.

Since we're using 'connect-modrewrite' here, we'll have to add this to our project:

npm install connect-modrewrite --save-dev

With this configuration every request starting will http://localhost:9000/api will be passed on to http://localhost:5000/api so we can just use /api in our AngularJS application. Now that we have this working, we can write some custom middleware to mock some of our requests.

Let's say we have a GET request /api/user returning some JSON data:

{"id": 1, "name":"Bob"}

Now we'd like to see what happens with our application in case the name is missing:

{"id": 1}

It would be nice if we could send a simple POST request to change the response of all subsequent calls. Something like this:

curl -X POST -d '{"id": 1}' http://localhost:9000/mock/api/user

We prefixed the path that we want to mock with /mock in order to know when we should start mocking something. Let's see how we can implement this. In the same Gruntfile that contains our middleware configuration we add a new function that will help us mock our requests.

var mocks = [];
function captureMock() {
  return function (req, res, next) {

    // match on POST requests starting with /mock
    if (req.method === 'POST' && req.url.indexOf('/mock') === 0) {

      // everything after /mock is the path that we need to mock
      var path = req.url.substring(5);

      var body = '';
      req.on('data', function (data) {
        body += data;
      req.on('end', function () {

        mocks[path] = body;

    } else {

And we need to add the above function to our middleware configuration:

middleware: function (connect, options) {
  return [
    require('connect-modrewrite')(['^/api http://localhost:5000/api [P L]']),


Our function will be called for each incoming request. It will capture each request starting with /mock as a request to define a mock request. Next it stores the body in the mocks variable with the path as key. So if we execute our curl POST request we end up with something like this in our mocks array:

mocks['/api/user'] = '{"id": 1}';

Next we need to actually return this data for requests to http://localhost:9000/api/user. Let's make a new function for that.

function mock() {
  return function (req, res, next) {
    var mockedResponse = mocks[req.url];
    if (mockedResponse) {
    } else {

And also add it to our middleware.

  require('connect-modrewrite')(['^/api http://localhost:5000/api [P L]']),

Great, we now have a simple mocking solution in just a few lines of code that allows us to send simple POST requests to our server with the requests we want to mock. However, it can only send status codes of 200 and it cannot differentiate between different HTTP methods like GET, PUT, POST and DELETE. Let's change our functions a bit to support that functionality as well.

 var mocks = {
  GET: {},
  PUT: {},
  POST: {},
  PATCH: {},
  DELETE: {}

function mock() {
  return function (req, res, next) {
    if (req.method === 'POST' && req.url.indexOf('/mock') === 0) {
      var path = req.url.substring(5);

      var body = '';
      req.on('data', function (data) {
        body += data;
      req.on('end', function () {

        var headers = {
          'Content-Type': req.headers['content-type']
        for (var key in req.headers) {
          if (req.headers.hasOwnProperty(key)) {
            if (key.indexOf('mock-header-') === 0) {
              headers[key.substring(12)] = req.headers[key];

        mocks[req.headers['mock-method'] || 'GET'][path] = {
          body: body,
          responseCode: req.headers['mock-response'] || 200,
          headers: headers


function mock() {
  return function (req, res, next) {
    var mockedResponse = mocks[req.method][req.url];
    if (mockedResponse) {
      res.writeHead(mockedResponse.responseCode, mockedResponse.headers);
    } else {

We can now create more advanced mocks:

curl -X POST \
    -H "mock-method: DELETE" \
    -H "mock-response: 403" \
    -H "Content-type: application/json" \
    -H "mock-header-Last-Modified: Tue, 15 Nov 1994 12:45:26 GMT" \
    -d '{"error": "Not authorized"}' http://localhost:9000/mock/api/user

curl -D - -X DELETE http://localhost:9000/api/user
HTTP/1.1 403 Forbidden
Content-Type: application/json
last-modified: Tue, 15 Nov 1994 12:45:26 GMT
Date: Wed, 18 Jun 2014 13:39:30 GMT
Connection: keep-alive
Transfer-Encoding: chunked

{"error": "Not authorized"}

Since we thought this would be useful for other developers, we decided to make all this available as open source library on GitHub and NPM

To add this to your project, just install with npm:

npm install mock-rest-request --save-dev

And of course add it to your middleware configuration:

middleware: function (connect, options) {
  var mockRequests = require('mock-rest-request');
  return [

Are You Going to ALE 2014?

NOOP.NL - Jurgen Appelo - Thu, 06/26/2014 - 15:57
ALE 2014

Did you know ALE is the only event where I pay so that I can attend?
I hope to see you in August in Kraków.

The post Are You Going to ALE 2014? appeared first on NOOP.NL.

Categories: Project Management