Software Development Blogs: Programming, Software Testing, Agile Project Management

### Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Feed aggregator

### Pragmatic Manager Posted: Standup or Handoff

I published a Pragmatic Manager yesterday to my subscribers. Normally, I let them enjoy the pleasure of being “in-the-know” about what I have to say this month for a while before I post the emails to my site.

Read the Pragmatic Manager here: Standup or Handoff.

However, I made a Big Mistake in where I will be speaking this week. I fat-fingered July 10 into July 19. What made it into the newsletter was July 19. Oops. I’m actually a panelist this Thursday, July 10, at Agile New England. The topic: Agile: Massive Success or Empty Buzzword?

My fellow panelists are Ken Schwaber and Linda Cook. We will have no shortage of opinions!

For example, I suspect that Ken and I might disagree on this very issue, of whether you can do agile with a geographically distributed team, and if you can have handoffs or you must do standups.

If you are near the Boston area, this Thursday, July 10, and want to see some sparks fly—or at least engage in lively debate—come to the Agile New England meeting July 10.

If you’re wondering how did this get past my reviewers, my copyeditor, and me? Well, we all make mistakes, don’t we?

Categories: Project Management

### Identifying Architectural Elements in Current Systems

Coding the Architecture - Simon Brown - Tue, 07/08/2014 - 10:27

Simon recently talked about the gap between Software Architecture and Code and how to close this with architecturally-evident coding. He's also creating tools to allow Software Architecture to be expressed as code.

If you're working on a greenfield project then including annotations to help with navigation is a great solution but what if you've inherited a large system with a model-code gap? Or if you only realise, sometime into a project, that you lack a model to help you understand its growing complexity? Well, Simon also had some thoughts on scanning Spring annotations to provide this data. This works quite well and it got me thinking about other artifacts in code that can be extracted for these diagrams.

(In more formal terms - for those of you that like to quote ISO42010 - we are trying to extract architectural elements from code that can be displayed within architectural views. Of course the elements may be from a variety of differing abstractions, levels and granularity and therefore need to be placed within differing views.)

So what can we extract from a current/legacy system to give us a view into it? Some suggestions include:

What do you think? Of course all of the above is very dependent on your codebase but if none of the above works then you have to question the quality and structure of the code! The data extracted may need filtering and manual correction as it won't give you exactly what you want. You might consider creating structurizr annotations using an initial scan and then maintaining them. One of my tasks for the next few weeks is to try this out on some legacy codebases I maintain.

What other ways of identifying architectural elements can you think of?

Categories: Architecture

### What is an Estimate?

Software Development Today - Vasco Duarte - Tue, 07/08/2014 - 04:00
If you don’t know what an estimate is, you can’t avoid using them. So here’s my attempt to define what is an estimate.
The "estimates" that I'm concerned about are those that can easily (by omission, incompetence or malice) be turned into commitments. I believe Software Development is better seen as a discovery process (even simple software projects). In this context, commitments remove options and force the teams/companies to follow a plan instead of responding to change.

Here's my definition: "Estimates, in the context of #NoEstimates, are all estimates that can be (on purpose, or by accident) turned into a commitment regarding project work that is not being worked on at the moment when the estimate is made."
The principles behind this definition of an estimateIn this definition I have the following principles in mind:
• Delay commitment, create options: When we commit to a particular solution up front we forego possible other solutions and, as a consequence we will make the plan harder to change. Each solution comes with explicit and implicit assumptions about what we will tackle in the future, therefore I prefer to commit only to what is needed in order to validate or deliver value to the customer now. This way, I keep my options open regarding the future.
• Responding to change over following a plan: Following a plan is easy and comforting, especially when plans are detailed and very clear: good plans. That’s why we create plans in the first place! But being easy does not make it right. Sooner or later we are surprised by events we could not predict and are no longer compatible with the plan we created upfront. Estimation up front makes it harder for us to change the plan because as we define the plan in detail, we commit ourselves to following it, mentally and emotionally.
• Collaboration over contract negotiation: Perhaps one of the most important Agile principles. Even when you spend time and invest time in creating a “perfect” contract there will be situations you cannot foresee. What do you do then? Hopefully by then you’ve established a relationship of trust with the other party. In that case, a simple conversation to review options and chose the best path will be enough. Estimation locks us down and tends to put people on the defensive when things don’t happen as planned. Leaving the estimation open and relying on incremental development with constant focus on validating the value delivered will help both parties come to an agreement when things don’t go as planned. Thereby focusing on collaboration instead of justifying why an estimated release date was missed.
Here are some examples that fit the definition of Estimates that I outlined above:
• An estimate of time/duration for work that is several days, weeks or months in the future.
• An estimate of value that is not part of an experiment (the longer the time-frame the more of a problem it is).
• A long term estimate of time and/or value that we can only validate after that long term is over﻿.

How do you define Estimates in your work? Are you still doing estimates that fit the definition above? What is making it hard to stop doing such estimates? Share below in the comments what you think of this definition and how you would improve it.

This definition of an estimate was sent to the #NoEstimates mailing list a few weeks ago. If you want to receive exclusive content about #NoEstimates just sign up below. You will receive a regular email on the topic of #NoEstimates as Agile Software Development.

#mc_embed_signup{background:#fff; clear:left; font:14px Helvetica,Arial,sans-serif; } /* Add your own MailChimp form style overrides in your site stylesheet or in this style block. We recommend moving this block and the preceding CSS link to the HEAD of your HTML file. */ Subscribe to our mailing list* indicates required Email Address * First Name Last Name

Picture credit: Pascal @ Flickr

### Project Charters

This team is eager to begin work on its charter.

Project charters play an extremely important role in the initiation of projects.  The Project Management Institute (PMI) has suggest that no project should begin without a charter.  A project charter creates a foundation to guide project activities.  The primary roles of the charter are to:

1. Authorize the project to move forward.  In many methodologies the charter acts as the authorizing document.  The charter will include items such as the business need, project scope, budget and the signature of sponsors.  In some organizations the sponsor or the sponsor’s organization will author the document and use it as a form of transmittal or work ticket. Note: While the literature is rich with processes in which the charter is written outside of the development team, I have never seen this process in action.  What is more typical is a charter written by IT and signed-off on by the stakeholders or some sort of electronic budget authorization. Regardless of methodology, in a large organization, an authorization to spend money is important.
2. Acts as a communication vehicle. The charter is often used to capture the initial thoughts of the sponsors, product owner and team about the goals of the project and the rules they will use to govern themselves (or be governed in classic command and control methodologies).
3. Provide structure for the team. Who is on the project team is not always as evident as it should be.  The charter should identify the team members (and some demographics, like time zone, if distributed) with the roles they are playing, if needed.  In many IT organizations, team members play specified specialist roles. While specialization is a feature mainly thought to be seen in organizations using classic development and project management techniques, it can also be seen, albeit to a lesser extent, in organizations using Agile. Depending on the method used the product owner and Scrum Master should be identified.  Other team attributes such as team norms and expectations should be identified and agreed upon.
4. Identify need resources. Identify any needed resources. Remember that the resources needed,like business needs and requirements continually evolve therefore overtime the charter must evolve or be replaced. For example, a charter I recently reviewed for a new Agile project and team identified the need for a team work room and licenses for a specific automated testing tool

The macro goal for the charter is to get the project going and in the right direction.  The needs and requirements of most projects of moderate to large size/duration are relatively dynamic.   The issue is that while charters are supposed to change and evolve, once created are prone to become shelfware.

A charter can contain additional information such as risks, constraints, project context and deadlines. The range of data that can be housed in a charter is limitless.  I have seem large projects spend months writing and wordsmithing charters that were hundreds of pages long. The process of thinking about all of the charter topics might be valuable, however rarely is the document used after the sponsors and other stakeholders sign off on the document. Whether Agile or a classic plan-based, rarely will the charter see the light of day (or an update cycle) unless the topics covered are important to the team.

Categories: Process Management

### Applying Little's Law in Agile Games

Xebia Blog - Mon, 07/07/2014 - 21:20

Have you ever used Little's Law to explain that lower WiP (work in progress) limits lead to shorter cycle times? Ever tried to illustrate Little's Law in an Agile game and found it doesn't hold? Then read this blog to discover that it is exactly true in Agile games and how it really works.

Some time ago I gave a kanban workshop. Part of the workshop was a game of folding paper airplanes to illustrate flow. To illustrate Little's Law we determined the throughput, cycle time and work in progress. To my surprise the law didn't hold. Not even close. In this blog I want to share the insight into why it does work!

Introduction

It is well known that the average number of items in progress is proportional to the average cycle time of completed work items. The proportionality is the average input rate (or throughput rate) of work items. This relation is known as Little's Law. It was discovered by Little in the 1960s and has found many applications.

In kanban teams this relationship is often used to qualitatively argue that it is favourable for flow to have not too much work in parallel. To this end WiP (work in progress) limits are introduced. The smaller the WiP the smaller the average cycle time which means better flow.

A surprise to me was that it is exactly true and it remains true under very relaxed conditions.

Little's Law

In mathematical form the law is often stated as:

(1)

$\bar{N} = \lambda \bar{W}$ \bar{N} = \lambda \bar{W}

Here $\bar{N}$ \bar{N} is the average number of work items in progress at a certain time, and $\bar{W}$ \bar{W} is the average cycle time. $\lambda$ \lambda is the average input rate (new work items per unit of time). In stable systems this also equals the average throughput. In this case Little's Law is often (re)stated as

(2)

$\frac{\mathrm{Work\, in\, progress}}{\mathrm{Throughput}} = \mathrm{Cycle Time}$ \frac{\mathrm{Work\, in\, progress}}{\mathrm{Throughput}} = \mathrm{Cycle Time}

Conditions

In practise one considers Little's Law over a finite period of time, e.g. 6 months, 5 sprints, 3 rounds in an Agile game. Also in practise, teams work on backlog items which are discrete items. After the work is done this results in a new product increment.

Under the following conditions (1) is exact:

• The system is observed over a finite period of time,
• The system is a queueing system.

A queuing system is a system that consists of discrete items which arrive at a certain rate, receive service after which they depart.

Examples of a queueing system. An agile team works on backlog items. A kanban team that works on production incidents. A scrum team.

Agile Game

An often used game for explaining the importance of flow to team is the game of folding paper airplanes. Many forms of this games exist. See e.g. [Heintz11].

For this blog's purpose consider a team that folds air planes. The backlog is a stack of white paper. 3 Rounds of folding are done. Airplanes that are folded and fly at least 2 meters are considered done.

At the end of each round we will collect the following metrics:

• number of completed airplanes
• number of airplanes in progress and not yet finished.

The result of the the 3 rounds are shown at the right. At the end of round 1 Team A completed 3 airplanes and having 8 unfinished airplanes. Likewise, Team B finished 4 airplanes in round 3 giving a total of 12 finished airplanes and having 6 unfinished airplanes in progress.

The cycle time are got by writing the round number of the sheet of paper when starting to fold the airplane. When done, write the round number of the paper. The cycle time for one airplanes is got by subtracting the two and adding 1.

Calculating Little's Law

The way I was always calculating the number for work in progress, throughput and cycle time has been

1. averaging cycle time for all completed airplanes,
2. averaging the throughput over all rounds,
3. averaging the work in progress over all rounds.

When calculated at the end of round 3, for Team A this amounts to:

• Average work in progress = (8+6+2)/3 = 16/3,
• Average throughput = 10 (completed airplanes)/3 = 10/3,
• Average cycle time = 22/10 = 11/5

Using (2) above we get: 16/3 / (10/3) = 8/5. This is not equal to the average cycle time of 11/5. Not even close. How come?

The Truth

The interpretation of work in progress, throughput and cycle time I got from working with cumulative flow diagrams. There are many resources explaining these, see e.g. [Vega2011].

The key to the correct interpretation is choosing the time interval for which to measure the quantities $\bar{W}$ \bar{W} , $\lambda$ \lambda , and $\bar{W}$ \bar{W} . Second, using the input rate instead of the throughput. Third, at the end of the time period include the unfinished items. Last, in calculating $\bar{N}$ \bar{N} consider all items that went through the system.

When we reinterpret the results for teams A and B we get

Team A

• Average work in progress
In round 1 3 airplanes were completed and left 8 unfinished; a total of 11 for work in progress (11 airplanes picked up as work)
In round 2 the team completed 2 airplanes and have 6 unfinished; a total of 8
In round 3 the team finished an additional 5 airplanes and left 2 uncompleted; a total of 7
When measured over 3 rounds an average of (11+8+7)/3 = 26/3
• Average input rate
Using the input rate:
In round 1 the team picked up 11 new airplanes
In round 2 the team picked up no additional airplanes
In round 3 one new airplanes was picked up.
An average input rate of (11+0+1)/3 = 4 airplanes per round
• Average cycle time
At the end of the third round 2 airplanes are left in progress; one taken up in the third round having a waiting time of 1 and one left from the first round having waiting time of 3 rounds. A total waiting time of 22 + 3 + 1 = 26 rounds.
Averaging over 12 airplanes we have an average cycle time of 26/12 = 13/6 rounds per airplane.

Dividing the average work in progress by the average input rate we get 26/3 divided by 4 = 26/12(!). This is exactly equal to the calculated average cycle time!

Team B

In a similar manner the reinterpreted results for team B are:

• Average work in progress = (13+14+10)/3 = 37/3 airplanes,
• Average input rate = (13+2+3)/3 = 6 airplanes per round,
• Average cycle time = (27 (completed) + 10 (unfinished))/18 (airplanes) = 37/18 rounds per airplane

Again, dividing the average work in progress by the average input rate we get 37/18 rounds per airplane, which again is exactly equal to the average cycle time or waiting time!

Note: the cycle time of 10 days is built up by (a) 1 airplane from round 1 (cycle time of 3), 2 airplanes picked up in round 2 (total of 4 rounds), 3 airplanes picked up in round 3 (total of 3 rounds).

What About Cumulative Flow Diagrams?

Now that we understand how to calculate the quantities in Little's Law, we go back to cumulative flow diagrams. How come Little's Law works in this case.

In the case of teams that have collected data on cycle time, work in progress and throughput Little's Law work when done as explained in the section 'Calculating Little's Law' because:

1. the teams are kept stable by having WiP limits on the left most column ("To Do"); then the throughput is more or less equal to the input rate,
2. the team has completed a fairly large amount of work items in which case the waiting time of unfinished work items can be neglected,
3. when measured over the (large part of the) value creation process, the completed items per time period can often be neglected in the calculation of the average work in progress.
Summary

Little's Law (1) holds under the conditions that (a) the system considered is a queueing system and (b) the observation or measurements are done over a finite time interval. It then holds independently of the stationaryness of the probability distributions, queuing discipline, emptiness of the system at the start and end of the time interval.

Calculate the quantities $\bar{N}$ \bar{N} , $\lambda$ \lambda , and $\bar{W}$ \bar{W} as follows:

• Average work in progress $\bar{N}$ \bar{N}
For each time interval considered count the total amount of work in the system and add any items completed in that time interval.
• Average cycle time $\bar{W}$ \bar{W}
Sum the cycle times for all completed items and include the waiting time for unfinished items and divide by the total number of items.
• Average input rate $\lambda$ \lambda
Add the total number of items that entered the system and divide by the total number of time intervals.
References

[Little61] Little, J. D. C. 1961. A proof for the queuing formula: L = ãW . Oper. Res. 9(3) 383–387.

[Heintz11] John Heintz, June 2011, Agile Airplane Game, GistLabs, http://gistlabs.com/2011/06/agile-airplane-game/

[Vega11] Vega Information System Services, Inc., September 2011, Basics of Reading Cumulative Flow Diagrams, http://www.vissinc.com/2011/09/29/basics-of-reading-cumulative-flow-diagrams/

### Scaling the World Cup - How Gambify runs a massive mobile betting app with a team of 2

This is a guest post by Elizabeth Osterloh and Tobias Wilke of cloudControl.

Startups face very different issues than big companies when they build software. Larger companies develop projects over much longer time frames and often have entire IT-departments to support them in creating customized architecture. It’s an entirely different story when a startup has a good idea, it gets popular, and they need to scale fast.

This was the situation for Gambify, an app for organizing betting games released just in time for the soccer World Cup. The company was founded and is run in Germany by only two people. When they managed to get a few major endorsements (including Adidas and the German team star Thomas Müller), they had to prepare for a sudden deluge of users, as well as very specific peak times.

The Gambify App: Basic Architecture
Categories: Architecture

### Do Software Developers Really Need Degrees?

Making the Complex Simple - John Sonmez - Mon, 07/07/2014 - 15:45

When I first started out my career as a software developer, I didn’t have a degree. I took my first real job when I was on summer break from my first year of college. By the time the summer was up and it was time to enroll back in school, I found that the salary […]

The post Do Software Developers Really Need Degrees? appeared first on Simple Programmer.

Categories: Programming

### Try to Be 50% Wrong from Both Sides

NOOP.NL - Jurgen Appelo - Mon, 07/07/2014 - 14:06

When extremists on both sides claim that you’re 50% wrong, you can feel confident that you’re probably right where you should be.

Vegetables are healthy, but I’ve never heard someone suggest that eating only vegetables is a wise approach to long-term health

The post Try to Be 50% Wrong from Both Sides appeared first on NOOP.NL.

Categories: Project Management

### R/plyr: ddply – Error in vector(type, length) : vector: cannot make a vector of mode ‘closure’.

Mark Needham - Mon, 07/07/2014 - 07:07

In my continued playing around with plyr’s ddply function I was trying to group a data frame by one of its columns and return a count of the number of rows with specific values and ran into a strange (to me) error message.

I had a data frame:

n = c(2, 3, 5)
s = c("aa", "bb", "cc")
b = c(TRUE, FALSE, TRUE)
df = data.frame(n, s, b)

And wanted to group and count on column ‘b’ so I’d get back a count of 2 for TRUE and 1 for FALSE. I wrote this code:

ddply(df, "b", function(x) {
countr <- length(x$n) data.frame(count = count) }) which when evaluated gave the following error: Error in vector(type, length) : vector: cannot make a vector of mode 'closure'. It took me quite a while to realise that I’d just made a typo in assigned the count to a variable called ‘countr’ instead of ‘count’. As a result of that typo I think the R compiler was trying to find a variable called ‘count’ somwhere else in the lexical scope but was unable to. If I’d defined the variable ‘count’ outside the call to ddply function then my typo wouldn’t have resulted in an error but rather an unexpected resulte.g. > count = 10 > ddply(df, "b", function(x) { + countr <- length(x$n)
+   data.frame(count = count)
+ })
b count
1 FALSE     4
2  TRUE     4

Once I spotted the typo and fixed it things worked as expected:

> ddply(df, "b", function(x) {
+   count <- length(x$n) + data.frame(count = count) + })  b count 1 FALSE 1 2 TRUE 2 Categories: Programming ### Quote of the Day Herding Cats - Glen Alleman - Mon, 07/07/2014 - 07:02 Movement without direction will create a hole in the ground — Sophia Bedford-Pierce Categories: Project Management ### SPaMCAST 297 – IFPUG Function Points http://www.spamcast.net Listen to SPaMCAST 297 now! SPaMCAST 297 features our essay on IPFUG Function Points. IFPUG Function Points are a measure of the functionality delivered by the project or application being counted based on a set of rules documented in the IFPUG Counting Practices Manual (CPM). The measure of delivered functionality is a proxy for size which can be used in estimating and measuring work. An analogy for function points is the measure of the number of square feet (or square meters) of a house. Want more? Listen to SPaMCAST 297 now! Next week we will feature our interview with Brian Federici. We discussed working in an environment with a nearly continuous delivery model from the point of view of a practitioner. Does what sounds good in theory really work when it is implemented? Upcoming Events Upcoming DCG Webinars: July 24 11:30 EDT – The Impact of Cognitive Bias On Teams Check these out at www.davidconsultinggroup.com I will be attending Agile 2014 in Orlando, July 28 through August 1, 2014. It would be great to get together with SPaMCAST listeners, let me know if you are attending. http://agile2014.agilealliance.org/ I will be presenting at the International Conference on Software Quality and Test Management in San Diego, CA on October 1. I HAVE A PROMO CODE! Send me an email and I will share. http://www.pnsqc.org/international-conference-software-quality-test-management-2014/ I will be presenting at the North East Quality Council 60th Conference October 21st and 22nd in Springfield, MA. http://www.neqc.org/conference/60/location.asp More on all of these great events in the near future! I look forward to seeing all SPaMCAST readers and listeners that attend these great events! The Software Process and Measurement Cast has a sponsor. As many you know I do at least one webinar for the IT Metrics and Productivity Institute (ITMPI) every year. The ITMPI provides a great service to the IT profession. ITMPI’s mission is to pull together the expertise and educational efforts of the world’s leading IT thought leaders and to create a single online destination where IT practitioners and executives can meet all of their educational and professional development needs. The ITMPI offers a premium membership that gives members unlimited free access to 400 PDU accredited webinar recordings, and waives the PDU processing fees on all live and recorded webinars. The Software Process and Measurement Cast some support if you sign up here. All the revenue our sponsorship generates goes for bandwidth, hosting and new cool equipment to create more and better content for you. Support the SPaMCAST and learn from the ITMPI. Shameless Ad for my book! Mastering Software Project Management: Best Practices, Tools and Techniques co-authored by Murali Chematuri and myself and published by J. Ross Publishing. We have received unsolicited reviews like the following: “This book will prove that software projects should not be a tedious process, neither for you or your team.” Support SPaMCAST by buying the book here. Available in English and Chinese. Categories: Process Management ### SPaMCAST 297 – IFPUG Function Points Software Process and Measurement Cast - Sun, 07/06/2014 - 22:00 SPaMCAST 297 features our essay on IPFUG Function Points. IFPUG Function Points are a measure of the functionality delivered by the project or application being counted based on a set of rules documented in the IFPUG Counting Practices Manual (CPM). The measure of delivered functionality is a proxy for size which can be used in estimating and measuring work. An analogy for function points is the measure of the number of square feet (or square meters) of a house. Next week we will feature our interview with Brian Federici. We discussed working in an environment with a nearly continuous delivery model from the point of view of a practitioner. Does what sounds good in theory really work when it is implemented? Upcoming Events Upcoming DCG Webinars: July 24 11:30 EDT - The Impact of Cognitive Bias On Teams Check these out at www.davidconsultinggroup.com I will be attending Agile 2014 in Orlando, July 28 through August 1, 2014. It would be great to get together with SPaMCAST listeners, let me know if you are attending. http://agile2014.agilealliance.org/ I will be presenting at the International Conference on Software Quality and Test Management in San Diego, CA on October 1. I HAVE A PROMO CODE! Send me an email and I will share. http://www.pnsqc.org/international-conference-software-quality-test-management-2014/ I will be presenting at the North East Quality Council 60th Conference October 21st and 22nd in Springfield, MA. http://www.neqc.org/conference/60/location.asp More on all of these great events in the near future! I look forward to seeing all SPaMCAST readers and listeners that attend these great events! The Software Process and Measurement Cast has a sponsor. As many you know I do at least one webinar for the IT Metrics and Productivity Institute (ITMPI) every year. The ITMPI provides a great service to the IT profession. ITMPI's mission is to pull together the expertise and educational efforts of the world's leading IT thought leaders and to create a single online destination where IT practitioners and executives can meet all of their educational and professional development needs. The ITMPI offers a premium membership that gives members unlimited free access to 400 PDU accredited webinar recordings, and waives the PDU processing fees on all live and recorded webinars. The Software Process and Measurement Cast some support if you sign up here. All the revenue our sponsorship generates goes for bandwidth, hosting and new cool equipment to create more and better content for you. Support the SPaMCAST and learn from the ITMPI. Shameless Ad for my book! Mastering Software Project Management: Best Practices, Tools and Techniques co-authored by Murali Chematuri and myself and published by J. Ross Publishing. We have received unsolicited reviews like the following: "This book will prove that software projects should not be a tedious process, neither for you or your team." Support SPaMCAST by buying the book here. Available in English and Chinese. Categories: Process Management ### I teach people how to draw pictures Coding the Architecture - Simon Brown - Sun, 07/06/2014 - 11:55 Regular readers will know that I'm a big fan of pictures, especially as a mechanism for communicating the structure of software systems. To this end, I sometimes introduce myself as somebody who teaches people how to draw pictures. This is often said with a smile, because on the face of it this is exactly what I appear to be doing. But there's less truth to this than there initially appears. Something that we do on the full 2-day training course and the 1-day sketching workshop is to get people drawing some pictures to communicate the structure of a software system. This is either a solution to the financial risk system or their own software. Over the years, I've done this for thousands of people and pretty much everybody struggles to draw something that effectively communicates the software. The same challenges crop up again and again. The diagrams that are produced during the initial 90-minute timebox are usually fairly confusing and exhibit problems ranging from a lack of clarity around the abstractions and notation used through to diagrams that show too little or too much detail. This is an aside, but the majority of the diagrams are informal "boxes and lines" rather than UML. After some reviews and feedback, I introduce people to my C4 model. In a nutshell, it's all about thinking of a software system as being made up of containers (web applications, databases, mobile devices, etc), each of which contains components, which in turn are made up of classes. There are some variations of this depending on the technology you're using, but that's basically it. With this structure in mind, you can then draw a diagram at each level in turn, which leads to a system context diagram, a container diagram, one or more component diagrams and (optionally) some class diagrams. If you want a slightly more detailed introduction to C4, take a look at Simple sketches for diagramming your software architecture. Back to the workshops, and iteration two is another 90-minute timebox in which teams can redraw their diagrams. I don't actually mandate that people should follow my C4 approach, but most do, and what happens next is really interesting. The conversations change. In stark contrast to the first timebox, the conversations are much more technical. Gone are the discussions about what to include on each diagram and instead the focus is on the technical aspects of the software. Instead of "what should we show on this diagram?", the questions I hear are more along the lines of "what are the responsibilities of this thing?" and "how does this thing talk to that thing?". In essence, the discussion returns to be about software design, which is exactly where it should be. There are a number of ways in which you can communicate the architecture of a software system, so I don't want to pitch my C4 approach as "the one true way". What I do want to say is that providing even a little guidance in this area can go a long way. I ask people for their thoughts after the second timebox and the consensus is that the diagrams are easier to draw and comprehend because of the "framework" (their words, not mine) that's been provided. All I'm doing is providing some constraints that people might want to work within, and this frees them up to focus on what's important again. Back to that comment about introducing myself as somebody who teaches people how to draw pictures. It turns out that this isn't actually what I do. I don't really care so much *how* people draw pictures. What I do care about is that they use a well-understood, consistent and meaningful set of *abstractions* to think about and therefore describe their software. It turns out that modelling our software isn't dead after all. ;-) Categories: Architecture ### Distributed big balls of mud Coding the Architecture - Simon Brown - Sun, 07/06/2014 - 10:27 If you want evidence that the software development industry is susceptible to fashion, just go and take a look at all of the hype around microservices. It's everywhere! For some people microservices is "the next big thing", whereas for others it's simply a lightweight evolution of the big SOAP service-oriented architectures that we saw 10 years ago "done right". I do like a lot of what the current microservice architectures are doing, but it's by no means a silver bullet. Okay, I know that sounds obvious, but I think many people are jumping on them for the wrong reason. I often show this slide in my conference talks, and I've blogged about this before, but basically there are different ways to build software systems. On the one side we have traditional monolithic systems, where everything is bundled up inside a single deployable unit. This is probably where most of the industry is. Caveats apply, but monoliths can be built quickly and are easy to deploy, but they provide limited agility because even tiny changes require a full redeployment. We also know that monoliths often end up looking like a big ball of mud because of the way that software often evolves over time. For example, many monolithic systems are built using a layered architecture, and it's relatively easy for layered architectures to be abused (e.g. skipping "around" a service to call the repository/data access layer directly). On the other side we have service-based architectures, where a software system is made up of many separately deployable services. Again, caveats apply but, if done well, service-based architectures buy you a lot of flexibility and agility because each service can be developed, tested, deployed, scaled, upgraded and rewritten separately, especially if the services are decoupled via asynchronous messaging. The downside is increased complexity because your software system now has many more moving parts than a monolith. As Robert says, the complexity is still there, you're just moving it somewhere else. There is, of course, a mid-ground here. We can build monolithic systems that are made up of in-process components, each of which has an explicit well-defined interface and set of responsibilities. This is old-school component-based design that talks about high cohesion and low coupling, but I usually sense some hesitation when I talk about it. And this seems odd to me. Before I explain why, let me quote something from a blog post that I read earlier this morning about the rationale behind a team adopting a microservices approach. When we started building Karma, we decided to split the project into two main parts: the backend API, and the frontend application. The backend is responsible for handling orders from the store, usage accounting, user management, device management and so forth, while the frontend offers a dashboard for users which accesses this API. Along the way we noticed that if the whole backend API is monolithic it doesn't work very well because everything gets entangled. The blog post also mentions scaling, versioning and multiple languages/frameworks as other reasons to choose microservices. Again, there are no silver bullets here, everything is a trade-off. Anyway, "everything getting entangled" is not a reason to switch from monoliths to microservices. If you're building a monolithic system and it's turning into a big ball of mud, perhaps you should consider whether you're taking enough care of your software architecture. Do you really understand what the core structural abstractions are in your software? Are their interfaces and responsibilities clear too? If not, why do you think moving to a microservices architecture will help? Sure, the physical separation of services will force you to not take some shortcuts, but you can achieve the same separation between components in a monolith. A little design thinking and an architecturally-evident coding style will help to achieve this without the baggage of going distributed. Many of the teams I've spoken to are building monolithic systems and don't want to look at component-based design. The mid-ground seems to be a hard-sell. I ran a software architecture sketching workshop with a team earlier this year where we diagrammed one of their software systems. The diagram started as a strictly layered architecture (presentation, business services, data access) with all arrows pointing downwards and each layer only ever calling the layer directly beneath it. The code told a different story though and the eventual diagram didn't look so neat anymore. We discussed how adopting a package by component approach could fix some of these problems, but the response was, "meh, we like building software using layers". It seems as if teams are jumping on microservices because they're sexy, but the design thinking and decomposition strategy required to create a good microservices architecture are the same as those needed to create a well structured monolith. If teams find it hard to create a well structured monolith, I don't rate their chances of creating a well structured microservices architecture. As Michael Feathers recently said, "There's a bit of overhead involved in implementing each microservice. If they ever become as easy to create as classes, people will have a freer hand to create trouble - hulking monoliths at a different scale.". I agree. A world of distributed big balls of mud worries me. Categories: Architecture ### Stakeholders Don’t Care How Hard A Story Is! The language they understand is months and dollars (or any other type of currency). Clients, stakeholders and pointy haired bosses really do care about how long a project will take, how much the project will cost and by extension the effort required to deliver the project. What clients, stakeholders and bosses don’t care about is how much the team needs to think or the complexity of the stories or features, except as those attributes effect the duration, cost and effort. The language they understand is months and dollars (or any other type of currency). Teams however, need to speak in terms of complexity and code (programming languages). Story points are an attempt to create a common understanding. When a team uses story points, t-shirt or other relative sizing techniques, they hash a myriad of factors together. When a team decomposes problem they have to assess complexity, capability and capacity in order to determine how long a story, feature or task will take (and therefor cost). The number of moving parts in this mental algebra makes the outcome variable. That variability generates debates on how rational it is to estimate at this level that we will not tackle in this essay. When the team translates their individual perceptions (that include complexity, capacity and capability) into story points or other relative sizing techniques, they are attempting to share an understanding with stakeholders of how long and at what price (with a pinch of variability). For example, if a team using t-shirt sizing and two week sprints indicate they can deliver 1 large story and 2 two medium or 1 medium and 5 small stories based on past performance, it would be fairly easy to determine when the items on the backlog will be delivered and a fair approximation on the number of sprints (aka effort, which equates to cost). Clients, stakeholders and bosses are not interested in the t-shirt sizes or the number of story points, but they do care about whether a feature will take a long time to build or cost a lot. The process of sizing helps technical teams translate how hard a story or a project is into words that clients, stakeholders and bosses can understand intimately. Categories: Process Management ### Celebration And Reflection Are Sides Of The Same Coin Happy 4th of July. Whether on a personal or a project level, the calendar is the most important measuring stick used to gauge progress because it is the measure everyone understands and can follow. We mark significant milestones with a celebration. For example, birthdays are always a milestone and every country, person or project has a birthday (whether they celebrate them or not is another story). Whether the celebration is annually or monthly (anyone with a newborn has celebrated birthdays monthly), isn’t as material as using the calendar milestone to create space for celebration and reflection. At milestones like birthdays we should remember and reflect on where we started, the path that we have taken and our goals. In Agile projects we build in time for reflection called retrospectives. While in classic waterfall projects we have post-implementation reviews. Regardless of which project management camp you find yourself in, introspection and the process adjustments generated from those changes are valuable. What is rarer is personal reflection. Sure we create resolutions on New Year’s Day, but how often do we review our progress against those goals? Milestones represent chance not only to celebrate, but, as importantly, a chance to step back and take time for introspection. For example, the team at the Software Process and Measurement Podcast and blog (yes there is a team) spent a bit of time during a recent retreat reflecting on a number of stalled projects and how we were going to get them going. Milestone evoke celebration and introspection. Celebration is the easy part, what is typically harder is to reflect on how we met our goals. In some cases to we accept accomplishment with asking whether means justify the ends. Early in my career I saw projects that, even though they delivered great outcomes, left project teams in shambles, or some cases, used creative accounting to hide budget overages. In the short run the celebration was exciting, however in the long run was cost worth the benefit? I think not. Tellingly, I do not know anyone from that stage of my career that is still in the business. A better approach is when the markers that show that time is passing are a signal to celebrate and find time to reflect and renew. In projects, those milestones include sprint reviews, demonstrations, sprint planning or classic phase gates. Each of those milestones generate feedback that helps teams and individuals change direction if needed. Feedback provides the impetus for change. It is usually very easy to mark those events by letting friends, family and stakeholders know what has been accomplished since the last milestone. Everyone likes a celebration, whether it is because of fireworks, a piece of cake or the demonstration of some tasty bit of promised functionality. After the celebration step back and reflect on the path that has been taken. Seek out feedback, just like a retrospective and make changes where needed to ensure that when you reflect back at the next milestone that you do not have to think about what you should have done. Categories: Process Management ### Create the smallest possible Docker container Xebia Blog - Fri, 07/04/2014 - 21:59 When you are playing around with Docker, you quickly notice that you are downloading large numbers of megabytes as you use preconfigured containers. A simple Ubuntu container easily exceeds 200MB and as software is installed on top of it, the size increases. In some use cases, you do not need everything that comes with Ubuntu. For example, if you want to run a simple web server, written in Go, there is no need for any tool around that at all. I have been searching for the smallest possible container to start with and found this one: docker pull scratch  The scratch image is perfect. Literally perfect! It is elegant, small and fast. It does not contain any bugs, security leaks, slow code or technical debt. And that is because it is basically empty. Except for a bit of metadata added by Docker. In fact, you could have created this scratch image yourself with this command as described in the Docker documentation: tar cv --files-from /dev/null | docker import - scratch  So that is it, the smallest possible Docker image. End of blog post! ... or is there something more we can say about this? For example, how do you use the scratch base image? It turns out this brings some challenges of its own. Creating content for the scratch image What can we run on an empty base image? An executable without dependencies. Do you have executables without dependencies? I used to write code in Python, Java and JavaScript. Each of these languages/platforms require a runtime installed. Recently, I started looking into the Go (or GoLang if you prefer) platform. And it seems (spoiler alert) like Go is statically linked. So I tried compiling a simple web server saying Hello World and running it within the scratch container. Here is the code for the Hello World web server: package main import ( "fmt" "net/http" ) func helloHandler(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, "Hello World from Go in minimal Docker container") } func main() { http.HandleFunc("/", helloHandler) fmt.Println("Started, serving at 8080") err := http.ListenAndServe(":8080", nil) if err != nil { panic("ListenAndServe: " + err.Error()) } }  Obviously, I cannot compile my webserver inside the scratch container as there is no Go compiler in it. And as I am working on a Mac, I also cannot compile a Linux binary just like that. (Actually, it is possible to cross-compile GoLang sources to different platforms, but that is material for another blog post) So I first need a Docker container with a Go compiler. Let's start simple: docker run -ti google/golang /bin/bash  Inside this container, I can build the Go web server, which I have committed in a GitHub repository: go get github.com/adriaandejonge/helloworld  The go get command is a variant of the go build command that allows fetching and building remote dependencies. You can start the resulting executable with: $GOPATH/bin/helloworld


This works. But it is not what we want. We need the hello world container to run inside the scratch container. So, in fact, we need a Dockerfile saying:

FROM scratch
CMD ["/helloworld"]


and then start that. Unfortunately, the way we started the google/golang container, there is no way to build this Dockerfile. So first, we need a way to access Docker from within the container.

Calling Docker from within Docker

When you use Docker, sooner or later you run into the need to control Docker from within Docker. There are multiple ways to accomplish this. You could use recursion and run Docker inside Docker. However, that seems overly complex and again leads to large containers. You can also provide access to the Docker server outside the instance with a few additional command line options:

docker run -v /var/run/docker.sock:/var/run/docker.sock -v $(which docker):$(which docker) -ti google/golang /bin/bash


Before you continue, please rerun the Go compiler, as Docker forgot our previous compilation during the restart:

go get github.com/adriaandejonge/helloworld


When starting the container, the -v flag creates a volume inside the Docker container and allows you to provide a file from the Docker machine as input. The /var/run/docker.sock is the Unix socket that allows access to the Docker server. The $(which docker) part is a clever way to provide the path for the docker executable inside the container without hardcoding it. However, be careful when you use this command on an Apple when using boot2docker. If the docker executable is installed in a different location than it is installed in boot2docker's virtual machine, this results in a mismatch. It will be the executable inside the boot2docker virtual server that gets inserted into the container. So you may want to replace$(which docker) with /usr/local/bin/docker which is hardcoded. Similarly, if you run a different system, there is a chance that the /var/run/docker.sock has a different location and you need to adjust it accordingly.

Now you can use the Dockerfile inside the google/golang container in the $GOPATH directory, which points to /gopath in this example. Actually, I already checked this Dockerfile into GitHub. So you can copy it from the Go build directory to the desired location like this: cp$GOPATH/src/github.com/adriaandejonge/helloworld/Dockerfile $GOPATH  You need to copy this as the compiled binary is now located in$GOPATH/bin and it is not possible to include files from parent directories when building a Dockerfile. So after copying, the next step is:

docker build -t adejonge/helloworld $GOPATH  And if all goes, well, Docker responds with something like: Successfully built 6ff3fd5a381d Which allows you to run the container: docker run -ti --name hellobroken adejonge/helloworld  But unfortunately, now Docker responds with: 2014/07/02 17:06:48 no such file or directory So what is going on? We have a statically linked executable inside a scratch container. Did we make a mistake? As it turns out, Go does not statically link libraries. Or at least not all libraries. Under Linux, we can see the dynamically linked libraries for an executable with the ldd command: ldd$GOPATH/bin/helloworld


Which responds with:

linux-vdso.so.1 => (0x00007fff039fe000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f61def84000)
/lib64/ld-linux-x86-64.so.2 (0x00007f61df530000)

So before we can run the Hello World webserver, we need to tell the Go compiler to actually do static linking.

Creating statically linked executables in Go

In order to create statically linked executables, we need to tell Go to use the cgo compiler rather than the go compiler. The command to do so is:

CGO_ENABLED=0 go get -a -ldflags '-s' github.com/adriaandejonge/helloworld


The CGO_ENABLED environment variable tells Go to use the cgo compiler rather than the go compiler. The -a flag tells Go to rebuild all dependencies. Otherwise you still end up with dynamically linked dependencies. And finally the -ldflags '-s' flag is a nice extra. It reduces the file size of the resulting executable by roughly 50%. You can also do this without the cgo compiler. The size reduction is a result from removing debug information.

Just to be sure, rerun the ldd command.

ldd $GOPATH/bin/helloworld  It should now respond with: not a dynamic executable You can also rerun the steps for creating the Docker container around the executable from scratch: docker build -t adejonge/helloworld$GOPATH


And if all goes well, Docker responds with something like:

Successfully built 6ff3fd5a381d

Which allows you to run the container:

docker run -ti --name helloworld adejonge/helloworld


And this time it should respond with:

Started, serving at 8080

Until so far, there were many manual steps and there is a lot of room for error. Let's exit from the google/golang container and continue from the surrounding machine:

<Press Ctrl-C>
exit


You can check the existence or absence of containers and images with:

docker ps -a
docker images -a


And you can do some cleaning of Docker with:

docker rm -f helloworld
docker rmi -f adejonge/helloworld


Creating a Docker container that creates a Docker container

The steps we took so far, we can also record in a Dockerfile and have Docker do the work for us:

FROM google/golang
RUN CGO_ENABLED=0 go get -a -ldflags '-s' github.com/adriaandejonge/helloworld
RUN cp /gopath/src/github.com/adriaandejonge/helloworld/Dockerfile /gopath
CMD docker build -t adejonge/helloworld gopath


I checked this Dockerfile into a separate GitHub repository called adriaandejonge/hellobuild. It can be built with this command:

docker build -t adejonge/hellobuild github.com/adriaandejonge/hellobuild


Providing the  -t flag names the image as adejonge/hellobuild and implicitly tags it as latest. These names make it easier for you to remove the image later on. Next,  you can create a container from this image while providing the flags that you have seen earlier in this post:

docker run -v /var/run/docker.sock:/var/run/docker.sock -v $(which docker):$(which docker) -ti --name hellobuild adejonge/hellobuild


Providing the --name hellobuild flag makes it easier to remove the container after running. In fact, you can do so right away, because after running this command, you already created the adejonge/helloworld image:

docker rm -f hellobuild
docker rmi -f adejonge/hellobuild


And now you can start a new container named helloworld based on the adejonge/helloworld image as you have done before:

docker run -ti --name helloworld adejonge/helloworld


Because all these steps are run from the same command line, without opening a bash shell inside a Docker container, you can add these steps to a bash script and run it automatically. For your convenience, I have added these bash scripts to the hellobuild GitHub repository.

Also, if you want to try the smallest possible Docker container running a Hello World web server without following all the steps described in this blog post, you can also use the pre-built image that I checked into the Docker Hub repository:

docker pull adejonge/helloworld


With docker images -a you can see that the size is 3.6MB. Of course, you can make it even smaller if you manage to create an executable that is smaller than the web server in Go that I wrote. In C or Assembly you may be able to do so. However, you can never make it smaller than the scratch image.

### Stuff The Internet Says On Scalability For July 4th, 2014

Hey, it's HighScalability time:

Beauty is everywhere. Household dust magnified 22 million times.
• Let's play a game of guess the company. They have: >100 billion searches per month; > 60 trillion known URLs; > 50 billion facts in knowledge graph; > 100 hours of video uploaded every minute; > 2 billion containers; > 6 trillion Cloud Datastore ops/month. Who is it? Why it's Google, of course.
• Billions of events every day: Twitter. One billion active users: Android.
• Quotable quotes:
• PeterGriffin: I don't know why the author called this "Multi-process architectures suck :(" when he really meant "I suck at multi-process architectures :("
• @khrabrov: Experienced startup engineers are looking for a full-stack Business Guy to be CEO, COO, PM, marketer, account manager, HR, and receptionist.
• @PatrickMcFadin: 30x perf over #hadoop by running #spark over #cassandra The crowd was stunned.
• @jcoglan: A programmer is someone who can simultaneously entertain the ideas that tight coupling is bad and fridges should be connected to the 'net
• @BenedictEvans: Consumers spend more on apps (~$20bn run rate) than on recorded music ($17bn).
• @solarce: "You achieve nirvana when all failures are viewed as normal operations and not as apocalyptic events"
• Rudiger Moller: Yup. As memory keeps getting cheaper, Java cannot profit except going off heap or use Azul Zing. Either improve concurrent GC or reduce the amount of references required to model data structures in Java.
• @PatrickMcFadin: OH: "idompotency is better than beefalo"
• I started listening to Songza about 6 weeks ago. Loved its emotional intelligence. And now I find Google went and acquired it. A coincidence? This is not a case of megalomania. It occurred to me that Google is in the perfect position to let some algorithms loose on its data to see if a service like Songza is gaining mind share. If you look at DNS access, G+, Gmail, Chrome, web trends, etc you have a pretty good proxy for actual usage data. In fact, your algorithms could just look at everything and identify acquisition targets by ranking what services are rising above the noise. And in double fact Google can probably estimate future growth trends better than Songza because they have historical data on many other services.
• Concurrency Improvements in HyperLevelDB. Taking single threaded code and making multithreaded is not for the faint of heart. Deadlocks await each new access pattern. By reducing time locks are held, using lock free data structures, and using fine grained locking HyperDex was able to reach 400K operations per second, better than LevelDB's 275K operations per second.
• The Lambda Architecture has nothing to do with The Secret, in case you were wondering. To see why Jay Kreps has an excellent article Questioning the Lambda Architecture based on his experiences at LinkedIn. The main objection is double processing, concluding: These days, my advice is to use a batch processing framework like MapReduce if you aren’t latency sensitive, and use a stream processing framework if you are, but not to try to do both at the same time unless you absolutely must. Great discussion in the comment section. For me it's as simple as never mix read and write streams. They have completely different purposes. More on Hacker News.
• Videos from the Velocity Conference 2014 on YouTube.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so keep on going)...

Categories: Architecture

### Quote of the Day

Herding Cats - Glen Alleman - Fri, 07/04/2014 - 16:23

It is the mark of an educated mind to rest satisfied with the degree of precision which the nature of the subject admits and not to seek exactness where only an approximation is possible —Aristotle (384 B.C - 322 B.C.)

When we hear someone say estimates are guessesWhen we estimate we act as if we believe the plan will not change, or similar uninformed nonsense, think of Aristotle. Without the understanding, from education, experience, and skill to realize that all project variables are random variables and vary naturally and vary from external events.

As such in order to determine the future impacts from decisions that involve cost, schedule, and performance, we need to estimate that impact of those random processes on the outcome of our decision.

This is the basis of all decision making in the presence of uncertainty. It's been claimed decisions can be made without estimates, but until someone comes up with the way to make decisions without estimating those impacts, statistical estimating is the way.

Categories: Project Management

### My Best Diagram Ever: the Celebration Grid

NOOP.NL - Jurgen Appelo - Fri, 07/04/2014 - 12:59

At the Spark the Change conference in London, someone tweeted, “This one slide was worth the ticket price!” which was then retweeted more than 30 times. I got similar great comments from people in my workshops. And yes, I agree. My celebration grid might be the best model I ever created. It is based on the work by Donald Reinertsen, but I tried to visualize what he said in many more words.

The post My Best Diagram Ever: the Celebration Grid appeared first on NOOP.NL.

Categories: Project Management