Today Engine Yard announced enhancements that will make its architecture increasingly flexible and modular. As we were preparing for the announcement, I took some time to reflect on where Engine Yard started, where we are headed and how the industry is evolving.

Companies today are literally running past existing players in the market by focusing on end user value instead of getting distracted by infrastructure and system software. For example, we have a customer in the sports reporting field who became the #3 entity in its market within five years. They passed several companies that have been in the space for decades. They achieved this by focusing on how to get content into their application from the outside world and they spent almost none of their time or money worrying about how the infrastructure works or how to create the right runtime environment - and they did this with Platform as a Service (PaaS).

PaaS helps them:

1. Focus on features and user experience without being dragged into the details of operating and managing infrastructure.

2. Use infrastructure and software as an on-demand utility to convert capital expenditures into variable costs.

From Managed Hosting to Platform as a Service

Six years ago, before the days of PaaS, Engine Yard started as a managed hosting company. When hosting applications we would configure servers, load system software, install application code and then run applications on behalf of clients. We did this enough times that we saw an opportunity to scale through automation. We also realized that hosting an application entails two businesses:

1. Owning the physical infrastructure (servers, storage, power, etc)

2. Operating the software that manages the hardware and empowers the application

We decided to focus on the software layer and grow it into the business now called PaaS.

Developers focus on apps, we handle DevOps

Let’s examine how PaaS is changing the way people build application businesses, by looking at how one of our customers, a well known media company, runs their web app (i.e. website) on our PaaS. The front-end of their app is primarily JavaScript that runs in a browser. In the backend they have a lot of Ruby and database code that helps serve all the pages, stream their videos, etc. To run their application, our automation system installs their code and then configures a runtime environment, including an operating system, language environment, database environment, and the web tier frameworks necessary to run a modern website.

Our customer points to their code repository literally with an HTTP call, and then we take care of everything necessary to run it. If they anticipate extra traffic they can easily add (or decommission) additional compute resources with just a couple clicks.  But not all PaaS options are created equal and it’s important to understand architectural differences and how those might constrain your business.

Choosing a PaaS: The balance between control and automation

Some providers deliver complete automation in a black box. Like all automation, you don’t necessarily have control, but it does exactly what it says it’s going to do. You can use it for relatively simple workloads that will not need much modification. Once deployed, the app will run and that’s a reasonable approach for some use cases. Another trade off with the black box approach is that infrastructure and virtual machines are typically used in a shared manner. Potential issues include:

• The “noisy neighbor” problem: one application impacts another’s performance

• Security issues: shared access to the platform introduces risk across all workloads

• Single point of failure: If the platform goes down, so does your application

At the other end, is the do it yourself approach where you’re responsible for building and managing the automation offered by a conventional PaaS. You procure hardware, or provision infrastructure from a hosting provider, but you still need to build, configure, and manage operating systems, middleware and databases. You end up spending a lot of time on operational and administrative tasks that are non-revenue generating.

The approach Engine Yard advocates - increase control and allow for choice - is the basis of the architecture we have brought to market and will continue updating throughout this year. Choice between different types of components and infrastructures to run applications on and the ability to go underneath the defaults and gain control and access to the actual virtual machine. Faster provisioning through our new clusters, a purpose built set of virtual machines for applications, databases and utility processes. We know from experience that such an architecture is best for serious commercial workloads.

Learn more about the vision for our architecture in this 4 min whiteboard video

Even with this approach, trusting a vendor to help you with something as important as running your application inevitably raises questions.

PaaS is great - but what about security?

It’s true that you give up some control in areas historically owned by IT departments; for instance, the server itself isn’t actually on premise. To address these concerns, our architectural updates are making it possible to implement and control security policy throughout the infrastructure, platform and application layers. At the infrastructure layer, our partners are now able to provide SOC and HIPAA certified solutions. At the software layer our philosophy is to provide well architected defaults while allowing developers to retain operational control.

For example, an important area of security is key management. To help developers feel comfortable, we provide visibility into the entire process and they can even access and manage the keys themselves. So, even though we handle the painful part of managing thousands of SSH and SSL keys, we’re not hiding it. In addition, we’re continuing to expand security capabilities to provide further transparency and visibility.

PaaS is great - but can’t I just do this myself?

If you’re a small company with a great idea, it’s very clear that PaaS lowers the entry cost tremendously. You don’t have to license a bunch of software, set up servers or negotiate with an infrastructure provider. You gain immediate access to a platform that’s capable of serving millions of end users. And you only need to spend on infrastructure as you grow, you don’t have to otherwise. When growth accelerates you can avoid hiring people to take care of infrastructure and instead put more money into what you do best - developing the application.

Because we’re automating your operational tasks, Engine Yard essentially joins your operations team and manages runtime optimizations, upgrades and security fixes that continually surface across environments. You just focus on your application.

The future of applications

Increasingly, what you think of as an application is really a user interface running in your pocket or on a tablet - while the app itself is running on a virtual server in the cloud. You may use the application on your laptop at work, with your cell phone on your commute and then perhaps with your TV or tablet at home. Same app, different interfaces - available to you wherever you want it - all running in the cloud. Cloud computing is making ubiquitous application experiences readily available, and yet it’s very early in the cycle where people learn how utility-based compute is going to change how experiences are delivered.

PaaS lets you launch a bold and risky idea without huge upfront investment. For example, there’s a whole set of social applications that didn’t exist five years ago and most were built on PaaS. Some of the giants in this space were bootstrapped at first but were able to get started with PaaS and rapidly built massively disruptive businesses that have changed the way companies connect with prospective customers. Corporate IT is another constituency that will benefit tremendously. They will use PaaS to deliver a suite of customized and integrated applications as a service that enable their workforces to be more productive.

This is why we’re enhancing our architecture. You can choose your infrastructure and hundreds of permutations of components in your software stack. You can integrate third-party add-ons such as application monitoring, email or any other capability with just a single click. With new user interface improvements, you can easily plan, deploy, monitor and manage applications and iterate several times a day.

That’s our mission - to help developers innovate and make applications that change the world. And this is just the beginning.

Learn more about how we are enhancing Engine Yard Cloud

Sign up for our live webcast on March 7th

A few years ago my wife and I were hiking in an obscure corner of the Rocky Mountain National Park. The trail we had chosen had a very significant change in elevation but the guide book promised a beautiful mountain lake.  We were not disappointed.  Not only was the spring fed lake nestled in the rugged peaks, quite a site, but the clarity and reflections in the lake were mesmerizing.

We spend a significant amount of time studying how code is written, functions tested and projects led and/or managed. Behavior is often a critical determinant of outcome of work. However, just observing a behavior and understanding the root of that behavior is rarely a straight forward trip. Systems Thinking is an important concept to help change agents understand that behavior is often a reflection of many factors.

Documentation on software projects is often a contentious topic, particularly since the agile manifesto says that we should value working software over comprehensive documentation. My take on lightweight documentation has now been included in my Software Architecture for Developers book and I've had a stream of e-mail requests since publishing the new version to ask whether I would create a sample software guidebook too.

A sample software guidebook

Yes is the answer ... and I've already started putting this together, based upon a side-project of mine called techtribes.je, which is basically a community portal for the IT, tech and digital industry in Jersey*. This is a fun little side-project rather than a huge high-performance distributed system, but it should be enough to illustrate the concepts in the book.

And some sample software architecture sketches

The sample guidebook will also include some software architecture diagrams, which will again serve as an example of the context, containers, components and classes sketches that I talk about. The following context diagram (included in the sample guidebook) provides an overview of techtribes.je:

The introduction and context sections of the sample guidebook are available in the published version of my book now and I'll be adding the other sections over the next few days.

Plus some open source code

As a final note, I'm planning to open-source the code behind techtribes.je on GitHub so that you can see the correlation between the code and the documentation. If there's anything else that you'd like to see in the book, please just drop me a line.

* just to avoid any confusion, I'm referring to Jersey in the Channel Islands, not New Jersey! :-)

Late 2011 a movie was shot where I explain the then version of PointZERO. As I wanted to share this one with you all I’ve uploaded it on YouTube.

http://www.youtube.com/watch?v=Pl7cQ9YcBpg

The original video from Sogeti can be found here.

Posted by Trevor Johns, Android Developer Relations team

Following our talk "Security and Privacy in Android Apps" at Google I/O last year, many people had specific questions about how to use cryptography in Android. Many of those revolved around which APIs to use for a specific purpose. Let's look at how to use cryptography to safely store user credentials, such as passwords and auth tokens, on local storage.

An anti-pattern

A common (but incorrect) pattern that we've recently become aware of is to use SecureRandom as a means of generating deterministic key material, which would then be used to encrypt local credential caches. Examples are not hard to find, such as here, here, here, and elsewhere.

In this pattern, rather than storing an encryption key directly as a string inside an APK, the code uses a proxy string to generate the key instead — similar to a passphrase. This essentially obfuscates the key so that it's not readily visible to attackers. However, a skilled attacker would be able to easily see around this strategy. We don't recommend it.

The fact is, Android's existing security model already provides plenty of protection for this kind of data. User credentials should be stored with the MODE_PRIVATE flag set and stored in internal storage, rather than on an SD card, since permissions aren't enforced on external storage. Combined with device encryption, this provides protection from most types of attacks targeting credentials.

However, there's another problem with using SecureRandom in the way described above. Starting with Android 4.2, the default SecureRandom provider is OpenSSL, and a developer can no longer override SecureRandom’s internal state. Consider the following code:

  SecureRandom secureRandom = new SecureRandom();
  byte[] b = new byte[] { (byte) 1 };
  secureRandom.setSeed(b);
  // Prior to Android 4.2, the next line would always return the same number!
  System.out.println(secureRandom.nextInt());

The old Bouncy Castle-based implementation allowed overriding the internally generated, /dev/urandom based key for each SecureRandom instance. Developers which attempted to explicitly seed the random number generator would find that their seed replaces, not supplements, the existing seed (contrary to the reference implementation’s documentation). Under OpenSSL, this error-prone behavior is no longer possible.

Unfortunately, applications who relied on the old behavior will find that the output from SecureRandom changes randomly every time their application starts up. (This is actually a very desirable trait for a random number generator!) Attempting to obfuscate encryption keys in this manner will no longer work.

The right way

A more reasonable approach is simply to generate a truly random AES key when an application is first launched:

public static SecretKey generateKey() throws NoSuchAlgorithmException {
    // Generate a 256-bit key
    final int outputKeyLength = 256;

    SecureRandom secureRandom = new SecureRandom();
    // Do *not* seed secureRandom! Automatically seeded from system entropy.
    KeyGenerator keyGenerator = KeyGenerator.getInstance("AES");
    keyGenerator.init(outputKeyLength, secureRandom);
    SecretKey key = keyGenerator.generateKey();
    return key;
}

Note that the security of this approach relies on safeguarding the generated key, which is is predicated on the security of the internal storage. Leaving the target file unencrypted (but set to MODE_PRIVATE) would provide similar security.

Even more security

If your app needs additional encryption, a recommended approach is to require a passphase or PIN to access your application. This passphrase could be fed into PBKDF2 to generate the encryption key. (PBKDF2 is a commonly used algorithm for deriving key material from a passphrase, using a technique known as "key stretching".) Android provides an implementation of this algorithm inside SecretKeyFactory as PBKDF2WithHmacSHA1:

public static SecretKey generateKey(char[] passphraseOrPin, byte[] salt) throws NoSuchAlgorithmException, InvalidKeySpecException {
    // Number of PBKDF2 hardening rounds to use. Larger values increase
    // computation time. You should select a value that causes computation
    // to take >100ms.
    final int iterations = 1000; 

    // Generate a 256-bit key
    final int outputKeyLength = 256;

    SecretKeyFactory secretKeyFactory = SecretKeyFactory.getInstance("PBKDF2WithHmacSHA1");
    KeySpec keySpec = new PBEKeySpec(passphraseOrPin, salt, iterations, outputKeyLength);
    SecretKey secretKey = secretKeyFactory.generateSecret(keySpec);
    return secretKey;
}

The salt should be a random string, again generated using SecureRandom and persisted on internal storage alongside any encrypted data. This is important to mitigate the risk of attackers using a rainbow table to precompute password hashes.

Check your apps for proper use of SecureRandom

As mentioned above and in the New Security Features in Jelly Bean, the default implementation of SecureRandom is changed in Android 4.2. Using it to deterministically generate keys is no longer possible.

If you're one of the developers who's been generating keys the wrong way, we recommend upgrading your app today to prevent subtle problems as more users upgrade to devices running Android 4.2 or later.

Join the discussion on

+Android Developers

By my latest count I have around 15 different micro services/applications checked out on my machine which comprise the system that I’m currently working on.

Most of these are Ruby related so it’s easy to figure out how to start up a local copy because it’s either bundle exec rails server if it’s a rails application or bundle exec backup if it’s a sinatra/rack application.

The clojure applications follow a similar convention and we use rake to run any offline tasks.

Despite these conventions there is still a reasonable amount of knowledge around how to actually get these applications to work that remains in people’s heads but doesn’t necessarily have to.

Although talking to each other is generally a good thing, an exercise that we tried at GDS was to try and document in the README file what someone new to an application would need to know to get it working.

They then tried to get it up and running while only referring to the instructions but noted down anything that didn’t make sense or any steps which seemed unnecessarily manual and could be automated.

This was almost certainly more frustrating than having someone show you what to do but it scales a bit better and reduces what I think isn’t a high value conversation.

I recently came across a blog post Tom Preston Werner wrote a couple of years ago titled ‘Readme Driven Development‘ in which he goes further in suggesting that we drive out our applications from the README.

This document should stand on its own as a testament to your creativity and expressiveness. The Readme should be the single most important document in your codebase; writing it first is the proper thing to do.

I don’t know if we need to go this far but as we more towards a future where we’ll be working with lots of different applications it would be nice not to have to remember everything about them in our head!

Anna and I were recently discussing the way that we get things done outside of work and since December I’ve been fairly religiously working through various ‘to-do’ lists with a pomodoro timer.

So far I’ve done 308 30 minute pomodoros in about 8 weeks which is just under 20 hours a week which is not bad but still leaves time for a ridiculous amount of procrastination.

These are some of the things that I’ve noticed from only doing things when it’s explicitly on a timer:

I waste a lot of time

I haven’t been keeping track of how many pomodoros I’ve been doing each day but the Easy Pomodoro app which I’ve been using to track my time allows me to see how many pomodoros I’ve done on a task so I have a rough idea.

On some days I’ve managed to go from 8pm to 1am and only done 3 pomodoros which leaves an impressive 3 1/2 hours of Facebook, twitter, gmail time.

I did try the StayFocusd application for a few weeks but it becomes really annoying when you do legitimately want to chill out!

Things need to be deleted from the ‘to-do’ list

The first to-do list I created had around 10 things on it initially and I added every new thing I wanted to do to it until I had around 25 things on the list and yet had only completed 3 of the original 10 things.

My rough rule of thumb is that if I re-write the ‘to-do’ list twice and the same item is still there then it gets deleted because I clearly don’t have the motivation to do it!

I thought I’d find it frustrating to delete things from the list but it’s actually more relaxing to know there’s a shorter list of things I’m actually interested in doing.

Once I get into something I spend ages doing it

I’ve been tracking how many pomodoros I spent on individual tasks and when sorted I’ve spent 12 pomodoros on average on the tasks in positions 2-7.

The item in position 1 is me playing around with football data which I’ve written about previously and has taken up 36 pomodoros or 18 hours of time in the last couple of weeks.

I generally try to spend my spare time learning new things but in this case most of it has been spent cleaning up/manipulating data into a format that I can do something useful with.

It’s now clear when I’m giving up without really trying

One of the habits I’ve been trying to shake is that I often give up learning something if I’m not able to grasp the basics reasonably quickly.

Timing things on a pomodoro has made it visible how quickly I tend to reach that state and when I see that I’ve only been trying to understand the topic for 1 hour it does make me realise how ridiculous I’m being.

It does seem to motivate me to try for at least a few hours and varying the approach before deciding that a topic maybe isn’t for me at least for now!

I last wrote about pomodoros about a year ago and it’s interesting to see that I’ve covered almost completely different things than I did last time!

I’m not sure exactly what that means but I’d recommend them as a tool for fighting procrastination at the very least!

A reasonable amount of the information that I consume comes either via scanning twitter or from my prismatic feed but I noticed that I’m quite biased to reading things in similar subject areas.

I tend to end up reading about data mining/science, functional programming and startups and while the articles are mostly interesting it does eventually start to feel like you’re in an echo chamber.

I have a subscription to the ACM mainly because I enjoy reading the ‘Communications of the ACM’ magazine which gets sent out every month and until recently I only read articles which I thought would be interesting.

This unfortunately meant that I was adding to the problem I mentioned earlier whereby everything I read is about similar topics.

I decided to try and change that by reading the magazine from cover to cover and although I haven’t finished yet it’s been an interesting experience and I’ve read about things that I wouldn’t have thought to read about including:

• Do more computer science papers get rejected than those in other subjects?
• How do we allow people to vote in areas where there’s been a natural disaster?
• Why are there currently so many post-docs in computer science?
• How should we go about analysing performance problems?

This seems to be a reasonably good way of diversifying what I read/learn because an editor has decided what I should read rather than me choosing or an algorithm choosing based on what I’ve previously read.

Having said that, I’d be intrigued to know what approaches/strategies others have for getting knowledge of a broader range of topics.

In my last blog on legacy systems I talked about what they were and weren't. In this one I'm going to expand on this and how they fit into the business process lifecycle.

Like most developers the centre of my work life is the software development process. This might involve a business analysis phase before and some support afterwards but this is a relatively small percentage of my time.

The 'software development centric' view of the world is below (this is the 'SDLC diagram' and your own process may differ but will share elements). There are two parts that interact with the outside world - requirements and deploy (which may be lightweight and frequent) and the external interactions are wrapped in that.

For some businesses (e.g. public facing web sites) the software IS the business and there is no significant period before or after the software development - it occurs constantly. These businesses are very interesting to us developers for this very reason (and therefore the projects often discussed at conferences) but are actually the exception. For the majority of organisations their individual IT systems perform a function that constitute only a small part of what they do.

Therefore users of our software view the world differently. The software development phase is a very small part of their business processes lifecycle and lifespan. They view the world a little more like this:

Most of the processes they execute will originally have been done ‘manually’ (I include generic software tools such as spreadsheets) and may been done this way from 6 months to 100 years. (If you think I'm exaggerating with '100 years' then you should speak to someone who's worked with an old insurance company!)

Eventually it will be decided that it is cost effective to develop bespoke software (or spend a lot of effort configuring BPM/CRM etc software). This process may be iterative and deliver value quickly but will be mostly complete after a relatively short period of time (compared to the organisation’s lifespan).

It is now fully live and a software team will consider it to now be in its maintenance phase. From the organisation's point-of-view this is where the real value is extracted from the system. Very little money is being spent on it but it is being used and either helping to generate revenue or increasing the efficiency of a process.

As time goes on there will be a decreasing number of changes made to it. Eventually it will be considered ‘legacy’. Ultimately the system will reach end-of-life and replaced when no longer fit for purpose.

If we were going to scale the diagram to indicate time then it might look like this:

These are kind of lifecycle times that I've experienced and yours may differ - this will depend largely on the industries you work within.

Therefore 'legacy' should be viewed as a phase in the lifecycle of the process. Note that I have a small arrow going back to Software Development from the maintenance/legacy phase as it may have upgrades or additions made many years after being deployed.

A comment on my previous blog post said that they refer to 'legacy' systems as 'mature' as this has more positive connotations. I think it also accurately reflects legacy as being a stage of a lifecycle rather than any particular indication of quality.

Lastly I'm just going to quickly plug my talk at QCon again (Modern Legacy Systems) which explores some legacy issues, makes suggestions about approaching them and how to leave a good legacy rather than a bad one. If you haven't booked your place at QCon yet then you can use code ANNE100 to get £100 off.

Should you use PaaS for your next project? Often the answer is no because you want control, but here's an example from SongPop showing why the promise of PaaS is not passé. SongPop was able to autoscale to 60 million users, 1 million daily active users, deliver 17 terabytes/day of songs and images worldwide, handle 10k+ queries/second, all with a 6 person engineering team, and only one engineer working full-time on the backend.

Unfortunately there aren't a lot of details, but what there is can be found in Scaling SongPop to 60 million users with App Engine and Google Cloud Storage. The outline follows the script. You start small. Let PaaS do the heavy lifting. And when you need to scale you just buy more resources and tune a little (maybe a lot). The payoff is you get to focus on feature development and can get by with a small team.

Here's a diagram of their architecture:

As I was doing some market research for my EA's competency driven strategy I ran across an interesting article from late 2012 that validates the importance of a well educated, fully rounded and even certified Enterprise Architect. The article, "23 IT Certifications That Mean Higher Pay" posted on CIO.com in September of 2012, shows that companies value these skills and are willing to pay more. 

The data was pulled from the Foote Research Group quarterly 2012 IT Skills Demand and Pay Trends Report and its CEO David Foote spoke with CIO.com about how you can use certifications to get employers to show you the money.

The article separates the list into three tiers of percent added to base pay from the lowest to highest:

• 8% - 13% Added to Base
• 10% - 15% Added to Base
• 12% - 16% Added to Base

 

In the highest tier of 12% to 16% Open Group Certified Architect (Open CA) gets top ranking in Enterprise Architecture certifications for largest percent premium added to your base salary. I received my Open CA certification in 2011 and was nominated to the certification board. It was quite the honor for both.

Afterwards I really wanted top share my experiences and write up a condensed version of what Open CA really was. You can find this information in the post entitled, "The Open Group Certified Architect (Open CA) Program Distilled".  

Two things I think this article says about our industry:

1. Baseline of Skills and Competencies - Whether it is Open CA or any other EA certification, I believe this shows a natural maturing in the EA discipline. The industry is bringing together a set of common and recognized set of skills and competencies through these certifications. In this case the EA market has recoginized Open CA as the most popular and perhaps the defacto standard for EA skills and competencies. 
2. Competencies of Skills - I liked seeing Open CA certification on this list rather than TOGAF or a Zachman certification (there are many others besides these two). The reason I prefer Open CA is because it is  competency based. Meaning it's about how and what you have done as a practitioner and not about what you know from reading a book or taking a class.  

 

For more information on Open CA: http://www.opengroup.org/openca/cert/ 

 

Also of note in that tier with Open CA is the Program Management Professional (PgMP) certification.

I’ve been asked to do a lot of Agile retrospectives around Microsoft over the years.  I don’t know how it started, but it started several years ago when somebody recommended that I lead a retrospective for their team, and then it caught fire from there.  

In this post, I’ll share a simple recipe you can use as a baseline to help shape your Agile retrospectives for building high-performance teams.

Agile retrospectives are a powerful way to help teams go from good to great, and to help less than good teams, get better fast.   The value of a retrospective is the learning and insight that you can carry forward.  A retrospective is a look back with an open mind on the collective learning around what to start doing, stop doing, and continue doing.

With that in mind, the true value is in the actual change in the people, process, or technology that supports your ability to flow value.

To drive a great Agile retrospective, it helps to know what gets in the way or what goes wrong:

1. Lack of clarity on outcomes
2. Lack of focus during the retrospective
3. Too much debate and not enough dialogue.

Below is a recipe that you can use that should help you get started or improve your retrospectives.  Nothing is set in stone, but it helps to see some things that have worked time and again (the timeless truths.)  Be sure to adapt as you see fit, but at the end of the day, remember that establishing is important, and that your best outcome is a short list of what to start doing, stop doing, and continue doing – that you actually implement.

Keys

• Set and agenda with a timebox (and allow plenty of time to really dive into the issues)
• Ask what went well
• Ask what could be improved

Outcomes

1. List of highlights and wins
2. Inefficiencies and improvement opportunities
3. Contextualized best practices

Flow

1. Agenda
2. Celebrate accomplishments
3. Brainstorm what went well
4. Identify what needs improvement
5. Talk about the details
6. Identify 3 actionable things to carry forward, with dates and owners (accountability)
7. Scrum Master sends out summary

Switching Perspectives

I’ve found two Edward de Bono techniques help deal with conflict during hot topics are:

• PMI (Plus Points, Minus Points, and Interesting Points) – What’s the upside? what’s the downside? and what’s interesting about this?
• Six Thinking Hats – A way to switch hats to switch perspective to see multiple angles on the same topic.

I find these techniques help keep an open and curious mind.  Especially if you use them in a question-driven way and keep it simple.  Rather than have people arguing different sides at the same time, have them argue the same side at the same time, and then switch perspectives (or “hats.”)

 

Category

Items

The Hats

· White Hat – the facts and figures

· Red Hat – the emotional view

· Black Hat – the “devil’s advocate”

· Yellow Hat – the positive side

· Green Hat – the creative side

· Blue Hat – the organizing view

Questions

· What are the facts and figures?

· What’s your gut reaction?  How do you feel about this?

· Why can’t we do this?  What prevents us?  What’s the downside?

· How can we do this?

· What are additional opportunities?

· How should we think about this? (what are the metaphors or mental models)

Dialogue, Debate, and Discuss

Sometimes, the best way to help people stay open and

• Dialogue - listening with an open spirit.
• Debate - listening to win an argument -- a verbal "fight."
• Discuss - listening to break apart an issue and pushing a winning idea.

How to Shift to Dialogue:

1. How might that be true?
2. What is it you see that I don’t?
3. How do you see this differently and why?

Highlights / Wins

1. Highlight 1 …
2. Highlight 2 …
3. Highlight 3 …

Affinity Exercise

You can skip doing an affinity diagram exercise, if folks are comfortable with each other and there's no recent new members.  Otherwise, it's overhead, but it helps for the following:

1. Get people to open up
2. Get more people contributing
3. Give people time to think since new members bring up more issues

There are other tricks of the trade, but if you focus on a clear agenda, compelling outcomes, and manage the conflict while driving an open dialogue, you’ll be in good shape.

May the power of Agile retrospectives serve you well.

You Might Also Like

• 10 Big Ideas from Getting Results the Agile Way
• Getting Results the Agile Way: The Book on Getting Results
• Productivity on Fire: 30 Days of Getting Results the Agile Way

Last month my blog was exactly 5 years old. I celebrated this moment by asking readers which blog post had been the most important for them. And I sent some free goodies (games and a booklet) to everyone who replied. Here are some more comments that poured in…

(Oh and sorry, the deadline is closed now!)

“A recent one that stuck with me, and is actually causing me to stop every now and then and think about it, is Why Not Delight the Supplier? It's actually not as much the idea of delighting the supplier, but the idea that there are multiple stakeholders around every organization and every one of them ought to be looked after.”
- Flavius

“My favorite post is the T-Shirt Test. Although I doubt I would proudly wear a t-shirt with the photo of my pet on it, I got your point. It's a great source of reflection about the pride you or your team has of working in the organization, which is at the end key to have successful implication from the team in the projects.”
- Pierre

“One of the posts that I liked was 360 Degrees Dinner. We are currently doing 360 degrees evaluation by email. I liked your post because it made me think of another view of an important process I do not like.”
- Boris

“I did some browsing back the history of your blog until I found the very first post that led me to your site. "Top 15 Systems Thinking Books" was the title of it - and it was almost two years ago.”
- Oliver

“One of the most important for me was the post Top 100 Agile Books (Edition 2012). It was probably one of easiest posts for you :). Also I would like to mention the funniest part in your blog posts: Nonviolent Communication (Stop It!) and the Video from Bob Newhart (stop It). Stop It method could also help to avoid long, aimless, tough discussions. In big projects you always have such. ;)”
- Maxim

“For me it was: Is Your Work an Expression of Your Life? The funny thing is that this blog post was written at exactly the moment when I needed it :) Since that time I believe in "changing work into life" and I'm quite comfortable to work anytime I feel like. It brought me lots of great ideas and makes me very happy.”
- Tomasz

“The most important post for me was The Big List of Agile Practices and for two things: At first it helps me to go deeper into the amazing world of Agile. It helped me for my thesis about Agile practices. In second, with it, I discovered the rest of your blog and then Management 3.0.
- Geoffrey

“I would choose "Why I Won't Take Your Call", because you are talking a little bit out of my own soul since I see it the same as you. Reachability has always been the virtue of the slaves and that's why I have an answering machine acting as a firewall to protect my free time. I even do not have a mobile phone! The other thing I like in your posts is your honest way of writing. This is refreshing in this day and age and it leaves me with a smile.”
- Heiko

“To me the most important blog post was Top 100 Blogs for Developers (Q1 2009). I know it's not a masterpiece in the sense of creativity (being just a collection of data), but to me it was exactly what I needed.”
- Gabriel

And that’s it.

Thanks everyone who participated. It’s great to know some of my work matters to you. Your feedback helps me to look forward to the next 5 years. :-)

 

If you want to organize an agile program, so you can manage the stream of features in your agile program, you have some options. It depends on the size of your program. The communication structures in your agile program matter.

How Large Is Your Agile Program?

I think of programs as small, medium, and large. Yes, it’s a generality, and it’s been helpful to me. A small program is up to three teams. A medium program is four to nine teams. A large program is ten teams or more. Your mileage will definitely vary. It varies because of the size of the teams.

Here’s why I find the these guidelines useful. Remember back when I wrote Why Team Size Matters? If you try to graph the communication paths for a 5 -person team, you have a graph that looks like the one to the left.

Now, you’ve only added one more person, and you’ve increased the communication paths by five. This is why the size of the team matters when you think about your program and whether you have a small, medium or large program.

If you have four-person teams and you have three teams, you have a small program. If you have ten-person teams, you might still have a small program. But you’re pushing it. You might have a medium size program. You know how sometimes you have to change the data structures in code when the size of the data changes? The same thing happens with a program. You want to think about the organization and communication structures you’re using, to make sure they enhance the product development you’re doing, and not holding you back.

Remember that an agile team depends on the micro-commitments every single day between its members to make progress. The more teams you have attempting to make progress on features across the organization, the more you want to enhance that communication. So you want to select a program organization that enhances that communication.

Programs Have Communication Paths, Too

Technical Program with Communities of Practice

In an agile program, the technical project teams make commitment to and with each other. They have communication paths not just inside the teams but between the teams. The way they communicate can vary depending on whether the program is small, medium or large. How you organize the program will change how you communicate. You have choices.

Many people use Scrum as their agile approach of choice. Scrum is a great project management framework. It’s designed for a 5-7 person collocated team.

If you want to scale Scrum, especially for small programs, many people use a technique called Scrum-of-Scrums.

Scrum of Scrums is a Hierarchy

I have nothing against Scrum-of-Scrums. In my opinion, it’s a lot of overhead for not a lot of return, but it works for a lot of people. And, more importantly, it’s a lot more effective than what they were doing. You know me. If it’s more effective and they are successful, rock on!

And, SoS is a hierarchy. The Scrum teams get together at their standups and then send their Scrum Masters to another standup where the Masters get together and discuss the cross-program risks on a daily basis. They then have to get back together with their teams. Yes, they do this every day.

So, the problems go up the chain and down the chain. Now, in a small program, such as three teams, especially if the teams are small and the people are trained well in agile, and they are accustomed to working in small features that are written end-to-end, the people solve problems themselves. Alice doesn’t have a problem walking over to someone and saying, “Hey Bob, I have a problem. Can you take a look at this and tell me what’s wrong?”

But, what I see, even in small programs, is that the teams are not collocated. Alice is not located in the same timezone as Bob. Alice needs her Scrum Master to coordinate with Bob’s Scrum Master. And, because it’s the job of the Scrum Masters to coordinate, it’s not anyone else’s job to coordinate.

Let me repeat that. Because it’s the Scrum Master’s job to facilitate and maintain the coordination job, it’s no one else’s job to do so. Why? Because everyone else is busy getting their features to done. It’s human nature. We push problems up a hierarchy for someone else to remove if that’s how it’s supposed to be.

This is not Scrum’s fault. It is a human thing. It’s the way we are made.

Add Communities of Practice

The Scrum Masters cannot and should not track the details of everything for the testing and the architecture and the UX and the databases and and and… That will depend on your program. You may well need people on each team connected with communities of practice. And, you need someone from each team connected with the program team. So, you have more of a messy picture. But, who cares if you have a messy picture if you have a more effective program?

This picture has a community of practice built in. You can do this with S-o-S. You don’t need permission.

Now, with a three-team program of small teams of only 5 people, you might not need it. But, if you have 10-person teams, or if you are geographically distributed, you might.

Ask yourself these questions: are we getting to done every iteration on every single feature on every single team? Do we have interdependency problems? And, if you are larger than three teams, you almost certainly need a different structure. If you cannot answer yes to these questions, or if you are having trouble with your features flowing through your program, you need a different structure.

Communities of practice help.

Lean Works, Too

For those of you wondering, yes, you can always move to a lean approach. This works for any size team. I’m going to talk more about this for medium and large teams, because lean really shines there. So hang in there for later.

This discussion is about communication structures between teams, not the process or lifecycle the individual teams should use in a program. As a program manager, I don’t care what the teams use as long as they meet their commitments to the program. I hope you are not surprised about that. More about that, later.

Use a Network Instead of Hierarchy

Instead of a hierarchy, you have a choice of using a network, even if you are using Scrum. Even if you just have a three-team program.

When you have a network of teams, you don’t have to have everyone interconnected with everyone else. And, you don’t just need the Scrum Masters connected to each other. You need teams tightly connected to themselves. And, you need teams with loose connections to each other. This is called a small world network. (Do not sing the Disney song. No, I told you not to!)

If you’ve heard of the Six degrees of separation from Kevin Bacon thing, this is how it works. You don’t need all of the teams connected to each other, as long as all of them are connected to some of them.

For a three-team program, all of the teams would be connected. That’s easy. Since, in agile, we want to encourage people to collaborate, the small world network pattern is a reasonable pattern to use.

With a small world network, if Bob and Alice have a question, they ask each other. It’s their obligation to do so. If they don’t know that they should ask each other, it’s their obligation to ask someone who might know. That someone is not necessarily a Scrum Master. Or an agile project manager. Or a program manager. It’s someone in their network. The question doesn’t go up the hierarchy. It goes across the network.

This is not anarchy. It’s collaboration. Here’s a quote from Clay Shirky from Here Comes Everybody: The Power of Organizing with Organizations:

Collaborative production, where people have to coordinate with one another to get anything done, is considerably harder than simple sharing, but the results can be more profound. New tools allow large groups to collaborate, by taking advantage of nonfinancial motivations and by allowing for wildly differing levels of contribution.

I’ve drawn my picture using five teams, because while the small world network is useful for small programs, you can really see how it starts to shine for medium programs. For a small program, use the simplest thing that works. (Do that anyway.) The real question is this: Does what you are doing scale, as your program grows?

For my clients, the answer has been no. In fact, for my clients, Scrum of Scrums has not even worked for three teams. Why? Because they have not gotten training or coaching. They have tried to learn Scrum out of books. They have sent one person to one class and attempted to learn it that way. This is not a successful way to learn anything about agile.

When my clients drop the Scrum of Scrums approach and revert to networks, which is how their work got done in their organization before, they get their work done. Don’t ask me why they thought all the questions had to be funneled through the Scrum Master. I have no idea. I do know that the network approach works for them. And, it’s a lot less overhead for the poor Scrum Master/Agile Project Manager

Why Does a Network Approach Work?

The small network pattern works because it puts the inherent rumor mill to work for you. The small world network engages people in a way that hierarchy does not. And, it decreases the transaction cost of just about everything. That makes a huge difference. You don’t have to wait for any standups to address problems, issues, or risks. People on teams solve problems when they have the problem. No need for a “master” or a “chief” to intervene. You know how much I dislike the chief titles business.

In the next parts, I’ll talk more about how networks help, especially for medium and large programs and how they help features move through programs.

Thanks for hanging in there with me. I tried to make this short, but I am not a good enough writer to write this short. Please, ask me questions.

Here is a list of software development related conferences and events that will take place in the coming weeks and that have media partnerships with the Methods & Tools software development magazine: SPTechCon, March 3-6 2013, San Francisco, USA EclipseCon 2013, March 25-28, 2013, Boston, USA STARCANADA, April 8–12 2013, Toronto, Canada Big Data TechCon, April 8-10 2013, Boston, USA GOTO Zurich, April 8-12, 2013, Zurich, Switzerland GOTO Zurich for Leaders, April 8-12, 2013, Zurich, Switzerland JAX, April 22-26 2013, Mainz, Germany ICSE 2013, May 18-26 2013, San Francisco, USA Android DevCon Spring, May 28-31 2013, Boston, USA International PHP ...

 

Caterpillars are amazing. They begin life as an egg and end life as butterfly with two steps in-between.  Their life cycle helps them blend into the environment. As change agents we need the ability to change to meet the needs of organizations. However, metamorphosing doesn’t mean blowing in the wind but rather the ability to change building on the abilities, skills and capabilities to match the needs of the environment.

A butterfly begins as an egg then changes to a caterpillar until it spins its cocoon and becomes a pupa in preparation for unfurling its wings as an adult butterfly. Don’t get stuck in any stage or risk loosing relevance.

The Software Process and Measurement Cast 226 features my interview with John Hunter.  We talked about his new book Management Matters: Building Enterprise Capability.  A great interview with someone who has an amazing perspective on the world!  Jo Ann Sweeney also stopped in to talk about the Worth Working Summit  . . . I was one of the speakers and highly recommend checking it out!

Listen: Audio Podcast

It seems just yesterday I was trying to push forward the idea of developing software in an Agile way, but somehow now it seems like that battle is over.

As if we won without a fight.

When I look around now, I don’t see software development shops doing big upfront design.  I don’t see consultants knocking down doors to certify you as a Scrum Master.

It seems that we have now entered a phase where Agile is accepted as the default and now instead of everyone trying to pitch the idea of Agile, everyone is trying to fix their broken Agile implementations.

The funny thing is, still no one even knows what Agile is

The big problem with Agile from the beginning has always been trying to define it.

Pretty early on, this problem was solved by calling it Scrum.  Scrum was something that was easily definable, and something you could get certified in.

Scrum was a set of rules you follow that makes you Agile.

At least that is how it was pitched too often.

I predicted that Scrum would die, and I am pretty ready to call that prediction as correct.

Sure, there are plenty of development shops still using Scrum today, but it isn’t really growing and less and less organizations are following it strictly.  (I can’t back this up, just my feel.)

I am a pretty firm believer in most of the value of Scrum being that it contains a firm set of rules that doesn’t require debate or judgment calls.  If most organizations are just taking the idea of a 2 week sprint and having daily scrum meetings, they are not likely getting much of the value out of Scrum.

But, the problem is that Scrum itself was never Agile.  Scrum was a defined set of process, that if you followed, would give you the framework you needed to actually be Agile.

To me Agile has always meant stopping the BS about software development.

To me Agile meant stop building this plan that you know is going to fail and rules lawyering your customers to death to get them to pay for something they didn’t want, because that is what they agreed to.

To me Agile meant to instead try to develop software as honestly as possible.  To go in and find out exactly what the customer wanted at the moment, try to build that thing and as openly as possible and as quickly as possible get further feedback to refine and improve.  To focus on doing a good job and know that if you are doing that everything else will fall into place.  To me that is what Agile has always been.

So when I say where is Agile now, I am probably asking a different question than most people

I have to ask myself: are software development shops doing what I define as Agile?  Has that idea permeated the software development community as a whole?

I don’t think so, but I don’t think it has died, nor will it ever.

But, I have seen some things that make me hopeful.

I’ve seen a large amount of talk about MVP, Minimum Viable Product.

I’ve seen many start-ups launching MVPs and being successful doing so.  And I’ve seen awesome companies like Buffer using this idea to build a product that is exactly what I want, because their plan is completely based on the customer and it adapts to the customer.

Why am I saying all this?

Simple, I think that what the world thought was Agile was two things:

1. Scrum
2. Iterative development

For the most part the software development world has ditched Scrum, at least the only form of useful Scrum, which is strict by-the-book Scrum, and adopted Scrum meetings and iterative development.  Honestly, I could do without the Scrum meetings, because although they are a good idea, no one actually does them correctly.

So, in essence, we won the wrong battle and we did so with major concessions.  But, that is ok, because what the consultants packaged up, certified people in and sold as Agile, wasn’t really Agile at all. 

Instead the real Agile movement has been gaining traction and it isn’t being sold by consultants, it is being championed by small start-up companies that are producing products that are 100 times more weighted on the side of results in the results to employees ratio than traditional software development shops and they are calling it MVP.

This is where the true spirit and ideas of Agile live and thrive and as more and more of these companies become successful and more and more researchers dissect their results, they are going to find that these small software boutiques were the ones who were actually practicing Agile, because they were cutting through all the BS of software development and focusing and developing and building exactly what the customer wanted—tools, process, contracts, plans be damned!

Follow @jsonmez !function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0],p=/^http:/.test(d.location)?'http':'https';if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src=p+'://platform.twitter.com/widgets.js';fjs.parentNode.insertBefore(js,fjs);}}(document, 'script', 'twitter-wjs');

The post Where Is Agile Now? appeared first on Making the Complex Simple.

It seems just yesterday I was trying to push forward the idea of developing software in an Agile way, but somehow now it seems like that battle is over.

As if we won without a fight.

When I look around now, I don’t see software development shops doing big upfront design.  I don’t see consultants knocking down doors to certify you as a Scrum Master.

It seems that we have now entered a phase where Agile is accepted as the default and now instead of everyone trying to pitch the idea of Agile, everyone is trying to fix their broken Agile implementations.

The funny thing is, still no one even knows what Agile is

The big problem with Agile from the beginning has always been trying to define it.

Pretty early on, this problem was solved by calling it Scrum.  Scrum was something that was easily definable, and something you could get certified in.

Scrum was a set of rules you follow that makes you Agile.

At least that is how it was pitched too often.

I predicted that Scrum would die, and I am pretty ready to call that prediction as correct.

Sure, there are plenty of development shops still using Scrum today, but it isn’t really growing and less and less organizations are following it strictly.  (I can’t back this up, just my feel.)

I am a pretty firm believer in most of the value of Scrum being that it contains a firm set of rules that doesn’t require debate or judgment calls.  If most organizations are just taking the idea of a 2 week sprint and having daily scrum meetings, they are not likely getting much of the value out of Scrum.

But, the problem is that Scrum itself was never Agile.  Scrum was a defined set of process, that if you followed, would give you the framework you needed to actually be Agile.

To me Agile has always meant stopping the BS about software development.

To me Agile meant stop building this plan that you know is going to fail and rules lawyering your customers to death to get them to pay for something they didn’t want, because that is what they agreed to.

To me Agile meant to instead try to develop software as honestly as possible.  To go in and find out exactly what the customer wanted at the moment, try to build that thing and as openly as possible and as quickly as possible get further feedback to refine and improve.  To focus on doing a good job and know that if you are doing that everything else will fall into place.  To me that is what Agile has always been.

So when I say where is Agile now, I am probably asking a different question than most people

I have to ask myself: are software development shops doing what I define as Agile?  Has that idea permeated the software development community as a whole?

I don’t think so, but I don’t think it has died, nor will it ever.

But, I have seen some things that make me hopeful.

I’ve seen a large amount of talk about MVP, Minimum Viable Product.

I’ve seen many start-ups launching MVPs and being successful doing so.  And I’ve seen awesome companies like Buffer using this idea to build a product that is exactly what I want, because their plan is completely based on the customer and it adapts to the customer.

Why am I saying all this?

Simple, I think that what the world thought was Agile was two things:

1. Scrum
2. Iterative development

For the most part the software development world has ditched Scrum, at least the only form of useful Scrum, which is strict by-the-book Scrum, and adopted Scrum meetings and iterative development.  Honestly, I could do without the Scrum meetings, because although they are a good idea, no one actually does them correctly.

So, in essence, we won the wrong battle and we did so with major concessions.  But, that is ok, because what the consultants packaged up, certified people in and sold as Agile, wasn’t really Agile at all. 

Instead the real Agile movement has been gaining traction and it isn’t being sold by consultants, it is being championed by small start-up companies that are producing products that are 100 times more weighted on the side of results in the results to employees ratio than traditional software development shops and they are calling it MVP.

This is where the true spirit and ideas of Agile live and thrive and as more and more of these companies become successful and more and more researchers dissect their results, they are going to find that these small software boutiques were the ones who were actually practicing Agile, because they were cutting through all the BS of software development and focusing and developing and building exactly what the customer wanted—tools, process, contracts, plans be damned!

There is another round of discussion of the Chaos aspects of software development projects and the models of this chaos.

• The chaotic ones that are totally unpredictable like a double pendulum
• A double pendulum (two pendulums attached to each other) is also a simple system. It is easy to make and easy to understand. And yet, it undergoes unpredictable chaotic motion due to a high sensitivity to the initial setup of the pendulum.
  

The common paradigm of the Double Pendulum is misunderstood and misapplied to the problems of modeling systems.

A double pendulum, is one pendulum suspended from another, shown below. The path of the bottom element (m2) is a potentially chaotic system. When certain parameter ranges undergo a slight change in one of the initial starting conditions there can be dramatic effect on the subsequent motion of the pendulum. As a result the motion of a double pendulum extremely difficult to predict without a model of the system. The external observation is the pendulum exhibits seemingly random or chaotic behavior.

The first step in solving for the motion of m2 is to understand the difference between chaotic motion and unpredictable motion. The double pendulum can behave in a chaotic manner. But the motion of m2 as well as m2 is predictable. These terms get mixed up and we need to keep them straight if we're going to apply them to paradigms outside of the double pendulum. 

Chaos can described as:

• Being sensitive to initial conditions. 
• Possessing topological mixing, meaning the system evolves over time. 
• Possessing a density of periodic orbits, meaning every point in the space is approached arbitrarily closely by periodic orbits.

Why Is This Important for Project Management

When we use analogies in project management to explain a situation, we need to actually have the right analogy. The Double Pendulum is a common explanation of chaotic behaviour. But the Double Pendulum, while behaving chaotically, is a deterministic system driven by the solution to the Hamiltonian equations of motion. This is called Hamiltonian Chaos. 

The Hamiltonian description of motion uses momentum as the variable.

Depending on the initial conditions the system behaves regularly or show chaotic behaviour. The solution to this equation can be found in Java at Physics Department of the University of Buffalo.

This solution plots the path of the double pendulum in its normal mode or in its chaotic mode. Solutions in Mathematica can be found in Matematica for Theoretical Physics: Classical Mechanics and Nonlinear Dynamics.

So we can see that the solution to the double pendulum is not unknown. The plot of the path can be known by running the simulation for the needed period of time. 

What this means is using the Double Pendulum as an analogy for chaotic behaviour of projects requires careful wording and application. Human systems don't not have Hamiltonian's, double pendulums do. With Hamiltonian, we can write a simualtion to show the motion of the pendulum, run it and see whatit does - Exactly what it does. We can run the simulation forward in time and Predict what it will do sometime in the future.

No mysteries about the path of the pendulum parts, they are completly specified by the solutions to the Hamiltonian. They may look chaotic, unpredictable, and wandering all over the place. But the equations of motion are explictly defined in the Hamiltonian solution and its simulaiton in our favorite language, Java or Mathematica.