Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Process Management

SPaMCAST 335 ‚Äď Critical Agile Definitions, Communication Content, Microservices and Granularity

Software Process and Measurement Cast - Sun, 03/29/2015 - 22:00

In this episode of the Software Process and Measurement Cast we feature three columns!  The first is our essay on the definitions of four critical words.  What do the words effectiveness, efficiency, frameworks and methodologies really mean?  These words get used ALL the time, however they really do have fairly specific meanings.  Meanings that, once understood and used to guide how we work, can help everyone to deliver more value and make our customers more satisfied!  The second column is from Jo Ann Sweeney with another of her stellar, Explaining Change columns.  In this segment, Jo Ann talks about content and a framework to guide the development of content.  Anchoring the Cast this week is Gene Hughson with another of his Forms Follows Function columns.  Gene extends his mini-series on microservices with a discussion of whether granularity is irrelevant.  Lots of content in this installment of the Software Process and Measurement Cast!

Call to action!

Reviews of the Podcast help to attract new listeners.  Can you write a review of the Software Process and Measurement Cast and post it on the podcatcher of your choice?  Whether you listen on ITunes or any other podcatcher, a review will help to grow the podcast!  Thank you in advance!

Re-Read Saturday News

The Re-Read Saturday focus on Eliyahu M. Goldratt and Jeff Cox’s The Goal: A Process of Ongoing Improvement began on February 21nd. The Goal has been hugely influential because it introduced the Theory of Constraints, which is central to lean thinking. The book is written as a business novel. Visit the Software Process and Measurement Blog and catch up on the re-read.

Note: If you don’t have a copy of the book, buy one.  If you use the link below it will support the Software Process and Measurement blog and podcast.

Dead Tree Version or Kindle Version 

I am beginning to think of which book will be next. Do you have any ideas?

Upcoming Events

Categories: Process Management

Re-Read Saturday: The Goal: A Process of Ongoing Improvement. Part 6

IMG_1249

The Goal: A Process of Ongoing Improvement, published in 1984, is a business novel. The Goal uses the story of Alex Rogo, plant manager, to illustrate the theory of constraints and how the wrong measurement focus can harm an organization. The focus of the re-read is less on the story, but rather on the ideas the book explores that have shaped lean thinking. Earlier entries in this re-read are:

Part 1                         Part 2                         Part 3                         Part 4                      Part 5

This week I attended the CMMI EMEA conference.  Kirk Botula, CEO of the CMMI institute delivered a great speech about developing and managing capability.  Capability is ability to do something.  Without capabilities, individuals, teams and organizations are powerless. Tools like the CMMI, Agile and lean provide frameworks to decide which capabilities are important and then to measure the level and impact of current capabilities. During Kirk’s talk he referenced two books.  The first was Out of the Crisis by Edwards Deming and The Goal: A Process of Ongoing Improvement by Jeff Cox and Eliyahu Goldratt.  Deming’s work defined the continuous process improvement movement, while Goldratt framed the lean movement.  Later during the week I asked a group of C-level executives if they were familiar with The Goal and the Theory of Constraints.  Unfortunately the answer was not a resounding yes.  The limited awareness is problematic because that leads to a focus on cost containment and efficiency as the most important goals over effectiveness and throughput. Over focusing on cost and efficiency leads sub-optimal performance and in the long run market failure. In this installment of Re-read Saturday we begin to encounter the nuts and bolts that make the Theory of Constraints an effective tool for structured process improvement.

Chapter 17

After dealing with the complications of a household in turmoil (Alex’s household is a reflection of a series of dependent events and statistical variability), Alex arrives at the plant to find an order for an important client is late. Alex explains how dependent events and statistical variability will lead to the sub-optimal performance of a system. Despite Alex’s revelations from observing the scout troop, his team¬†believes they can deliver the parts in time to make the last truck at 5 PM. Alex places a¬†ten dollar bet that the parts will not be ready ship that day due to variations in the production process. ¬†The production of the parts requires two steps. The first step in the production process averages 25 parts per hour. In this case the production started slowly, however by the end of four hours 100 parts had been completed. Therefore the process had averaged 25 parts per hour. The production ranged from 19 parts to 32 parts. While the second step in the process had the same 25 parts per hour capacity, because of the variability of the first processes the capacity of the second process was exceeded on two occasions, therefore all 100 parts were not completed.

Chapter 18

Alex and his now attentive staff (the performance on late order has gotten their attention) attempt to determine how to control the flow of work thought the steps in the process (dependent events) as they are subject to statistical variability.They consider longer lead-time as a method to smooth the flow, because longer lead times would generate higher inventories to reduce efficiency. As a group they elect to call Johan, who introduces another new concept: bottleneck and non-bottleneck resources. A bottleneck resource is any resource that has a capacity that is equal to or less than the demand placed upon it. Any variability in the flow of work that is beyond the capacity of the bottlenecked resources will generate a backlog. While The Goal was written from the point of view of a manufacturing plant, bottlenecks can occur in any type of project. Any resource that is planned to 100% capacity is a bottlenecked resource. Through discussion with Johan the team discovers that like the scout troop the capacity of the plant is governed by the step with the least capacity. In the plant, the team needs to discover the bottlenecks that are blocking their ability to deliver effectively.

Bottlenecks are neither good or bad in their own right. A process with infinite capacity at all steps will not be efficient because resources would be under utilized. Bottlenecks can be used to generate a predictable process. This a reflection of how Alex used the slow scout in Chapter 15. Also understanding the capacity and variability of the bottleneck step will help make the process predictable.

When the staff discovers the bottleneck, it generates a process of discovery in which the team discovers that they really don’t know the process well and that the time accounting data is less than perfect. The big revelation is that looking for steps in the process with build-ups of work-in-process waiting to be addressed are an indication of a bottleneck. In this case the team finds two culprits. The first is the industrial robot that is darling of senior management and the second is the heat treatment step. The bottlenecks cannot be moved to the head of the column like Alex did to solve the problem with the boy scouts, nor can extra capacity be generated by increasing staff or buying machines. The workday ends with Alex searching for a solution.

The affect of the combination dependent events and statistical variability on the plant floor is obvious. In a manufacturing plant where processes are heavily automated variability might be low, however it still exists. In software or product development same interaction of process steps and variability exits although because the work tends to be more discovery-oriented, the variability is probably higher making the impact more substantial and less predictable. The variability in flow through the process exposes bottlenecks that limit our ability to catch up, making projects and products late or worse generating technical debt when corners are cut in order to make the date or budget.

Summary of The Goal so far:

Chapters 1 through 3 actively present the reader with a burning platform. The plant and division are failing. Alex Rogo has actively pursued increased efficiency and automation to generate cost reductions, however performance is falling even further behind and fear has become central feature in the corporate culture.

Chapters 4¬†through¬†6¬†shift the focus from steps in the process to the process as a whole. Chapters 4 ‚Äď 6 move us down the path of identifying the ultimate goal of the organization (in this book). The goal is making money and embracing the big picture of systems thinking. In this section, the authors point out that we are often caught up with pursuing interim goals, such as quality, efficiency or even employment, to the exclusion of the of the ultimate goal. We are reminded by the burning platform identified in the first few pages of the book, the impending closure of the plant and perhaps the division, which in the long run an organization must make progress towards their ultimate goal, or they won‚Äôt exist.

Chapters 7 through 9 show Alex‚Äôs commitment to change, seeks more precise advice from Johan, brings his closest reports into the discussion and begins a dialog with his wife (remember this is a novel). In this section of the book the concept ‚Äúthat you get what you measure‚ÄĚ is addressed. In this section of the book, we see measures of efficiency being used at the level of part production, but not at the level of whole orders or even sales. We discover the corollary to the adage ‚Äėyou get what you measure‚Äô is that if you measure the wrong thing ‚Ķyou get the wrong thing. We begin to see Alex‚Äôs urgency and commitment to make a change.

Chapters 10 through 12 mark a turning point in the book. Alex has embraced a more systems view of the plant and that the measures that have been used to date are more focused on optimizing parts of the process to the detriment to overall goal of the plant.  What has not fallen into place is how to take that new knowledge and change how the plant works. The introduction of the concepts of dependent events and statistical variation begin the shift the conceptual understanding of what measure towards how the management team can actually use that information.

Chapters 13 through 15 drive home the point that dependent events and statistical variation impact the performance of the overall system. In order for the overall process to be more effective you have to understand the capability and capacity of each step and then take a systems view. These chapters establish the concepts of bottlenecks and constraints without directly naming them and that focusing on local optimums causes more trouble than benefit.

Note: If you don’t have a copy of the book, buy one.  If you use the link below it will support the Software Process and Measurement blog and podcast. Dead Tree Version or Kindle Version


Categories: Process Management

Internet of things

Xebia Blog - Sat, 03/28/2015 - 11:26

Do you remember the time that you were not connected to the internet? When you had to call your friend instead of sending a message via Whatsapp to make an appointment. The times you had to apply for a job by writing a letter with  a pen instead of sending your Linkedin page? We all have gone through a major revolution in which we've all got a digital identity. The same revolution has started for things, right now!

Internet of Things

Potential

Things we use all day, like your watch or thermostat, get suddenly connected to the internet. These things are equipped with electronics, sensors and connectivity. Whether you like it or not, you will not escape this movement. According to Gartner there will be nearly 26 billion devices on the Internet of Things by 2020.
The potential of the Internet of Things isn’t just linking millions of devices. It is more like the potential of Google or Amazon had when building massive IT infrastructure. The IoT is about transforming business models and enabling companies to sell products in entirely new and better ways.

Definition

The definition of IoT is fairly simple. The internet of things involves `every thing that is connected to the internet`. You should exclude mobiles and computers, because they've already become mainstream.

In fact, the Internet of things is a name for a set of trends. First of all we have the trend that objects, we use all day,  are suddenly connected to the internet. For example bathroom scales that compares your weight with others.
Another trend is that devices that were already shipped with CPUs have become much more powerful when they’re connected to a network. Functionality of the hardware depends on local computing as well as powerful cloud computing. Fitness tracker, can carry out some functionality, like basic data cleaning and tabulation, on local hardware, but it depends on sophisticated machine-learning algorithms on an ever improving cloud service to offer prescription and insight.
Besides that we have the makers movement. Creating hardware isn’t for the big companies like Sony or Samsung anymore, just like creating software in the 80’s and 90’s was a privilege for IBM and Microsoft.

What’s new?

Devices that are connected to the internet aren’t new. For many years, we've connected devices to the internet, although not at that size. I think we have to compare it concepts like 'the cloud'. Before I heard the name 'the cloud' I was already using services that run on the internet, like Hotmail. But one swallow doesn't make a summer. When suddenly lots of services are running on the web, we speak of a trend and the market will give it a name to it. The same is happening with the internet of things. Suddenly all kinds of 'normal' devices are connected to the internet and we have a movement towards that direction that is unstoppable. And the possibilities it will bring are endless.

Splitting up in pieces

We can divide the internet of things into five smart categories. First of all we have the smart wearable. It is designed for a variety of purposes as well as for wear on a variety of parts of the body. Think of a bicycle helmet that detects a crash. Next to it we have the smart home. It’s goal is to make the experience of living at home more convenient and pleasant. A thermostat that learns what temperatures users like and builds a context-aware personalized schedule. A third and fourth category are the smart city and smart environment. It focuses on finding sustainable solutions to the growing problems. The last category is very interesting. It is the smart enterprise. One can think of real-time shipment tracking or smart metering solution that manages energy consumption at the individual appliance and machine level.

Changes of business models

There are plenty of new business models due to the internet of things. An example is a system that provides a sensor-embedded trash, so it’s capable of real-time context analysis and alerting the authorities when it is full and needs to be emptied. Another example is the insurance world, which will adopt the internet of things. There's already behaviour-based car insurance pricing available. Customers who agree to install a monitoring device on their car's diagnostic port, get a break on their insurance rates and pricing that varies according to usage and driving habits.

Big data

Other disciplines in software development also have interest in the new movement, like big data. Internet of things will not generate big data, but obesitas data. Did you know that within every flight, there are  3 terabytes of data generated with all the sensors in an airplane? Think about how much data will be generated, that has to be analyzed, when all buildings in a town are equipped with sensors to measure the quality of the air?

Conclusion

The intersection between software and the physical world becomes smaller every day and that is the result of the unstoppable movement of internet of things. It is still at the beginning. Business models are changing and there are lots of opportunities. You just have to find them...

Personas: Nuts and Bolts Without Navel Gazing

Two completely different personas using the same system simultaneously.

Two completely different personas using the same system simultaneously.

Personas are often used in software development to generate and group requirements. For example, one of the classic constructions of user stories is: <Persona> <Goal> <Benefit>. The use of personas has become nearly ubiquitous. A persona represents one of the archetypical users that interacts with the system or product. Every system or product will have at least one archetypical user that can be identified. The concept of personas was originally developed by Alan Cooper in his book The Inmates Are Running the Asylum. Personas represent different behaviors based on research into the jobs, lifestyles, spare time activities, attitudes, motivations, behaviors and social roles. I am often asked where a team should get a set of personas given the level of usage (as if they were a set of playing cards), and once a team gathers the data, how that data should be captured.

In most internal development enhancement and maintenance projects, personas are developed using some form of brainstorming process. Brainstorming with affinity diagraming is a great tool to develop personas. The Software Process and Measurement Blog has described the process of Affinity diagraming. The people involved in generating personas are critical. The team you use to generate personas needs to have a broad experiential base. It needs to be cross-functional and diverse. The team needs to be able to take systems views of who uses the systems or product, otherwise the list of personas and their definitions will be incomplete. Second, when generating ideas in the brainstorming session someone needs to act as a facilitator and provide seed questions to the team. The seed questions will help the team to uncover groups of users and potential users, their needs, behaviors, attitudes and both work and non-work activities. In the grouping portion of the affinity diagraming process, the team should first think about personas when grouping and then when naming the groups ask the team to give the groups persona names. A further twist on the tried and true affinity session mechanism that I have recently found to be effective is to have the team break up in to smaller sub-teams to review what was done and re-brainstorm entries that could fill the gaps after the grouping and naming exercise. I generally time box the second step to fifteen minutes, although time-boxing may constrain innovation. When the sub-teams return integrate the new entries into the affinity diagram. The named and grouped entries from the affinity diagram will form the basis for the documentation needed to capture the personas.

Documenting personas might be the most contentious part of the process of establishing and using personas. Documentation smacks of overhead to many in the development community. I have found that if you don’t take the time to document a persona, it is far less useful. Documenting personas does a number of things. First the act of documentation allows the team to flesh out their ideas and drive out gaps and inconsistencies. Second, documentation helps the team to establish collective memory. Third, fleshing out the nuances fleshed out and establishing institutional memory makes it more likely that the team will refer to the persona after development because the investment in time and effort generates a perception of value. When documenting personas, ensure that you keep the documentation to the absolute minimum. I use a single sheet of flip chart paper per persona; not only does the single sheet feel lean, it also allows teams to post the personas around the room. Templates are good to for maintaining repeatability.

In Agile Story Maps: Personas we discussed and referenced a template defined by Roman Pitchler. There are a wide variety of templates for capturing information about a persona. In general they all include the following information:

  1. Persona Name
  2. Picture: a picture gives the persona visual substance and personality.
  3. Job: What does a person in this archetypical group do for a living (as it relates to the product or system).
  4. Goal: A definition of why the persona would want to use the product or system. Support of the goal can be found in the more personality and lifestyle (often written as backstory) of the persona.
  5. Personality: How does the group that the persona represents think, act, or react. This attribute includes motivation, attitudes, behaviors and social roles.
  6. Lifestyle: How does the group that the persona represents actually act? What does the persona do both at work and in their spare time?

Every template that I have seen (or created) is nuanced based on the needs of the team and the project. For example, the personas generated to guide developing requirements for an enhancement to a data entry application would be different from the personas needed to plan a major Agile conference. In the latter case more emphasis might be needed around motivation and lifestyle which would influence the speakers that might be accepted, social events planned and when the speakers and events might be slotted in the schedule. Teams use personas to help them think conceptually about the problem they are trying to solve without getting bogged down by having to think about and address each individual that uses the system or product as a special, individual case.

Personas are a tool help teams to understand a class of users by creating an archetypical user. Personas need to be identifiable and have back-story that includes needs and goals. The use of fictional people allows the team to dissociate themselves from the large number of individuals in the user base so they don’t get trapped in paralyzing detail. Personas need to be written down or you will not remember the nuances of personalities and lifestyles that make any two personas different.


Categories: Process Management

Quote of the Month March 2015

From the Editor of Methods & Tools - Wed, 03/25/2015 - 09:10
Competencies versus Roles: We’ve seen a positive move toward emphasizing competencies in a team rather than roles or titles. As teams make that change, we see fewer “It’s not my job” excuses and more “How can I help?” conversations. Team members will continue to have core competencies in some areas more than others, but they may not identify as strongly with a particular role. For example, saying, “I am a tester” really means, “I perform mainly testing activities because that is my primary passion and strength. I can provide leadership ...

CMMI: The Easy Way or The Hard Way? A Simple CMMI Readiness Checklist

Are you ready to build change?

Are you ready to build change?

Many times an organization or individual will start a change program because they deem it necessary for survival. But change is never easy. Survival and pain avoidance, while powerful, can lead to pursuing change as a reaction to pain rather than as pursuit of value. Pain avoidance and generation of business value both are necessary pieces of knowledge as the intellectual benefits persuade, while pain avoidance sells. In order to ensure that both sides of the change are addressed a framework can be useful to generate focus. The simple CMMI Readiness Checklist can be used for any major change initiative, but is tailored toward the testing whether the requirements for implementing a framework like the CMMI have been addressed.

I have broken this checklist into three categories: resources, plans and attitudes.  Each can be leveraged separately; however using the three components will help you to focus on the big picture.

Scale

The simple checklist can be used as a tool to evaluate how well you have prepared for you CMMI journey using the questions as evaluation criteria.  To use the checklist, evaluate each question on a scale of high, medium, low and not present (with one exception). Each question will potentially contribute points toward the total that can be used to evaluate preparation.

Section and Question Weights:

Resources: Forty-two total points. Each component contributes up to 7 points (7, 3, 1, 0).

Plans: Eighteen total points. Each component contributes up to 6 points (6, 3, 1, 0).

Attitude: Forty total points. Each component contributes up to 8 points (8, 4, 2, 0).

Resources

Resources are the raw materials that you will consume on your journey.  As with any journey having both the correct resources and correct amount of resources will make the journey easier.  Just think of trying to canoe from New York to London for a meeting; the wrong resources can make the trip difficult.

Management Support

Support from management is critical as we have discussed in past checklists, but so is support from your peers and from the teams that will be using the processes.

Score

7 ‚Äď Senior management is actively involved in guiding and using the outputs of the CMMI.¬† Senior managers stop people in the hall to discuss progress and recently process implementations. Discussion of progress is an agenda item at all managers staff meetings.

3 ‚Äď Senior and middle managers attend formal CMMI informational meetings and talk about the need to support the CMMI initiative.

1 ‚Äď Senior managers attended the kick-off meeting, then relocating in mass to Aruba, leaving the middle managers in charge.

0 ‚Äď The change initiative is a grass-roots effort.

Cash

Change costs money. Costs can include consultants, training, travel and an odd late-night pizza or two.

7 ‚Äď A reasonable budget has been established and the implementation team can draw from the budget for planned expenditures.¬† Emergency funding can be attained to handle issues.

3 ‚Äď A reasonable budget has been established and approved; however, access must be requested and justified for all expenditures.

1 ‚Äď Any time that money is required funding must be requested and approved.

0 ‚Äď Donations are sought in the organization‚Äôs lunchroom on a periodic basis (consider a PayPal donation button on your homepage).

Effort

Even if you have bales of cash, developing and implementing processes will require effort. Effort will be required from many constituencies including the process-improvement team, management and from the teams using the process, just to name a few.

7 ‚Äď A reasonable staffing plan has been established and the change program is the only project the assigned resources have been committed to.

4 ‚Äď A reasonable staffing plan has been established and the change initiative is the highest priority for the assigned resources.

1 ‚Äď All resources are shared between the¬†change initiative and are also assigned to other projects with high priority.

0 ‚Äď You have all the effort you need after 5 PM and before 8 AM and during company holidays.

Change Specialist

Organizational change requires skills that are not generally found in an IT department. The skills needed include sales, marketing and communication.

7 ‚Äď An organizational-change specialist has been assigned as a full-time resource for the project.

3 ‚Äď An organizational-change specialist is available within the organization and works on many projects simultaneously. The specialist may or may not have had experience with IT change programs.

1 ‚Äď Someone on the team has helped craft an organizational change plan in the past.

0 ‚Äď Organizational change websites are blocked and your best bet is buying a book on Amazon using your own cash.

Projects

Change requires something to impact.  The organization needs to have a consistent flow of projects so that changes are not one-shot attempts.

7 ‚Äď Projects are constantly beginning that will provide a platform for implementing process changes.

3 ‚Äď There are numerous projects in the organization; however they typically begin early in the year or on some other periodic basis that makes waiting a necessity if you are not ready exactly on time.

1 ‚Äď The organization does only a small number of projects every year.

0 ‚Äď The organization does one large project every year.

Calendar Time

Calendar time is a resource that is as important as any other resource. Severe calendar constraints can lead to irrational or bet-the-farm behaviors which increase risk.

7 ‚Äď The schedule for implementing the CMMI is in line with industry norms and includes time for tweaking the required processes before appraising.

3 ‚Äď The schedule is realistic but bare bones. Any problems could cause delay.

1 ‚Äď Expectations have been set that will require a compressed schedule; however, delay will only be career limiting rather than a critical impact on the business.

0 ‚Äď The CMMI implantation is critical for the organization‚Äôs survival and is required on an extremely compressed schedule.

Expertise

A deep understanding of the CMMI (or any other framework for that matter) will be needed to apply the model in a dynamic environment.¬† Experience is generally hard won. ‚ÄúDoing‚ÄĚ it once generally does not provide enough expertise to allow the level of tailoring needed to apply the model in more than one environment. Do not be afraid to get a mentor if this is a weakness.

7 ‚Äď The leaders and team members working to implement the CMMI have been intimately involved in successfully implementing the framework in different environments.

3 ‚ÄďThe leader and at least one of the team members have been involved in implementing the CMMI in the past in a similar environment.

1 ‚Äď Only the leader of the CMMI program has been involved with implementing the CMMI in another environment.

0 ‚Äď All of the team members have taken the basic CMMI course and can spell CMMI assuming they can buy a vowel.

Plans

Planning for the implementation of change can take many forms ‚ÄĒ from classic planning documents and schedules to backlogs.¬† The structure of the plan is less of a discussion point than the content.¬† You need several plans when changing an organization. While the term ‚Äúseveral‚ÄĚ is used this does not mandate many volumes of paper and schedules, rather that the activities required are thought through and recorded, the goal is known and the constraints on the program have been identified (in other words the who, what, when, why and how are known to the level required).

Scale and Scoring

Plans: Eighteen total points. Each component contributes up to 6 points (6, 3, 1, 0).

Organizational Change Plan

The Organizational Change Plan includes information on how the changes required to implement the CMMI will be communicated, marketed, reported, discussed, supported, trained and, if necessary escalated.

6 ‚Äď A full change management plan has been developed, implemented and is being constantly monitored.

3 ‚ÄďAn Organizational Change Plan is planned but is yet to be developed. .

1 ‚Äď When created, the Organizational Change Plan will be referenced occasionally.

0 ‚Äď No Organizational Change Plan has or will be created.

Backlog

The backlog records what needs to be changed in prioritized order. The backlog should include all changes, issues and risks. The items in the backlog will be broken down into tasks as they are selected to be worked on.  The format needs to match corporate culture and can range from an Agile backlog to in a waterfall organization, a requirements document.

6 ‚Äď A prioritized backlog exists and is constantly maintained.

3 ‚Äď A prioritized backlog exists and is periodically maintained.

1 ‚Äď A rough list of tasks and activities is kept on whiteboard.

0 ‚Äď No backlog or list of tasks exists.

Governance

Any change program requires resources, perseverance and political capital. In most corporations these types of requirements scream the need for oversight (governance is a code word for the less friendly word oversight). Governance defines who decides which changes will be made, when they will be made and who will pay for the changes. I strongly recommend that you decide how governance will be handled and write it down and make sure all of your stakeholders are comfortable on how you will get their advice, counsel, budget and in some cases permission.

6 ‚Äď A full-governance plan has been developed, implemented and is being constantly monitored.

3 ‚ÄďA governance plan is planned, but is yet to be developed.

1 ‚Äď When created, the governance plan will be used to show the process auditors.

0 ‚Äď Governance . . . who needs it!

Attitude

When you talk about attitude it seems personal rather than organizational, but when it comes to large changes I believe that both the attitude of the organization and critical individuals are important.  As you prepare to address the CMMI, the onus is on you as a change leader to develop a nuanced understanding of who you need to influence within the organization. The checklist will portray an organizational view; however, you can and should replicate the exercise for specific critical influencers.

Scale and Scoring

Attitude: Forty total points. Each component contributes up to 8 points (8, 4, 2, 0).

Vision of tomorrow

Is there a belief that tomorrow will be demonstratively better based on the actions that are being taken? The organization needs to have a clear vision that tomorrow will be better than today in order to positively motivate the team to aspire to be better than they are.

8 ‚Äď The organization is excited about the changes that are being implemented.¬† Volunteers to help or to pilot are numerous.

4 ‚Äď Most of the organization is excited about most of the changes and their impact on the future.

2 ‚Äď A neutral outlook (or at least undecided) is present.

0 ‚Äď Active disenchantment with or dissension about the future is present.

Minimalist

The view that the simplest process change that works is the best is important in today’s lean world.  In many cases heavy processes are wearing on everyone who uses them and even when the process is okay today, entropy will add steps and reviews over time, which adds unneeded weight.  Score this attribute higher if the organization has a process to continually apply lean principles as a step in process maintenance.

8 ‚Äď All processes are designed with lean principles formally applied.¬† Productivity and throughput are monitored to ensure that output isn‚Äôt negatively impacted.

4 ‚Äď All processes are designed with lean principles formally applied; however, they are not monitored quantitatively.

2 ‚Äď All processes are designed with lean principles informally applied.

0 ‚Äď Processes are graded by the number of steps required, with a higher number being better.

Learner

A learner is someone that is learning understands that they don‚Äôt know everything and that mistakes will be made. They understand that when made, mistakes are to be examined and corrected rather than swept under the carpet. Another attribute of a learner is the knowledge that the synthesis of data and knowledge from other sources is required for growth.¬† In most organizations, an important source of process knowledge and definition are the practitioners ‚ÄĒ but not the sole source.

8 ‚Äď New ideas are actively pursued and evaluated on an equal footing with any other idea or concept.

4 ‚Äď New ideas are actively pursued and evaluated, but those that reflect the way work is currently done are given more weight.

2 ‚Äď The ‚Äúnot invented here‚ÄĚ point of view has a bit of a hold on the organization, making the introduction of new ideas difficult.

0 ‚Äď There is only one way to do anything and it was invented here sometime early last century.¬† Introduction of new ideas is considered dangerous.

Goal Driven

The organization needs to have a real need to drive the change and must be used to pursuing longer-term goals. The Process Philosopher of Sherbrooke once told me that being goal-driven is required to be serious about change.  In many cases a good, focused near-death experience increases the probability of change, but waiting that long can create a negative atmosphere. A check-the-box goal rarely provides more than short-term localized motivation.

8 ‚Äď The organization has a well-stated positive goal and that the CMMI not only supports, but is integral to attaining that goal.

2 ‚Äď The pursuit of the CMMI is about checking a box on a RFP response.

0 ‚Äď CMMI is being pursued for no apparent purpose.

Conviction

Belief in the underlying concepts of the CMMI (or other change framework) provides motivation to the organization and individuals.¬† Conviction creates a scenario where constancy of purpose (Deming) is not an after-thought but the way things are done. Implementing frameworks like the CMMI are long-term efforts ‚ÄĒ generally with levels of excitement cycling through peaks and valleys.¬† In the valley when despair becomes a powerful force, many times conviction is the thread that keeps things moving forward. Without a critical mass of conviction it will be easy to wander off to focus on the next new idea.

8 ‚Äď We believe and have evidence that from the past that we can continue to believe over time.

4 ‚Äď We believe but this is the first time we‚Äôve attempted something this big!

2 ‚Äď We believe¬† . . . mostly.

0 ‚Äď No Organizational-Change Plan has been created.

Scoring

Sum all of the scores and apply the following criteria.

100 ‚Äď 80¬†¬† You have a great base; live the dream.

79 ‚Äď 60¬†¬† Focus on building your change infrastructure as you begin the CMMI journey.

59 ‚Äď 30¬†¬† Remediate your weaknesses before you start wrestling with the CMMI.

29 ‚Äst¬† 0¬†¬† Run Away! Trying to implement the CMMI will be equivalent to putting your hand in the garbage disposal with it running; avoid if you absolutely can!


Categories: Process Management

A High Available Docker Container Platform using CoreOS and Consul

Xebia Blog - Tue, 03/24/2015 - 11:35

Docker containers are hot, but containers in themselves are not very interesting. It needs an eco-system to make it into  24x7 production deployments. Just handing your container names to operations, does not cut it.

In the blog post, we will show you how  CoreOS can be used to provide a High Available Docker Container Platform as a Service, with a box standard way to deploy Docker containers. Consul is added to the mix to create a lightweight HTTP Router to any docker application offering a HTTP service.

We will be killing a few processes and machine on the way to prove our point...

Architecture

The basic architecture for our Docker Container Platform as a Service, consists of the following components

coreos-caas

  • CoreOS cluster
    The CoreOS cluster will provide us with a cluster of Highly Available Docker Hosts. CoreOS is an open source lightweight operating system based on the Linux kernel and provides an infrastructure for clustered deployments of applications. The interesting part of CoreOS is that you cannot install applications or packages on CoreOS itself. Any custom application has to be packed and deployed as a Docker container. At the same time CoreOS provides only basic functionality for managing these applications.
  • Etcd
    etcd is the CoreOS distributed key value store and provides a reliable mechanism to distribute data through the cluster.
  • Fleet
    Fleet is the cluster wide init system of CoreOS which allows you to schedule applications to run inside the Cluster and provides the much needed nanny system for you apps.
  • Consul
    Consul from Hashicorp is a tool that eases service discovery and configuration. Consul allows services to be discovered via DNS and HTTP and provides us with the ability to respond to changes in the service registration.
  • Registrator
    The Registrator from Gliderlabs will automatically register and deregister any Docker container as a service in Consul. The registrator runs on each Docker Host.
  • HttpRouter
    Will dynamically route HTTP traffic to any application providing a HTTP services, running anywhere in the cluster.  It listens on port 80.
  • Load Balancer
    An external load balancer which will route the HTTP traffic to any of the CoreOS node listening on port 80.
  • Apps
    These are the actual applications that may advertise HTTP services to be discovered and accessed. These will be provided by you.

 

Getting Started

In order to get your own container platform as a service running, we have created a Amazon AWS CloudFormation file which installs the basic services: Consul, Registrator, HttpRouter and the load balancer.

In the infrastructure we create two autoscaling groups: one for the Consul Servers which is limited to 3 to 5 machines and one from the Consul clients which is basically unlimited and depends on your need.

The nice thing about the autoscaling group is that it will automatically launch a new machine if the number of machines drops below the minimum or desired number.  This adds robustness to the platform.

The Amazon Elastic Load Balancer balances incoming traffic to any port machine in either autoscaling group.

We created a little script that creates your CoreOS cluster. This has prerequisite that you are running MacOS and have installed:

In addition, the CloudFormation file assumes that you have a Route53 HostedZone in which we can add Records for your domain. It may work on other Linux platforms, but that I did not test.

 

git clone https://github.com/mvanholsteijn/coreos-container-platform-as-a-service
cd coreos-container-platform-as-a-service
./bin/create-stack.sh -d cargonauts.dutchdevops.net

...
{
"StackId": "arn:aws:cloudformation:us-west-2:233211978703:stack/cargonautsdutchdevopsnet/b4c802f0-d1ff-11e4-9c9c-5088484a585d"
}
INFO: create in progress. sleeping 15 seconds...
INFO: create in progress. sleeping 15 seconds...
INFO: create in progress. sleeping 15 seconds...
INFO: create in progress. sleeping 15 seconds...
INFO: create in progress. sleeping 15 seconds...
INFO: create in progress. sleeping 15 seconds...
INFO: create in progress. sleeping 15 seconds...
INFO: create in progress. sleeping 15 seconds...
INFO: create in progress. sleeping 15 seconds...
INFO: create in progress. sleeping 15 seconds...
INFO: create in progress. sleeping 15 seconds...
CoreOSServerAutoScale 54.185.55.139 10.230.14.39
CoreOSServerAutoScaleConsulServer 54.185.125.143 10.230.14.83
CoreOSServerAutoScaleConsulServer 54.203.141.124 10.221.12.109
CoreOSServerAutoScaleConsulServer 54.71.7.35 10.237.157.117

Now you are ready to look around. Use one of the external IP addresses to setup a tunnel for fleetctl.

export FLEETCTL_TUNNEL=54.203.141.124

fleetctl is the command line utility that allows you to manage the units that you deploy on CoreOS.


fleetctl list-machines
....
MACHINE		IP		METADATA
1cdadb87...	10.230.14.83	consul_role=server,region=us-west-2
2dde0d31...	10.221.12.109	consul_role=server,region=us-west-2
7f1f2982...	10.230.14.39	consul_role=client,region=us-west-2
f7257c36...	10.237.157.117	consul_role=server,region=us-west-2

will list all the machines in the platform with their private IP addresses and roles. As you can see we have tagged 3 machines for the consul server role and 1 machine for the consul client role. To see all the docker containers that have started on the individual machines, you can run the following script:

for machine in $(fleetctl list-machines -fields=machine -no-legend -full) ; do
   fleetctl ssh $machine docker ps
done
...
CONTAINER ID        IMAGE                                  COMMAND                CREATED             STATUS              PORTS                                                                                                                                                                                                                                      NAMES
ccd08e8b672f        cargonauts/consul-http-router:latest   "/consul-template -c   6 minutes ago       Up 6 minutes        10.221.12.109:80->80/tcp                                                                                                                                                                                                                   consul-http-router
c36a901902ca        progrium/registrator:latest            "/bin/registrator co   7 minutes ago       Up 7 minutes                                                                                                                                                                                                                                                   registrator
fd69ac671f2a        progrium/consul:latest                 "/bin/start -server    7 minutes ago       Up 7 minutes        172.17.42.1:53->53/udp, 10.221.12.109:8300->8300/tcp, 10.221.12.109:8301->8301/tcp, 10.221.12.109:8301->8301/udp, 10.221.12.109:8302->8302/udp, 10.221.12.109:8302->8302/tcp, 10.221.12.109:8400->8400/tcp, 10.221.12.109:8500->8500/tcp   consul
....

To inspect the Consul console, you need to first setup a tunnel to port 8500 on a server node in the cluster:

ssh-add stacks/cargonautsdutchdevopsnet/cargonauts.pem
ssh -A -L 8500:10.230.14.83:8500 core@54.185.125.143
open http://localhost:8500

Consul Console

You will now see that there are two services registered: consul and the consul-http-router. Consul registers itself and the http router was detected and registered by the Registrator on 4 machines.

Deploying an application

Now we can to deploy an application and we have a wonderful app to do so: the paas-monitor. It is a simple web application which continuously gets the status of a backend service and shows who is responding in a table.

In order to deploy this application we have to create a fleet unit file. which is basically a systemd unit file. It describes all the commands that it needs to execute for managing the life cycle of a unit. The paas-monitor unit file looks like this:


[Unit]
Description=paas-monitor

[Service]
Restart=always
RestartSec=15
ExecStartPre=-/usr/bin/docker kill paas-monitor-%i
ExecStartPre=-/usr/bin/docker rm paas-monitor-%i
ExecStart=/usr/bin/docker run --rm --name paas-monitor-%i --env SERVICE_NAME=paas-monitor --env SERVICE_TAGS=http -P --dns 172.17.42.1 --dns-search=service.consul mvanholsteijn/paas-monitor
ExecStop=/usr/bin/docker stop paas-monitor-%i

It states that this unit should always be restarted, with a 15 second interval. Before it starts, it stops and removes the previous container (ignoring any errors) and when it starts, it runs a docker container - non-detached. This allows systemd to detect that the process has stopped. Finally there is also a stop command.

The file also contains %i: This is a template file which means that more instances of the unit can be started.

In the environment settings of the Docker container, hints for the Registrator are set. The environment variable SERVICE_NAME indicates the name under which it would like to be registered in Consul and the SERVICE_TAGS indicates which tags should be attached to the service. These tags allow you to select the  'http' services in a domain or even from a single container.

If the container would expose more ports for instance 8080 en 8081 for  http and administrative traffic, you cloud specify environment variables.

SERVICE_8080_NAME=paas-monitor
SERVICE_8080_TAGS=http
SERVICE_8081_NAME=paas-monitor=admin
SERVICE_8081_TAGS=admin-http

Deploying the file goes in two stages: submitting the template file and starting an instance:

cd fleet-units/paas-monitor
fleetctl submit paas-monitor@.service
fleetctl start paas-monitor@1

Unit paas-monitor@1.service launched on 1cdadb87.../10.230.14.83

Now the fleet report that it is launched, but that does not mean it is running. In the background Docker has to pull the image which takes a while. You can monitor the progress using fleetctl status.

fleetctl status paas-monitor@1

paas-monitor@1.service - paas-monitor
   Loaded: loaded (/run/fleet/units/paas-monitor@1.service; linked-runtime; vendor preset: disabled)
   Active: active (running) since Tue 2015-03-24 09:01:10 UTC; 2min 48s ago
  Process: 3537 ExecStartPre=/usr/bin/docker rm paas-monitor-%i (code=exited, status=1/FAILURE)
  Process: 3529 ExecStartPre=/usr/bin/docker kill paas-monitor-%i (code=exited, status=1/FAILURE)
 Main PID: 3550 (docker)
   CGroup: /system.slice/system-paas\x2dmonitor.slice/paas-monitor@1.service
           ‚ĒĒ‚ĒÄ3550 /usr/bin/docker run --rm --name paas-monitor-1 --env SERVICE_NAME=paas-monitor --env SERVICE_TAGS=http -P --dns 172.17.42.1 --dns-search=service.consul mvanholsteijn/paas-monitor

Mar 24 09:02:41 ip-10-230-14-83.us-west-2.compute.internal docker[3550]: 85071eb722b3: Pulling fs layer
Mar 24 09:02:43 ip-10-230-14-83.us-west-2.compute.internal docker[3550]: 85071eb722b3: Download complete
Mar 24 09:02:43 ip-10-230-14-83.us-west-2.compute.internal docker[3550]: 53a248434a87: Pulling metadata
Mar 24 09:02:44 ip-10-230-14-83.us-west-2.compute.internal docker[3550]: 53a248434a87: Pulling fs layer
Mar 24 09:02:46 ip-10-230-14-83.us-west-2.compute.internal docker[3550]: 53a248434a87: Download complete
Mar 24 09:02:46 ip-10-230-14-83.us-west-2.compute.internal docker[3550]: b0c42e8f4ac9: Pulling metadata
Mar 24 09:02:47 ip-10-230-14-83.us-west-2.compute.internal docker[3550]: b0c42e8f4ac9: Pulling fs layer
Mar 24 09:02:49 ip-10-230-14-83.us-west-2.compute.internal docker[3550]: b0c42e8f4ac9: Download complete
Mar 24 09:02:49 ip-10-230-14-83.us-west-2.compute.internal docker[3550]: b0c42e8f4ac9: Download complete
Mar 24 09:02:49 ip-10-230-14-83.us-west-2.compute.internal docker[3550]: Status: Downloaded newer image for mvanholsteijn/paas-monitor:latest

Once it is running you can navigate to http://paas-monitor.cargonauts.dutchdevops.net and click on start.

Screen Shot 2015-03-24 at 10.05.20

 

 

You can now add new instances and watch them appear in the paas-monitor! It definitively takes a while because the docker images have to be pulled from the registry before the can be started, but in the end it they will all appear!

fleetctl start paas-monitor@{2..10}
Unit paas-monitor@2.service launched on 2dde0d31.../10.221.12.109
Unit paas-monitor@4.service launched on f7257c36.../10.237.157.117
Unit paas-monitor@3.service launched on 7f1f2982.../10.230.14.39
Unit paas-monitor@6.service launched on 2dde0d31.../10.221.12.109
Unit paas-monitor@5.service launched on 1cdadb87.../10.230.14.83
Unit paas-monitor@8.service launched on f7257c36.../10.237.157.117
Unit paas-monitor@9.service launched on 1cdadb87.../10.230.14.83
Unit paas-monitor@7.service launched on 7f1f2982.../10.230.14.39
Unit paas-monitor@10.service launched on 2dde0d31.../10.221.12.109

to see all deployed units, use the list-units command

fleetctl list-units
...
UNIT MACHINE ACTIVE SUB
paas-monitor@1.service 94d16ece.../10.90.9.78 active running
paas-monitor@2.service f7257c36.../10.237.157.117 active running
paas-monitor@3.service 7f1f2982.../10.230.14.39 active running
paas-monitor@4.service 94d16ece.../10.90.9.78 active running
paas-monitor@5.service f7257c36.../10.237.157.117 active running
paas-monitor@6.service 7f1f2982.../10.230.14.39 active running
paas-monitor@7.service 7f1f2982.../10.230.14.39 active running
paas-monitor@8.service 94d16ece.../10.90.9.78 active running
paas-monitor@9.service f7257c36.../10.237.157.117 active running
How does it work?

Whenever there is a change to the consul service registry, the consul-http-router is notified, selects all http tagged services and generates a new nginx.conf. After the configuration is generated it is reloaded by nginx so that there is little impact on the current traffic.

The consul-http-router uses the Go template language to regenerate the config. It looks like this:

events {
    worker_connections 1024;
}

http {
{{range $index, $service := services}}{{range $tag, $services := service $service.Name | byTag}}{{if eq "http" $tag}}

    upstream {{$service.Name}} {
	least_conn;
	{{range $services}}server {{.Address}}:{{.Port}} max_fails=3 fail_timeout=60 weight=1;
	{{end}}
    }
{{end}}{{end}}{{end}}

{{range $index, $service := services}}{{range $tag, $services := service $service.Name | byTag}}{{if eq "http" $tag}}
    server {
	listen 		80;
	server_name 	{{$service.Name}}.*;

	location / {
	    proxy_pass 		http://{{$service.Name}};
	    proxy_set_header 	X-Forwarded-Host	$host;
	    proxy_set_header 	X-Forwarded-For 	$proxy_add_x_forwarded_for;
	    proxy_set_header 	Host 			$host;
	    proxy_set_header 	X-Real-IP 		$remote_addr;
	}
    }
{{end}}{{end}}{{end}}

    server {
	listen		80 default_server;

	location / {
	    root /www;
	    index index.html index.htm Default.htm;
	}
    }
}

It loops through all the services and selects all services tagged 'http' and creates a virtual host for servicename.* which sends all request to the registered upstream services. Using the following two commands you can see the current configuration file.

AMACHINE=$(fleetctl list-machines -fields=machine -no-legend -full | head -1)
fleetctl ssh $AMACHINE docker exec consul-http-router cat /etc/nginx/nginx.conf
...
events {
    worker_connections 1024;
}

http {

    upstream paas-monitor {
	least_conn;
	server 10.221.12.109:49154 max_fails=3 fail_timeout=60 weight=1;
	server 10.221.12.109:49153 max_fails=3 fail_timeout=60 weight=1;
	server 10.221.12.109:49155 max_fails=3 fail_timeout=60 weight=1;
	server 10.230.14.39:49153 max_fails=3 fail_timeout=60 weight=1;
	server 10.230.14.39:49154 max_fails=3 fail_timeout=60 weight=1;
	server 10.230.14.83:49153 max_fails=3 fail_timeout=60 weight=1;
	server 10.230.14.83:49154 max_fails=3 fail_timeout=60 weight=1;
	server 10.230.14.83:49155 max_fails=3 fail_timeout=60 weight=1;
	server 10.237.157.117:49153 max_fails=3 fail_timeout=60 weight=1;
	server 10.237.157.117:49154 max_fails=3 fail_timeout=60 weight=1;

    }

    server {
	listen 		80;
	server_name 	paas-monitor.*;

	location / {
	    proxy_pass 		http://paas-monitor;
	    proxy_set_header 	X-Forwarded-Host	$host;
	    proxy_set_header 	X-Forwarded-For 	$proxy_add_x_forwarded_for;
	    proxy_set_header 	Host 			$host;
	    proxy_set_header 	X-Real-IP 		$remote_addr;
	}
    }

    server {
	listen		80 default_server;

	location / {
	    root /www;
	    index index.html index.htm Default.htm;
	}
    }
}
[/code]

This also happens when you stop or kill and instance. Just stop an instance and watch your monitor respond.
1
fleetctl destroy paas-monitor@10
...
Destroyed paas-monitor@10.service
Killing a machine

Now let's be totally brave and stop and entire machine!

ssh core@$FLEETCTL_TUNNEL sudo shutdown -h now.
...
Connection to 54.203.141.124 closed by remote host.

Keep watching your paas-monitor. You will notice a slow down and also notice that a number of backend services are no longer responding. After a short while (1 or 2 minutes) you will see new instances appear in the list.
paas-monitor after restart
What happened is that Amazon AWS restarted a new instance into the cluster and all units that were running on the stopped node have been moved to running instances with only 6 HTTP errors!

Please note that CoreOS is not capable of automatically recovering from loss of a majority of the servers at the same time. In that case, manual recovery by operations is required.

Conclusion

CoreOS provides all the basic functionality to manage Docker containers and provided High Availability to your application with a minimum of fuss. Consul and Consul templates actually make it very easy to use custom components like NGiNX to implement dynamic service discovery.

Outlook

In the next blog we will be deploying an multi-tier application that uses Consul DNS to connect application parts to databases!

References

This blog is based on information, ideas and source code snippets from  https://coreos.com/docs/running-coreos/cloud-providers/ec2, http://cargonauts.io/mitchellh-auto-dc and https://github.com/justinclayton/coreos-and-consul-cluster-via-terraform

Impact-Driven Scrum, Code Review & #NoEstimates in Methods & Tools Spring 2015 issue

From the Editor of Methods & Tools - Mon, 03/23/2015 - 16:13
Methods & Tools ‚Äď the free e-magazine for software developers, testers and project managers ‚Äď has just published its Spring 2015 issue that discusses Impact-Driven Scrum, Code Review, #NoEstimates,Self-Selecting Teams, Software Laws, Kanboard and ConQAT. * Impact-Driven Scrum Delivery * Code Review: Why It Matters * #NoEstimates – Alternative to Estimate-Driven Software Development * Self-Selecting Teams Part 2 – Keeping the Momentum * Laws for Software Development Teams * Kanboard – Open Source Kanban Board * ConQAT ‚Äď The Continuous Quality Assessment Toolkit 50 pages of software development knowledge that you can freely download from http://www.methodsandtools.com/mt/download.php?spring15

How to write an Amazon RDS service broker for Cloud Foundry

Xebia Blog - Mon, 03/23/2015 - 10:58

Cloud Foundry is a wonderful on-premise PaaS  that makes it very easy to build, deploy while providing scalability and high availability to your stateless applications. But Cloud Foundry is really a Application Platform Service and does not provide high availability and scalability for your data. Fortunately, there is Amazon RDS, which excels in providing this as a service.

In this blog I will show you how easy it is to build, install and use a Cloud Foundry Service Broker for Amazon RDS.  The broker was developed in Node.JS using the Restify framework and can be deployed as a normal Cloud Foundry application. Finally,  I will point you to a skeleton service broker which you can use as the basis for your own.

Cloud Foundry Service Broker Domain

Before I race of into the details of the implementation, I would like to introduce you into the Cloud Foundry lingo. If you are aware of the lingo, just skip to the paragraph 'AWS RDS Service Broker operations'.

Service - an external resource that can be used by an application. It can be a database, a messaging system or an external application.  Commonly provided services are mysql, postgres, redis and memcached.

Service Plan - a plan specify the quality of the service and governs the amount memory, disk space, nodes etc. provided with the service.

Service Catalog - a document containing all services and service plans of a service broker.

Service Broker - a program that is capable of creating services and providing the necessary information to applications to connect to the service.

Now a service broker can provide the following operations:

Describe Services - Show me all the services this broker can provide.

Create Service - Creating an instance of a service matching a specified plan. When the service is a database, it depends on the broker what this means: It may create an entire database server, or just a new database instance, or even just a database schema.   Cloud Foundry calls this 'provisioning a service instance'.

Binding a Service - providing a specific application with the necessary information to connect to an existing service.  When the service is a database, it provides the hostname, portname, database name, username and password. Depending on the service broker, the broker may even  create specific credentials for each  bind request/application. The Cloud Controller will store the returned credentials in a JSON document stored as an UNIX environment variable (VCAP_SERVICES).

Unbind service - depending on the service broker, undo what what done on the bind.

Destroy Service - Easy, just deleting what was created. Cloud Foundry calls this 'deprovisioning a service instance'.

In the next paragraph I will map these operations to Amazon AWS RDS services.

AWS RDS Service Broker operations

Any Service Broker has to implement a REST API of the Cloud Foundry specification.  To create the Amazon AWS RDS service broker, I had to implement four out of five methods:

  • describe services -¬†returns available services and service plans
  • create service -¬†call the createDBInstance operation and store generated credentials as tags in with the instance.
  • bind service - call the¬†describeDBInstances¬†operation and return the stored credentials.
  • delete service - just¬†call the deleteDBInstance¬†operation.

I implemented these REST calls using the Restify framework and the Amazon AWS RDS API for Javascript. the skeleton looks like this:

// get catalog
server.get('/v2/catalog', function(request, response, next) {
    response.send(config.catalog);
    next();
});

// create service
server.put('/v2/service_instances/:id', function(request, response, next) {
        response.send(501, { 'description' : 'create/provision service not implemented' });
        next();
    });

// delete service
server.del('/v2/service_instances/:id', function(req, response, next) {
        response.send(501, { 'description' : 'delete/unprovision service not implemented' });
        next();
    });

// bind service
server.put('/v2/service_instances/:instance_id/service_bindings/:id', function(req, response, next) {
        response.send(501, { 'description' : 'bind service not implemented' });
        next();
});

// unbind service
server.del('/v2/service_instances/:instance_id/service_bindings/:id', function(req, response, next) {
    response.send(501, { 'description' : 'unbind service not implemented' });
    next();
});

For the actual implementation of each operations on AWS RDS,  I would like to refer you to the source code of aws-rds-service-broker.js on github.com .

Design decisions

That does not look all too difficult does it?  Here are some of my design decisions:

Where do I store the credentials?

I store the credentials as tags on the  instance.  I wanted to create service broker that was completely stateless so that I could deploy it in Cloud Foundry itself. I did not want to be dependent on a complete database for a little bit of information. The tags seemed to fit the purpose.

Why does bind return the same credentials for every bind?

I wanted the bind service to be as simple as possible. I did not want to generate new user accounts and passwords, because if I did, I had even more state to maintain.  Even more, I found  that if I bind two applications to the same MySQL service, they could see each others data. So why bother creating users for binds? Finally, making the bind service simple, kept the unbind service even simpler because there is nothing to undo.

How to implement different service plans?

The createDBInstance operation of AWS RDS API operation, takes a JSON object as input parameter that is basically the equivalent of a plan. I just had to add an appropriate JSON record to the configuration file for each plan. See the description of the params parameter of the createDBInstance operation.

How do I create a AWS RDS service within 60 seconds?

Well, I don't.  The service broker API states that you have to create a service within the timeout of the cloud controller (which is 60 seconds), but RDS takes a whee bit more time. So the create request is initiated within seconds, but before you can bind an application to it may take a few minutes. Nothing I can do about that.

Why store the service broker credentials in environment variables?

I want the service broker to be configured upon deployment time. When the credentials are in the config file, you need to change the files of the application on each deployment.

Installation

In these instructions, I presume you have access to an AWS account and you have an installation of Cloud Foundry. I used  Stackato which is a Cloud Foundy implementation by ActiveState.  These instructions assume you are too!

  1. Create a AWS IAM user
    First create a AWS IAM user (cf-aws-service-broker) with at least the folllowing privileges
  2. Assign privileges to execute AWS RDS operations
    The newly created IAM user needs the privileges to create RDS databases. I used the following permissions:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [
             "rds:AddTagsToResource",
             "rds:CreateDBInstance",
             "rds:DeleteDBInstance",
             "rds:DescribeDBInstances",
             "rds:ListTagsForResource"
          ],
          "Resource": [
             "*"
          ]
        },
        {
          "Effect": "Allow",
          "Action": [
             "iam:GetUser"
          ],
          "Resource": [
              "*"
          ]
        }
      ]
    }
    
  3. Generate AWS access key and secret for the user 'cf-aws-service-broker'
  4. Create a Database Subnet
    Create a  database subnet 'stackato-db-subnet-group' in the AWS Region where you want to have the databases to be created.
  5. Check out the service broker
    git clone https://github.com/mvanholsteijn/aws-rds-service-broker
    cd aws-rds-service-broker
    
  6. Add all your parameters as environment variables to the manifest.yml
    applications:
       - name: aws-rds-service-broker
         mem: 256M
         disk: 1024M
         instances: 1
         env:
           AWS_ACCESS_KEY_ID: <fillin>
           AWS_SECRET_ACCESS_KEY: <fillin>
           AWS_REGION: <of db subnet group,eg eu-west-1>
           AWS_DB_SUBNET_GROUP: stackato-db-subnet-group
           SERVICE_BROKER_USERNAME: <fillin>
           SERVICE_BROKER_PASSWORD: <fillin>
         stackato:
           ignores:
             - .git
             - bin
             - node_modules
    
  7. Deploy the service broker
    stackato target <your-service-broker> --skip-ssl-validation
    stackato login
    push
    
  8. Install the service broker
    This script is a cunning implementation which create the service broker in Cloud Foundry and makes all the plans publicly available. In stackato we use the curl commands to achieve this. This script requires you to have installed jq, the wonderful JSON command line processor by Stephen Dolan.

    bin/install-service-broker.sh
    

Now you can use the service broker!

Using the Service Broker

Now we are ready to use the service broker.

  1. Deploy a sample application
    $ git clone https://github.com/mvanholsteijn/paas-monitor
    $ stackato push -n 
    
  2. Create a service for the mysql services
    $ stackato create-service
    1. filesystem 1.0, by core
    2. mysql
    3. mysql 5.5, by core
    4. postgres
    5. postgresql 9.1, by core
    6. redis 2.8, by core
    7. user-provided
    Which kind to provision:? 2
    1. 10gb: 10Gb HA MySQL database.
    2. default: Small 5Gb non-HA MySQL database
    Please select the service plan to enact:? 2
    Creating new service [mysql-844b1] ... OK
    
  3. Bind the service to the application
    stackato bind-service mysql-844b1 paas-monitor
      Binding mysql-844b1 to paas-monitor ... Error 10001: Service broker error: No endpoint set on the instance 'cfdb-3529e5764'. The instance is in state 'creating'. please retry a few minutes later (500)
    

    retry until the database is actually created (3-10 minutes on AWS)

    stackato bind-service mysql-844b1 paas-monitor
     Binding mysql-844d1 to paas-monitor ...
    Stopping Application [paas-monitor] ... OK
    Starting Application [paas-monitor] ...
    OK
    http://paas-monitor.<your-api-endpoint>/ deployed
    
  4. Check the environment of the application
    curl -s http://paas-monitor.<your-api-endpoint>/environment | jq .DATABASE_URL
    "mysql://root:e1zfMf7OXeq3@cfdb-3529e5764.c1ktcm2kjsfu.eu-central-1.rds.amazonaws.com:3306/mydb"
    

    As you can see the credentials for the newly created database has been inserted into the environment of the application.

Creating your own service broker

If you want to create your own service broker in Node.JS you may find the Skeleton Service Broker  a good starting point. It includes a number of utilities to test your broker in the bin directory.

  • catalog.sh - calls the catalog operation
  • provision.sh - calls the create¬†operation
  • unprovision.sh - call the delete¬†operation
  • bind.sh - calls the bind operation on a specified instance
  • unbind.sh - calls the unbind operation on a specified instance and bind id.
  • list.sh - calls the list all service instances operation
  • getenv.sh - gets the environment variables of an CF applications as sourceable output
  • install-service-broker.sh - installs the application and makes all plans public.
  • docurl.sh - calls the stackato CURL operation.

getenv.sh, install-service-broker.sh and provision.sh require jq to be installed.

Conclusion

As you can see, it is quite easy to create your own Cloud Foundry service broker!

SPaMCAST 334 ‚Äď Mario Lucero, It‚Äôs All About Agile Coaching

www.spamcast.net

                       www.spamcast.net

Listen Now

Subscribe on iTunes

In this episode of the Software Process and Measurement Cast we feature our interview with Agile coach Mario Lucero.  Mario and I discussed the nuts and bolts of coaching Agile teams, what is and isn’t Agile and the impact of coaching on success. Mario provided insights on Agile that span both Americas!

Mario describes himself as an Agile evangelist (including Kanban) delivering coaching for Agile transformations and Scrum mastery. He performs as a Scrum Master for several teams while mentoring and coaching other teams, Scrum Masters and product owners.

Mario is as comfortable advising senior management on the Agile transformation strategy and implementation as he is working with teams.

Email: metlucero@gmail.com

Twitter: @metlucero

Blog:  http://mariolucero.cl/

LinkedIn: http://cl.linkedin.com/in/luceromet/en

Call to action!

Can you tell a friend about the podcast? If your friends don’t know how to subscribe or listen to a podcast, show them how you listen and subscribe them!  Remember to send us the name of you person you subscribed (and a picture) and I will give both you and the horde you have converted to listeners a call out on the show.

Re-Read Saturday News

The Re-Read Saturday focus on Eliyahu M. Goldratt and Jeff Cox’s The Goal: A Process of Ongoing Improvement began on February 21nd. The Goal has been hugely influential because it introduced the Theory of Constraints, which is central to lean thinking. The book is written as a business novel. Visit the Software Process and Measurement Blog and catch up on the re-read.

Note: If you don’t have a copy of the book, buy one.  If you use the link below it will support the Software Process and Measurement blog and podcast.

Dead Tree Version or Kindle Version 

I am beginning to think of which book will be next. Do you have any ideas?

Upcoming Events

CMMI Institute Conference EMEA 2015
March 26 -27 London, UK
I will be presenting ‚ÄúAgile Risk Management.‚ÄĚ
http://cmmi.unicom.co.uk/

QAI Quest 2015
April 20 -21 Atlanta, GA, USA
Scale Agile Testing Using the TMMi
http://www.qaiquest.org/2015/

DCG will also have a booth!

Next SPaMCast

The next Software Process and Measurement Cast will feature our essay on the definitions of four critical words.  What do the words effectiveness, efficiency, frameworks and methodologies really mean?  These words get used ALL the time, however they really do have fairly specific meanings.  Meanings that, once understood and used to guide how we work, can help everyone to deliver more value and make our customers more satisfied!

Shameless Ad for my book!

Mastering Software Project Management: Best Practices, Tools and Techniques¬†co-authored by Murali Chematuri and myself and published by J. Ross Publishing. We have received unsolicited reviews like the following: ‚ÄúThis book will prove that software projects should not be a tedious process, neither for you or your team.‚ÄĚ Support SPaMCAST by buying the book¬†here.

Available in English and Chinese.


Categories: Process Management

SPaMCAST 334 ‚Äď Mario Lucero, It‚Äôs All About Agile Coaching

Software Process and Measurement Cast - Sun, 03/22/2015 - 22:00

In this episode of the Software Process and Measurement Cast we feature our interview with Agile coach Mario Lucero.  Mario and I discussed the nuts and bolts of coaching Agile teams, what is and isn’t Agile and the impact of coaching on success. Mario provided insights on Agile that span both Americas!

Mario describes himself as an Agile evangelist (including Kanban) delivering coaching for Agile transformations and Scrum mastery. He performs as a Scrum Master for several teams while mentoring and coaching other teams, Scrum Masters and product owners.

Mario is as comfortable advising senior management on the Agile transformation strategy and implementation as he is working with teams.

Email: metlucero@gmail.com

Twitter: @metlucero

Blog:  http://mariolucero.cl/

LinkedIn: http://cl.linkedin.com/in/luceromet/en

Call to action!

Can you tell a friend about the podcast? If your friends don’t know how to subscribe or listen to a podcast, show them how you listen and subscribe them!  Remember to send us the name of you person you subscribed (and a picture) and I will give both you and the horde you have converted to listeners a call out on the show. 

Re-Read Saturday News

The Re-Read Saturday focus on Eliyahu M. Goldratt and Jeff Cox’s The Goal: A Process of Ongoing Improvement began on February 21nd. The Goal has been hugely influential because it introduced the Theory of Constraints, which is central to lean thinking. The book is written as a business novel. Visit the Software Process and Measurement Blog and catch up on the re-read.

Note: If you don’t have a copy of the book, buy one.  If you use the link below it will support the Software Process and Measurement blog and podcast.

Dead Tree Version or Kindle Version 

I am beginning to think of which book will be next. Do you have any ideas?

Upcoming Events

CMMI Institute Conference EMEA 2015
March 26 -27 London, UK
I will be presenting ‚ÄúAgile Risk Management.‚Ä̬†
http://cmmi.unicom.co.uk/

QAI Quest 2015
April 20 -21 Atlanta, GA, USA
Scale Agile Testing Using the TMMi
http://www.qaiquest.org/2015/

DCG will also have a booth!

Next SPaMCast

The next Software Process and Measurement Cast will feature our essay on the definitions of four critical words.  What do the words effectiveness, efficiency, frameworks and methodologies really mean?  These words get used ALL the time, however they really do have fairly specific meanings.  Meanings that, once understood and used to guide how we work, can help everyone to deliver more value and make our customers more satisfied! 

Shameless Ad for my book!

Mastering Software Project Management: Best Practices, Tools and Techniques¬†co-authored by Murali Chematuri and myself and published by J. Ross Publishing. We have received unsolicited reviews like the following: ‚ÄúThis book will prove that software projects should not be a tedious process, neither for you or your team.‚ÄĚ Support SPaMCAST by buying the book¬†here.

Available in English and Chinese.

Categories: Process Management

Re-Read Saturday: The Goal: A Process of Ongoing Improvement. Part 5

IMG_1249

The Goal: A Process of Ongoing Improvement, published in 1984, is a business novel. The Goal uses the story of Alex Rogo, plant manager, to illustrate the theory of constraints and how the wrong measurement focus can harm an organization. The focus of the re-read is less on the story, but rather on the ideas the book explores that have shaped lean thinking. Earlier entries in this re-read are:

Part 1                         Part 2                         Part 3                         Part 4

In the next 4 chapters Alex stumbles on the how the concepts of dependent events and variability affect the flow of work, nearly literally.

Chapter 13

Chapter 13 begins with Alex awaking to his son, Dave, in his Boy Scout uniform waiting to go on a weekend hike and camping trip. Alex ends up as the leader due to the absence of the normal scout master.¬†¬† The column of scouts sets out with an adult, Ron, leading the way. Alex asks Ron to set a pace that is consistent and maintainable. The scouts create a queue based on the arcane rationale of young boys, and Alex anchors to the column to ensure the troop stays together. The column spreads out immediately even though everyone is moving at the same ‚Äúaverage‚ÄĚ speed. The interaction amongst the hikers is a series of dependent events. Scouts speed up and slow impacting those behind them. The act of speeding up and slowing down is the statistical variation described in Chapter 12. The speed of any individual hiker is influenced by the person directly ahead them in the line. Finally Alex realizes that the speed of the overall hike is less a reflection of first person in line than the last person in line. In the software-testing world, testing is not complete when the first test is done, but rather when the last test is completed.

Side Note: Anyone that wants to understand why every effort should be made to remove or reduce dependencies needs to read these this of chapters carefully. Dependencies make any process A LOT more complicated.

Chapter 14

Rogo considers how to reduce the statistical variation in the column. While stopping for lunch, Alex recruits a few of the scouts to play a game using match sticks, bowls and a die (they are boy scouts . . . ready for anything, including dice games). The game is played by moving match sticks between bowls. The number match sticks moved in each step is based on the roll of the die. As Alex gathers statistics by repeating the game, the combination of dependent events (movement of the match sticks from one bowl to another) and statistical variation (die roll) show him how build ups of inventory occurs between steps. The flow of work becomes irregular and unmanageable.

Side Note: This is a great game to play with software teams to drive home the point of the impact of variability.

Chapter 15

As the troop starts out from lunch, Alex considers the concept of reserves as a mechanism to fix the flow problem (spreading out) the troop is having. He watches the slowest kid in the troop fall behind and then sprint to catch up over and over, generating a large gaps in the line. The troop is utilizing all of its energy to stay together meaning that it has no spare capacity to recover when gaps appear. Consider software teams that generate plans with 100% utilization. As any developer or tester knows nothing goes exactly as planned, and as soon as a problem is encountered you are immediately behind if you are 100% utilized.

Another option he considers is to have everyone hike as fast as they can individually. In this scenario everyone would optimize their individual performance. The outcome would be chaos with scouts strung out all over the trail. With the troop spread out on the it would be impossible to know when the last person would get to the camp for the night. ¬†Remember the hike is only complete when the last person gets to camp, therefore chaos does not promote predictability.¬†In Alex’s¬†plant, even though they are using robot and each step is running at high levels of efficiency orders are not completing on-time. Similar problems can be seen in many software projects with developers and testers individually running at 100% capacity and high levels of efficiency while functionality is delivered well after it was promised.

When Rogo realizes that the process is only as fast as the slowest person, he decides to re-adjust the line of scouts so that the slowest is in front. The gaps immediately disappear. With the process now under control he can shift to helping the slowest person speed up, therefore improving the whole process.

Chapter 16 moves the novel plot forward with Julie, Alex’s wife, dumping Alex‚Äôs daughter at Alex‚Äôs mother‚Äôs house and leaves Alex.

Chapters 13 ‚Äď 15 drive home the point that dependent events and statistical variation impact the performance of the overall system. In order for the overall process to be more effective you have to understand the capability and capacity of each step and then take a systems view. These chapters establish the concepts of bottlenecks and constraints without directly naming them and that focusing on local optimums causes more trouble than benefit.

Summary of The Goal so far:

Chapters 1 through 3 actively present the reader with a burning platform. The plant and division are failing. Alex Rogo has actively pursued increased efficiency and automation to generate cost reductions, however performance is falling even further behind and fear has become central feature in the corporate culture.

Chapters 4¬†through¬†6¬†shift the focus from steps in the process to the process as a whole. Chapters 4 ‚Äď 6 move us down the path of identifying the ultimate goal of the organization (in this book). The goal is making money and embracing the big picture of systems thinking. In this section, the authors point out that we are often caught up with pursuing interim goals, such as quality, efficiency or even employment, to the exclusion of the of the ultimate goal. We are reminded by the burning platform identified in the first few pages of the book, the impending closure of the plant and perhaps the division, which in the long run an organization must make progress towards their ultimate goal, or they won‚Äôt exist.

Chapters 7 through 9 show Alex‚Äôs commitment to change, seeks more precise advice from Johan, brings his closest reports into the discussion and begins a dialog with his wife (remember this is a novel). In this section of the book the concept ‚Äúthat you get what you measure‚ÄĚ is addressed. In this section of the book, we see measures of efficiency being used at the level of part production, but not at the level of whole orders or even sales. We discover the corollary to the adage ‚Äėyou get what you measure‚Äô is that if you measure the wrong thing ‚Ķyou get the wrong thing. We begin to see Alex‚Äôs urgency and commitment to make a change.

Chapters 10 through 12 mark a turning point in the book. Alex has embraced a more systems view of the plant and that the measures that have been used to date are more focused on optimizing parts of the process to the detriment to overall goal of the plant.  What has not fallen into place is how to take that new knowledge and change how the plant works. The introduction of the concepts of dependent events and statistical variation begin the shift the conceptual understanding of what measure towards how the management team can actually use that information.

Note: If you don’t have a copy of the book, buy one.  If you use the link below it will support the Software Process and Measurement blog and podcast. Dead Tree Version or Kindle Version


Categories: Process Management

Consul: the end of CNAMEs and PuppetDB?

Xebia Blog - Fri, 03/20/2015 - 16:24

Do you use CNAME records to identify services on your network? Do you feel life is impossible without PuppetDB and exported resources? In this blog I will explain how Consul can be used to replace both, and jump-start your transition towards container-based infrastructure in the process.

Cadence: Twists and Turns

Different songs have a different cadence.

Different songs have a different cadence.

If you listen to the radio, talk or music, you will notice that the cadence you hear varies song-to-song, show-to-show or podcast-to-podcast. One cadence simply does not fit all people or situations. Songs, talk or software development the concept is the same.  In Agile, there are three types of cadence variation. They are innovation or recovery sprints, slow then fast and irregular sprints. Organizations tend to push towards a standardized cadence however they a few twists and turns might add a lot of value.

Innovation or Recovery Sprints. Innovation or recovery sprints are sometimes used to demarcate periods of significant development effort. These types of sprints are generally part of larger cadence cycles, just like the changes of seasons mark the boundaries of solar cycles within the overall annual cycle. Activities in these types of sprints often include training, vacations, planning for future sprints or roadmap building. Knowledge generation activities in these sprints often include infrastructure building, prototyping, spikes, building out the architectural runway for the project and hackathons. Sometimes these innovation and recovery sprints are used for catchup or overall integration testing (if not possible during standard sprints). A Scaled Agile Framework Enterprise (SAFe) program increment is generally 8 ‚Äď 12 weeks long made up of 4 to 6 2-week sprints. The last sprint is an Innovation Planning (IP) Sprint. All of the teams involved in the program increment participate in the IP sprint. In February 2014, Mike Cohn described an overall cadence of 6 x 2 +1, 6 2-week development sprints followed by a 1-week sprint for catchup, recovery and improving the team‚Äôs capabilities. The primary goal of innovation or recovery sprints is to improvement the capability of the team (or teams) involved. Learning, planning, exploration, reduction of technical debt and recovery improves the ability to deliver value when the next overall cycle begins.

Slow Then Fast Sprints. This variation is similar to the ebb and flow of cadence in a concert. The overall cadence at the beginning of many concerts I have attended are  generally slower than the cadence at the end. The cadence builds up to generate excitement so that then end of the concert is a crescendo. Recently at a monthly meeting of the Cleveland Agile Group, Chris Bohatka described a project that began with 2-week sprints, however 2 months before the end of the project shifted to 1-week sprints. The shift in cadence used the knowledge and team esprit de corps that had been built to accelerate the cadence and provide new functionality on a weekly basis. Interestingly it was noted in the presentation that the team probably would never go back to the longer/slower cadence. This is one of the VERY few times I have ever seen this behavior. Typically managers indicate that their teams will start with three or four week sprints and then accelerate, then the teams never accelerate their cadences. In theory this behavior seem to make sense however follow through is generally poor because team settle into a rhythm and get comfortable.

Irregular Sprints. While I have heard of Agile teams with irregular cadences, I have never actually seen a team perform in this manner over any significant period of time. Teams that are alleged to have irregular cadences often modify their cadence based on the story size and their inability to break them down. Sometimes implementation of continuous delivery, Kanban or a kanbanny-form Scrumban are confused with irregular cadence. These techniques, heavily influenced by lean, either don’t leverage or deemphasize the idea of cadence. They focus more on the continuous flow of work than on the time box that cadence generates. This is not the same thing as irregular cadence. Irregular cadence is most often a reflection of lack of discipline or a problem with how stories are being groomed.

A sustainable pace is one of the benefits of Agile. Sustainable is sometimes thought of as delivering prioritized user stories every ‚Äúx‚ÄĚ weeks, month in and month out. Innovation and recovery sprints interspersed at regular intervals are a great idea to ensure sustainability. Teams need to learn, innovate and in some cases catch-up to their promises. Interspersing innovation or recovery sprint at regular intervals is reflection of a higher order cadence. Using the knowledge gathered by working as a team, delivering functionality and innovation team can accelerate¬†the cadence, or at least has the option so that functionality can be delivered sooner AND feedback cna be gained sooner. Irregular sprints rarely make sense unless teams are ready to move to continuous delivery. Cadence in Agile projects promotes predictability and discipline which are attributes valued by both IT management and the business.


Categories: Process Management

Microservices: coupling vs. autonomy

Xebia Blog - Wed, 03/18/2015 - 14:35

Microservices are the latest architectural style promising to resolve all issues we had we previous architectural styles. And just like other styles it has its own challenges. The challenge discussed in this blog is how to realise coupling between microservices while keeping the services as autonomous as possible. Four options will be described and a clear winner will be selected in the conclusion.

To me microservices are autonomous services that take full responsibility for one business capability. Full responsibility includes presentation, API, data storage and business logic. Autonomous is the keyword for me, by making the services autonomous the services can be changed with no or minimal impact on others. If services are autonomous, then operational issues in one service should have no impact on the functionality of other services. That all sounds like a good idea, but services will never be fully isolated islands. A service is virtually always dependent on data provided by another service. For example imagine a shopping cart microservice as part of a web shop, some other service must put items in the shopping cart and the shopping cart contents must be provided to yet other services to complete the order and get it shipped. The question now is how to realise these couplings while keeping maximum autonomy.  The goal of this blog post is to explain which pattern should be followed to couple microservices while retaining maximum autonomy.

rr-ps

I'm going to structure the patterns by 2 dimensions, the interaction pattern and the information exchanged using this pattern.

Interaction pattern: Request-Reply vs. Publish-Subscribe.

  • Request-Reply means that one service does a specific¬†request¬†for information (or to take some action). It then expects a response. The requesting service therefore needs to know what to aks and where to ask it. This could still be implemented asynchronously and of course your could put some abstraction in place such that the request service does not have to know the physical address of the other service, the point still remains that one service is explicitly asking a for¬†specific¬†information (or action to be taken) and functionally waiting for a response.
  • Publish-Subscribe: with¬†this pattern a service registers itself as being interested in certain information, or being able to handle certain requests. The relevant information or requests will then be delivered to it and it can decide what to do with it. In this post we'll assume that there is some kind of middleware in place to take care of delivery of the published messages to the subscribed services.

Information exchanged: Events vs. Queries/Commands

  • Events are facts that cannot be argued about. For example, an order with number 123 is created. Events only state what has happened. They do not describe what should happen as a consequence of¬†such an event.
  • Queries/Commands: Both convey what should happen. Queries are a specific request for information, commands are a specific request to the receiving service to take some action.

Putting these two dimensions in a matrix results into 4 options to realise couplings between microservices. So what are the advantages and disadvantages for each option? And which one is the best for reaching maximum autonomy?

In the description below we'll use 2 services to illustrate each pattern. The Order service which is responsible for managing orders and the Shipping service which is responsible for shipping stuff, for example the items included in an order. Services like these could be part of a webshop, which could then also contain services like a shopping cart, a product (search) service, etc.

1. Request-Reply with Events:rre

In this pattern one service asks a specific other service for events that took place (since the last time it asked). This implies strong dependency between these two services, the Shipping service must know which service to connect to for events related to orders. There is also a runtime dependency since the shipping service will only be able to ship new orders if the Order service is available.

Since the Shipping service only receives events it has to decide by itself when an order may be shipped based on information in these events. The Order service does not have to know anything about shipping, it simply provides events stating what happened to orders and leaves the responsibility to act on these events fully to the services requesting the events.

2. Request-Reply with Commands/Queries:

rrcIn this pattern the shipping Order service is going to request the Shipping service to ship an order. This implies strong coupling since the Order service is explicitly requesting a specific service to take care of the shipping and now the Order service must determine when an order is ready to be shipped. It is aware of the existence of a Shipping service and it even knows how to interact with it. If other factors not related to the order itself should be taken into account before shipping the order (e.g. credit status of the customer), then the order services should take this into account before requesting the shipping service to ship the order. Now the business process is baked into the architecture and therefore the architecture cannot be changed easily.

Again there is a runtime dependency since the Order service must ensure that the shipping request is successfully delivered to the Shipping service.

3. Publish-Subscribe with Events

pseIn Publish-Subscribe with Events the Shipping service registers itself as being interested in events related to Orders. After registering itself it will receive all events related to Orders without being aware what the source of the order events is. It is loosely coupled to the source of the Order events. The shipping service will need to retain a copy of the data received in the events such that is can conclude when an order is ready to be shipped. The Order service needs to have no knowledge about shipping. If multiple services provide order related events containing relevant data for the Shipping service then this is not recognisable by the Shipping service. If (one of) the service(s) providing order events is down, the Shipping service will not be aware, it just receives less events. The Shipping service will not be blocked by this.

4. Publish-Subscribe with Commands/Queries

pscIn Publish-Subscribe with Command/Queries the Shipping service registers itself as a service being able to ship stuff. It then receives all commands that want to get something shipped. The Shipping service does not have to be aware of the source of the Shipping commands and on the flip side the Order service is not aware of which service will take care of shipping. In that sense they are loosely coupled. However, the Order service is aware of the fact that orders must get shipped since it is sending out a ship command, this does make the coupling stronger.

Conclusion

Now that we have described the four options we go back to the original question, which pattern of the above 4 provides maximum autonomy?

Both Request-Reply patterns imply a runtime coupling between two services and that implies strong coupling. Both Command/Queries patterns imply that one service is aware of what another service should do (in the examples above the order service is aware that another service takes care of shipping) and that also implies strong coupling, but this time on functional level. That leaves one option: 3. Publish-Subscribe with Events. In this case both services are not aware of each others existence from both runtime and functional perspective. To me this is the clear winner for achieving maximum autonomy between services.

The next question pops up immediately, should you always couple services using Publish-Subscribe with events? If your only concern is maximum autonomy of services the answer would be yes, but, there are more factors that should be taken into the account. Always coupling using this pattern comes at a price, data is replicated, measures must be taken to deal with lost events, events driven architectures do add extra requirements on infrastructure, their might be extra latency, and more. In a next post I'll dive into these trade-offs and put things into perspective. For now remember that Publish-Subscribe with Events is a good bases for achieving autonomy of services.

Reducing the size of Docker Images

Xebia Blog - Wed, 03/18/2015 - 01:00

Using the basic Dockerfile syntax it is quite easy to create a fully functional Docker image. But if you just start adding commands to the Dockerfile the resulting image can become unnecessary big. This makes it harder to move the image around.

A few basic actions can reduce this significantly.

Three Cadence Complaints

Longer isn't necessarily better.

Longer isn’t necessarily better.

In Agile, cadence is the number days or weeks in a sprint or release. Stated another way, it is the length of the team’s development cycle. The cadence that a project or organization selects is based on a number of factors that include: criticality, risk and the type of project. As a general rule, once a team or a team of teams has settled on a specific cadence they tend not to vary significantly. While In today’s business environment a plurality of teams and organizations use a two-week sprint cadence, there is often a lot of angst over the finding the exact number of weeks in a sprint any specific team.  Many organizations adopt a standard and take the discussion off the table. With any specified cadence duration, there are typically three complaints: too much overhead, not getting stories done and not being able to commit resources on full-time basis to the work. In almost every case the complaint is coupled with a request to lengthen the duration of the sprint and therefore to slow the cadence.

  1. The sprint structure requires too much overhead: Sprints begin with a planning event and typically complete with both a demonstration and retrospective. Short stand-up meetings occur daily. Some team participants view these events as overhead. Scrum practice and observation of teams strongly suggests that the effort for the Scrum events increase or decrease depending on the length of the sprint. Longer sprints tend to require more planning and lengthier demos and retrospectives. ¬†As a rule I suggest two hours of planning per week and 30 minutes each for the demonstration and retrospective per week in a sprint. Stand-ups should be approximately 15 minutes per day. For example, I would expect to spend 3 hours 45 minutes (2¬†hours for planning, 30 minutes for demo, 30 minutes for the retrospective and 45¬†minutes for 3 stand-up meetings) on events in a one-week sprint and 8 hours (4 hours for planning, 1 hour for the demo, 1 hour for the retrospective and 2 hours for 8 stand-up meetings). I have noticed that the effort for sprint events that are more than¬†three weeks long¬†tend to take longer than¬†one would expect (four weeks sometimes being more than twice what is expected). Realistically, sprint size should not significantly affect overhead until you get to sprints duration’s of four weeks or more therefore overhead is not an obstacle to shorter sprints. Remember that in earlier posts we have shown that shorter sprints deliver feedback and value sooner so therefore are preferred all things being equal.
  2. Our stories can’t be completed during the sprint. This is typically not a problem with the duration of sprint, but either an issue of how the stories are split or a process problem. One typical corollary to this complaint is that the team can’t break stories into thin enough slices to complete. Most of the time this is a training or coaching problem rather than a technical problem, however in highly regulated environments or systems that affect human life I have seen scenarios where stories tend to require longer sprints due to very specific variation and validation. One common cause of this problem is assigning and hard wiring roles (testers can only test for example), which can cause bottlenecks and constraints if there is any imbalance in capacity. This is illustrated by the Theory of Constraints (take a look at our Re-read Saturday entries about The Goal for more on the Theory of Constraints). Typically, longer sprints will not solve the problem. Unless the underlying capacity issue is addressed longer sprints typically equate to worse performance because more stories are started and are subject to bottlenecks and constraints.
  3. Our organization can’t commit full-time resources to a sprint. Part-time team members typically have to time slice, switching between projects and pieces of work leading to some loss of efficiency. This is a reflection too much work and not enough capacity causing delays, constraints and bottlenecks. Similar to issue 2 above, typically longer sprints will not solve the problem. Unless the underlying capacity issue is addressed longer sprints typically equate to worse performance for the same reasons as noted above.

Many problems with cadence are either a reflection of process problems that generate overhead or poorly split user stories. For example, teams that do not have a groomed backlog will need to spend more time planning. Some team members see planning as avoidable overhead (the just wing it mentality). Overly large teams tend to have long daily stand-up meetings, again typically seen as overhead. Stories that are not thinly sliced will take longer to complete and have higher propensity to get stuck giving the team a feeling that a longer sprint is better. In almost every case, thinly sliced stories and committing to less work tends to improve the flow of work so that the team can actually deliver more. The duration of the sprint and cadence are usually not the root cause of the team’s problems.


Categories: Process Management

Mile High Agile Keynote

Mike Cohn's Blog - Tue, 03/17/2015 - 15:00

I’m going to be giving the conference keynote this spring at Mile High Agile  in Denver on April 3, 2015.

I’ll be talking about the importance of being able to let go of some of our firmly held beliefs in order to move on. Here’s the official session title and description. I hope you can join me at this event!

Let Go of Knowing: How Holding onto Views May Be Holding You Back

You undoubtedly have a firmly held set of convictions about what is necessary to do agile well. These convictions have served you well—your teams have delivered better products more quickly and more economically than before they were agile. But could some of your firmly held convictions be holding you back? And have you ever wondered why some of your most agile friends are similarly firm in their own opinions—even ones that are the exact opposite of your own?

In this session, you’ll see ways that biases may be preventing you from questioning your assumptions, why being open to new views is hard but vital, and why beginners so often think they know it all.

After this session, you will know how to discern the inviolate rules of Scrum from its merely good practices. You’ll know why you feel certain of some aspects of agile, less so about others. You’ll leave with the confidence to let go of knowing. And when we let go of knowing, we open ourselves to learning, which is the heart of agile.

Software Architecture versus Code

From the Editor of Methods & Tools - Mon, 03/16/2015 - 17:35
Software architecture and coding are often seen as mutually exclusive disciplines, despite us referring to higher level abstractions when we talk about our software. You’ve probably heard others on your team talking about components, services and layers rather than objects when they’re having discussions. Take a look at the codebase though. Can you clearly see these abstractions or does the code reflect some other structure? If so, why is there no clear mapping between the architecture and the code? Why do those architecture diagrams that you have on the wall say ...

SPaMCAST 333 ‚Äď What is Agile, Selling Defect Control, Planning Communication

www.spamcast.net

                      http://www.spamcast.net

Listen Now

Subscribe on iTunes

This week’s Software Process and Measurement Cast is a magazine feature with three columns. This week we have columns from Kim Pries, The Software Sensei, and Jo Ann Sweeney’s Explaining Change.  In this installment Kim discusses the ins and outs of selling defect control.  In Explaining Change, Jo Ann tackles the concept of planning for communication (protip: it is better than winging it). The SPaMCAST essay this week tackles the topic of what is and isn’t Agile.  Does just saying you are Agile make you Agile?  We think not!

Call to action!

Can you tell a friend about the podcast? If your friends don’t know how to subscribe or listen to a podcast, show them how you listen and subscribe them!  Remember to send us the name of you person you subscribed (and a picture) and I will give both you and the horde you have converted to listeners a call out on the show.

Re-Read Saturday News

The Re-Read Saturday focus on Eliyahu M. Goldratt and Jeff Cox’s The Goal: A Process of Ongoing Improvement began on February 21nd. The Goal has been hugely influential because it introduced the Theory of Constraints, which is central to lean thinking. The book is written as a business novel. Visit the Software Process and Measurement Blog and catch up on the re-read.

Note: If you don’t have a copy of the book, buy one.  If you use the link below it will support the Software Process and Measurement blog and podcast.

Dead Tree Version or Kindle Version 

I am beginning to think of which book will be next. Do you have any ideas?

Upcoming Events

CMMI Institute Conference EMEA 2015
March 26 -27 London, UK
I will be presenting ‚ÄúAgile Risk Management.‚ÄĚ
http://cmmi.unicom.co.uk/

QAI Quest 2015
April 20 -21 Atlanta, GA, USA
Scale Agile Testing Using the TMMi
http://www.qaiquest.org/2015/

DCG will also have a booth!

CANCELED -International Conference on Software Quality and Test Management
Washington D.C. May 31 – June 5, 2015

Next SPaMCast

The next Software Process and Measurement Cast will feature our interview with Agile coach Mario Lucero.  Mario and I discussed the nuts and bolts of coaching Agile teams, what is and isn’t Agile and the impact of coaching on success.

Shameless Ad for my book!

Mastering Software Project Management: Best Practices, Tools and Techniques¬†co-authored by Murali Chematuri and myself and published by J. Ross Publishing. We have received unsolicited reviews like the following: ‚ÄúThis book will prove that software projects should not be a tedious process, neither for you or your team.‚ÄĚ Support SPaMCAST by buying the book¬†here.

Available in English and Chinese.


Categories: Process Management