Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Feed aggregator

Ask HighScalability: Choose an Async App Server or Multiple Blocking Servers?

Jonathan Willis, software developer by day and superhero by night, asked an interesting question via Twitter on StackOverflow

tl;dr Many Rails apps or one Vertx/Play! app?


I've been having discussions with other members of my team on the pros and cons of using an async app server such as the Play! Framework (built on Netty) versus spinning up multiple instances of a Rails app server. I know that Netty is asynchronous/non-blocking, meaning during a database query, network request, or something similar an async call will allow the event loop thread to switch from the blocked request to another request ready to be processed/served. This will keep the CPUs busy instead of blocking and waiting.

I'm arguing in favor or using something such as the Play! Framework or Vertx.io, something that is non-blocking... Scalable. My team members, on the other hand, are saying that you can get the same benefit by using multiple instances of a Rails app, which out of the box only comes with one thread and doesn't have true concurrency as do apps on the JVM. They are saying just use enough app instances to match the performance of one Play! application (or however many Play! apps we use), and when a Rails app blocks the OS will switch processes to a different Rails app. In the end, they are saying that the CPUs will be doing the same amount of work and we will get the same performance.

What do you think? The marketplace has seemingly moved, in the form of node.js, Golang, Akka, and even Java, to the async server model. Does that mean it's the only right way?

Here's my attempt at a response:

Categories: Architecture

My Agile 2015 Roundup

Agile 2015 was the week of Aug 3-7 this year. It was a great week. Here are the links to my interviews and talks.

Interview with Dave Prior. We spoke about agile programs, continuous planning, and how you might use certifications. I made a little joke about measurement.

Interview with Paul DuPuy of SolutionsIQ. We also spoke about agile programs. Paul had some interesting questions, one of which I was not prepared for. That’s okay. I answered it anyway.

The slides from Scaling Agile Projects to Programs: Networks of Autonomy, Collaboration and Exploration. At some point, the Agile Alliance will post the video of this on their site.

The slides from my workshop Agile Hiring: It’s a Team Sport. Because it was a workshop, there are built-in activities. You can try these where you work.

My pecha kucha (it was part of the lightning talks) of Living an Agile Life.

I hope you enjoy these. I had a great time at the conference.

 

Categories: Project Management

Are YOU In Control of Your Life?

Making the Complex Simple - John Sonmez - Mon, 08/24/2015 - 13:00

Are YOU in Control of Your Life? Perhaps not. If you’ll allow me to put my philosopher’s cap on for a minute, I’ll try to explain why—oh, and it’s related to self-discipline. Free will or determinism? For a long time, a debate has raged over whether or not we, as human beings, actually have free […]

The post Are YOU In Control of Your Life? appeared first on Simple Programmer.

Categories: Programming

Managing in the Presence of Uncertainty

Herding Cats - Glen Alleman - Mon, 08/24/2015 - 05:43

A Tweet caught my eye this weekend

Screen Shot 2015-08-23 at 1.39.09 PMBefore moving to risk let's look at what Agile is

Agile development is a phrase used in software development to describe methodologies for incremental software development. Agile development is an alternative to traditional project management where emphasis is placed on empowering people to collaborate and make team decisions in addition to continuous planning, continuous testing and continuous integration.

Next the notion that Agile is actually risk management is very misunderstood. Agile provides raw information for risk management, but risk management has little to do with what software development method is being used. The continuous nature of Agile provides more frequent feedback on the state of the project. That is advantageous to risk management. Since Agile mandates this feedback on fine grained boundaries - weeks not months - the actions in the risk management paradigm below are also fine grained.

Where Does Risk Come From?

All risk comes from uncertainty. Uncertainty comes in two types. (1) Aleatory (naturally occurring in the underlying process and therefore irreducible) and (2) Epistemic (a probability that something unfavorable will happen). 

Risk results from uncertainty. To deal with the risk from Aleatory uncertainty we can only have margin, since the resulting risk is irreducible. This is schedule margin, cost margin, and product performance margin. This type of risk is just part of the world we live in. Natural variances in the work performed developing products needs margin. Natural variances in the performance is a server's throughput needed margin.

We can deal directly with the risk from Epistemic uncertainty by buying down the uncertainty. This is done with experiments, trials, incremental development and other risk reduction  activities that lower the uncertainty in the processes.

By the way many use the notion that risk is both positive and negative. This is not true. It's a naive understanding of the mathematics of risk processes. PMI does this. It is not allowed in our domain(s).

Agile and Incremental Delivery

There is a popular myth in the agile community that they have a lock on the notion of incremental delivery. This is again not true. Many product development lifecycles use incremental and iterative processes to produce products. Spiral development, Integrated Master Plan/Integrated Master Schedule, Incremental Commitment. All applicable to Software Intensive Systems and System of Systems domains, like Enterprise ERP.

Managing in the Presence of Uncertainty and the Resulting Risk

Here's how we manage SIS and SoS in the presence of uncertainty.

The methods used to collect requirements, turn those requirements into products and ship those products produces are of little concern to the Risk Management Process. They are of great concern to those engineering those products, but the Risk Management Process is above that activity.  You can start with the SEI Continuous Risk Management Guidebook for a framework of managing software development in the presence of risk. The management of risk in agile is very close to the management of risk in any other product or service development process. For any risk management process to work it needs to have these processes as a minimum. So when you here agile manages risk, confirm that is actually taking place by having the person making that system show clearly in a process description how each of the process areas below are implemented. Without that connection it just ain't true. Screen Shot 2015-08-23 at 3.03.44 PM   And here's how to put these processes together to ensure risk is being managed to increase the probability of success of your project.
Screen Shot 2015-08-23 at 3.42.33 PM And how to put these process areas to work on a project Screen Shot 2015-08-23 at 4.20.33 PM For those interested in managing projects in the presence of uncertainty and the risk that uncertainty creates, independent of any development methodology or development framework, here's a collection from the office library, in no particular order:

And a short white paper on Risk Management in Enterprise IT

Information Technology Risk Management from Glen Alleman Related articles IT Risk Management Making Conjectures Without Testable Outcomes Why Guessing is not Estimating and Estimating is not Guessing
Categories: Project Management

Every Employee is a Digital Employee

“The questions that we must ask ourselves, and that our historians and our children will ask of us, are these: How will what we create compare with what we inherited? Will we add to our tradition or will we subtract from it? Will we enrich it or will we deplete it?”
― Leon Wieseltier

Digital transformation is all around us.

And we are all digital employees according to Gartner.

In the article, Gartner Says Every Employee Is a Digital Employee, Gartner says that the IT function no longer holds a monopoly on IT.

A Greater Degree of Digital Dexterity

According to Gartner, employees are creating increasing digital dexterity from the devices and apps they use, to participating in sharing economies.

Via Gartner Says Every Employee Is a Digital Employee:

"'Today's employees possess a greater degree of digital dexterity,' said Matt Cain, research vice president at Gartner. 'They operate their own wireless networks at home, attach and manage various devices, and use apps and Web services in almost every facet of their personal lives. They participate in sharing economies for transport, lodging and more.'"

Workers are Streamlining Their Work Life

More employees are using technology to simplify, streamline, and scale their work.

Via Gartner Says Every Employee Is a Digital Employee:

"This results in unprecedented numbers of workers who enjoy using technology and recognize the relevance of digitalization to a wide range of business models. They also routinely apply their own technology and technological knowledge to streamline their work life."

3 Ways to Exploit Digital Dexterity

According to Gartner, there are 3 Ways the IT organization should exploit employees' digital dexterity:

  1. Implement a digital workplace strategy
  2. Embrace shadow IT
  3. Use a bimodal approach
1. Implement a Digital Workplace Strategy

While it’s happening organically, IT can also help shape the digital workplace experience.  Implement a strategy that helps workers use computing resources in a more friction free way and that play better with their pains, needs, and desired outcomes.

Via Gartner Says Every Employee Is a Digital Employee:

“Making computing resources more accessible in ways that match employees' preferences will foster engagement by providing feelings of empowerment and ownership. The digital workplace strategy should therefore complement HR initiatives by addressing and improving factors such as workplace culture, autonomous decision making, work-life balance, recognition of contributions and personal growth opportunities.”

2. Embrace shadow IT

Treat shadow IT as a first class citizen.  IT should partner with the business to help the business realize it’s potential, and to help workers make the most of the available IT resources.

Via Gartner Says Every Employee Is a Digital Employee:

“Rather than try to fight the tide, the IT organization should develop a framework that outlines when it is appropriate for business units and individuals to use their own technology solutions and when IT should take the lead. IT should position itself as a business partner and consultant that does not control all technology decisions in the business.”

3. Use a bimodal approach

Traditional IT is slow.   It’s heavy in governance, standards, and procedures.   It addresses risk by reducing flexibility.   Meanwhile, the world is changing fast.  Business needs to keep up.  Business needs fast IT. 

So what’s the solution?

Bimodal IT.  Bimodal IT separates the fast demands of digital business from the slow/risk-averse methods of traditional IT.

Via Gartner Says Every Employee Is a Digital Employee:

“Bimodal IT separates the risk-averse and ‘slow’ methods of traditional IT from the fast-paced demands of digital business, which is underpinned by the digital workplace. This dual mode of operation is essential to satisfy the ever-increasing demands of digitally savvy business units and employees, while ensuring that critical IT infrastructure and services remain stable and uncompromised.”

Everyone has technology at their fingertips.  Every worker has the chance to re-imagine their work in a Mobile-First, Cloud-First world. 

With infinite compute, infinite capacity, global reach, and real-time insights available to you, how could you evolve your job?

You can evolve your digital work life right under your feet.

You Might Also Like

Empower Every Person on the Planet to Achieve More

Satya Nadella on a Mobile-First, Cloud-First World

We Help Our Customers Transform

Categories: Architecture, Programming

What Life is Like with Agile Results

“Courage doesn't always roar. Sometimes courage is the little voice at the end of the day that says I'll try again tomorrow.” -- Mary Anne Radmacher

Imagine if you could wake up productive, where each day is a fresh start.  As you take in your morning breath, you notice your mind is calm and clear.

You feel strong and well rested.

Before you start your day, you picture in your mind three simple scenes of the day ahead:

In the morning, you see yourself complete a draft you’ve been working on.

In the afternoon, you see yourself land your idea and win over your peers in a key meeting.

In the evening, you see yourself enjoying some quiet time as you sit down and explore your latest adventures in learning.

With an exciting day ahead, and a chance to rise and shine, you feel the day gently pull you forward with anticipation. 

You know you’ll be tested, and you know some things won’t work out as planned.   But you also know that you will learn and improve from every setback.  You know that each challenge you face will be a leadership moment or a learning opportunity.  Your challenges make you stronger.

And you also know that you will be spending as much time in your strengths as you can, and that helps keeps you strong, all day long. 

You motivate yourself from the inside out by focusing on your vision for today and your values.  You value achievement.  You value learning.  You value collaboration.  You value excellence.  You value empowerment.   And you know that throughout the day, you will have every chance to apply your skills to do more, to achieve more, and to be more. 

Each task, or each challenge, is also a chance to learn more.  From yourself, and from everyone all around you.  And this is how you never stop learning.

You may not like some of the tasks before you, but you like the chance to master your craft.  And you enjoy the learning.  And you love how you get better.  With each task on your To-Do list for today, you experiment and explore ways to do things better, faster, and easier.

Like a productive artist, you find ways to add unique value.   You add your personal twist to everything you do.  Your twist comes from your unique experience, seeing what others can’t see from your unique vantage point, and applying your unique strengths.

And that’s how you do more art.  Your art.  And as you do your art, you feel yourself come alive.  You feel your soul sing, as you operate at a higher level.  As you find your flow and realize your potential, your inner-wisdom winks in an approving way.  Like a garden in full bloom on a warm Summer’s day, you are living your arĂȘte.

As your work day comes to an end, you pause to reflect on your three achievements, your three wins, for the day.   You appreciate the way you leaned in on the tough stuff.  You surprised yourself in how you handled some of your most frustrating moments.  And you learned a new way to do your most challenging task.  You take note of the favorite parts of your day, and your attitude of gratitude feels you with a sense of accomplishment, and a sense of fulfillment.

Fresh and ready for anything, you head for home.

Try 30 Days of Getting Results.  It’s free. Surprise yourself with what you’re capable of.

Categories: Architecture, Programming

SPaMCAST 356 - Steve Turner, Time and Inbox Management

Software Process and Measurement Cast - Sun, 08/23/2015 - 22:00

The Software Process and Measurement 356 features our interview with Steve Turner.  Steve and I talked time management and email inbox tyranny! If you let your inbox and interruptions manage your day you will deliver less value than you should and feel far less in control than you could! 

Steve’s Bio:

With a background in technology and nearly 30 years of business expertise, Steve has spent the last eight years sharing technology and time management tools, techniques and tips with thousands of professionals across the country.  His speaking, training and coaching has helped many organizations increase the productivity of their employees. Steve has significant experience working with independent sales and marketing agencies. His proven ability to leverage technology (including desktops, laptops and mobile devices) is of great value to anyone in need of greater sales and/or productivity results. TurnerTime℠ is time management tools, techniques and tips to effectively manage e-mails, tasks and projects.  

Contact Information:
Email: steve@getturnertime.com
Web: www.GetTurnerTime.com

Call to Action!

For the remainder of August and September let’s try something a little different.  Forget about iTunes reviews and tell a friend or a coworker about the Software Process and Measurement Cast. Let’s use word of mouth will help grow the audience for the podcast.  After all the SPaMCAST provides you with value, why keep it yourself?!

Re-Read Saturday News

Remember that the Re-Read Saturday of The Mythical Man-Month is in full swing.  This week we tackle the essay titled “Why Did the Tower Babel Fall”!  Check out the new installment at Software Process and Measurement Blog.

 

Upcoming Events

Software Quality and Test Management 
September 13 – 18, 2015
San Diego, California
http://qualitymanagementconference.com/

I will be speaking on the impact of cognitive biases on teams.  Let me know if you are attending! If you are still deciding on attending let me know because I have a discount code.

Agile Development Conference East
November 8-13, 2015
Orlando, Florida
http://adceast.techwell.com/

I will be speaking on November 12th on the topic of Agile Risk. Let me know if you are going and we will have a SPaMCAST Meetup.

Next SPaMCAST

The next Software Process and Measurement feature our essay on mind mapping.  To quote Tony Buzan, Mind Mapping is a technique for radiant thinking.  I view it as a technique to stimulate creative thinking and organize ideas. I think that is what Tony meant by radiant thinking. Mind Mapping is one of my favorite tools.  

We will also feature Steve Tendon’s column discussing the TameFlow methodology and his great new book, Hyper-Productive Knowledge Work Performance.

Anchoring the cast will be Gene Hughson returning with an entry from his Form Follows Function blog.

Shameless Ad for my book!

Mastering Software Project Management: Best Practices, Tools and Techniques co-authored by Murali Chematuri and myself and published by J. Ross Publishing. We have received unsolicited reviews like the following: “This book will prove that software projects should not be a tedious process, neither for you or your team.” Support SPaMCAST by buying the book here. Available in English and Chinese.

Categories: Process Management

HTTP/2 Server Push

Xebia Blog - Sun, 08/23/2015 - 17:11

The HTTP/2 standard was finalized in May 2015. Most major browsers support it, and Google uses it heavily.

HTTP/2 leaves the basic concepts of Requests, Responses and Headers intact. Changes are mostly at the transport level, improving the performance of parallel requests - with few changes to your application. The go HTTP/2 'gophertiles' demo nicely demonstrates this effect.

A new concept in HTTP/2 is Server Push, which allows the server to speculatively start sending resources to the client. This can potentially speed up initial page load times: the browser doesn't have to parse the HTML page and find out which other resources to load, instead the server can start sending them immediately.

This article will demonstrate how Server Push affects the load time of the 'gophertiles'.

HTTP/2 in a nutshell

The key characteristic of HTTP/2 is that all requests for a server are sent over one TCP connection, and responses can come in parallel over that connection.

Using only one connection reduces overhead caused by TCP and TLS handshakes. Allowing responses to be sent in parallel is an improvement over HTTP/1.1 pipelining, which only allows requests to be served sequentially.

Additionally, because all requests are sent over one connection, there is a Header Compression mechanism that reduces the bandwidth needed for headers that previously would have had to be repeated for each request.

Server Push

Server Push allows the server to preemptively send a 'request promise' and an accompanying response to the client.

The most obvious use case for this technology is sending resources like images, CSS and JavaScript along with the page that includes them. Traditionally, the browser would have to first fetch the HTML, parse it, and then make subsequent requests for other resources. As the server can fairly accurately predict which resources a client will need, with Server Push it does not have to wait for those requests and can begin sending the resources immediately.

Of course sometimes, you really do only want to fetch the HTML and not the accompanying resources. There are 2 ways to accomplish this: the client can specify it does not want to receive any pushed resources at all, or cancel an individual push after receiving the push promise. In the latter case the client cannot prevent the browser from initiating the Push, though, so some bandwidth may have been wasted. This will make deciding whether to use Server Push for resources that might already have been cached by the browser subtle.

Demo

To show the effect HTTP/2 Server Push can have, I have extended the gophertiles demo to be able to test behavior with and without Server Push, available hosted on an old raspberry pi.

Both the latency of loading the HTML and the latency of loading each tile is now artificially increased.

When visiting the page without Server Push with an artificial latency of 1000ms, you will notice that loading the HTML takes at least one second, and then loading all images in parallel again takes at least one second - so rendering the complete page takes well above 2 seconds.

With server push enabled, you will see that after the DOM has loaded, the images are almost immediately there, because they have been Push'ed already.

All that glitters, however, is not gold: as you will notice when experimenting (especially at lower artificial latencies), while Server Push fairly reliably reduces the complete load time, it sometimes increases the time until the DOM-content is loaded. While this makes sense (the browser needs to process frames relating to the Server Push'ed resources), this could have an impact on the perceived performance of your page: for example, it could delay running JavaScript code embedded in your site.

HTTP/2 does give you tools to tune this, such as Stream Priorities, but this might need careful tuning and be supported by the http2 library you are choosing.

Conclusions

HTTP/2 is here today, and can provide a considerable improvement in perceived performance - even with few changes in your application.

Server Push potentially allows you to improve your page loading times even further, but requires careful analysis and tuning - otherwise it might even have an adverse effect.

SPaMCAST 356 – Steve Turner, Time and Inbox Management

 www.spamcast.net

                             www.spamcast.net

Listen Now

Subscribe on iTunes

The next Software Process and Measurement features our interview with Steve Turner.  Steve and I talked time management and email inbox tyranny! If you let your inbox and interruptions manage your day you will deliver less value than you should and feel far less in control than you could!

Steve’s Bio:

With a background in technology and nearly 30 years of business expertise, Steve has spent the last eight years sharing technology and time management tools, techniques and tips with thousands of professionals across the country.  His speaking, training and coaching has helped many organizations increase the productivity of their employees. Steve has significant experience working with independent sales and marketing agencies. His proven ability to leverage technology (including desktops, laptops and mobile devices) is of great value to anyone in need of greater sales and/or productivity results. TurnerTime℠ is time management tools, techniques and tips to effectively manage e-mails, tasks and projects.

Contact Information:
Email: steve@getturnertime.com
Web: www.GetTurnerTime.com

 

Call to Action!

For the remainder of August and September let’s try something a little different.  Forget about iTunes reviews and tell a friend or a coworker about the Software Process and Measurement Cast. Let’s use word of mouth will help grow the audience for the podcast.  After all the SPaMCAST provides you with value, why keep it yourself?!

Re-Read Saturday News

Remember that the Re-Read Saturday of The Mythical Man-Month is in full swing.  This week we tackle the essay titled “Why Did the Tower Babel Fall”!  Check out the new installment at Software Process and Measurement Blog.

 

Upcoming Events

Software Quality and Test Management
September 13 – 18, 2015
San Diego, California
http://qualitymanagementconference.com/

I will be speaking on the impact of cognitive biases on teams.  Let me know if you are attending! If you are still deciding on attending let me know because I have a discount code.

Agile Development Conference East
November 8-13, 2015
Orlando, Florida
http://adceast.techwell.com/

I will be speaking on November 12th on the topic of Agile Risk. Let me know if you are going and we will have a SPaMCAST Meetup.

Next SPaMCAST

The next Software Process and Measurement feature our essay on mind mapping.  To quote Tony Buzan, Mind Mapping is a technique for radiant thinking.  I view it as a technique to stimulate creative thinking and organize ideas. I think that is what Tony meant by radiant thinking. Mind Mapping is one of my favorite tools.

We will also feature Steve Tendon’s column discussing the TameFlow methodology and his great new book, Hyper-Productive Knowledge Work Performance.

Anchoring the cast will be Gene Hughson returning with an entry from his Form Follows Function blog.

Shameless Ad for my book!

Mastering Software Project Management: Best Practices, Tools and Techniques co-authored by Murali Chematuri and myself and published by J. Ross Publishing. We have received unsolicited reviews like the following: “This book will prove that software projects should not be a tedious process, neither for you or your team.” Support SPaMCAST by buying the book here. Available in English and Chinese.


Categories: Process Management

Why did the Tower of Babel fall? Re-Read Saturday: The Mythical Man-Month, Part 7

The Mythical Man-Month

The Mythical Man-Month

In the seventh essay of The Mythical Man-Month, Brooks returns to the topics of organization and communication using story of the Tower of Babel as a metaphor for many of the engineering fiascoes seen in software development. In the story of the Tower of Babel, when communication broke down, coordination failed and the project ground to a halt. As projects become larger and more complex, communication and organization always becomes more difficult and more critical.

The essay Why Did the Tower of Babel Fall? presents three mechanisms to facilitate communication.

  1. Informal Communication, which includes all of the telephone calls and conversations that occur in and among teams. Healthy teams and teams of teams talk, interact and share information and ideas.
  2. Meetings are more formal and include reviews, technical briefings, demonstrations, sprint planning and release planning events. Done well formal meetings, whether part of an Agile framework or not, can find problems and share information and knowledge.
  3. Project Workbooks are the repository of the evolving intellectual property that defines any significant endeavor.

Brooks goes into depth on the definition and structure of the project workbook. While the workbook that Brooks describes is a physical construct we should consider that the idea of workbook is less of a document then a structure for grouping information. All the project documents need to fit into the workbook and be part of the overall structure of project information so that it can be reused and inherited from. The ultimate goal of the workbook is ensure that everyone who needs information has access to the information and gets the information.

Workbook Mechanics: In order to meet the goal of ensuring of everyone access to the information they need, the workbook needs to be maintained and organized. For example, Brooks suggests ordering and numbering all project memorandums (we would call these emails now) so that people can look through the list and actually see if they need information. Brooks points out that as size of the effort, project or program goes up, the complexity of maintaining and indexing the project communications increases in a non-linear fashion. Today’s technical tools like wikis, SharePoint and hypertext markup make curating and sharing information easier than paper and microfiche (if microfiche is before your time, Google it). As a reminder, capturing and reusing IP is required in all types of work. As a coder, it is easy to fall prey to the idea that the code and a functional system is the only documentation required; however, this is false. For most large efforts, capturing decisions, designs, architectures and even the occasional user manual is an option that can be ignored only if you want to court a communication failure. Remember however that over-reliance on documentation is not the goal.

Brooks uses the formula (n2-n)/2, where n is the number of people working on an effort, to define the possible communication interfaces. As the number of people increase the number of interfaces increases very quickly. Organization is a tool to reduce the amount of interfaces. This is one of the reasons why small teams are more effective than large teams. Scaling Agile is often accomplished by assembling teams of small teams. We use the metaphor of trees or flowers are often to describe the visualizations of team of teams. Communications spreading from a central point can look like a tree or flower. Organizations leverage techniques like the specialization and division of labor to reduce the number of communication interfaces.

There is often push back on division of labor and specialization in Agile. This push back is often misplaced. Within a team the concept of T Shaped people, which eschews specialization, increases capabilities; however, teams often specialize thus becoming capability teams. Capability teams often tackle specific types of work more effectively and efficiently by reducing learning curves. Capability teams fit into Brooks’ ideas on organizing to reduce the communication overhead.

Brooks proposes that each grouping of work have six essential needs.

  1. A Mission,
  2. A Producer,
  3. A Technical Director or An Architect,
  4. A Schedule,
  5. A Division of Labor, and
  6. Interface Definitions Among the Parts.

Four of these essentials are easily understood; however, the roles of the producer and technical broke new ground and extended the ideas Brooks began in The Surgical Team. The role of the producer is to establish the team, establish the schedule and continually acquire the resources needed to deliver the project. The role of the producer includes many of the classic project and program management activities, but also much of the outside communication and coordination activities. When small effort with a single team the roles of the producer and technical director can be held by the same person. As efforts become larger and more complex the roles are generally held by separate people. The role of the producer in the Scaled Agile Framework (SAFe) is handled by the Release Train Engineer. The technical director focuses on the design and the technical components (inside focus). In SAFe this role is held by the System Architect.

Brooks leverages the metaphor of the Tower of Babel to focus on the need for communication and coordination. Projects, team and organizations need to take steps to ensure communication happens effectively. Techniques like meetings and project workbooks are merely tools to make sure information is available and organized. Once we have information organization provides a framework to ensure information is communicated effectively. Without communication any significant engineering effort will end up just like the Tower of Babel.

Are there other mechanisms for organization, communication and coordination that have worked for you? Please share and lets discuss!

Previous installments of the Re-read of The Mythical Man-Month

Introductions and The Tar Pit

The Mythical Man-Month (The Essay)

The Surgical Team

Aristocracy, Democracy and System Design

The Second-System Effect

Passing the Word


Categories: Process Management

Anecdotes versus Numbers (Statistics)

Herding Cats - Glen Alleman - Sat, 08/22/2015 - 04:08

In the world of project management and the process improvement efforts needed to increase the Probability of Project Success anecdotes appear to prevail when it comes to suggesting alternatives to observed dysfunction. 

If we were to pile all the statistics for all the data for the effectiveness or not effectiveness of all the process improvement methods on top of each other they would lack the persuasive power of a single anecdote in most software development domains outside of Software Intensive Systems. 

Why? because most people working in small groups, agile, development projects, compared to Enterprise, Mission Critical can't fail, that must show up on time, on budget, with not just the minimum viable products, the the mandatorily needed viable capability - rely on anecdotes to communicate their messages.

I say this not from just personal experience, but from research for government agencies and commercial enterprise firms tasked with Root Cause Analysis, conference proceedings, refereed journal papers, and guidance from those tasked with the corrective actions of major program failures.

Anecdotes appeal to emotion. Statistics, numbers, verifiable facts appeal to reason. It's not a fair fight. Emotiona always wins without acknowledging that emotion is seriously flawed when making decisions.

Anecdotal evidence is evidence where small numbers of anecdotes are presented. There is a large chance - statistically - this evidence is unreliable due to cherry picking or self selection (this is the core issue with the Standish Reports or anyone claiming anything without proper statistical sampling processes). 

Anecdotal evidence is considered dubious support of any generalized claim. Anecdotal evidence is no more than a type description (i.e., short narrative), and is often confused in discussions with its weight, or other considerations, as to the purpose(s) for which it is used.

We've all heard stories, œ of all IT projects fail. Waterfall is evil, hell even estimates are evil stop doing them cold turkey. They prove the point the speaker is making right? Actually they don't. I just used an anecdote to prove a point. 

If I said The Garfunkel Institute just released a study showing 68% of all software development projects did not succeed because of a requirements gathering process failed to define what capabilities were needed when done, I've have made a fact base point. And you'd become bored reading the 86 pages of statistical analysis and correlation charts between all the causal factors contributing to the success or failure of the sample space of projects. See you are bored.

Instead if I said every project I've worked on went over budget and was behind schedule because we were very poor at making estimates. That'd be more appealing to your emotions, since it is a message you can relate to personally - having likely experienced many of the same failures.

The purveyors of anecdotal evidence to support a position make use of a common approach. Willfully ignoring a fact based methodology through a simple tactic...

We all know what Mark Twain said about lies, dammed lies, and statistics

People can certainly lie with statistics, done all the time. Start with How to Lie With Statistics But those types of Lies are nothing compared to the able to script personal anecdotes to support a message. From I never seen that work, to what now you're telling me - the person that actually invented this earth shattering way of writing software - that it doesn't work outside my personal sphere of experience?

An anecdote is a statistic with a sample size of one. OK, maybe a sample size of a small group  of your closest friends and fellow travelers. 

We fall for this all the time.  It's easier to accept an anecdote describing a problem and possible solution from someone we have shared experiences with, than to investigate the literature, do the math, even do the homework needed to determine the principles, practices, and processes needed for corrective action.

Don’t fall for manipulative opinion-shapers who use story-telling as a substitute for facts. When we're trying to persuade, use facts, use actual example based on those facts. Use data that can be tested outside personal anecdotes used to support an unsubstantiated claim without suggesting both the rot cause and the testable corrective actions.  

Related articles Making Conjectures Without Testable Outcomes Deadlines Always Matter Root Cause of Project Failure
Categories: Project Management

Beacons, the Internet of things, and more: Coffee with Timothy Jordan

Google Code Blog - Sat, 08/22/2015 - 00:31

Posted by Laurence Moroney, Developer Advocate

In this episode of Coffee With a Googler, Laurence meets with Developer Advocate Timothy Jordan to talk about all things Ubiquitous Computing at Google. Learn about the platforms and services that help developers reach their users wherever it makes sense.

We discuss Brillo, which extends the Android Platform to 'Internet of Things' embedded devices, as well as Weave, which is a services layer that helps all those devices work together seamlessly.

We also chat about beacons and how they can give context to the world around you, making the user experience simpler. Traditionally, users need to tell you about their location, and other types of context. But with beacons, the environment can speak to you. When it comes to developing for beacons, Timothy introduces us to Eddystone, a protocol specification for BlueTooth Low Energy (BLE) beacons, the Proximity Beacon API that allows developers to register a beacon and associate data with it, and the Nearby Messages API which helps your app 'sight' and retrieve data about nearby beacons.

Timothy and his team have produced a new Udacity series on ubiquitous computing that you can access for free! Take the course to learn more about ubiquitous computing, the design paradigms involved, and the technical specifics for extending to Android Wear, Google Cast, Android TV, and Android Auto.

Also, don't forget to join us for a ubiquitous computing summit on November 9th & 10th in San Francisco. Sign up here and we'll keep you updated.

Categories: Programming

How To Lie With Statistics

Herding Cats - Glen Alleman - Fri, 08/21/2015 - 22:51

How-to-lie-with-statisticsHow To Lie With Statistics is a critically important book to have on your desk if you're involved any decision making. My edition is a First Edition, but I don't have the dust jacket, so not worth that much beyond the current versions.

The reason for this post is to lay the ground work for assessing reports, presentations, webinars, and other selling documents that contain statistical information. 

The classic statistical misuse if the Standish Report, describing the success and failure of IT projects.

Here's my summation on the elements of How To Lie in our project domain

  • Sample with the Built In Bias - the population of the sample space is not defined. The samples are self selected in that those who respond are the basis of the statistics. No adjustment for all those who did not respond to a survey for example. 
  • The Well Chosen Average - The arithmetical average, Median, and Mode are estimators of the population statistics. Any of these without a variance is of little value for decision making. 
  • Little Figures That Are Not There - the classic is use this approach (in this case #NoEstimates) and your productivity will improve 10X, that 1000% by the way. A 1000% improvement. That's unbelievable, literally unbelievable. The actual improvements are stated, only the percentage. The baseline performance is not stated. It's unbelievable.
  • Much Ado About Practically Nothing -  the probability of being in the range of normal. This is the basis of advertising. What's the variance?
  • Gee-Whiz Graphs - using graphics and adjustable scales provides the opportunity to manipulate the message. The classic example of this is the estimating errors in a popular graph used by the No Estimates advocates. It's a graph showing the number of projects that complete over there estimated cost and schedule. What's not shown is the credibility of the original estimate.
  • One Dimensional Picture - using a picture to show numbers, where the picture is not in the scale as the numbers provides a messaging path for visual readers. 
  • Semi-attached Picture - If you can't prove what you want to prove, demonstrate something else and pretend that they are the same thing. In one example, the logic is inverted. Estimating is conjectured to be the root cause of problems. With no evidence of that, the statement we don't see how estimating can produce success, so not estimating will increase the probability of success. 
  • Post Hoc Rides Again - posy hoc causality is common in the absence of a cause and effect understanding. The correlation and causality differences are many times not understood. 

Here's a nice example of How To Lie

There's a chart from an IEEE Computer article showing the numbers of projects that exceeded their estimated cost. But let's start with some research on the problem. Coping with the Cone of Uncertainty.

There is a graph, popularly used to show that estimates 

Screen Shot 2015-08-14 at 3.09.33 PM

 

This diagram is actually MISUSED by the #NoEstimates advocates.

The presentation below shows the follow on information for how estimates can be improved the increase the confidence in the process and improvements in the business. As well shows the root causes of poor estimates and their corrective actions. Please ignore any ruse of Todd's chart without the full presentation. 

 

My mistake was doing just that.

So before anyone accepts any conjecture from a #NoEstimates advocate using the graph above, please read the briefing at the link below to see the corrective actions for making poor estimates.

Screen Shot 2015-08-14 at 3.24.50 PM

Here's the link to Todd's entire briefing not just the many times misused graph of estimates not representing the actuals Uncertainty Surrounding the Cone of Uncertainty. 

Related articles Root Cause of Project Failure Estimating and Making Decisions in Presence of Uncertainty Are Estimates Really The Smell of Dysfunction?
Categories: Project Management

Neo4j: Summarising neo4j-shell output

Mark Needham - Fri, 08/21/2015 - 21:59

I frequently find myself trying to optimise a set of cypher queries and I tend to group them together in a script that I fed to the Neo4j shell.

When tweaking the queries it’s easy to make a mistake and end up not creating the same data so I decided to write a script which will show me the aggregates of all the commands executed.

I want to see the number of constraints created, indexes added, nodes, relationships and properties created. The first 2 don’t need to match across the scripts but the latter 3 should be the same.

I put together the following script:

import re
import sys
from tabulate import tabulate
 
lines = sys.stdin.readlines()
 
def search(term, line):
    m =  re.match(term + ": (.*)", line)
    return (int(m.group(1)) if m else 0)
 
nodes_created, relationships_created, constraints_added, indexes_added, labels_added, properties_set = 0, 0, 0, 0, 0, 0
for line in lines:
    nodes_created = nodes_created + search("Nodes created", line)
    relationships_created = relationships_created + search("Relationships created", line)
    constraints_added = constraints_added + search("Constraints added", line)
    indexes_added = indexes_added + search("Indexes added", line)
    labels_added = labels_added + search("Labels added", line)
    properties_set = properties_set + search("Properties set", line)
 
    time_match = re.match("real.*([0-9]+m[0-9]+\.[0-9]+s)$", line)
 
    if time_match:
        time = time_match.group(1)
 
table = [
            ["Constraints added", constraints_added],
            ["Indexes added", indexes_added],
            ["Nodes created", nodes_created],
            ["Relationships created", relationships_created],
            ["Labels added", labels_added],
            ["Properties set", properties_set],
            ["Time", time]
         ]
print tabulate(table)

Its input is the piped output of the neo4j-shell command which will contain a description of all the queries it executed.

$ cat import.sh
#!/bin/sh
 
{ ./neo4j-community-2.2.3/bin/neo4j stop; } 2>&1
rm -rf neo4j-community-2.2.3/data/graph.db/
{ ./neo4j-community-2.2.3/bin/neo4j start; } 2>&1
{ time ./neo4j-community-2.2.3/bin/neo4j-shell --file $1; } 2>&1

We can use the script in two ways.

Either we can pipe the output of our shell straight into it and just get the summary e.g.

$ ./import.sh local.import.optimised.cql | python summarise.py
 
---------------------  ---------
Constraints added      5
Indexes added          1
Nodes created          13249
Relationships created  32227
Labels added           21715
Properties set         36480
Time                   0m17.595s
---------------------  ---------

…or we can make use of the ‘tee’ function in Unix and pipe the output into stdout and into the file and then either tail the file on another window or inspect it afterwards to see the detailed timings. e.g.

$ ./import.sh local.import.optimised.cql | tee /tmp/output.txt |  python summarise.py
 
---------------------  ---------
Constraints added      5
Indexes added          1
Nodes created          13249
Relationships created  32227
Labels added           21715
Properties set         36480
Time                   0m11.428s
---------------------  ---------
$ tail -f /tmp/output.txt
+-------------+
| appearances |
+-------------+
| 3771        |
+-------------+
1 row
Nodes created: 3439
Properties set: 3439
Labels added: 3439
289 ms
+------------------------------------+
| appearances -> player, match, team |
+------------------------------------+
| 3771                               |
+------------------------------------+
1 row
Relationships created: 10317
1006 ms
...

My only dependency is the tabulate package to get the pretty table:

$ cat requirements.txt
 
tabulate==0.7.5

The cypher script I’m running creates a BBC football graph which is available as a github project. Feel free to grab it and play around – any problems let me know!

Categories: Programming

Project Tango I/O Apps now released in Google Play

Google Code Blog - Fri, 08/21/2015 - 20:15

Posted by Larry Yang, Lead Product Manager, Project Tango

At Google I/O, we showed the world many of the cool things you can do with Project Tango. Now you can experience it yourself by downloading these apps on Google Play onto your Project Tango Tablet Development Kit.

A few examples of creative experiences include:

MeasureIt is a sample application that shows how easy it is to measure general distances. Just point a Project Tango device at two or more points. No tape measures and step ladders required.

Constructor is a sample 3D content creation tool where you can scan a room and save the scan for further use.

Tangosaurs lets you walk around and dig up hidden fossils that unlock a portal into a virtual dinosaur world.

Tango Village and Multiplayer VR are simple apps that demonstrate how Project Tango’s motion tracking enables you to walk around VR worlds without requiring an input device.

Tango Blaster lets you blast swarms of robots in a virtual world, and can even work with the Tango device mounted on a toy gun.

We also showed a few partner apps that are also now available in Google Play. Break A Leg is a fun VR experience where you’re a magician performing tricks on stage.

SideKick’s Castle Defender uses Project Tango’s depth perception capability to place a virtual world onto a physical playing surface.

Defective Studio’s VRMT is a world-building sandbox designed to let anyone create, collaborate on, and share their own virtual worlds and experiences. VRMT gives you libraries of props and intuitive tools, to make the virtual creation process as streamlined as possible.

We hope these applications inspire you to use Project Tango’s motion tracking, area learning and depth perception technologies to create 3D experiences. We encourage you to explore the physical space around the user, including precise navigation without GPS, windows into virtual 3D worlds, measurement of spaces, and games that know where they are in the room and what’s around them.

As we mentioned in our previous post, Project Tango Tablet Development Kits will go on sale in the Google Store in Denmark, Finland, France, Germany, Ireland, Italy, Norway, Sweden, Switzerland and the United Kingdom starting August 26.

We have a lot more to share over the coming months! Sign-up for our monthly newsletter to keep up with the latest news. Connect with the 5,000 other developers in our Google+ community. Get help from other developers by using the Project Tango tag in Stack Overflow. See what others are creating on our YouTube channel. And share your story on Twitter with #ProjectTango.

Join us on our journey.

Categories: Programming

SE-Radio Episode 236: Rebecca Parsons on Evolutionary Architecture

Johannes Thönes talks to Rebecca Parsons, Chief Technology Officer at ThoughtWorks, about evolutionary architecture. The practice of evolutionary software architecture means making decisions as late as possible (last responsible moment) and setting up cross-functional requirements that the architecture has to meet (architectural fitness function). In the beginning, Parsons and Thönes introduce the term evolutionary architecture and […]
Categories: Programming

Hungry for some Big Android BBQ?

Android Developers Blog - Fri, 08/21/2015 - 17:20

Posted by Colt McAnlis, Head Performance Wrangler

The Big Android BBQ (BABBQ) is almost here and Google Developers will be there serving up a healthy portion of best practices for Android development and performance! BABBQ will be held at the Hurst Convention Center in Dallas/Ft.Worth, Texas on October 22-23, 2015.

We also have some great news! If you sign up for the event through August 25th, you will get 25% off when you use the promotional code "ANDROIDDEV25". You can also click here to use the discount.

Now, sit back, and enjoy this video of some Android cowfolk preparing for this year’s BBQ!

The Big Android BBQ is an Android combo meal with a healthy serving of everything ranging from the basics, to advanced technical dives, and best practices for developers smothered in a sweet sauce of a close knit community.

This year, we are packing in an unhealthy amount of Android Performance Patterns, followed up with the latest and greatest techniques and APIs from the Android 6.0 Marshmallow release. It’s all rounded out with code labs to let you get hands-on learning. To super-size your meal, Android Developer instructors from Udacity will be on-site to guide users through the Android Nanodegree. (Kinda like a personal-waiter at an all-you-can-learn buffet).

Also, come watch Colt McAnlis defend his BABBQ “Speechless” Crown against Silicon Valley reigning champ Chet Haase. It'll be a fist fight of humor in the heart of Texas!

You can get your tickets here, and we look forward to seeing you in October!

Join the discussion on

+Android Developers
Categories: Programming

Stuff The Internet Says On Scalability For August 21st, 2015

Hey, it's HighScalability time:


Hunter-Seeker? Nope. This is the beauty of what a Google driverless car sees. Great TED talk.
  • $2.8 billion: projected Instagram ad revenue in 2017; 1 trillion: Azure event hub events per month; 10 million: Stack Overflow questions asked; 1 billion: max volts generated by a lightening strike; 850: apps downloaded every second from the AppStore; 2000: years data can be stored in DNA; 60: # of robots needed to replace 600 humans; 1 million: queries per second with Nginx, Ubuntu, EC2

  • Quotable Quotes:
    • Tales from the Lunar Module Guidance Computer: we landed on the moon with 152 Kbytes of onboard computer memory.
    • @ijuma: Included in JDK 8 update 60 "changes GHASH internals from using byte[] to long, improving performance about 10x
    • @ErrataRob: I love the whining over the Bitcoin XT fork. It's as if anarchists/libertarians don't understand what anarchy/libertarianism means.
    • Network World: the LHC Computing Grid has 132,992 physical CPUs, 553,611 logical CPUs, 300PB of online disk storage and 230PB of nearline (magnetic tape) storage. It's a staggering amount of processing capacity and data storage that relies on having no single point of failure.
    • @petereisentraut: Chef is kind of a distributed monkey-patching festival running as root.
    • @SciencePorn: If you were to remove all of the empty space from the atoms that make up every human on earth, all humans would fit into an apple.
    • SDN for the cloud: Most of the concepts presented in the papers have been put into practice in Microsoft cloud infrastructures. As a result of these improvements, modern Azure services can carry up to 1,400,000 SQL databases. Moreover, a typical Azure event hub sees as high as 1 trillion events per month.

  • On the Alphabet Google reorg...what Horace Dediu has to say on functional vs divisional organizations may provide insight. A functional organization, which is used by the Army and Apple, prevents cross divisional fights for resources and power. Those are the kind of internal politics that kill a company. Why not just sidestep all that?

  • Here's how Pinterest shards MySQL to scale: All data needed to be replicated to a slave machine for backup, with high availability and dumping to S3 for MapReduce...You never want to read/write to a slave in production...Slaves lag, which causes strange bugs; I still recommend startups avoid the fancy new stuff — try really hard to just use MySQL. Trust me. I have the scars to prove it...We created a 64 bit ID that contains the shard ID...To create a new Pin, we gather all the data and create a JSON blob...A mapping table links one object to another...there are three primary ways to add more capacity...more RAM...open up new ranges...move some shards to new machines...This system is best effort. It does not give you Atomicity, Isolation or Consistency in all cases...We stored the shard configuration table in ZooKeeper...This system has been in production at Pinterest for 3.5 years now and will likely be in there forever. 

  • Nobody expects the quadruple fault! Google loses data as lightning strikes: four successive lightning strikes on the local utilities grid that powers our European datacenter caused a brief loss of power to storage systems...only a very small number of disks remained affected, totalling less than 0.000001% of the space of allocated persistent disks...full recovery is not possible.

  • Flxone upgraded to Go version 1.5 and reduced their 95-percentile garbage collector from 279 milliseconds down to just 10 ms, a 96% decrease in garbage collection pause time. Average request latency dropped by 53%. I wonder now if they can reduce the number of nodes required to meet their SLA? And would the results hold if they wrote their app more natively, that is to generate garbage?

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: Architecture

You Are Not Agile . . . If You Do Waterfall

The spiral method is just one example of a Agile hybrid.

The spiral method is just one example of a Agile hybrid.

Many organizations have self-titled themselves as Agile. Who wouldn’t want to be Agile? If you are not Agile, aren’t you by definition clumsy, slow or dull? Very few organizations would sign up for those descriptions; however, Agile in the world of software development, enhancements and maintenance means more than being able to move quickly and easily. Agile means that a team or organization has embraced a set of principles that shape behaviors and lead to the adoption of a set of techniques. When there is a disconnect between the Agile walk and the Agile talk, management is often a barrier when it comes to principles and practitioners are when it comes to techniques. Techniques are often deeply entrenched and require substantial change efforts. Many organizations state they are using a hybrid approach to Agile to transition from a more classic approach to some combination of Scrum, Kanban and Extreme Programming. This is considered a safe, conservative approach that allows an organization to change organically. The problem is that this tactic rarely works and often organizations get stuck. Failure to spend the time and effort on change management often leads to hybrids frameworks that are neither fish nor fowl.  Those neither fish nor fowl frameworks are rarely Agile. Attributes of stuck (or potentially stuck) organizations are:

The iterative waterfall. The classic iterative waterfall traces its roots to the Boehem Spiral Model. In the faux Agile version of iterative development, short, time-boxed iterations are used for each of the classic waterfall phase. A requirements sprint is followed by a design sprint, then a development sprint and you know the rest. Both the classic spiral model or the faux Agile version are generally significantly better than the classic waterfall model for generating feedback and delivering value faster; therefore, organizations stop moving toward Agile and reap the partial rewards.

Upfront requirements. In this hybrid approach to Agile, a team or organization will gather all of the requirements (sometimes called features) at the beginning of the project and then have them locked down before beginning “work.” Agile is based on a number of assumptions about requirements. Two key assumptions are that requirements are emergent, and that once known, requirements decay over time. Locking product backlogs flies in the face of both of these assumptions, which puts teams and organizations back into the age of building solutions that when delivered don’t meet the current business needs. This approach is typically caused when the Agile rollout is done using a staggered approach beginning with the developers and then later reaching out to the business analysts and business. the interface between groups who have embraced Agile and those that  have not often generates additional friction, often blamed on Agile making further change difficult.

Testing after development is “done.” One of the most pernicious Agile hybrids is testing the sprint after development is complete. I have heard this hybrid called “development+1 sprint.” In this scenario a team will generate a solution (functional code if this is a software problem), demo it to customers, and declare it to be done, and THEN throw it over the wall to testers. Testers will ALWAYS find defects, which requires them to throw the software back over the wall either to be worked on, disrupting the current development sprint, or to be put on the backlog to be addressed later. Agile principles espouse the delivery of shippable software (or at least potentially shippable) at the end of every sprint. Shippable means TESTED. Two slightly less pernicious variants of this problem are the use of hardening sprints or doing all of the testing at the end of the project. At least in those cases you are not pretending to be Agile.

How people work is the only cut and dry indicator of whether an organization is Agile or not. Sometimes how people work is reflection of a transition; however, without a great deal of evidence that the transition is moving along with alacrity, I assume they are or will soon be stuck. When a team or organization adopts Agile, pick a project and have everyone involved with that project adopt Agile at the same time, across the whole flow of work. If that means you have to coach one whole project or team at a time, so be it. Think of it as an approach that slices the onion, addressing each layer at the same time rather than peeling it layer by layer.

One final note: Getting stuck in most of these hybrids is probably better than the method(s) that was being used before. This essay should not be read as an indictment of people wrestling with adopting Agile, but rather as a prod to continue to move forward.


Categories: Process Management

Polymer Summit Schedule Released!

Google Code Blog - Thu, 08/20/2015 - 20:12

Posted by Taylor Savage, Product Manager

We’re excited to announce that the full speaker list and talk schedule has been released for the first ever Polymer Summit! Find the latest details on our newly launched site here. Look forward to talks about topics like building full apps with Polymer, Polymer and ES6, adaptive UI with Material Design, and performance patterns in Polymer.

The Polymer Summit will start on Monday, September 14th with an evening of Code Labs, followed by a full day of talks on Tuesday, September 15th. All of this will be happening at the Muziekgebouw aan ‘t IJ, right on the IJ river in downtown Amsterdam. All tickets to the summit were claimed on the first day, but you can sign up for the waitlist to be notified, should any more tickets become available.

Can’t make it to the summit? Sign up here if you’d like to receive updates on the livestream and tune in live on September 15th on polymer-project.org/summit. We’ll also be publishing all of the talks as videos on the Google Developers YouTube Channel.

Categories: Programming