Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!
Software Development Blogs: Programming, Software Testing, Agile Project Management
Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!
There is a quote from George Box that is used - misused actually - by many populist¬†blogers and authors that goes like this...
All models are, some are useful
Of course this quote is used to avoid asking and answering questions around models, forecasting, and assessment of possible future states of systems, a project being a system.
The actual quote is from Science and Statistics, George E. P. Box, Journal of the American Statistical Association, December 1976, pp. 791-799.
But how simple? That's always the challenge.¬†
The nonsense that #NoEstimates is the answer is of cource just as foolish as overparameterized and overelaborated models. This of course is lost on both sides of the modeling¬†discussion.¬†Related articles Models ... Calling BS on "All Models Wrong" Everything is a Random Variable Models All models are wrong
I've concluded the #NoEstimates advocates have difficulty stating the domain and context in which their exploration¬†is applicable. Without a Domain and a Context in that Domain, it's difficult to have a conversation about the merits of the idea.
Can I apply it to embedded flight control systems? How about emerging databases for nuclear safety and safeguards? Enterprise health insurance processing? How about pipeline right of way document management? Maybe editorial systems in media? Turbine controls, emergency shutdown, fire and gas monitoring? Radar signal processing?
You see the problem, right? General claims like making estimates is a waste of time¬†seem to beg the question - in what domain? Answer comes back Software Development is a domain. Ah, not really. Kinda ends the discussion doesn't it.Related articles Complex Problems Require Better Solutions Estimates and all that... Estimate Anything
Shim has asked a question - a recurring question - about having a PMP. Since everyone is selling¬†all the time - it's part of being human. There are those in the community that encourage a PMP. There are those who oppose the PMP on principles. There are those selling alternatives.
Here's my sales pitch for a PMP
The #NoEstimates notion has merit in the right domain. If you're looking for motivation for estimates outside of the domain where the customer doesn't really care about the final cost, just the final product developed inside a "level of effort" budget, look here and remember.
When you are spending other people's money - billions of dollars of other people's money - you'd better have a credible answer when they ask "how much is this going to cost and when will you be done? This is separate from "will I get what I asked for?" That's another topic all together.
PMI and most other project management educators¬†use the notion of the Triple Constraint - Cost, Schedule, and Scope. This is sometimes called the Iron Triangle. It is call Iron¬†because when you change one thing, the other two change as well. This has been talked about here before.
Cost we know about. Cost starts with the Budget and then includes the Actual Cost¬†of performing the work - consuming that budget. Schedule is the same. We have a Plan¬†and we execute the project against that Plan. As the project proceeds, we are either on plan¬†or off plan. That is we are ahead¬†or behind¬†schedule.
Scope is supposed to be a description of the deliverables from the project. The Outcomes¬†of the projects schedule and budget. But in the PMI vernacular, scope fails to explicitly include several things.¬†
These are all wrapped into the word scope. Here's a better way to look at the Triple Constraint.
Here is a handbook for developing credible¬†Integrated Master Schedules. There are many suggestions but this one comes from the place where schedules are critical.The Train Wreck of all Projects Starts With These Problems Scheduling Guides Unrealistic Cost and Schedule Estimates Uncertainty is the Source of Risk Probabilistic Cost and Schedule Processes
There is a popular quote from George Box about all models are wrong, but some models are useful. This quote is many times used by people suggesting some new or innovative approach to a problem that has been around long time.
While this quote has a pithy ring to it, I'm almost certain those using the quote do so without actually reading the book where is was used.
In the early 1970s econometric models had been constructed for a number of countries using time series data. They were largely static, with responses to ¬†change assumed to take place within one period, irrespective of whether it was a year or a quarter.
The time series models of Box and Jenkins stood in stark contrast to these naive and static models. The Box and Jenkins used a single variable in isolation and dynamics played the central role. A few studies compared the two approaches and concluded that univariate time series models provided the more accurate short-term forecasts.
This was a turning point because it implied that dynamics were more important for understanding short-run movements than economic relationships as then understood. The emphasis in time series econometrics therefore shifted from modelling large simultaneous systems to taking account of dynamic interactions.
Box and Jenkins were influential not so much because what they said was new, but because they said it well. Their contribution was to show how the dynamic properties of observed series could be matched to those of theoretical models.¬†
The Project Management Point
Models are a critical component of any credible forecast of cost, schedule, and technical performance. Without these models it is actually Guessing¬†as so many critics of estimating are fond of stating. With credible models, forecasting becomes a tool used to increase the probability of project success.
So next time some self-proclaimed person uses Box's quote, ask if they have his book in the shelf and on what page that quote appears.Related articles Models All models are wrong George Box, the Accidental Statistician
I was presenting at the College of Performance Management conference in Florida this week. The topic of credible estimating is always at the top of the list. Here's a phrase that says it all for anyone thinking twice about estimating, not making estimates #NoEstimates, or simply being lazy about how they are estimating
We don't make predictions about the future because we want them to come true. But because we want them to not come true. -¬†Professor Frank Anbari. "Earned Value Project Management Method and Extensions," Project Management Journal, December 2003.
I'll get the attribution from the person who told me this. In the end, to be credible we must manage with our eyes wide open. If you don't have a credible estimate of what it is going to cost, when you are going to be done, and what features you will be able to deliver, in what order, then you'd better find a customer that doesn't really care about those things.Related articles Probabilistic Cost and Schedule Processes Credible Estimating Processes Measurement of Uncertainty How to be a Master Scheduler
The more a man is imbued with the ordered regularity of all events, the firmer becomes his conviction that there is no room left by the side of this ordered regularity for causes of a different nature¬†- Albert Einstein, in Ideas and Opinions, from an address at Princeton Theological Seminary, May 19, 1939.
When I hear people talking at #NoEstimates¬†I have to smile. When someone is paying you to spend their money and produce something of value with that money. When that someone has an expectation of that value arriving sometime in the future. You're response, probably shouldn't be we don't do estimates, just give us the money and we'll spend it.
That's a bit over the top I know. Ron Jeffries has a rational approach to actually developing the software, ONCE¬†the needed capabilities have been defined.
But without some high level estimating processes, the software development efforts are considered Level of Effort. Which means keep going until the money runs out. You can certainly¬†do this using the methods defined by Ron. It's called maintenance. PayPal does this. But you have to pick your projects carefully.
How many legs does a dog have if you call the tail a leg? Four. Calling a tail a leg doesn't make it a leg
- Abraham Lincoln
The notion that we can rename things and then assume they are the same doesn't work, just like Lincoln said. The notion of measuring progress with the passage of time and calling it Earned Value, doesn't mean we're doing Earned Value.
Doing Agile without doing Scrum, XP, Crystal, or DSDM is not doing agile.
The LinkedIn forum The Project Manager Network - #1 Group for Project Managers¬†which I sometimes wonder if that is actually the case, has a conversation going on around project failure. A Master's student asked about project failure modes. That led to the use of Earned Value. One participants made some pretty boneheaded statement around EV:
5% (2 hrs)for applying the border and title
20% (8 hrs) for defining the area - boundaries and overall dimensions
30% (12 hrs) for showing all the equipment
5% (2 hrs) for the drawing being checked
10% (4 hrs) for it being Issued for Approval
15% (6 hrs) for it being Issued for Bid
15% (6 hrs) for it being Issued for Construction.
100% = 40 hrs.¬†
So, silly me, I responded that determining physical percent complete was a straight forward process once you have Quantifiable Backup Data¬†or measures of Physical Percent Complete, or even better, Technical Performance Measures. Let's look at each concept
Then the final come back from the OP.
If you're doing a drawing and it takes 40 hours to do the drawing and no two drawings are alike, how do you determine what percentage complete you are? When estimates of the work to be completed are put together, they will estimate X drawings at 40 hrs/drawing. They will have different standards by discipline. Will a person ever report 41% complete on a drawing? Most likely not. It will be shown as 40%. Collectively all the drawings may be 41%, but not individual. No matter how you look at it, it is still an estimate.
First measuring progress to plan by the number of hours consumed is called Level of Effort. Second LOE is a very poor measure of anything other than the passage of time and consumption of resources.
In the end the poster was insistent that ANSI-748-B is just a guide, and doing EV was a local¬†issue. No doubt that is true in his domain. But as a practitioner of EVM using ANSI-748-B for programs subject to DCMA Validation, as well as engagements with the DOD office that own the¬†Earned Value Management processes for the DOD, I'd be laughed out of the room with an approach like that.
For a summary Earned Value in a Nutshell¬†using a similar drawing example.
So Here's The Way TO Do Earned Value
That's all there is - essentially - for doing Earned Value Management. Of course if you read ANSI-748-B, the NDIA Earned Value Management Intent Guide, and the DCMA Intent Guide, there is a lot more.
You Can't Make Stuff Up
OK, you can. But if you do, it makes having a conversation about EV much harder.¬†Related articles What is Earned Value? Definitions, Yes They Are Important In Earned Value, Money Spent is Not a Measure of Progress! Rings of Success Earned Value says its Name Issues with Deploying Earned Value Management
The conversation on Twitter around No Estimates #NoEstimates brings a smile. Here's some samples
That's called an estimate. A rough estimate for sure, but it's an estimate
From the same blog (a good one BTW)...
Now we're into the realm of actual estimating
Are No Estimates Better Than Estimates?
Before answering this another question needs to be answered...
What is the value at risk?
This means how much money and time are you willing to risk without understanding how much time and money is at risk.
So before anyone can answer the question about estimating value, that question needs to be answered by the owner¬†of the money. NOT to consumer¬†of the money.¬†
This at risk¬†question rarely comes up in the agile world. It doesn't come up enough in our formal FAR/DFARS acquisition work either. The at risk¬†question is a risk management¬†question. Agile claims to be a risk management¬†process - which of course is laughable in our DOD/NASA/NRC/FDA/NNSA risk management world. All software intensive by the way.
So before venturing further, what's the value at risk¬†must be answered. Then the owner¬†of that risk, can tell those who should¬†be working in the best interest of the owner of the risk, if an estimate is needed. Not the other way aorund.
Without this answer, the discussion on #NoEstimate is just sharing of personal opinions in the absence of any value at risk.Related articles Time to Revisit The Risk Discussion When We Say Risk What Do We Really Mean? More Uncertainty and Risk Measurement of Uncertainty Estimates in Software Development. New Frontiers. Risk Management
There is a twitter discussion going on around Neil's post of People Need Estimates. This is a typical agile approach. No real domain or context, just a principle looking for a problem to solve.¬†
First let's establish some ground rules:
And the final one ...
Neil says "People need certainty about what they will get and how much they have to spend. " This is not factually true in many domains, not is it actually possible. Those with the money need to know the confidence in the spend plan and the project duration. Certainty is not possible. But don't use that as an excuse for not having a confidence interval estimate:
This is what those spending the money need to know. Otherwise the project is Level of Effort. Work producing good things till the money runs out, get some more money continue doing good things. No problem. That is called maintenance. This is how PayPal uses Scrum. They budget annually for features, develop those features within that budget, repeat forever.
I love the notion "When estimating a date or cost you are creating uncertainty around those things, because you are guessing. "
Well stop guessing. Start being a software professional and start estimating. Have you done this before? How long did it take. Has anyone you know done it before? Go ask them. Guessing is lame. No estimates is even lamer. Worse, it's just being lazy.¬†
One last thing, "People still¬†need¬†500-page business requirement documents." This is bogus. We work multi billion programs, with 100's of millions of dollars of embedded sofwtare and don't have 500 page requirements documents. Nonsense.¬†
If you don't know where you are going, if you don't have some notion of what done looks like, if you can't tell me approximately how much and how long, please don't start. You can be a binary search process for duration and cost for any tangible piece of software and within five question get without 20%. That's good enough to start. So start and take corrective actions with ne estimate along the way.
Finally "However, I would argue that #NoEstimates gives¬†greater certainty¬†than estimating does." There is absolutely no evidence for this. This is pure speculation and therefore also nonsense. Think about it,
Are you out of your !@#$'ing mind. It is no different for software. Here's a nice short intro from Target Process, that actually makes sense.
Let's say the CIO comes and says "I need these features", and asks "how much and how long?" You say "Oh we get better outcomes when we don't estimates." Really, and some one believes that nonsense? I guess so.Related articles Probabilistic Cost and Schedule Processes Estimates and all that... Cost (and Schedule) Estimating Foundations A Point Measures Need A Variance
When the role of the project manager is to push the rock up the hill¬†and then let it roll down again, there is little chance of success, and in the end little need for project management or the project manager. Cancel the project, everyone just go home.
Risk Management is how adults management projects - Tim Lister
There have several rounds of how to use analogies and how not to use analogies in the past few years.¬†
These involved notions like agile is an untended garden. Actually and untended garden and called a weed patch.
Or we can't really stop and develop a strategy, because we're always putting out fires. Actually firemen rarely put out fires. They spend the majority of their time preventing¬†fire with fire safety, inspections, fire inspections. Their job is to never have fires start.
And of course for false analogies of the double pendulum as a stand in for chaos and unpredictability. Since the equations of motion are easily defined - an exercise for any upper division physics student - and a MathLab plugin, you can plot the path of the double pendulum.¬†
And my favorite the attractor¬†analogy, in which it is presumed that some how in chaos theory the attractor¬†attracts. Without understanding ¬†that those pretty pictures of the attractor¬†are the result of the underlying equations for the system.
So with that in mind, there is a new book from one of my favorite authors, Douglas Hofstadter, Surfaces and Essences. In the book Hofstadter makes the case for analogy as the fuel for creative thinking. Using Robert Oppenheimer's quote...¬†
whether or not we talk about discovery or of invention, analogy is inevitable in human thought, because we come to new things in science with what equipment we have, which is how we have learned to think, and above all how we learned to think about the relatedness of things.
But as always we need to take care to assure that those analogies we use to expand the conversation, don't violate the laws of physics, gardening, or mathematics.Related articles The Misunderstanding of Chaos - Again Happy 50th Anniversary, Chaos Managing in the presence of all things
More discussion on LinkedIn about risk, risk management, types of risk, and definitions of risk and opportunity. The "risk management" community in the domains I work is based on Department of Defense, Department of Energy, Department of Homeland Security,¬†NASA, ITIL,¬†AACEI (Association for the Advancement¬†of Cost Engineering International), Nuclear¬†Regulatory Commission,¬†INCOSE (International Council of Systems Engineering), OGC Management of Risk, GAO, and Deloitte's Risk Management Framework.
Several core concepts get confused. So before taking any deep dive into "managing risk," these need to be established.
What is Risk
There are many definitions of risk from a variety of sources. In the end these definitions are all pretty much the same.¬†
Here's a good definition that can be used in probably any domain.
Risk refers to the uncertainty that surrounds future events and outcomes. It is the expression of the likelihood (probability of occurrence)¬†and (probability of)¬†impact of an event with the potential to influence the achievement of an organization‚Äôs objectives. - Managing Risk in Government: An Introduction to Enterprise Risk Management, IBM Center for The Business of Government.
I've edited¬†this a bit to remove the likelihood¬†measure and replace it with the probability¬†just to keep the math straight. You'll see why below when it comes to ordinal¬†values in the risk matrix¬†- a bad idea.
There is a time honored¬† approach to ranking a¬†risk, by calculating some number. This number is meanigless of course, because it has no scale, it is just a ordinal¬†number. We need cardinal¬†numbers for actual risk management.
Risk = Probability of Occurrence x Impact ¬†
This is a trick question. The risk matrix approach that results from this calculation treats these numbers as likelihood¬†not probabilities. But in fact they are probabilities. As well they are Ordinal¬†numbers (1, 2, 3, 4, 5 usually) and as such¬†represents the relative¬†ranking of the likelihood¬†of occurrence, not the actual probability¬†of occurrence.
This approach starts with a major problem - a show stopper problem. The two values on either side of the multiplication sign are probability distributions, not real numbers. There is no¬†MULTIPLY¬†operator between two probability distributions. The is the CONVOLUTION ¬†operator, but that is not likely to be of much value to ordinary project management staff. One starting point for a better understanding of the problems with the risk multiplier and risk matrix¬†approach is the work of Tony Cox. Optimal Design of Qualitative Risk Matrices to Classify Quantitative Risks. Tony's source paper provides the basis for this discussion, "What's Wrong with Risk Matricies."
To put it in blunt terms.
In this link, it is stated again, so I'll restate it here
It is a category mistake to multipy ordinal¬†numbers
By category error it means the objects, the categories, do not have the proper properties to work with the operators. For some more discussion of this topic and a bit of counter discussion, see What's Right with Risk Matricies.
Where does risk come from?
Risk comes from uncertainty. There are two kinds of uncertainty.
The critical point here is the reducibility and the irreducibility¬†of these uncertainties.¬†
How is Risk Maanged?
We must start with a risk management process. There are several, but they are essentially the same. Here's my favorite, there are others of course, but this one covers all the steps.
Cost estimating methods have been around for a long time. The current processes found in agile use a points¬†system, sometimes a Fibonacci series to bin¬†the values of the points.
The challenge with this approach is the estimate in agile is not monetized so we can't really tell if the Total Allocated Budget (TAB) is sufficient for the project at any point in time, unless the capacity for and the quality of the ourcomes is steady - that is Level of Effort.¬†
With the LOE approach, the capacity for work is the critical measurement needed for estimating the cost at completion. As well continuous updating of this capacity for work is needed and correctly done on good agile projects.
But there are other issues with this LOE approach on larger projects:
So the examples like that found at Projects @ Work, don't really consider any of the underlying uncertainties in estimating. Without the next level down - statistically adjusted estimates of the work effort, the capacity for work, the quality of that work, and the interdependencies between those work activities and their products, the simple and maybe even simple minded approaches to estimating have limits to scaling.
This is one of those topics where everyone is right in some way, depending on the domain, context in the domain, and scale of that domain. As agile enters the larger acquisition community, where we're spending other peoples money - maybe 100's of millions of dollars, care needs to be taken when applying un-monetized, non-probabilistic, non-joint probability (cross correlations between work element) and non-stochastic forecasting models. The real¬†world is not that simple.Related articles Deterministic versus Stochastic Trends in Earned Value Management Data Probabilistic Cost and Schedule Processes Uncertainty is the Source of Risk Time to Revisit The Risk Discussion When We Say Risk What Do We Really Mean? Complex Problems Require Better Solutions A Point Measures Need A Variance
Over the past weeks there have been several discussions on the forums and Blogs amount risk and risk management. Here's a short post on those topics and their impact on project performance.
Risk Comes from Uncertainty
Risk does not exist by itself. Risk is created when there is uncertainty. If I am certain that it is going to rain this afternoon, then there is no risk of rain. It's going to rain with 100% probability. There is no uncertainty about the forecast of rain.
So first we need uncertainty to have a risk. But there are two classes of uncertainty:
Aleatory uncertainty refers to the inherent uncertainty due to the probabilistic variability.
This type of uncertainty is Irreducible, in that there will always be variability¬†in the underlying variables.
These uncertainties are characterized by a probability distribution.
The parameter that is being measured - duration, RPM, discharge from a river flow, is stochastic and cannot be reduced.
Handling Risks Created from Uncertainty
Aleatory uncertainty and the resulting risk is modeled with a Probability Distribution Function. This PDF describes all the possible values the process can take and the probability of each value. For a single toss of a coin, there is a 50% probability it will be either heads or tails. For multiple tosses of a fair¬†coin the probability distribution of the total number of heads or the total number of tails is a binomial distribution that ¬†looks like this for the numbers of HEADs from¬†fair¬†coin¬†being tossed 20 times.
The PDF for the possible durations for the work in the project can be determined in several ways. It turns out we can buy¬†knowledge about aleatory uncertainty through Reference Class Forecasting¬†and past performance modeling. This new information then allows us to update - adjust - our past performance on similar work will provide information about our future performance. But the ¬†underlying processes is still random, and our new information simply created a new aleatory uncertainty PDF.
Epistemic risk is modeled by defining the probability that the risk will occur, the time frame in which that probability is active, and the probability of an impact or consequence from the risk when it does occur.
Risk statements are used to define and model these event based risk:
Risk Management is how Adults Manage Projects
One of the posters stated what would be considered a Lame¬†response to the processes and seeming conplexity of managing risks on non-trivial projects, by stating you're making this to complex - Just Do It. It was lame. Here's the response to those who objective in what ever way to doing risk management.
First answer the question what is the value at risk for your project? Don't know? Go find out. Then ask the project sponsor or the person giving you money to manage the project, if they would be willing to lose that money outright. Just write it off¬†when the risk comes true. Probably not would be the answer. So go do the risk management process.¬†
Here's Tim Lister's advice. The section title is Lister's quote and should be used every time some lame response comes back about risk management.Four components of risk Time to Revisit The Risk Discussion Uncertainty is the Source of Risk Deterministic versus Stochastic Trends in Earned Value Management Data