Warning: Table './devblogsdb/cache_page' is marked as crashed and last (automatic?) repair failed query: SELECT data, created, headers, expire, serialized FROM cache_page WHERE cid = 'http://www.softdevblogs.com/?q=aggregator/sources/6' in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc on line 135

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 729

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 730

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 731

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 732
Software Development Blogs: Programming, Software Testing, Agile, Project Management
Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Herding Cats - Glen Alleman
warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/common.inc on line 153.
Syndicate content
Deep Learning Resources about Performance-Based Project Management¬ģ And the Principles, Practices, and Processes to Increase Probability of Success
Updated: 2 hours 35 min ago

Quote of the Day

Thu, 07/21/2016 - 19:03

A skeptic will question claims, then embrace the evidence. A denier will question claims, then reject the evidence. - Neil deGrase Tyson

Think of this whenever there is a conjecture that has no testable evidence of the claim. And think ever more when those making the conjectured claim refuse to provide evidence. When that is the case, it is appropriate to ignore the conjecture all together 

Categories: Project Management

Assessing Value Produced in Exchange for the Cost to Produce the Value

Wed, 07/20/2016 - 13:35

A common assertion in the Agile community is we focus on Value over Cost.

Both are equally needed. Both must be present to make informed decisions. Both are random variables. As random variables, both need estimates to make informed decisions.

To assess value produced by the project we first must have targets to steer toward. A target Value must be measured in units meaningful to the decision makers. Measures of Effectiveness and Performance that can monetized this Value.

Value cannot be determined without knowing the cost to produce that Value. This is fundamental to the Microeconomics of Decision making for all business processes.

The Value must be assessed using...

  • Measures of Effectiveness - is an Operational measure of success that is¬†closely related to the achievements of the mission or operational objectives evaluated in the operational environment, under a specific set of conditions.
  • Measures of Performance - is a Measure that characterize physical or functional attributes relating to the system operation, measured or estimated under specific conditions.
  • Key Performance Parameter - is a Measure that represents the capabilities and characteristics so significant that failure to meet them can be cause for reevaluation, reassessing, or termination of the program.
  • Technical Performance Measures - are Attributes that determine how well a system or system element satisfies or expected to satisfy a technical requirement or goal.

Without these measures attached to the Value there is no way to confirm that the cost to produce the Value will breakeven. The Return on Investment to deliver the needed Capability is of course.

ROI = (Value - Cost)/Cost

So the numerator and the denominator must have the same units of Measure. This can usually be dollars. Maybe hours. So when we hear ...

The focus on value is what makes the #NoEstimates idea valuable - ask in what units of measure is that Value? Are those units of measure meanigful to the decision makers? Are those decision makers accountable for the financial performance of the firm?

 

Categories: Project Management

Quote of the Day

Tue, 07/19/2016 - 15:08

Where You Stand Depends On Where You Sit

This notion of the basis of all good discussions. It is also the basis of discussions that get us in trouble. For example, I sit in a FAR 34.2/DFARS 234.2 Federal Procurement paradigm and a similar paradigm for Enterprise IT based in ISO 12207, ITIL V3.1, CMMI,  or similar governance processes.

Both these domains are guided by a Governance framework for spending other people's money, planning for that spend, performing the work with that money, reporting the progress to plan for that spend to produce the needed value, and taking corrective actions when the outcomes don't match the plan to increase the probability that the needed value (Capabilities) will be delivered for the planned cost to keep the Return On Investment on track.

This paradigm is independent of the software development method - traditional or agile.

If you work where the customer has a low need to know the cost, schedule, or what will be produced for the money, then you likely sit somewhere else. 

Categories: Project Management

Systems, Systems Engineering, Systems Thinking

Mon, 07/18/2016 - 17:17

On our morning ride, the conversation came around to Systems. Some of our group or like me - a techie - a few others are business people in finance and ops. The topic was what's a system and how does that notion impact or world. The retailer in the group had a notion of a system - grocery stores are systems that manage the entire supply chain from field to basket.

Here's my reading list that has served me well for those interested in Systems

  • Systems Engineering: Coping with Complexity, Richard Stevens, Peter Brook, Ken Jackson, Stuart Arnold
  • The Art of Systems Architecting, Mark Maier and Eberhardt Rechtin
  • Systems Thinking: Coping with 21st Century Problems, John Boardman and Brian Sauser
  • Systemantics: How Systems Work and Especially How They Fail, John Gall
  • The Art of Modeling Dynamic Systems: Forecasting for Chaos, Randomness and Determinism, Foster Morrison
  • Systems Thinking: Building Maps for Worlds of Systems, John Boardman and Brian Sauser
  • The Systems Bible: The Beginner's Guide to Systems Large and Small, John Gall
  • A Primer for Model-Based Systems Engineering, 2nd Edition, David Long and Zane Scott
  • Thinking in Systems: A Primer, Donella Meadows

These are all actionable outcomes books. 

Systems of information-feedback control are fundamental to all life and human endeavor, from the slow pace of biological evolution to the launching the latest space satellite ... Everything we do as individuals, as an industry, or as a society is done in the context of an information-feedback system. - Jay W. Forrester

Categories: Project Management

Symptoms versus Root Causes

Mon, 07/18/2016 - 01:01

There is a popular graph showing project performance versus the estimated project performance in "Schedule Estimation and
Uncertainty Surrounding the Cone of Uncertainty," Todd Little, IEEE Software, May/June 2006. 

Screen Shot 2016-07-17 at 4.52.40 PM

This chart (above)  shows data from samples of software development projects and is used by many in the agile community and by #NoEstimates advocates to conjecture that estimates are usually wrong. In fact estimates can easily be wrong for many reasons. But knowing why they are wrong is rarely the outcome of the discussion. 

This approach is missing a critical understanding about assessing the effectiveness of any estimating process. It starts with the notion of an ideal estimate. That is a post hoc assessment of the project. The idea estimate  is only known after  the project is over.

Next is another critical issue. 

Did the projects (in the graph above) overrun the estimate because the estimate was wrong or because the project was executed wrongly?

In our domain of complex projects, many of which are Software Intensive System of Systems, there are four primary Root Causes of Unanticipated Cost and Schedule Growth, shown below.

Screen Shot 2016-07-17 at 5.02.08 PM

The top graph shows samples from the programs in the business. But it does not show WHY those programs ended up  greater than the initial estimates. The graph is just a collection of data after the fact. What was the Root Cause of this unanticipated cost (and schedule) growth. The paper goes on to speak about a Estimating Quality Factor but that is only One of the four core Root Causes of cost growth from the PARCA research. As well the paper mentions other causes of growth, similar some PARCA causes.

But each project in the graph is not assigned a specific (maybe even more than one) cause. Without that assignment it is not possible to determine if estimating was the Root Cause, or as stated above one or more of the three other causes was the reason the project was above the Ideal line. 

In the notion of Ideal I will assume the estimate was Ideal if the project actuals matched the estimate.

The problem here is those advocating estimates are flawed, a waste, not needed, do harm, or are even evil use this chart to support their claim. Without stating what the Cause for the deviation from the Ideal is. Just that it IS. Not Why. There is no basis to make any claims about the issues with estimating. 

Without the Why, NO corrective action can be taken to improve the performance of both the project or any of the four listed (and many other) processes including  the estimating process for the project. This is the fundamental basis of Root Cause Analysis. And it comes down to this ...

Without the Root Cause for the undesirable outcome, no corrective action can be taken. You're just treating the Symptom. Those conjecturing a Cause (say Estimating is the smell of Dysfunction) have no basis on which to make that statement. It's a house built on sand.  Same for the Cone  of Uncertainty - also popularly misused, since the vertical scale is rarely calibrated for the specific domain, instead some broad and uncalibrated range provided for notational purposes.

And as a notional concept the Cone of Uncertainty is useful. As a practical concept, only when the vertical scale is calibrated (Cardinal versus Ordinal) can it be used to assess the uncertanty of the estimates during specific periods of the project. This is another misuse of statistical data analysis.

There are databases that provide information needed to calibrate that chart as well. But that's another post.

For now, take care when seeing the first chart, to ask do you know the cause for the projects that is above the Ideal Line?  No, then all you can say is we don't know why, but there are a lot of projects in our organization that have a Symptom of cost overrun, but we have no way to know why. And therefore we have no way to know how to fix this symptom. Just that we've observed it. But can't fix it.

To start to apply Root Cause Analysis, here is an introduction to the method we use on our Software Intensive Systems.

Root cause analysis master plan from Glen Alleman

 

 

Categories: Project Management

A collection of #NoEstimates Responses

Sun, 07/10/2016 - 19:40

The question of the viability of #NoEstimates starts with a simple principles based notion.

Can you make a non-trivial (NOT de minimis) decision in the presence of uncertainty without estimating?

The #NoEstimates advocates didn‚Äôt start there. They started with ‚ÄúEstimates are a waste, stop doing them.‚ÄĚ Those advocates also started with the notion that estimates are a waste for the developers. Not considering those who pay their salary have a fiduciary obligation to know something about cash demands and profit resulting from that decision in the future.

The size of the ‚Äúvalue at risk‚ÄĚ is also the starting point for estimates. If the project is small (de minimis) meaning if we over run significantly no one cares, then estimating is likely a waste as well. The size of the project, whether small or multi-million‚Äôs doesn't influence the decision to estimate. The decision is determined by ‚Äúvalue at risk,‚ÄĚ and that is determine by those paying NOT¬†by those consuming. So the fact I personally¬†work on larger projects does not remove the principle of ‚Äúvalue at risk.‚ÄĚ Any¬†client‚Äôs (internal or external) V@R may be much different than my personal experience¬†‚Äď but it‚Äôs not our money.

Next comes the original post from Woody ‚Äď ‚Äúyou can make decisions with No Estimates.‚ÄĚ If we are having a principled based conversation (which NE‚Äôer don‚Äôt) then that statement violates the principles of Microeconomics. Making decisions in the presence of uncertainty (and I‚Äôm assuming all projects of interest have uncertainty), then estimates are needed to make decision. Those decisions are based in MicroEcon on the Opportunity Cost and the probability of making the best choice for the project involves assessing the probability outcome of those choices, estimating is required.

Real options is a similar process in IT based on estimating. Vasco stated long ago he was using RO along with Bayesian Decision Making. I suspect he was tossing around buzz words without knowing what they actually mean.

From the business side, the final principle is Managerial Finance. This is the basis of business management of its financial assets. The balance sheet is one place these are found. Since the future returns from the expenses of today and the ‚Äúpotential‚ÄĚ expenses of the future are carried in that balance sheet, estimating is needed there as well for the financial well being of the firm.

These three principles Value at Risk, MicroEconomics of Decision Making, and Managerial Finance are ignored by the NE advocates when they start with the conjecture that ‚Äúdecisions can be made without¬†estimates,‚ÄĚ and continue on with ‚Äúestimates are a waste of developers time, they should be coding not estimating.‚ÄĚ

It‚Äôs the view of the world, that as a developer ‚Äúit‚Äôs all about me.‚ÄĚ Never looking at their paycheck and asking where did the money come from. That‚Äôs a common process and one I did when I started my career 35 years ago as a FORTRAN developer for Missile Defense radar systems and our boss had us take¬†out our paychecks (a paper check in those days) and look at the upper left hand corner. ‚ÄúThat money doesn‚Äôt come from the Bank of America, El Segundo Blvd, Redondo Beach, it comes from the US Air Force. You young pups need to stay on schedule and make this thing work as it says in our¬†contract.‚ÄĚ

In the end, the NE conversation can be about the issues in estimating and there are many - and Steve McConnell speaks to those. I work large federal acquisition programs ‚Äď ¬†IT and embedded systems. And many times the ‚Äúover target baseline‚ÄĚ root cause is from ‚Äúbad estimating.‚ÄĚ But the Root Cause of those bad estimates is not corrected by No¬†Estimating as #Noestimates would have us believe.

As posted on this¬†blog before and sourced from the Director of ‚ÄúPerformance Assessment and Root Cause Analysis,‚ÄĚ unanticipated growth in cost has 4 major sources:

1. Unrealistic performance expectations ‚Äď that will cost more money.
2. Unrealistic cost and schedule estimates ‚Äď the source of poor estimating.
3. Inadequate assessment of risk and unmitigated risk exposure.
4. Unanticipated technical issues.

Research where I work some times (www.ida.org) has shown these are core to problems of cost overruns in nearly every domain from ERP to embedded software intensive systems. It is unlikely those 4 root causes are not universal.

So what‚Äôs #NoEstimates trying to fix? They don‚Äôt say, just ‚Äúexploring‚ÄĚ new ways.‚ÄĚ In what governance frameworks? They don‚Äôt say. They don‚Äôt actually say much of anything, just ‚Äúestimates are waste, stop doing them and get back to coding.‚ÄĚ

As my boss in 1978 reminded us newly minted Master‚Äôs degreed coders, ‚Äúit ain‚Äôt your money it‚Äôs the USAF‚Äôs money, act accordingly ‚Äď give me an estimate of this thing you‚Äôre building can be used to find SS-9‚Äôs coming our way?‚ÄĚ Since then I‚Äôve never forgotten his words, ‚Äúbusiness (in that case TRW) needs to know how much, when, and what, if it‚Äôs going to stay in business.‚ÄĚ

 

Categories: Project Management

Baseball as Incremental and Iterative Development

Sat, 07/09/2016 - 16:50

20160707_190852

Sitting in our seats at last night's Rockie v. Phillies game and dawned on me the analogy between Moneyball strategy and good management of software development. In Moneyball, Billy Beane was faced with a limited budget for players.  He hired a statistics guy and they figured out the getting on base was just as important as hitting a homerun and a hell of alot cheaper.

Last night there were a few home runs. But most of the action were singles and a few doubles. If you do the simple minded math, when the rotation all get singles with batting averages of 0.303, then there'll be runs scored every inning. 4 singles equals a home run. Getting on base is the key to winning games.

Getting incrementally more software out the door - assuming it's the right software needed by the customer and that customer can put the software to work - then progress toward the win is being made.

So the Strawman of Waterfall and Big Bang is just that a Strawman. The Straw Man of No Projects is also nonsense, along with No estimates. The manager of the Rockies has a plan in the presence of uncertanty and emerging situations. That's why he's called the MANAGER because he manages in the presence of uncertainty. And in doing that job he makes estimates on the probability of success of the emerging play options. 

We can learn a lot from Baseball about managing projects. First get on base. You can't score unless you're on base. First, Second, Third, then Home. You can't count on hitting Home Runs to win the game. It doesn't work that way. Offense of good, but so is defense. Managing the risks is defense. Defense in baseball is more than just putting players on the field. It's how those players react when the ball is hit. Go for the out at first? Try for a double play? Hold the ball after a single bounce in the outfield?

While baseball is not a contact sport, it still requires teamwork. I played competitive Volleyball in College - the ultimate Team Sport, since you're only as strong as the weakest player. Much like the software development team. But all teams have a strategy, a game play that changes as the game emerges and most of all - as Peter Kretzman has lambasted some NE advocates who have not likely ever played baseball - all the players are making estimates all the time in order to catch the ball, keep control of the ball and the emerging situation of the game.

Categories: Project Management

Accounting for Software Development - Products or Projects

Fri, 07/08/2016 - 15:33

There appears to be a resurgence in the No Projects conversation, similar to the No Estimates notion that has been around for awhile.

I’m going to suggest that most of the disconnects around ideas of software development ‒ from No Estimates to No Projects to whatever ‒ starts with Developers and the assumption It’s their money. It’s not their money. If it is they can do with it as they please, no one cares.

There are standard business accounting processes in any business that creates products or services -- including Software - for money. Unless the developers are Staff Aug (just labor) to another firm, the accounting processes are defined in several places. FASB 86, FASB 10, even GAAP for capitalization and expenses of the cost of developing that product for use internally and for sale externally.

Here's a start …

The separation of Products from Projects at the software development process level is understandable. I work a Program that has both. Both for good reasons for both. A Product Line enhancement is usually on continuing system¬†‚Äď Version 9 of a legacy system is a Product Line extension of a system that as be in place for 11 years. A ‚Äúnew‚ÄĚ product‚ÄĚ Version 1.0 in the same domain is treated as a Project. The Project is to establish Version 1.0 which will then be extended over its life.

If the No Projects approach goes that same way as the No Estimates approach, those paying for the work will be intentionally excluded from the conversation. When I asked one of the originators of the #NoEstimates conjecture to go check your idea with the CFO, I got silence. 

Follow the Money is advice I received from my colleague ‚Äď former NASA Cost Director

This is good advice for any anyone proposing an initiative that pretends to change the status quo, tilt at the windmill of supposed bad management, or any suggested evil as seen by those spending the money provided by someone else. Remember this when making suggestions...

It's not Your Money, act accordingly. Your opinion should be considered as appropriate, but it's not your money. Those whose money it is have a fiduciary responsibility to spend it in a manner compliant with the accounting principles of the firm, be that private or public.

Categories: Project Management

The Broad Range of Domains of Agile

Thu, 07/07/2016 - 17:18

I work on Agile software development programs. Most everyone in Boulder Colorado works on Agile development programs. We meet once a month or so for coffee and talk about Agile. We have formal MeetUps hosted by local vendors - Rally, Scaled Agile, and other agile development shops.

The range of projects at our morning coffee clutch go from one man shows to multi-billion dollar space flight programs and back again. There are times when we get pissed off at each other for all the right reasons - this is the big ended littled ended argument of Gulliver's Travels that was actually an argument about the bit order in microprocessors when the 32 bit machines started with floating point calculations - Holy Wars and a Plea for Peace. When I was working on embedded systems and we had to choose a 32 bit machine for our new signal processing system, we got in huge fights about how the floating points software was going to be moved over to the floating point hardware.

So the question is what are the shared principles across a broad range of projects, business and technical domains. Here's some background on the domain I work and the application of Agile in that domain. I'm speaking at the Boulder Agile Meetup July 27th if anyone is interested in hearing about Agile in Government.

Here's some background on Agile in the domain I work

Typically these are also Software Intensive System of Systems. Here's some background on those

So when we hear something about exploring new approaches, ask first - in what domain have you tested that idea? By what Principles can that new idea be accepted into a domain outside your domain? Is there any evidence that your new idea can work outside your personal experience? How would I test your idea before spending my customer's money? What would be those test cases?

Categories: Project Management

The Misuse Hofstadter's Law

Thu, 07/07/2016 - 02:35

The misuse of Hofstadter's Law is common in many agile development domains. The quote is ...

It always takes longer than you expect, even when you take into account Hofstadter's Law

This quote is misused to suggest that estimating can't be done. On page 152 of Gödel, Escher, Bach: an Eternal Golden Braid, Hofstadter explains the context and meaning of Hofstadter's Law.

He's speaking about the development of a Chess playing program - and doing so from the perspective of 1978 style software development. The game playing programs use a look ahead tree with branches of the moves and countermoves. The art of the program is to avoid exploring every branch of the look ahead tree down to the terminal nodes. In chess - actual chess, people - not the computer - have the skill to know what branches to look down and what branches to not look down. 

In the early days (before 1978) people used to estimate that it would be ten years until the computer was a world champion, But after ten years (1988) it was still estimated that day was still ten years away. 

This notion is part of the recursive Hofstadter's Law which is what the whole book is about. The principle of Recursion and Unpredictability is described at the bottom of page 152. 

For a set to be recursively enumerable (the condition to traverse the look ahead tree for all position moves), means it can be generated from a set of starting points (axioms), by the repeated application of rules of inference. Thus, the set grows and grows, each new element being compounded somehow out of previous elements, in a sort of mathematical snowball. But this is the essence of recursion - something being defined in terms of simpler versions of itself, instead of explicitly. 

Recursive enumeration is a process in which new things emerge from old things by fixed rules. There seem to be many surprises in such processes ...

So if you work on the development of recursive enumeration based software designs, then yes - estimating when you'll have your program working is likely going to be hard. Or if you work on the development of software that has no stated Capabilities, no Product Roadmap, no Release Plan, no Product Owner or Customer that may have even the slightest notion of what Done Looks like in units of measure meaningful to the decision makers, then probably you can apply Hofstadter's Law. Yourdan calls this type of project A Death March Project - good luck with that.

If not, then DO NOT fall prey to the misuse of Hofstadter's Law by those likely to not have actually read Hofstadter's book, nor have the skills and experience to understand the processes needed to produce credible estimates.

So once again, time to call BS, when quotes are misused

Related articles Agile Software Development in the DOD Risk Management is How Adults Manage Projects I Think You'll Find It's a Bit More Complicated Than That Architecture -Center ERP Systems in the Manufacturing Domain IT Risk Management Why Guessing is not Estimating and Estimating is not Guessing Making Conjectures Without Testable Outcomes Deadlines Always Matter Two Books in the Spectrum of Software Development Seven Principles of Agile Software Development in the US DOD
Categories: Project Management

Mathematical Definitions - Again Misused to Convey Unsubstantiated Information

Tue, 07/05/2016 - 21:58

1280px-Galileos_Dialogue_Title_PageIt's popular in the #NoEstimates community to claim Forecasting is not Estimating. Using English Dictionaries, they build a case using logical like this. It's been repeating nearly continuously since the start of the discussion about how to make decisions in the presence of uncertanty without estimating. 

Salviati: If you have a number of uniform sized tasks & know velocity you can predict ttc (Time to Complete) Simplicio responds: Indeed...gives us much stronger forecasting ability than estimation 

Let's use the Oxford Dictionary of Mathematics, not the High School English Dictionary.

It ain't so much the things we don't know that gets us in trouble. It's the things we know that ain't so - Artemus Ward

Estimate

  • Point Estimate -¬†A single value of a statistic derived from a sample that is taken as an approximation for the population value of that statistic. For example, the sample mean , x , is a point estimate¬†of the population mean, őľ
  • Interval Estimate -¬†An inference made about the range of values within which a population parameter will lie, drawn from values obtained from a random sample.
  • Interval Estimate -¬†An interval within which a parameter under study (such as a relative risk ) is stated to lie with a particular degree of confidence, likelihood, or probability based on an analysis of a study or multiple studies. See also confidence interval.
  • Maximum Likelihood Estimate -¬†The value for an unknown parameter in a model that maximizes the probability of obtaining exactly the data that were observed.
  • Estimation is used to calculate an unknown value.
  • Estimation is the calculated approximation of a result.¬†

Forecast

  • Forecast -¬†Also referred to as prediction. In econometrics, a point forecast is the expected value of the variable of interest, or the dependent variable, conditional on the given values of the exogenous and predetermined variables. An interval forecast is the confidence interval for the point¬†forecast.
  • Forecast -¬†To predict or estimate a future event or trend.
  • Forecast -¬†A statistical synthesis of probabilities and expert opinion that attempts to define an outcome either in terms of numbers or actual courses of action.
  • Sale Forecast -¬†An estimate of future sales volumes and revenue. It is usually based on past trends and takes into account current and future directions.
  • Forecasting -¬†prediction of future events.
  • Forecasting is the process of making a forecast or prediction. The terms forecast and prediction are often used interchangeably but sometimes forecasts are distinguished from predictions in that forecasts often provide explanations of the pathways to an outcome.
  • Forecasting -¬†to calculate or predict (some future event or condition) usually as a result of study and analysis of available pertinent data predict.

Estimating is about the past, present, and future outcomes of a process, a model, or some external observation. We can estimate the number leaves that have fallen and need to be raked come fall, to size how many bags have to be bought at Home Depot (past). We need to estimate the number of people on the Pearl Street Mall right now to assess how long the walk will be to Starbucks from our parking spot (present). We can estimate the number of minutes we'll need to reach our favorite parking spot at Denver International Airport from the office parking lot (future).

Forecasting is Estimating about the future. The weather forecast for tomorrow is 70 degrees and a 30% chance of rain in northern Boulder County (where we live). With this forecast, we can estimate what time we need to tee off to have a high probability of getting all 18 holes of golf on our course in before the rain starts.

Forecasting is estimating about the future

Related articles What does it mean when we say 80% confidence in a number? Taxonomy of Logical Fallacies All Project Numbers are Random Numbers - Act Accordingly How We Make Decisions is as Important as What We Decide. Who Builds a House without Drawings? Managing Projects By The Numbers Critical Success Factors of IT Forecasting Do The Math Humpty Dumpty and #NoEstimates
Categories: Project Management

July 4th 2016

Mon, 07/04/2016 - 15:09

Declaration_Engrav_Pg1of1_doctored_0

Categories: Project Management

Decision Making in the Presence of Uncertainty

Sun, 06/26/2016 - 19:08

There's another rash of Twitter posters supporting the conjecture that decisions can be made about how to spend other people's money without estimating the impact and outcome of that decision. This is a core premise of #NoEstimates from the Original Poster

No Estimates

Here's a starting point to learn that is simply not true...

There are 100's more books, papers, conference proceedings on the topic of decision making in the presence of uncertanty.

It comes down to a simple concept

All project work is uncertain. Reducible uncertainties (epistemic) and irreducible uncertainties (aleatory) are present on all projects. When money is being provided to develop software, those providing the money have a reasonable expectation to know something about when you will be providing the value in exchange for that money, how much money will be required to provide that value, and what capabilities will be provided in exchange for that money. The field of study where these questions and answers live is microeconomics of decision making. Software development decision making is a well studied subset of that field.

When it is conjectured decisions can be made in the presence of uncertanty without estimates - like the Original Poster did, and there is no supporting theory, principle, practices, or evidence of how that can be done - run away - it's a bogus claim. 

Related articles Making Decisions In The Presence of Uncertainty Essential Reading List for Managing Other People's Money Herding Cats: Decision Making in the Presence of Uncertainty Herding Cats: Thoughts on Suggestions That Violate Principles of Finance, Probability, and Statistics Making Conjectures Without Testable Outcomes
Categories: Project Management

#NoEstimates Book Review - Part 1

Sat, 06/25/2016 - 01:17

I've started reading Vasco's book #NoEstimates and will write a detailed deconstruction. I got the Kindle version, so I have a $10 investment at risk. Let's start with some graphs that have been around and their misinformation that forms the basis of the book.

The Chaos Report graph,is the 1st one. This graph is from old 2004 numbers. That's 12 year old numbers. Many times the books uses 12, 16, even 25 year old reports as the basis of the suggestion that Not Estimating fixes the problems in the reports. The Chaos reports have been thoroughly debunked as self selected samples using uncalibrated surveys for the units of measure for project failure. Here's a few comments on the Standish reporting process. But first remember, Standish does not say what the units of measure are for Success, Challenged, or Failure. Without the units of measure, the actual statistics of the projects and the statistical ranges of those projects for each of the three categories, the units as essentially bogus. Good fodder for selling consulting services or for use by those with an idea to sell, but worthless for decision making about the root cause of Failure, Challenged, or even Success. Any undergraduate design of experiments class would have all that information in the public. 

So the 1st thing to read when you encounter data like this is Project Success: A Multidimensional Strategic Concept, Aaron J. Shenhar, Dov Dvir, Ofer Levy and Alan C. Maltz. Only then start to assess the numbers. Most likely, like the numbers in the book, they're not credible to support the hypothesis. Which by the way there is no hypothesis for you can make decisions in the presence of uncertanty without estimating

So let's look further at the difficulties with Standish and why NOT to use it as the basis of a conjecture

A simple google search would have found all this research and many many more. I get the sense V didn't do his homework. The bibliography has very few references to actually estimating, no actual estimating books, papers, or research sites. Just personal anecdotes from a  set of experiences as a developer.

The Standish Report failure mode is described in Darrell Huff's How to Lie With Statistics - self select the samples from the survey. Standish does not provide any population statistics for their survey.

  • How many surveys were set out?
  • How many responded?
  • Of those responding, how many were statistically significant?
  • What does it means in terms of actual measures of performance to the¬†troubled?
  • If the project was over budget, was the needed budget estimate correct?
  • If the project was late, was the original schedule credible?

None of these questions are answered in Standish reports. No Estimate picks these serious statistical sampling error up and uses the, as the basis of the pure-conjecture that Not Estimating is going to fix the problems. This would garner a high school D is the Stats class.

Screen Shot 2016-06-24 at 2.08.13 PM

Next comes a chart that makes a similar error. This reference is from Steve McConnell's book, but is actually from another source. The No Estimates book does a poor job of keeping the references straight. It is common to misattribute a report, a graph, even a phrase. The book needs a professional editor.

The graph below is used to show that estimates are usually wrong. But there is a critical misunderstanding about the data.

  • It starts with a straight line called¬†perfect accuracy/. First there are two attributes of any estimate.¬†Accuracy¬†and¬†precision. There is no such thing as perfect accuracy in the estimating business. An estimate is an approximation of reality.
  • The sample projects show they did not meet the¬†perfect accuracy - whatever that might have been. This knowledge can only be obtained¬†after the work has been done. Either at the end or during the project - cumulative to date.¬†
  • But there are two sources of the variances of¬†estimates and actuals

Screen Shot 2016-06-24 at 2.08.06 PM

I'm in the early parts of the book and already have a half dozen pages of notes for either fallacies, incorrect principles, 30 year old references, and other serious mistakes on understanding on how decisions are made in the presence of uncertainty. My short assessment is 

It's a concept built on sand.

Related articles Myth's Abound All Project Numbers are Random Numbers - Act Accordingly The Failure of Open Loop Thinking How to "Lie" with Statistics Statistical Significance How to Estimate Software Development Capabilities Based Planning IT Risk Management Why Guessing is not Estimating and Estimating is not Guessing Making Conjectures Without Testable Outcomes
Categories: Project Management

Control, Stability, Short Term, Long Term all needed for Success

Tue, 06/21/2016 - 02:27
Single Track

When riding a single track like this one behind our neighborhood, I came to an understand of the tradeoffs between stability and control. In our conference sessions, we speak about how the Wright Brothers learned this concept as well.

Nationals Here's another trail being ridden by our son at Nationals in 2013. who's massively better than I will ever be. But same tradeoff between stability and control is needed to be ranked 33rd Cross Country and 23rd Short Track at DII in the nation, with team taking 2nd overall.

In both cases, nationally ranked and rank amaetur, control of bike is the key. Stability is a relative term, depending on speed, terrain, skill, risk taking tolerance,  experience, technical equipment, and other intangible factors.

We speak at conference where program planning and controls is the topic. Starting 2 years ago, our foundation for speaking about managing in the presence of  uncertanty is the Wright Brothers.

Most earlier experimenters in powered flight focused only on one or two of the primary problems - (a) A set of lifting surfaces, or wings, (b) A method of balancing and controlling the aircraft, and (c) A means of propulsion -  and did not consider the final design from the outset. The Wrights recognized that each of these areas had to be successfully addressed to build a working airplane. They believed that the aerodynamic and propulsion problems would be comparatively easier to solve, so they first concentrated on how to maintain balance and control.

Many believed that air currents were too swift and unpredictable for human reflexes. Therefore, an aircraft had to be inherently stable for the pilot to be able to maintain control.¬†Because of the Wrights‚Äô extensive experience with the bicycle‚ÄĒa highly unstable but controllable machine (just like the mountain bike)‚ÄĒthey saw no reason why an airplane could not be unstable yet controllable as well.

The notion of Control is missed used in software development projects. Misused and misunderstood. Control inside the upper and lower control bounds is a critical success factor. What are those upper and lower control bounds? Good question. They have to be estimated in the start. They become cleared as the project progresses. They have to be adjusted as new information emerges.

In projects, the question for Stability is a false quest. All project work is uncertain. Stability is short lived. New inputs arrive every day. Just like new inputs arrive while riding down the flat trail for cruising around the open space in the neighborhood or more so on the bumpy high speed trail of collegiate racing.

Even if the trail is defined in from of you, the small disruptions and many times big distributions input your control of the bike. The key is Control over Stability. Staying in control will get you across the finish line, down the trail, and home again to ride another day.

Skills of Maintaining Control on the Mountain Bike and the Project

Starting from our house there is a nice loop Left Hand Trail, that is easy, has a few climbs, and a few descents, uncrowded, and provides a nice view of the small hills before the real mountains start . Combine that with the Eagle Trail, Sage Trail, and North Rim Trail and it's a nice 7 mile loop

  1. Short term feedback - what's happening right in front of me? What do I need to do NOW to maintain stability, to survive for the next 10 feet on the trail 
    • On projects,¬†short term feedback is for the¬†next day or week.
    • What's coming due?
    • A Plan of the Week or even a Plan of the Day is ¬†very useful.
  2. Long term feedback - I see things coming on the trail. A big drop. A huge climb. What am I going to do to prepare for both? Do I have a plan of action to get through these?
    • On projects, I need to see this coming before I am¬†Overcome¬†By¬†Events¬†(OBE). The term OBE is used often in our domain. It says, they didn't see it coming. They didn't have a plan to respond to emerging events.
    • The naive term in agile of¬†respond to emerging requirements means you have to have a plan to meet that emerging requirement. If you don't, just like on¬†the bike, you're going to end ¬†up on the side¬†to the trail, landing on a cactus¬† or worse a snake where we live.
  3. Corrective actions - what corrective actions am I capable of performing? On the bike, I must know my limits. Can I power through the steep climb? Maybe if it's short. No likley if it's 1/2 mile at 18% grade.
    • On projects knowing the limits of the team is a critical success factor.
    • Who can work the problem with the most rapid response? Who's got skills and experience needed to¬†power through till we get back in control
  4. Alternative choices - as the trial unfolds in front of me, choices present themselves. Some are optional, some are mandatory alternatives. Other riders coming up the trail have to be addressed. If their coming up hill and I'm going down hill, they have the right-of-way. I can easily start again going down hill. Stopping and starting again going up hill is a real pain. Obstacles on the trail have to be dealt with. In the spring branches hang in th trail from the trees and bushes. Many have sharp leaves or thorns, riding through them is painful.
    • On project having alternatives is also mandatory.¬†
    • Nothing ever goes right as planned. Having an alternative ready to go, means when trouble arrives, there is no delay in making a choice to take another path.
  5. Estimate what's right in front of you - the trail is rough, bumpy, rutted, off camber, muddy, thorny, and many times some animal runs across my path - rabbitt, prairie dog, a snake. A quick estimate is needed to decide what to do. Keep going, speed up, slow down, stop? All depends on the situation.
    1. Estimating on projects is part of the close loop control system.
    2. Both project controls and business control depend on making estimates in the presence of uncertanty about future outcomes.
    3. These estimates keep the project Green, just like estimates on the bike keep me rubber side down.
  6. Estimates of needed for solutions coming up the trail - with short term estimating there are a limited number of choices, bounded by the physical terrain. For longer term choices, there are more options. Do I turn left at the next opportunity, go over the mesa, then back down to rejoin the trial at the parking lot of the hiking trailhead? 
    • For projects a longer view of what's coming is needed.
    • Points of integration or incorporation need to be in the Plan.
    • Alternative Points of Incorporation (APOI) are needed as well.
    • Many time these APOI are part of the master plan, since external facilities may be booked, resources become scarce, technology changes.¬†
    • A ¬†Plan B is needed as part of the normal process
  7. Estimates of needed resources - food, water, spare equipment are part of the normal riding gear. For really big rides, Bear Spray is common, since we live in Bear Country. Energy resources must be managed. Knowing how fast you can go, when is the next resting spot, next shady spot?
    • The resource plan for the project is needed no matter the method used to develop the project.
    • What skills, how many of those skills, the availability of those skills?¬†
  8. Estimates to complete - rarely do we go out without some plan of when we'll be back. This means some estimate of when we'll be back. Telling wife's we're going up to Hygiene and we'll be back in 2 hours is always good protocol. 
    • On projects knowing something about when you plan to finish is part of managing the project.
    • Anyone working on a project that doesn't have some type of deadline is working on a de minimis project - it's too small for anyone to care about when you'll be down.
    • Same for the budget
    • Same for the needed technical performance.
  9. A Plan B and even a Plan C - planning in depth is a good idea when riding anywhere. The longer the ride, the steeper the terrain, the higher the exposure, means having more plans to deal with the emerging situations
    • Without a plan, you're driving in the dark with the lights off.
    • What ever comes up is likely to be a surprise.
    • While surprises are nice for birthday parties, surprises 25 miles out from home, with a 25 mile return can turn unpleasant real quick.
    • Planning in depth is always a good idea.
    • Knowing the¬†Value at Risk¬†defines the magnitude and detail of the plan. If the impact is minimal and can be handled with a few changes, no problem. If the impact is major, having a detailed plan when it occurs is the basis of success
    • Risk Management is How Adults Manage Projects Tim Lister should never be forgotten
  10. Knowledge of capacity for work - how big is a big ride? Good question. When we have friends ask hey want to go for a ride, I have to make sure I have the capacity for work. Our neighborhood has many cyclist. A Silver Medalist rider. A semi-pro road rider. A former pro road rider. A semi-pro mountain biker all the way down to casual riders like me and our group. Know what the ride will demand and what I'm capable of doing is a starting point. Along the way reassessing both those informs decisions about routes, pace, and overall strategy to get home in one piece
    • On projects, if we don't our capacity for work, we can't make informed estimates if we can get to the end in¬†one piece.
    • If we¬†bite off more than we can chew, then we're going to be late and over budget before we start.
    • Know our capacity for work comes from past performance.
    • But it also means estimating what we can do, knowing that past performance, and he future demands for work.
Related articles Late Start = Late Finish Eyes Wide Shut - A View of No Estimates All The World's A Random Process Project Risk Management, PMBOK, DoD PMBOK and Edmund Conrow's Book Architecture -Center ERP Systems in the Manufacturing Domain Staying on Plan Means Closed Loop Control Strategy is Not the Same as Operational Effectiveness Estimating Processes in Support of Economic Analysis Qualitative Risk Management and Quantitative Risk Management
Categories: Project Management

Quick Summary of Estimating Advice

Mon, 06/20/2016 - 23:23

There's lots of misinformation going around again in the #NoEstimates community

  • We can't estimate things we've never done before - this is simply not true. There is not much that hasn't been done before in some form. If you truly haven't done the work before, you're probably not the right person to be estimating for those wanting to pay you.¬†
  • Estimates are guesses, because we don't know that the future is - this is a fundamental error,¬†either of commision¬†or¬†omission) on how estimates are made.
  • Just deliver often in small chunk and we won't need to estimate - this of course ignores the need to produce an Estimate To Complete and an Estimate At Completion.
  • Estimates are a waste, we should be coding to generate value for the customer -¬†the¬†customer has a fiduciary¬†need to know how much it will cost to get that value and when that value will be delivered. This is core business. If there is no need to know estimated cost and schedule, the project is likely a de minimis effort, or the customer has enough money to burn to not care about how much and when.
  • Deadlines kill innovation and reduce quality - this is only true of your a really bad planner. All project work is probabilistic. This probabilistic (and statistical) behaviour mandates a plan to manage in the presence of uncertainty. Bout reducible and irreducible uncertainties have specific¬†handling plans. Either¬†margins¬†to protect from the naturally occurring ¬†variances or specific¬†buy down¬†activities to protect against probabilistic occurrences of undesirable outcomes.¬†

Here's the starting point to clarify and debunk those concepts

  • How To Estimate Almost Any Software Deliverable in¬†90 Seconds¬†- this the starting point for learning how to estimate for all software projects. Is it a¬†bigger than a breadbox, or¬†Twenty Questions approach. It's actually a Binary Search approach, that will¬†get you an¬†85%¬†confidence number in 90- seconds or¬†less. No more excuses for not estimating anything, including things you know nothing about - assuming you are a developer with any experience in the domain. If you have no experience in the domain, then you shouldn't be estimating to begin with - go find someone who is.

Here's some more details, once it's been confirmed you have domain experience

  • How Do You Estimate Work That's Never Been Done Before?¬†- This is a common lament in the #Noestimates community.¬†We've never done this before, so how can we¬†possibly¬†estimate how long it will take?¬†Lot's of ways - Reference Classes, Parametric models. Monte Carlo models of activity networks.
  • Let's Stop Guessing and Learn How to Estimate¬†- estimates are NOT guesses. Estimate - it is an approximation of some value for some purpose, even if the data is incomplete, uncertain, or even unstable. Typically the estimate is derived from a statistical source of data - either observed or referenced in some way.
  • How ¬†to Estimate if you really want to¬†(Updated)¬†- the key here¬†is if you want to. There is NO reason to not estimate. Estimates are need to make decisions in presence of uncertainty.¬†
  • How to Estimate Almost Anything if You Really Want To¬† - this starts with¬†what's the¬†coastline¬†of England¬†problem and uses a similar red herring about a hike down the coast of California.

In the end it's simple. Anyone speaking about the difficulties or the waste of estimating is likely on the spend side of the software development equation. Those on the pay side know (or should know) they needed estimates. They can get the estimates from those most qualified to provide them - the developes. Or they can just make them up. Probably better to have a bottom up estimate.

Related articles Humpty Dumpty and #NoEstimates Want To Learn How To Estimate? Let's Stop Guessing and Start Estimating
Categories: Project Management

Software Development Process Improvement Opportunities

Wed, 06/15/2016 - 15:56

 When we hear about all the suggested ways to improve the effectiveness of our development effort, if we're to going work on improvements, let's go where the REAL money is. 

Here's the IT budget for the Federal Government. This is larger than all the IT systems found everywhere else in the world, plus all their custom built IT stuff. This is not the embedded systems. These are business systems.

So if we're going to make improvements in the spend of IT for better value, let's start here.

Screen Shot 2016-06-13 at 1.55.39 PM

 

 

Related articles Estimating and Making Decisions in Presence of Uncertainty Intentional Disregard for Good Engineering Practices? What's in a Domain?
Categories: Project Management

Deadlines Require Time and Care For Success

Tue, 06/14/2016 - 16:10

The latest thread in agile is ...

the continued paradigm of deadline-driven development is killing the benefits that Agile Software Development can bring.

It is suggested by Neil Killick that ...

... using genuine time constraints as a factor in the prioritisation of work rather than as a focus for its execution, the odds of meeting those "deadlines" are actually improved.

Not sure what a genuine time constraint is versus any time constraint. But the conjecture that Executives of software product and service companies always want stuff delivered faster, cheaper and better. Agile principles and methods are believed to be a way to achieve this ignores several fundamental principles of all work and especially software development work.

 All project work has uncertanty. Reducible uncertanty and irreducible uncertanty.

Agile does not remove this uncertanty. Agile is NOT a risk management process. Genuine constraints don't remove this uncertanty., I've spoken many times in the past about how to Manage in the Presence of Uncertainty.

These principles are always in place. More so on Agile projects, where emergent requirements are encouraged, which in turn drive uncertanty further to the right in the Probability Distribution Function of the possible range of durations, costs, and technical performance shortfalls.

When those uncertainties are not considered and handled, any project is going to have an increased chance of being late, over budget, and have technical issues. 

Setting genuine constraints may make this issue visible, but does not remove the risk to the project's probability of success. Only active risk management and the necessary margin can increase this probability.

The only protection for irreducible uncertanty is Margin and the only protection for reducible uncertainty is active risk management. Both of these activities require careful planning and execution of the plan, along with the estimate of the probability of occurrence of a reducible event and the statistical distribution of the naturally occurring variances and the probabilistic impact on the success of the project.

This is the Reason for Planning

It's been suggested I work in a unique domain, where deadlines and need dates are themselves unique. This is False.

No credible business, no matter the size, Doesn't have a need date for the Value produced by the software project. If there was no need date, the developers would show up whenever they wanted after spending whatever they wanted, with whatever they thought the customer needed.

Ignoring the simple time cost of money and the time phased Return of Investment of (Value-Cost)/Cost, any business that intends to stay in business is spending money for software - either developed or purchased - to provide some value to the business. Not having a need date for the production of that Value means the business is ignoring the core principle of retained earnings. Even non-profits and not-for-profit business (and I've worked there as well) have a time value of money economic model.

The End

  • No process can remove the irreducible uncertainties that create risk (naturally occurring variances) of project work - only margin can protect the project from these variances. Schedule margin, cost margin, technical margin.¬†If you're building software with Scrum, add more sprints to cover the margin. Or be prepared to deprecate the¬†needed Capabilities and the Features and Stories that implement those Capabilities.¬†You do know what needed business or mission¬†capabilities¬†will be produced for the money you're¬†spending¬†right?¬†NO? You're on a death march project from day one.¬†Genuine¬†constraints¬†aren't going to do anything for you. Nothing's going to help that project.
  • For reducible uncertainties that create risk (probability of occurrence uncertainties), specific actions need to be taken to reduce the risk from these probabilistic event. This can be¬†buying two, running extra tests on test beds, formal verification and validation, external surveillance‚Ć of the product (DB engineering, security, performance assessments).

So if you're going to produce value for your customer, that value is most always time sensitive other wise it's de minimis value. If it's time sensitive, there is a deadline. If there's a deadline, reducible and irreducible uncertanty and the risk it produces must be handled. 

Risk Management is How Adults Manage Projects - Tim Lister

† The naive notion that scrum teams are self contained and need no external support is only the case when there is little at risk for the resulting code. Cyber security, Database integrity, performance validation, operational integrity are external surveillance roles on any Software Intensive System of Systems. This is called Governance and guided by documents like ISO 12207, ITIL V3.1, COBIT, DoDAF, ToGAF.

Related articles Risk Management is How Adults Manage Projects Essential Reading List for Managing Other People's Money Agile Software Development in the DOD The Art of Systems Architecting Seven Principles of Agile Software Development in the US DOD Deadlines Always Matter
Categories: Project Management

Software Economics

Mon, 06/13/2016 - 19:37

Here's some thoughts on the economics of software development using other people's money, after 3 weeks of working a proposal for a major software intensive system of systems using Agile.

  • With the advent of Agile, the linear spend planning and delivery of capabilities was altered to iterative and incremental spend and delivery planning.
  • Time boxed, drip funding, fixed budget are ¬†funding profiles that might be added to the economic model once they've been verified and validated in practice to provide better approaches to managing funding in the presence of uncertainty.
  • Core business processes are still in effect, no matter the funding profiles
    • Money consumed provides capabilities produced.
    • Capabilities enables value from the use of money.
    • Capability defined by the¬†user¬†through some form of¬†value assessment, not by the provider.
    • Emergent needs can be addressed.
  • When we merge business value with development cost without monetizing that business value, we've lost the ability to make economic decisions.
    • Both the business value and the development cost operate in the presence of uncertainty.
    • This uncertainty is always present.
    • To make those economic decisions, we need to estimate both business value and development cost.
    • There is no way out of this in any credible development environment.
  • Monetized value allows the decision process to use ROI, IRR, Analysis of Alternatives.
  • Without monetized value cost and value decisions are simply¬†made up and are arbitrary.

So to come full circle 

Why Do We Need Estimates?

  • It's not the developers that need the estimates - they take the money and turn that money into value.
    • They should estimate if the needed value can be produced by that money.
    • But if the developers decided they don't need to estimate, then they'll be subject to the whims of management, just like Dilbert.
  • The developers are certainly closed to the work and have the information needed to best contribute to the estimate.
  • Estimates are primarily used to support decisions.
    • Product margin.
    • Cost target for business management.
    • ROI, IRR, AOA.
    • Staffing, release date, launch date - literally and figuratively.
  • Knowing the cost of a product or service produced is a fundamental piece of information needed by the business if they intend to stay in business.

It can't be any clearer than this - if you  don't know what something costs or is going to cost in the future, you can't make a business decision about it's value

When you read estimates are a waste, estimates are non-transferable, estimates are wrong, estimates are temporary think again. Go ask those paying if they need estimates to make decisions for the business. If they don't, then continue to spend your customers money. If they do, they may consider looking for someone who knows the difference between willful ignorance of how to estimate software development and someone who can provide that information needed to stay in business. On our proposal team, this would get you a 2nd place prize in the estimating capabilities contest - which is a one way trip home with weekends off.

Related articles Capabilities Based Planning First Then Requirements Incremental Delivery of Features May Not Be Desirable Quantifying the Value of Information Essential Reading List for Managing Other People's Money Decision Making Without Estimates? Economics of Software Development Risk Management is How Adults Manage Projects Process is King Deadlines Always Matter
Categories: Project Management