How to Manage Risk

Also see "On Risk and Estimating and Agile Methods". This post is yet another angle on a common theme.

Orders of Ignorance and Risk Management.

Software risk management has two sides.  First, there's the classical meaning of risk, we'll call that "casino risk" because it's really random events.  This includes fire, flood, famine, conquest, war, pestilence, death, etc.  Actual risks.

The second meaning of risk is a load of malarkey.  It's a code word that includes two things: "bad management" and "ignorance".  Some things called project risks are just plain old bad management—generally driven by a nonexistent process for handling ignorance.  The events aren't random.  

There are five orders of ignorance, and each of them leads to project management problems.  None of these are "random events"; none of this is like casino gambling.  There aren't any odds; most of these things are certainties.

0 Order Ignorance: Things We Know.

There are two sides to the things we think we know about a project.  There are the things we know which are true, and things which are false.  Falsehoods come from at least two places:  we assumed something or we were actually lied to.  (Other choices, like illusions and hallucinations, are too creepy to pursue.)

Our assumptions aren't facts.  This sounds so obviously stupid, but projects get into trouble based on assumptions that were never checked to see if they were true or not.  Managers insist on doing "risk analysis" and then pad their project estimates with time and money instead of simply challenging their assumptions.

Some "assumptions" are explicit placeholders for facts to be found out later.  These formally documented assumptions are a different thing, they're 1st order ignorance, something we know we don't know.

Example.  The customer says they need an app to do [X].  There's 8 people in the department who are likely actors.  We assume the user population is 8.  With no fact checking.  We don't put it in the plan as a formal, documented assumption, we just assume it.

Bottom line.  What are the odds that a plan is based on false knowledge?  This isn't casino gambling.  There aren't any odds that we assumed something, or odds that we were lied to.  This is simple fact-checking, simple bad management. Every unchecked fact is going to be false.

1st Order Ignorance: Things We Don't Know.

There are two ways to deal with things we don't know.  Make a guess and document this, or actually ask a question.

The formally documented guesses (usually called "assumptions") are the hallmark of software project plans.  Documents are often full of lists of assumptions on which the plan and associated estimates rest.

Each one of these "assumptions" is a question that—for some reason—couldn't be asked or couldn't be answered.  Some questions are "politically sensitive" and can't be asked.  Some questions require lots of research to develop an answer.

The answers, of course, change the project.  In most cases they change the project dramatically.  If they didn't have a big impact, we wouldn't spend any time documenting them so carefully, would we?

And all project managers are punished for making any changes.  We can't expand the scope without a lot of accusatory conversations where people keep repeating the original price back to us.  

The canonical line is something like "I thought this was only 1.8 million dollars, how can you change it now?" or "I've already committed to $750K, we can't change the price, something else has to be changed.  We have to work smarter not harder."  Bleah.

This doesn't involve casino-like odds of finding answers that will change the scope of the project.  We know we have questions.  We know we made guesses and documented them as "assumptions".  There were no odds; it was absolutely certain there would be changes.

Bottom line.  What are the odds that a plan is based on things we guessed at?  Typically, this is a fact of life: parts of the estimate are guesses.  

What are the odds that the real answer will be different from the guess?  This, too, is absolutely certain. It's merely a question of magnitude.

For first-order ignorance problems, we should create a contingency budget for each answer that will diverge from the guesses.  This isn't a book-making exercise, it's a list of alternate guesses (or assumptions).

It isn't enough to simply detail the assumptions.  We have to provide alternative answers and the associated costs when the assumptions turn out to be false.

2nd Order Ignorance: Things We Didn't Know to Ask.

If we didn't know to ask, what's that?  Is that a risk?  What are the odds we forgot to ask something?

Here's the canonical quote: "Are there any other unforeseen problems?"

What? If they're unforeseen, then, uhhh, we can't identify them now.

If we can identify them, then, uhhh, they're not unforeseen.

There aren't any "odds" of an unforeseen problem.  It's an absolute certainty that there will be unforeseen problems.

Remember, these are things we didn't know to ask.  Things that didn't make the list of "assumptions".  These are things that completely escaped our attention.

What will the impact be?  We have no way of knowing. No. Way. Of. Knowing.

We can't even put a contingency in place for these things.  We didn't know to ask.  So we don't know what it will cost for rework when we figure out what we should have asked.

All we can do here is use a good, transparent management process.  Each new piece of information -- each thing that's learned that we didn't know to ask -- will change scope, schedule, cost, deliverables, everything.

This isn't "casino risk".  There are no odds associated with this.  This is just change management. It's inevitable. Calling it a project risk is lying about it.

3rd Order Ignorance: No Process for Managing Ignorance.

When we have second-order ignorance (we didn't know to ask) there are two responses: an organized change-management process, or a leap down to 3rd order ignorance.  Third order ignorance slips from simply not knowing into denying that the level of knowledge changes through time.

When we learn something unexpected, we can either deny that it is something new, or we can expose it.  When a business analyst learns that the "simple" calculation involves a magical MS-Access database with no known author, magical numbers and no discernible calculations, this is going to change the scope of the work.  Or make it impossible to make progress until someone explains the MS-Access database.

Denying this kind of unexpected information is common, it's done by playing the management trump card of "schedule is sacred.  Once the schedule is sacred, all learning is either trivially denied or learning turns into ways of shaving scope or quality to make the schedule.  

3rd Order Ignorance means there's no change process and the "schedule is sacred".  If the only thing that matters is schedule, then buggy, useless software will be delivered on time and on budget.

What are the odds of 3rd order ignorance?  Either 1.0 or 0.0.  Either the organization has an effective change management process (in which case, we don't have 3rd order ignorance) or there will be problems in delivering software that works on time. 

Bottom Line.

Here's the summary of ignorance and mitigation.

0th order ignorance: do basic fact checking to validate your assumptions.

1st order ignorance: do contingency planning. Define specific contingencies around each specific unknown fact.  Don't just document an "assumption", plan for alternatives when the assumption is invalidated.

2nd order ignorance: have a change management process.

3rd order ignorance (i.e., no change management): stop using waterfall-style methodologies. Switch to Agile methods so that change and the management of ignorance become essential features of the overall process.

4th order ignorance is the state of not being aware that ignorance is one of the most significant driving forces behind project failure.  A symptom of 4th-order ignorance is conflating "risk analysis" for a project with "casino risks" (or "insurance risks.")  With rare exceptions, all project risk analysis is just ways of coping with bad management.

When there's 4th order ignorance, folks are told that it's helpful or meaningful to try and assign odds to the veracity of the facts, the presence of things which were forgotten, and the change management process itself.

Avoiding 4th order ignorance means recognizing that software project management "risks" are just bad management (with minor exceptions for fire, flood, famine, conquest, war, pestilence, and death.)

Here's how to manage risk:

• Check the facts, 
• plan specific contingencies, 
• use Agile methods because of their built-in ability to manage change.