Dan North: Embracing uncertainty
Dan North has been coaching, coding and consulting for over 20 years, he is the originator of Behaviour-Driven Development. He gave the following talk at the Norwegian Developers Conference in June 2012.
You can read the outline of the talk here.
Fear leads to Risk,
Risk leads to Process,
Process leads to Hate, Suffering and Gantt charts.
Risk - the business-friendly word for uncertainty
We are uncertain of the future, but that is not something you talk about on a business meeting, it doesn’t sound professional. Nevertheless, the business dictionary defines a phrase for it: we will be managing risk. In addition to that, we will create processes, as the ultimate tools to resist the uncertainty, with defined validations and checkpoints.
If we are uncertain of the scope, we create plans, backlogs, todo-lists, and we schedule them way ahead.
If we are uncertain of the technology, we create blueprints, patterns, we ask architects to hand down toolchains and define corporate standards.
If we are uncertain of the effort, we keep ourselves busy with local optimizations and chase local efficiency, but we rarely look at the global optimum.
If we are uncertain of the structure, we create hierarchies and approval rules, in the hope that the other guy above us has a better clue about the direction and events. Oh, the unlucky ones at the top, who need to act as they were the ultimate source of truth :)
Waterfall software development is a good example how business imagined an ideal world: it is a gated model, and we have well-defined rules, with checkpoints when everyone knows and accepts that certain things are true. In reality, everyone is frightened to deliver a bad feedback or bad news, in the hope that eventually problems will disappear, or at least someone else will be blamed for it. Until the end, when everyone switches to a new project or goes to a different company.
The Agile Manifesto was created to uncover better ways of developing software. Yet, after ~10 years, we tend to defy its goals and principles:
Teams are arguing about which process or tool to use? (vs: individuals and interactions)
Teams are aiming for executable specification (== comprehensive documentation) (vs: working software)
Sprint planning and arguing its scope become the new way of contract negotiation (vs: customer collaboration)
Following a plan is more than important, especially keeping 600+ stories in a backlog (vs: responding to change)
It seems that we have a good idea here, but its interpretation becomes a dogma, our complex problems demand simplistic answers. We would rather be wrong than uncertain.
The risk analysis lie: likelihood is never zero, impact is never infinite
Analyzing risk in a one-dimensional model is easy, but risk is more a multi-dimensional variable, than a simple numerical value. However, to simplify this uncertainty, analysis tend to put events in the extremes, e.g. considering impact as infinite. After such assumption, the only way to mitigate the risk is to minimize the likelihood, preferably to zero. Everyone knows it is a lie, but at least we can find someone to blame if the event happens.
It will happen at some point, no matter what.
Dan North gives a few examples how we can embrace the uncertainty and face our fears.
If we are uncertain of the scope: scrap the backlog, forget the 600+ items, instead do a rolling planning: every time only the next 2-3 biggish themes. With just that, we can have surprising amount of certainty in the delivery.
If we are uncertain of the technology: accept that you don’t know if something is going to be good. Contract 2-3 vendors in isolation (all paid), and after a point, stay with one.
If we are uncertain of effort: think of throughput and results instead of efforts. We are obsessed with activities, but we should measure and balance everything else too. Self-organizing teams are better, as the team sees the bigger vision, more efficient solutions might came out of nowhere.
If we are uncertain of structure: don’t work with specialists, instead put people in different roles in different times (e.g. developer, designer, tester, support).
Oh, that craving for certainty!
A good way to think about it to treat decisions like buying a financial option. Like options, any decision has value (opportunity factor, different cost model, capital expense vs operating expense), which changes over time, and eventually it expires. (Read more about real options.)
Early commitment signals our fear of uncertainty. If a sprint-planning closes down options, it means we will learn nothing in the next 4-6 weeks.
A good methodology allows us to defer decisions, let them be about scope, tooling or technologies.
Estimation as a process isn’t useful in itself. It becomes useful, because while we are doing it, we learn something. But these discoveries are accidental, and usually because some business guy happens to be in the same room. Having these "oh-crap" moments earlier plays a critical part of our success (read more about deliberate discovery).
Assume: some unexpected bad things will happen.
- some == nonzero
- unexpected == you cannot plan for them
- bad things == will affect the project, we cannot sidestep them
What can we do?
Don’t make lists (as the ultime source of truth). No matter how long list you make, things will be missing. Assume you are second order ignorant (lack of awareness): you don’t know what you don’t know. But you can act actively to reduce your ignorance.
Use group activities to detect and point out other’s blind spots.
Making something better with planning, deciding, doing, checking and measuring is good, but sometimes you need to step back and ask: is the cycle the right cycle?
Convincing others might be hard:
Attribution bias: when bad things happen to other people, they should have seen it coming. When it happens to me, no one could have seen it.
Confirmation bias: I will "remove" data that doesn’t confirm my position. I’d rather be liked than right. I like to be approved.
Bias bias: protecting our ego, we are "better than average". I’m better than average driver. I’m not as biased as the guy next to me.
Expect the unexpectable.
Embrace uncertainty - it’s inevitable.