Framing the question frames the answers – my crown jewels


Today I’m giving away my crown jewels. Once you have read this post I can’t run my favourite exercise with you. There is a bit of experiential learning I can’t give you. So if you’ve rather have the experience stop reading and go and book yourself on my May workshop.

I’m describing an exercise that models what happens in “the real world(tm).” Plenty of the people who have done the exercise recognise it was a real life problem. The insights are many, and some a little disturbing.

Dozens of teams and the answers are always the same. I live in dread that someone will guess and ruin the exercise but it never happens. Now I’m telling the world that might change.

On screen I put a story something like:

As a widget maker I want an online store to sell my widgets so that I can make money

I separate the room into teams. Each team represents a technology supplier – an agency, an outsourcer, whatever. I want each team to competitively bid on a piece of work. Each team gets to think about the problem and estimate the work. At the end I want each team to be ready to name their price, how long it will take and how many people they need. They may add any more details they like, e.g. staging, design, technology, etc. (and most do).

The teams on the right get a story which says:

As an international widget maker I want to sell direct to customers so that I can cut out distributors. I anticipate $10million turnover within 3 years and have budgeted $1.2m for this project.

15 minutes later the teams on the right read out their bids. They always want a million give or take. They want months, if not years. They want teams of half a dozen or more engineers, testers, UXD, analysts and project managers. They may propose an ongoing maintenance contract too.

Very disconcerting for these teams is that while they are answering and taking questions the other teams, those on the left, often burst out laughing – literally – when they hear these proposals.

What neither side knows is that they have different stories. The teams on the left get a story saying:

As an artisan widget maker I want to sell my widgets to customers so that I can give up my day job. When I make $100,000 a year in sales I can live my dream. My accountant tells me a WordPress website will cost $5,000.

These teams want a week or two, an engineer or two and perhaps $10,000 tops. In some cases they say “We can do it this afternoon, we’ll set up Etsy.” Even if they don’t want to use Esty or eBay they probably propose an OpenSource solution.

So what do you think?

True, it is a semi-artificial set-up but I would argue that these situations happen all the time in “the real world” work environment. However they are usually well disguised and hard to see. Maybe now you will spot them.

That aside there are many potential lessons this exercise illustrates, almost everyone is worth a discussion in its own right. To keep things brief I’ll just highlight some of them:

  • Anchoring (cognitive bias): individuals are anchored to those numbers, bigger number lead them to frame their answers as bigger numbers.
  • Assumptions: people jump to assumptions, people automatically fill in the blanks when they lack information and the information they fill in flows from the numbers mentioned. Few questions get asked.
  • Different solutions: the key lesson for me, confronted with similar problems, people (especially engineers) are capable of formulating very different solutions. Those solutions have different time and cost implications.
  • Problem bounding: presenting the same problem with different bounding constraints results in massively different solutions.
  • Effort estimates, and therefore cost estimates, flow from value: whether through anchoring assumptions or solution designs the estimates (time and money) flow from the value available NOT the other way around.
  • Prior experience often goes out the window. In one run a low-end teams told me: “We did this last week. A digital consultant showed us how to install WordPress and Magento for online retail in the morning and in the afternoon we did it ourselves.” While this team came up with a low cost proposal their colleagues who were given the $1m story forgot everything they learned last week.
  • People don’t ask questions: I answer questions while teams are creating their answers but people rarely challenge what is asked for. Maybe its because I’m usually in some position of authority as a consultant or workshop trainer and my word should be followed.

Occasionally a team with the million dollar story say “We could do this with eBay/WordPress/Shopify.” I keep a poker face and let them be. Inevitably left alone for long enough they talk themselves into a much more complex and expensive bid.

In fact, the longer I give teams the higher the estimates go. I heard a team in Australia say three times “Those estimates look low, lets double them.” And they did. (Again, planning has diminishing returns.)

So far nobody has offered two solutions: you could offer up a Shopify solution and a custom build solution but nobody has.

While we are going through the exercise the minimal viable product idea often gets mentioned – usually by the teams on the right. So recently I introduced a third story. This read the same as the international widget maker but has an extra paragraph underneath:

MacAllan consulting has advised the company to take an iterative and agile approach to this work using a minimally viable product model.

How do you think teams respond?

Think for a minute… your answer is?

It makes no difference.

Or rather, so far I’ve not had any of the million dollar teams propose anything close to the $5,000 solution. In one case a team with the MVP story actually came in more expensive – and longer – than the million dollar team without the MVP clause.

My learning here: talking MVP makes no difference. If you want an MVP you have to impose constraints. (Hence try an MVT.)

People continue to fill in the blanks after the charade is exposed. I’ve heard software architects argue forcefully they these are different problems because of the money involved and therefore require different architecture. They clearly feel cheated and want to justify the proposal they have made. I suspect they feel I’ve made them look silly and want to undo that impression, I’m sorry if I’ve made anyone feel silly.

I wonder how often that happens in the work place? How many of us would back down in real life? I’d like to think I would but I would probably first try and justify my position.

The architects have a point, in many ways the stories are functionally the same but the differences lie in the non-functional requirements: load, throughput, security and so on. But all of that is inferred by people from the price tag without question.

It makes me sad that teams ask so few questions. People easily see themselves as a tailor not as a consultant (my Tailor or Image consultant post.)

Then there are the questions about the bidding process and companies bidding on the work.

Imagine you are the buyer: one supplier bids really low, the others were much higher but inline with your expectations. Would you trust the low bid? Have they blow their credibility?

And as a bidder: if you know the client plans to spend $1,000,000 why bid lower? The engineers cost estimates and designs aren’t relevant. Ideally you bid just below the competition. You are the lowest price with all the credibility and maximum revenue.

For that matter, should you be bidding on this at all?

If you work for a small e-commerce provider in rural Cornwall you may never know about, let alone, bid on a million dollar piece of work from an American multi-national. And if you did, would anyone take you seriously?

Suppose you got your big break: you walk in and offer a quick, low cost solution based on something like Shopify. Would they take you seriously? Would they want to listen?

Do corporations increase their own costs simply by being?

Conversely, if you work for a big consultancy and bid on million dollar deals every week will you be interested in bidding on a $5,000 piece of work? Of course not!

But that also means that if a corporation approaches you for a million dollar online shop, even if you could deliver it in a week’s time running on a third party platform do you have any incentive?

I don’t have answers to these questions. Indeed, there aren’t any right answers. All answers are valid, just some answers are “better” in some places than others.

Ultimately the lesson I take away from this is: we craft solutions within constraints.

More specifically: Engineers engineer within constraints, that is what engineers do.

That reinforces my belief that one needs to really understand benefit (value) and how that changes with time. From there we can work back to a solution.

If you would like to see this exercise for real then book yourself my Requirements, Backlogs and User Stories workshop. If you are in London Learning Connexions are running this again in May.

Like this post? – Like to receive these posts by e-mail?


Subscribe to my newsletter & receive a free eBook “Xanpan: Team Centric Agile Software Development”

Leave a Reply