Estimation, planning, teams and money, some data

PlannedMay17-2018-05-17-11-46.jpg

When I deliver Agile training for teams I run an exercise called “The Extended XP Game”. It is based on the old “XP Game” but over the years I’ve enhanced it and added to it. We have a lot of fun, people are laughing and they still talk about it years later. The game illustrates a lot of agile concepts: iteration, business value, velocity, learning by doing, specification by example, quality is free, risk, the role of probability and some more.

When I run the exercise I divide the trainees into several teams, usually three or four people to a team. I show them I have some tasks written on cards which they will do in a two minute iteration. They do two minutes or work, review, retrospect then do another two minutes of work – and possibly repeat a third time.

The first thing is for teams to Get Ready: I hand out the tasks and ask them to estimate, in seconds how long it will take to do each task: fold a paper airplane that will fly, inflate a balloon, deflate a balloon, roll a single six on a dice, roll a double six on two dice, find a two in a pack of cards and find all the twos in the pack of cards. Strictly speaking, this estimate is a prospective estimate, “how long will it take to do this in future?”

Once they have estimated how long each task will take someone is appointed product owner and they have to plan the tasks to be done (with the team).

What I do not tell the teams is that I’m timing them at this stage. I let the teams take as long as they like to get ready: estimate and plan. But I time how long the estimation takes and how long the following planning takes.

Once all the teams are “ready” I ask the teams: “how long did that take?”

At this point I am asking for a retrospective estimate: how long did it take. The teams have perfect estimation conditions: they have just done it, no time has elapsed and no events have intervened.

Typically they answer are 5 or 6 minutes, maybe less, maybe more. Occasionally someone gets the right number and they are then frequently dismissed by their colleagues.

Although I’ve been running this exercise for nearly 10 years, and have been timing teams for about half that time I’ve only been recording the data the last couple of years. Still it comes from over 65 teams and is consistent.

The total time to get ready to do 2 minutes of work is close to 13 minutes – the fastest team took just 5.75 minutes but the slowest took a whopping 21.25.

The average time spent estimating the tasks is 7 minutes. The fastest team took 2.75 minutes and the slowest 14 minutes.

The average time planning once all tasks are estimated is just short of 6 minutes. One team took a further 13.5 minutes to plan after estimates while another took just 16 seconds. While I assume they had pretty much planned while estimating it is also interesting to note that that team contained several people who had done the exercise a few years before.

(For statistics nuts the mean and median are pretty close together and I don’t think the mode makes much sense in this scenario.)

So what conclusions can we draw from this data?

1) Teams take longer to estimate than do

Everyone taking part in the exercise has been told – several times – that they are preparing to do a 2 minute iteration. Yet on average teams spend 12.75 minutes preparing – estimating and planning – to do 2 minutes of work!

Or to put it another way: teams typically spend six times longer to plan work than to do work.

The slowest team ever took over 10 times longer to plan than to do.

In the years I’ve been running this exercise no team has ever done a complete dry run. They sometimes do little exercises and time themselves but even teams which do this spend a lot of time planning.

This has parallels in real life too: many participants tell me their organization spend a long time debating what should be done, planning and only belatedly executing. One company I met had a project that had been in planning for five years.

TeamSize-2018-05-17-11-46.jpg

2) Larger teams take longer to estimate than small teams

My second graph shows there is a clear correlation between team size and the time it takes to estimate and plan. I think this is no surprise, one would expect this. In fact, this is another piece of evidence supporting Diseconomies of Scale: the bigger the team the longer it will take to get ready.

This is one reason why some people prefer to have an “expert” make the estimate – it saves the time of other people. However this itself is a problem for several reasons.

Anyone who has read my notes on estimation research (and the later more notes on estimation research) may remember that research shows that those with expert knowledge or in a position of authority underestimate by more than those who do the work. So having an expert estimate isn’t a cure.

But, those same notes include research that shows that people are better at estimating time for other people than they are at estimating time for themselves, so maybe this isn’t all bad.

However, this approach just isn’t fair. Especially when someone is expected to work within an estimate. One might also argue that it is not en effective use of time because the first person – the estimator – has to understand the task in sufficient detail to estimate it but rather than reuse this learning the task is then given to someone else who has to learn it all over again.

PlanningDelta-2018-05-17-11-46.jpg

3) Post estimation planning is pretty constant

This graph shows the planning delta, that is: after the estimates are finished how long does it take teams to plan the work?

It turns out that the amount time it takes to estimate the task has little bearing on how long the subsequent planning takes. So whether you estimate fast or slow on average it will take six more minutes to plan the work.

Perhaps this isn’t that surprising.

(If I’ve told you about this data in person I might have said something different here. In preparing the data for this blog I found an error in my Excel graphs which I can only attribute to a bug in Excel’s scatter chart algorithm.)

4) Vierordt’s Law holds

People underestimate longer periods of time (typically anything over 10 minutes), and overestimate short period of time (typically things less than two minutes).

Not only do trainees consistently underestimate how long it has taken them to get ready – which is over 10 minutes – but teams which record how long it takes to actually do each task find that their estimates are much higher than the actual time it takes. Even when teams don’t time themselves observation shows that they do the work far faster than they thought they would.

TimeVMoney-2018-05-17-11-46.jpg

5) Less planning makes more money

One of my extensions to the original game is to introduce money: teams have to deliver value, measured in money. This graph shows teams which spend less time planning go on to make more money.

I can’t be as sure about this last finding as the earlier ones because I’ve not been recording this data for so long. To complicate matters a lot happens between the initial planning and the final money making, I introduce some money and teams get to plan for subsequent iterations.

Still, there are lessons here.

The first lesson is simply this: more planning does not lead to more money.

That is pretty significant in its own right but there is still the question: why do teams which spend less time planning make more money?

I have two possible explanations.

I normally play three rounds of the game. When time is tight I sometimes stop the game after two rounds. In general teams usually score more money in each successive round. Therefore, teams who spend longer in planning are less likely to get to the third round so their score comes from the second round. If they had time to play a third round they would probably score higher than in round two.

This has a parallel in real life: if extra planning time delays the date a product enters the market it is likely to make less money. Delivering something smaller sooner is worth more.

This perfectly demonstrates that doing creates more learning than planning: teams learn more (and therefore increase their score) from spending 2 minutes doing than spending an extra 2 minutes planning.

The second possible explanation is that the more planning a team does the more difficult they might find it to rethink and change the way they are working.

The $1,600 shown was recorded by a Dutch team this year but the record is held by a team in Australia who scored over $2,000: to break into these high scores teams need to reinterpret the rules of the game.

One of the points of the game is to learn by doing. I suspect that teams who spend longer in planning find it harder to break away from their original interpretation of the rules. How can you think outside the box when you’ve spent a lot of time thinking about the box?

In one training session in Brisbane last year the teams weren’t making the breakthrough to the big money. Although I’d dropped hints of how to do this nobody had made the connection so I said: “You know, a team in Perth once scored over $2,000.” That caused one of the players to rethink his approach and score $1,141.

I’ve since repeated the quote and discovered that simply telling people that such high scores are possible causes them to discover how to score higher.

* * *

I’m sure there is more I could read into all this data and I will carry on collecting the data. Although now I have two problems…

First, having shared this data I might find people coming on my agile software training who change their behaviour because they have read this far.

Second: I need more teams to do this to gather data! If you would like to do this exercise – either as part of a full agile training course or as a stand alone exercise – please call (+44 20 3286 4292) or mail me, contact@allankelly.net, my rates are quite reasonable!

Want to receive these posts by e-mail? – join the newsletter today and receive a free eBook: Xanpan: Team Centric Agile Software Development

Product Owner or Backlog Administrator?

3337233_thumbnail-2018-03-20-18-08.jpg

In the official guides all Product Owners are equal. One size fits all.

In the world I live in some Product Owners are more equal than others and one size does not fit all.

The key variable here is the amount of Authority a Product Owner has. In my last post I said that Authority is one of the four things every product owner needs – the others being legitimacy, skills and time. However there is a class of Product Owner who largely lack authority and who I have taken to calling Backlog Administrators.

About the only thing a Backlog Administrator owns is their Jira login. They are at the beck and call of one or more people who tell them what should be in the backlog. Prioritisation is little more than an exercise in decibel management – he who shouts loudest gets what they want.

A Backlog Administrator rarely throws anything out of the backlog, they don’t feel they have the authority to do so. As a result their backlogs are constipated – lots of stories, many of little value. Fortunately Jira knows no limits, it is a bottomless pit – just don’t draw a CfD or Burn-Up chart!

If the team are lucky the Backlog Administrator can operate as a Tester, they can review work which is in progress or possibly “done.” They may be able to add acceptance criteria. If the team are unlucky the Backlog Administrator doesn’t know enough about the domain to do testing.

I would be the first to say that the Product Owner role can be vary a great deal: different individuals working with different teams in different domains for different types of company mean there that apart from backlog administration there is inherently a lot of variability in the role.

The Product Owner role should be capable of deciding what to build and/or change.

So Product Owners need to know what the most valuable thing to do is. Part of the job means finding out what is valuable. While Backlog Administration is part of the job the question one should ask is:

How does the Product Owner know what they need to know to do that?

Backlog Administrators are little more than gophers for more senior people.

True Product Owners take after full Product Managers and Senior Business Analysts – or a special version of Business Analysts sometimes called Business Partners.

Product Owners should be out meeting customers and observing users. They should be talking about technology options with the technical team and interface design options with UXD.

Product Owners should understand commercial pressures, how the product makes (or saves) money for the company. Product Owners are responsible for Product Strategy so they should both understand company strategy and input into company strategy. Product Strategy both supports company strategy and feeds into company strategy.

Product Owners may need to observe the competitor landscape and keep an eye on competitors and understand relevant technology trends. That probably means attending trade shows and even supporting sales people if asked.

Frequently Product Owners will require knowledge of the domain, i.e. the field in which your product is used. Sometimes – like in telecoms or surveying that may require actual hands on experience.

And apart from backlog administration there is a lot of work to do to deliver the things they want delivered: they need to work with the technical team to explain stories, to have the conversations behind the story, write acceptance criteria, attend planning meetings, perhaps help with interviewing new staff and sharing all the things they learn from meeting customers, analysing competitors, debating strategy, attending shows, etc. etc.

I sure there are many who would rush to call the Backlog Administrator an “anti-pattern” but since I don’t believe in anti-patterns I don’t. I just think Product Owners should be more than a Backlog Administrator.

Kanban paradox

iStock_000014068443XSmall-2017-10-31-14-08.jpg

For a while now I’ve been seeing a paradox with Kanban. Specifically, Kanban compared to Scrum.

For a team new to Agile – although some regard this as heretical I place Kanban under the Agile umbrella, yes I know its more about Lean than Agile but of cause Agile is itself a Lean method, anyway…

For team, specifically a software team, looking to adopt a new process there is a choice:

  • Kanban has a very low barrier to entry, to get started Kanban essentially says “visualise your work and manage the result.” Starting Kanban can be as simple as putting up a board and tracking work items. In Kanban visualisation should drive improvement. Change can be incremental and gradual. Change is rooted in learning.
  • Scrum has a far higher barrier to entry: essentially Scrum says, “Adopt Sprints, designate a Product Owner, appoint a Scrum Master and build out a backlog.” Once these changes are done you can run with Scrum and then the Scrum Master and retrospectives will kick-in and drive further improvement.

Interestingly, neither method says explicitly “Improve your quality.” Yet I always believe a lot of the success of Agile methods is down to good old quality improvement: writing fewer bugs and having fewer bugs to fix means greater predictability and more time to deliver valuable software. But I digress.

It is easier to start with Kanban because it requires less up front change. However that does mean the improvements are slower coming.

Conversely, Scrum drops in, changes a lot and most teams see an immediate improvement. Scrum relies less on subsequent change.

Because Kanban relies more on ongoing change it is more difficult. It is easy to get stuck at the “we built a Kanban board so we are doing Kanban stage.” Change in Kanban requires one to see the need to change, understand what will fix a problem and then follow the change through. That often requires experience. Thus in teams adopting Kanban there is a greater need for a coach, a consultant, someone who has done it before.

Scrum on the other hand makes far more changes upfront and the recipe for improvement is more straight forward. And of cause there are a lot more books on Scrum, blogs on Scrum, Certified Scrum Masters and Scrum experience out there. So while it is harder to get started with Scrum (because more needs to change) there is less need for further change and you change does not require the same level of knowledge.

You see this specifically when you look at statistics. Watching the numbers should be important in both processes but with Kanban it is near essential. Anyone with real understanding of Kanban knows that queuing theory, lead times, possibly weighted lead times, and a bunch of other numbers need to be examined.

Scrum on the other hand doesn’t go much further than a burn-down chart. Yes, making more improvement with Scrum will also benefit from understanding lead times, queuing theory and the rest but you can quite happily use Scrum, and even improve Scrum, a fair bit without understanding these ideas.

So here is the paradox:

It is easier to start with Kanban than it is Scrum without expert knowledge, but it is harder to improve Kanban than Scrum without expert knowledge.

In many ways I prefer Kanban but I find this need for expert knowledge troubling. I suppose I shouldn’t, I’m a consultant, I am that expert, people hire me to help improve their Kanban processes so it does make more work for me.

In the longer run, the Kanban approach is more likely to lead to a genuine all inclusive culture of improvement and is less likely to get stuck in a sub-optimal position – yes Scrum fixes things, but is it the best fix possible?

Looked at like this gives me a new perspective on Xanpan.

I wanted Xanpan to be two things:

  • An understandable description of actually following an Agile process, specifically a Kanban/XP hybrid processes
  • An example of how, and why, teams should create their own processes.

The same paradox is here: Xanpan should be easy to start but allow you to improve; creating your own process requires a bit more knowledge that only really comes with experience.

To step back a minute and ask another question: What amount of change can a team handle to start with?

I find that I advocate more initial change than I used to. Because I’m fearful of creating a learned dependency I really want teams to learn to change and improve themselves. But… once a team has decided to change I want to seize the opportunity and install a bunch of changes while enthusiasm is there.

Read more? Subscribe to my newsletter – free updates on blog post, insights, events and offers.