Purpose over Backlog

Backlogs are a good idea. Backlogs ease the transition from the old “requirements up front” world to the new more dynamic agile world. Backlogs provide a compatibility layer for agile teams to interface to more traditional project management and governance. Backlogs even allow you to take a stab at done date!

Backlogs allow you to even out work between the quiet periods and the busy times. Backlogs give you a place to store good ideas which you can’t do just now. And because stakeholders can see their request is not forgotten they don’t need to shout for it today.

Yes backlogs are good. I’ve seen them work well myself and I’ve taught many teams to effectively use backlogs.

But – you knew there was a but coming didn’t you? – but…

Backlogs have problems, too many teams are labouring under the Tyranny of the Backlog, they have become backlog-slaves and practice something we might call BLDD – Back Log Driven Development.

(To be clear, when I say “backlog” I am primarily thinking of the product backlog – the long list of all the things the team (might) do in the future. This is different to the sprint backlog (iteration backlog). The sprint backlog is a shorter list of things the team aims to do this iteration. I am using Scrum terminology but the ideas are pretty much “generic agile” and I’m thinking more broadly than Scrum. Many implementations of Kanban feature a product backlog of sorts so while Kanban is less prone to these problem it is not immune.)

1) Lump of Work Fallacy

There is usually an assumption that the backlog represents all the work to be done – an impression reinforced by early implementations of Scrum. In the short term that leads to agile teams being seen as inflexible and prioritising process over need because new work is not allowed in.

In some cases teams even struggle to get started on work because a big-up-front requirements gathering and analysts activity is required to create a backlog. In the worst cases that work is even estimated and end-dates forecast before a line of code is cut or developers hired.

In the longer term it is simply unrealistic to assume the backlog is fixed. Even with more and better analysis it is impossible to foreseen future requests. The agile adage “it is in doing the work that we understand the work” cuts both ways: coders understand what they need to build and customers/stakeholders/analysts understand what they want.

Work will arrive after you begin, any system that does not incorporate that truth will fail one-way or another.

2) Bigger than you think

Not only does the backlog grow with completely new work the work in it changes – and grows. There are many reasons this happens: new opportunities appear, hidden ones become clear, requests require more work than expected and so on.

Humans are very bad at estimating, especially about the future, and, it turns out, they are also very bad at estimating time spent in the past. If you want accurate forecasts you need to invest in them, you need to make structural changes and you need to use statistics.

However, because of the lump of work fallacy and the belief that humans can make estimates, poor end-date projections get made and when they are missed (because they were wrong to start with) everyone gets upset.

3) Fallacy of Done

Backlogs come with burn-down charts and an assumption that there is an end; and that end is when everything is “done.” The team will be done when the backlog is empty. That assumption is baked into BLDD, traditional project management and even governance.

I have long argued that software is never done. I’ll accept that I might be wrong, but in the digital age, when business runs on digital technology (i.e. software) your products are only done when you business is done. The technology is the business, and the business is the technology. Stop the backlog growing, stop growing you technology and you kill the business.

4) Backlog Bottomless pit

Put all those reasons together and the backlog becomes a bottomless pit. In the early days of agile, when I managed teams myself, the backlog would often sit on my desk, written out on index cards and held together with rubber bands. I could get a sense of how big the backlog was my looking.

Today everyone uses electronic tracking systems. Not only do these allow an infinite number of items they rob us of perspective. To paraphrase Comrade Stalin: “2 outstanding backlog items is tragedy, 200 is a statistic.”

5) Backlogs obscure strategy & purpose

With so many backlog items it is easy to get lost – you can’t see the wood for the trees. Arguments over what will be done next start to resemble deciding who should get a lifeboat place on a sinking ship, add in the demands “when will you be done?” (plus explaining why the date has changed) and “the bigger picture” gets lost.

In Back Log Driven Development the sense of purpose and strategic goals is lost as teams struggle with the day-to-day demands of just doing stuff.

6) Powerless product owner (i.e. backlog administrators)

Tyranny of the backlog seems worst where product owners lack real authority and skills. They are little more than backlog administrators. They spend most of the week adding requests to the backlog, then passing a few chosen items to developers in planning meetings. A vicious circle develops, the product owner can’t win so people trust them less, their authority wanes, and the backlog spirals.

Few organisations give product owners the power needed to get a grip on this situation. Indeed, many product owners are plucked from the ranks for development or support and given a battlefield promotion to product owner but lack the skills required. (See The problem with Product Owners.)

A solution?

For years I’ve been suggesting teams throw away the backlog – you will not forget the important things. But then how do you know what to do?

Take a step back, start with your purpose, your mission, the reason you team, your company, your organisation exists. What should you be doing? How can you fulfil that purpose and serve your customers?

This is where I see a role for OKRs and jobs to be done. Both these techniques – together, or separately – can be used as story generators. Every time you need to more work, more stories, you return to your OKRs and ask “what can we do now to move us towards our objective?”

When writing Succeeding with OKRs in Agile I became more and more convinced this is the path to take. Increasingly I sum this up as Purpose over Backlog.

Post publication addition: great suggestion fro Jari Mäkelä on Twitter to call this: Purpose Driven Development, PDD.

Step 1: Clarify your purpose – what is your overarching reason for existing?
Step 2: Clarify how your existing strategy builds towards that purpose, and if you don’t have a strategy create one.

Repeat steps 1 & 2 annually.

Step 3: Think broadly, set your OKRs as a team so you build towards your purpose by following your strategy.
Step 4: Spend the next 12 weeks executing against those OKRs

Repeat steps 3 & 4 every 3 months.

Step 5: In each planning meeting take stock of what you have done and progress against OKRs
Step 6: Ask “what do we need to do next to move towards the OKRs?”

Succeeding with OKRs in Agile

Repeat steps 5 and 6 every 2 weeks

And if you are Kanban’ing then keep steps 1, 2, 3 and 4, adjust 5 and 6 as appropriate.

Having finished, completed, published Succeeding with OKRs I really wish I had been clearer in the book. The ideas are there but with time they have become so much clearer… maybe I need another book.

Buy Succeeding with OKRs in Agile at Amazon today.


Subscribe to my blog newsletter and download Project Myopia for Free

Best seller – Succeeding with OKRs in Agile

Succeeding with OKRs in Agile book cover

I’m delighted that my new book Succeeding with OKRs in Agile went on sale at Amazon yesterday. By this morning it was the number #1 best seller in Amazon’s IT Project Management category – and not doing badly in Computer Programming and Business Management & Leadership either. (Although the publisher has some power over which categories a book is in it is still a black-art.)

It is hard to express just how great it is to see the book in the number #1 slot. While I hope it stays at #1 for a while I expect it will drop down before long.

Print and audio versions of the book are in the works and should be released in the next few weeks so if you would rather read a physical version or listen, watch this space as they say.

The book has taken a little under a year to write and a few more months to make production ready. The wonders of LeanPub mean many readers have already been enjoying early versions of the book. If you would like to read the book on iBooks or as a PDF then LeanPub is the place to buy from.

I recorded the little video below to explain why I wrote the book.

Technical Debt: Engineers, you are not alone

Debt help sign

I don’t read many books about software or technology these days, I tend to read outside the domain: economics, business and management – which after all is much of what I do in the technology world these days.

Recently I’ve been reading Winning now, winning later by David Cote and find really interesting. He hardly mentions software and never mentions agile but he is giving me a new perspective on technical issues, particularly technical debt (or technical liabilities as I prefer to call them). He talks about issues which have similar characteristics to tech debt but are completely different, legal issues for example. He sees these issues as conflicts between short-term thinking and long-term thinking.

Cote’s argument is that short term actions should support, not conflict, with long term goals. I agree. It might not be easy but if actions in the hear-and-now conflict with longer term goals then the chances of reaching those goals is diminished.

Cote is writing about his time as CEO of Honeywell – a US industrial conglomerate if you don’t know. Unusually Cote is honest about many of the dirty problems the company faced when he took over – a lot of business books glossy over such problems or talk about “challenges” or “opportunities”.

For example, Cote describes how Honeywell managers were chasing numbers and targets every quarter. They had no time for long term improvements because they were so busy “making the numbers”. One of his managers cut down a forest to sell as timber in order to make the end of quarter numbers. Sales people would give products away to new customers or offer large discounts at the end of the quarter. However customers knew this would happen so delayed orders until they were sweetened.

Making the short term numbers meant the company undercut itself so lost revenue next quarter. Management time was spent finding accounting tricks to “make the numbers” rather than improving the business. And since targets ratcheted up the next quarter was more difficult and required more diversions.

Other examples included legal cases Honeywell was fighting: spending time and money on lawyers, building up bad will with customers, politicians and local people. This in turn made it more difficult to get support when the company needed it.

I read these examples, and others, and I hear an engineer saying “Technical debt.” That is exactly what it is.

A software engineer who does a dirty job on a code change because they feel under pressure stores up problems for themselves and future engineers who need to do the next change. Which is exactly the same as a factory which dumps waste into a lake as a quick fix and then needs to clean up the lake later.

Actually economists have a term for this: externalities. These are the costs which are forced onto other payer, e.g. the factory saves money on waste disposal but the local government has to pay to clean the lake. I’ve long thought a lot of “technical debt” could be considered an externality because it pushed the cost onto someone else.

Today it is probably harder than ever to escape these cost – in code, in law, in financing – because there are more and more people out there looking for these things. Environmentalists look at waste in lakes, society expects companies to pay if they pollute and courts make companies pay. Smart investors will look closely at a firms accounts and discount the firm, or short them, if they see dubious practices.

This is Cote’s argument: in the short term it might save or generate money to fight legal cases (deny deny deny), sell off forests, discount sales and such, but, in the longer term – and the longer term might just be weeks – it will costs. And when it costs it will damage growth.

Doesn’t that sound just like technical debt/liabilities?

Naturally it is hard to see a company that chases numbers, pollutes and fights all legal claims caring about the quality of code. Engineers will have a hard job fighting for technical excellence there.

Cote argues, and I agree with him, that it doesn’t have to be this way. Acting responsibly and thinking about tomorrow – whether that is pollution, sales, accounting, code quality – will make it easier to grow later. Just because it is difficult to act in a manor that meets todays needs and make the world better for tomorrow does not allow use to ignore it: all of us need to think harder and find a solution that doesn’t mortgage tomorrow.

And sometimes the right answer is to accept the slow path, take it on the chin, pay the cost you’ve been avoiding. For Cote that mean settling legal cases and accepting some costs, for software teams that means doing the refactoring, rewriting a module or just saying No to more changes.

As I’ve said before: in software the long term comes along very soon.

As as I’ve blogged before there is no such thing as quick and dirty, only dirty and slow.

We might talk about debt/liabilities but really we are talking short-term v. long-term, a pay-day loan v. investment. Engineers have an unfortunate habit of talking about technical debt as a binary good v. evil debate with no other options.

Finding these less obvious paths which satisfy the short-term and long term is hard(er) but also offers the opportunity for higher, and longer, term improvement, something which is itself a competitive advantage.


Subscribe to this blog by e-mail and download Project Myopia ebook for Free