Blog

People or the system?

“the two view-points are always tenable. The one, how can you improve human nature until you have changed the system? The other, what is the use of changing the system before you have improved human nature?”

George Orwell, “Charles Dickens” essay in Shooting and Elephant and Other Essays, Penguin Books

I am sure I am not alone in exhibiting another of Orwellian trait: Double think.

For several years I have been guilty of agreeing with two contradictory points of view:

  • The Deming idea that the system governs how people will perform and act
  • That no matter what system (e.g. Scrum, XP, Prince2, DSDM, etc.) you use to develop software the motivation and quality of the people is the overriding factor
It wasn’t until John Harvey tweeted about the latter recently that I realised In was double thinking, my positions were not compatible with one another.

I should have noticed the cognitive dissonance sooner. I’ve been known to say, and still believe: “The true test of any software development method (e.g. Kanban, Scrum, etc. etc.) is whether it can help mediocre people perform, i.e. deliver, be more productive”

In that one statement I epitomise the contradiction. I’m hoping that writing this blog will allow me to resolve this!

This isn’t an academic problem: depending on which side of the argument you come down on will determine how you go about improving things. It also explains why some people in the Agile moment emphasis: trust, communication and other inter-personal issues while others emphasis a method, technical practices, etc.

I’m thinking of my experiences:

  • I have worked in environments where the system defeated the people: Railtrack privitisation, Sema had an ISO-9000 waterfall process and 120 people. Of course in 120 people some were good and some not so good.
  • I have worked in places where (good) people have overcome the system: two years after leaving Railtrack I returned, a much smaller team were in maintenance mode. The same process was in place but we circumvented it, we faked the documentation and so on.
  • And I have worked places where everyone was treading water and complying enough with the system
I have seen different things happen within the same organisation, even within what the company regarded as “one team.” I saw Reuters destroy its own processes in an effort to impose a CMMI level 2 (going to 3) process. (Managers were a bit like the American General in Vietnam who supposedly said “we have to destroy this village to save it”. Like the American’s Reuters ultimately lost.)

I have worked places were the concept of “system” would have been laughed it, things might be more accurately characterised as “Chaos.” But even in this chaos there was a system of sorts.

At Dodge Group it was hard to define any system or process, but there was somewhere a process. The company was trying to sell, sell, sell so they could IPO and the software team just had to keep up as best they could.

The good people at Dodge won out for a while. I instigated a simple process we could now recognise as a form of Agile but ultimately the company, the system, killed the success. Partly by hiring bad people, partly by not recognising the good people and partly by letting the system run its course.

In my “Agile will never work in investment banks” posting a while back I made a similar point: Good people in banks can install a good process which can deliver. But in time the bank – the systems, culture and contradictions – will kill the system and dispose of the good people.

Then there is the question of whether “good people” are themselves the result of the system. Can you pick up a “good person” from company A, put them into company B and let them do their good stuff?

An article in the MIT Sloan Management Review a few years back suggested that when “star players” move to a new team they don’t necessarily, or even normally, keep their “star player” performance. (When ‘Stars’ Migrate, Do They Still Perform Like Stars?, Fall 2008 issue.)

So where does all this lead us?

I think there is truth in both the System-first and People-first point of views. Things are not so black-and-white. Sometimes the System-first view is right and sometimes the People-first is right. There are forces at work in different contexts.

Some of these forces are:

1) The strength of the system: the stronger the system the more likely it is to win out.

The strength of the system derives form multiple sources including: the effectiveness of the system, the degree to which it is enforced and the degree to which people see their interests being served by the system.

Reuters CMMI initiative also put a lot of effort into policing. At Dodge the system was weak day-to-day, developers got on with it and they made good stuff.

At Sema Railtrack (first time) a lot of effort was put into policing the system, the system delivered ISO-9000 which helped the company win contracts. As a programmer woking in the system it didn’t seem particularly effective but I guess it was effective at generating money.

2) The System affects the Good People

At Reuters the system encouraged people to leave, I was one. At Dodge the system eventually recruited the wrong people, forced others out and wore down the good people who submitted.

3) Good people can make an poor system work

Primarily through ignoring it, compensating or changing it. Which leads to…

4) Good people can change the system

I’ve seen it happen. In fact you need at least one good person to cause they system to change or it will not.

Again a strong system is harder to change.

5) How “good” are the people?

Yes “good people” can do things change thing but how “good” do you need to be? Is it real a question of having super-stars? Most people aren’t super-stars. Surely we should be more concerned with the mediocre.

And before anyone rushes to say “Good developers are 10 times (or 28 times) better than average” please read Laurent Bossavit book “Leprechauns of Software Development” where he shows the evidence for this claim doesn’t hold up.

I guess these conclusions are going in the Deming direction.

Yet good people can make a bad system work; but if they do not succeed in actually changing the system it will eventually impose itself.

Can I suggest it comes down to a question of time scales: In the short-run good people can overcome a poor system, but in the long-run, unless they change the system for something better the system will overwhelm them.

And the mediocre people? – How about we say this:

Over the short-run, the better your people are the more likely they will overcome a poorer system, but in the long-run the hard it will be for people to sustain the effort if they do not change the system.

A good system will help everyone work the best. It will allow star-performers to shine and mediocre people to contribute and feel good, I’d even like to think a good system will improve the mediocre people.

Conway's Law v. Software Architecture

I’ve written about Conway’s Law before (Return to Conway’s Law (2006) and a Focus Group I ran at EuroPLoP “What do we think of Conway’s Law Now?”)

but I make no apologies for revisiting it again because I still think it hold true.

There are times when I wonder if there is any point to the discipline called “Software Architecture.” Sure design can make a difference in the small but when it comes to the big then, for my money, Conway’s Law is a far more powerful force than the plans, designs and mandates of an enterprise architect.

Conway’s Law predates Agile and Waterfall but it applies to both:

“Any organization that designs a system (defined more broadly here than just information systems) will inevitably produce a design whose structure is a copy of the organization’s communication structure.” Conway, 1968

The original paper “How do committees invent?” is available from Conway’s own website. The original publication was in Datamation, April 1968.

While Conway’s Law can be seen at the macro level, the company level, it is also observed in in the small, at the micro level. The law if often restated in the small as:

“If you have four developers writing a compiler you will get a four pass compiler”

Or

“If you have three developers writing a UI you will get three ways of doing everything (mouse click, menu item, short-cut key)”

For example, if a development team is set up with a SQL database specialist, a JavaScript/CSS developer and a C# developer you they will produce a system with three tiers: a database with stored procedures, a business middle tier and a UI tier. This design will be applied whether it is the best or not. Indeed, the system might not even need a relational database or a graphical interface – a flat file might be quite adequate for data storage and a command line interface perfectly acceptable.

It is a little like a children’s cartoon where the superheroes need to rescue somebody: Superman says “I can use my super strength to lift the ice boulders out of the way”, while Fireman says “I can use my fire breath to melt the ice” and Drillerman says “I use my drill hands to make a rescue tunnel”. Once added to the picture each specialist finds a way to utilise his or her special powers. For a software team this can lead to poor architecture, over complicated designs and competing modules.

In setting up a team architectural decisions are being made even if no architecture diagrams are being drawn. Just deciding to staff the team with four developers rather than three will influence the architecture because work now needs to be found for the fourth team member.

These decisions are made at the time when the least information is known – right at the beginning. They may made by planners – project managers – who have little knowledge of the technologies or by people who will not actually be involved in the development effort, e.g. enterprise architects.

To avoid this my advice is to start a team as small as possible. Try to avoid making architectural decisions in staffing the team by understaffing it. Keeping the team hungry will reduce the possibility of building more than is needed or over architecting it.

There are those who would quickly point out the risk of under architecting a system, not looking to the future and not building a system that can change and grow. But the risks of over architecting are if anything worse. Too much architecture can equally prevent a system changing and growing, and too much architecture leads to more time consuming and expensive code to cut. Then there is the risk of not shipping at all, too long spent producing the “right” design may result in a system too late to be viable.

Second staff the team with people who are more generalist than specialist, people who have more than one skill. Yes have a Java developer on the team but have one who knows a bit of SQL and isn’t scared of a little UI work. In time you might need to add database and UI specialists but delay this until it is clear they are needed.

These suggestions echo Conway’s own conclusion at the end of his paper:

“Ways must be found to reward design managers for keeping their organizations lean and flexible. There is need for a philosophy of system design management which is not based on the assumption that adding manpower simply adds to productivity.”

It is worth remembering that Conway was writing over 20 years before Womack and Jones coined the term Lean to describe Toyota.

Reverse Conway’s Law & Homomorphic force

Since Conway wrote this a lot more systems have been developed. Many of these systems are now referred to as “Legacy Systems.” In some cases these systems force certain structure on the team who maintain the systems and even on the wider company.

For example, take the 3-tier database, business, GUI system from above. Years go by, the original developers leave the company and new staff are brought in. Perhaps some of these have been and gone in the years. The net effect is a system so complex in all three tiers that each requires a specialist. The database is so rich in stored procedures, triggers and constraints that only a SQL expert can understand it. The GUI is crammed full of JavaScript, CSS and not HTML 5 that only someone dedicated to interfaces can keep it all working. And the middle tier isn’t any different either.

Given this situation the company has no option but to staff the team with three experts. Conway’s Law is now working in reverse: the system is imposing structure on the organization.

Again this happens not just at the micro-level but at the macro-level. Entire companies are constrained by the systems they have. Economists might call this path-dependency: you are where you are because of how you got here not because of any current, rational, forces.

Underlying both Conway’s Law and Reverse Conway’s Law is the Homomorphism, or as I prefer to think of it: The Homorphic Force.

This force acts to preserve the structure of system even as the system itself moves from one technology to another, from one platform to another.

Both forms of Conway’s Law and the Homomorphic Force pose a dilemma for any organization. Should they work with the force or try to break it?

I can’t answer this question generically, there is too much context to consider. However I tend towards saying: Work with Conway’s Law, not against it – like woodworking, work with the grain not across it. Be aware of Conway’s Law and Learn to play it to your advantage.

Conway’s Law does contain a get out clause: the system that will be created will be a copy of an existing system, if you can create a new system, a system not pre-loaded with assumptions and well-trodden communication paths then maybe you can create something new and of a better design.

Thus I sometimes view myself as an architect. Now I don’t think I’d accept membership of the Architects-Club even if it was offered, and probably my software design skills are not up to the job but it doesn’t stop me dreaming.

But I architect by (trying) to bring about a good organizational architecture which will in turn bring about a better system architecture via Conway’s Law.

Links! – 2 conferences, 1 week

I’ve been to two conferences this week!

The first was Agile Dev Practices in Potsdam, outside of Berlin. At I presented my Retrospective Dialogue Sheets (www.dialoguesheets.com), well I say presented, it was 10 minutes of introduction, 60 minutes of attendees doing Dialogue Sheets and 20 minutes wrap up.

Anyone who has attended one of my Dialogue Sheet sessions will recognise the format. Which also means there aren’t a lot of slides for download. There are as few slides, essentially the same as Agile Cambridge last September so if you grab those slides you’ll be fine – Dialogue Sheets presentation as PDF download or Dialogue Sheets presentation on SlideShare, take your pick.

The second conference was DevWeek 2013 here in London. This was a completely new presentation: 10 Tips to make your Agile Adoption more successful – so far I only have this up on SlideShare. This presentation was based on an article I wrote for InfoQ last year My 10 things for making your Agile adoption successful.

If you check out the Allan Kelly InfoQ page you will find more on Dialogue Sheets too – I say this because I’m expecting InfoQ to publish another piece on Dialogue Sheets before long.

11 Agile Myths and 2 Truths

I deliver a lot of Agile training courses and I give a lot of talks about Agile (BCS Bristol tonight). There are some questions that come up again and again which are the result of myths people have come to believe about Agile. Consequently I spend my time debunking these myths again and again.

I’ve been keeping a little list and there are 11 reoccurring myths. There are also two truths which are a bit more difficult for teams and companies to accept.

Agile Myths

  1. Agile is new: No, the Agile manifesto was published in 2001, the Scrum Pattern language was works hoped at PLoP 1998, the Episodes pattern language (the forerunner of XP) was workshopped at PLoP 1995, Tom Gilb’s Evo method dates back to 1976 and there are some who trace things back further.
  2. Agile means No Documentation: You can have as much documentation as you like in Agile. Documentation is just another deliverable, if it brings you value then schedule it and product it like anything else. Please be aware: documentation is often unread, often fails to communicate, is used as a defensive tool and is typically the second most expensive think on a large software project (after rework).
  3. Agile means No Design: No, Agile probably means MORE DESIGN. Design is inherent all the way through development, at every planning meeting and more. Agile does mean the end to big-up-front design which is invalidated five minutes after someone starts coding.
  4. Agile means No Planning: No, again, Agile probably has more planning. Again planning is spread out through the whole development exercise rather than at the front and it is the work of everybody rather than one or two anointed individuals.
  5. Developers get to do what they like: No, if this is true for you then you are doing it wrong, please call me. Agile needs more discipline from the team and what gets done should be lead from a specific role usually designated the Customer or Product Owner and usually played by a Product Manager or Business Analyst. If developers are doing what they like then there is a failure of in this role.
  6. There is a right size for a User Story: There is no right size for a user story. Every team is different, get over it.
  7. Work must fit in a Sprint: If you are doing Hard Core Scrum then Yes. If you are playing Agile the way I do (which I now call Xanpan) then No. In fact I advise letting stories span sprints in order to improve flow. You can have stories spanning sprints but we won’t let them continue for ever and we will try and break them down into smaller pieces of work.
  8. Scrum and Kanban are sworn enemies: No but the marketing efforts behind each they can get a lot of eyeballs by playing it that way. Xanpan is a Kanban/XP hybrid, and XP isn’t that different to Scrum so there you go.
  9. Agile doesn’t work for fixed deadline projects: No, Agile works best in fixed deadline project environments.
  10. Agile doesn’t work on Brownfield projects: No, Agile works best in brownfield environments. Granted retrofitting automated unit tests is harder but it is far from insurmountable.
  11. Agile doesn’t work on Greenfield projects: No but your first objective is to get yourself to a steady state where you can think like a brownfield project.
To my mind the ideal project to start an Agile initiative with is a brownfield system with a fixed deadline in about 3 to 6 months where development has started but requirements are still unclear.

  1. Now two truths about Agile:
  1. Agile will not work for us because… (complete this sentence for yourself)
  2. Agile is a good idea but … we should wait until we have finished X, got Y to buy in, bought Z and the (new) Pope has given it his blessing
You can always talk yourself out of it or find a good reason for not doing it today.

Agile Advice for Aviva (and many other big companies)

I’m starting this blog entry on the train returning to London after speaking at Sync Conference in Norwich – and I’m finishing sitting in my local coffee shop on Tuesday. (My conference presentation Agile in 90 minutes is online (PDF) or via Slideshare.)

An awful lot of people at SyncConf worked for Aviva and during the day I spoke to many of them. Aviva are trying to be “Agile” but like every other big company are struggling.

Personally I’m pessimistic about their chances. Someone asked me if I could name a big company which had made the Agile change and I struggled. Yes, many big companies are trying Agile, and yes there are many successful teams in these companies. But has an entire company made the entire change? I don’t know of any.

All the banks are trying it, and all are failing in similar ways (see Reverse-Tolstoy and Why Agile will Never work in Investment Banking). Many other big corporations are trying and most of what I here is not positive.

During the day I did build up a picture of much that is happening inside Aviva and I did give some advice in conversations. Now I’ve thought about it there is some more, bullet points below.

(I can say this because Aviva are not my client – OK, I did do a little work with them years about but it was insignificant. If Aviva were my client I wouldn’t be writing this blog, I don’t speak openly like this about clients. I try to avoid talking about active clients at all in this blog. Nor am I in the habit of giving unsolicited free advice to clients.)

Advice for Top

  • There are no Silver Bullets: Conway’s Law, Reverse-Conway’s Law, Brooks Law, Goodhart’s Law and dis-economies of scale constrain what can be done.
  • Take Software Craftsmanship seriously: Start and apprenticeship programme – speak to Jason Gorman about how the BBC do it
  • Let a thousand flowers bloom: dump any attempts at standardised working and let people experiment, let them have fun as they learn and change. In a few years time when you’ve made lots of progress start to mine out your own “Good practices” and sharing them amount teams.
  • (That also means be prepared to work with multiple difference consultants, coaches and advisors. Accept they will give you different advice. Yes engage with some big players but also work with the small players.)
  • Embrace dis-economies of scale – you don’t have time to be involved with the details so delegate
  • Get with Quality: the Quanlity v. Time trade-off is the reverse of what you think; higher quality (i.e. reduced bug count) makes for shorter scheduled, shorter schedules makes for lower costs, together this make for happier customers and more responsive business
  • Get clear about what you want: if it is cutting costs say so, you can use some of the Agile toolkit for this but don’t expect to use it all or achieve Agility. There is nothing wrong with not being Agile if you have a better alternative.
  • Pay the SyncConf Norwich team to re-run the conference again (with a few tweaks) inside the company. (You’ll have to pay the speakers but this will be a drop in your ocean of your IT budget.)
  • Go further and let give employees attending seed money, let them spend it to sign up for courses and bring in some of the consultants they liked into the company. Yes this is radical, no apologies.
  • If you want to be Agile then accept that it won’t be the lowest costs, it should result in the greatest value overall so focus on value not costs
  • Your business and IT operations need to be more joined up. Maybe you want to dismantle your IT group, at least embed it in the business.
  • Your annual planning cycle needs to relax, you need to learn to start small pieces of work without having every T crossed an i dotted.
  • In the long run you want to look to Beyond Budgetting.
  • Don’t worry about your CMMI or ISO or whatever accreditation, my clients have FDA and ISO-13485 approval, if they can do it so can you.
  • In the meantime give up on strategic planning: that idea died in the paddy fields of Vietnam thanks to Robert MacNamara. Go read Henry Mintzberg “Rise and Fall of Strategic Planning”.
Advice for the Middle

  • Visualise, visualise, visualise: boards, graphs, everything!
  • You have to lead the top, they don’t know how to spell TDD so educate them
  • Involve “the business”, their representatives in everything
  • Get with the Quality Agenda: ditto the above, embrace and push quality to eliminate bugs. Give your team support, send them on training, hire consultants to come and coach, buy them books.
  • Architects must also code
  • Celebrate every little success; make sure the whole company knows about a realise
  • Spend on travel: Indian outsourcing doesn’t work because you paid TCS, you actually have to go there and see, they have to come here and see, you need to talk, talk, talk. Skype, VOIP, everything
  • Don’t spend money on expensive test tools
  • Manage teams as a whole: not a Test Team and a Test Manager there, and a Dev Team and a Dev Manager there, and a BA…. the whole team wins or looses together.
  • There is no long term future for Test Managers, you options are: a) focus on Test and work to improve the Testing Process, specifically Automation; b) focus on management and manage the whole development team; c) disentangle Test from Quality and become get Quality throughout the process
  • Form self-support book study groups: spend an hour, meet for lunch every other week, agree a book, agree a chapter in advance, someone volunteer to summarise it. Open each meeting by going once around the table and having someone say something positive (“Tell me something that made you smile today?” “Tell me something you learned this week?”).
  • Good luck: you have the hardest task, and its lonely. Some of you will need to lay down your jobs to make this happen
Advice for the Workers

  • You control the means of product, Agile is in your hands, Make it So. Your greatest weapon is Quality, you have the power to improve it today.
  • Embrace Test Driven Development, Refactoring (i.e. emergent design.) Dump big-up-front design, get coding on day-1 and work out the design later.
  • You have to lead the middle and they will lead the top. Neither of these two groups knows as much about your work as you do. Lead by showing them what works. Which means…
  • You have to do it, make those improvements. Developers start learning TDD, spend your own money if you need to. Testers: download the open source test automation tools and enrol developers to help you get them working.
  • Testers: Automation, Automation, Automation – get with the Open Source tools – Selenium, FIT/Fitnesse, Cucumber, etc. etc.
  • Form your own study groups
  • Make sure retrospectives happen and action items are actioned
Most of this advice is applicable to any large company. I really don’t know that much about the inside of your company, I’m assuming. Nor is this list really that different from the one I published on InfoQ last year – 10 things for making your Agile adoption successful.

Finally, Aviva, you don’t have to take any of my advice but a) I guess your paid advisors are saying similar things and b) if you don’t change new competitors will kill you; you are big so it will take time but it is never nice to watch a big animal in pain.

Speaking at PAM – Project & Analysis Management Summit (Poland)

I’ve never been to Poland so when the invitation arrived to speak at the Project and Analysis Management Summit in Cracow it was hard to resist. However, I probably should have trodden more carefully. After a talking it over with the organisers I agreed to give a talk entitled:

“Is there a role for Project Managers and Business Analysts on Agile?”

For many Project Managers and Business Analysts this is The Big Question. Indeed, their jobs are on the line. While I’ve talked about both halts of this question before I’ve never put it all together. Now I’ll have to, a challenge, I’m part intimidated, part enthused.

So to start off with I thought I’d take a slightly different approach to the synopsis, let me know what you think:

Pity the poor Project Manager

There have no place in Agile,

Or so they say

Scrum Masters rule! Or rather they don’t

Facilitate, Teach, Inspire, anything but Manage!

Or so they say

And where is the Business Analyst?

A void in every Agile book, blog and Tweet

The silence is defining

So tell me why, Why do many Agile teams have PMs and BAs?

Tell me, Do they add value? What should they do?

And tell me why, Why do Agilists dislike them so?

In this session Allan Kelly will attempt to untangle this conundrum, answer these questions and more.

SOA is software equivalent of a fast breeder reactor

A fast breeder nuclear reactor is a wonderful idea. Basically, you put in used nuclear fuel from a conventional reactor, burn it, produce useful electricity and at the end of the process the used fuel has changed into a form you can put back into a conventional reactor.

Alternatively, another use for the final product is put in nuclear bombs but we don’t like to mention that.

Fast breeders have been shown to work but have failed to take over the world. They are very expensive to build and operate, pose security dangers and are hideously complex to operate – as you might imagine of a device cooled by liquid sodium, lead or mercury.

In other words: they are not commercially viable.

They aren’t even sensible to Governments.

I have come to the conclusion that in the software world Service Oriented Architecture, SOA for short, is the equivalent of a fast breeder.

SOA works in the laboratory. The technology can be shown to work by big service companies – IBM, Accenture and co. It seems like a great idea.

But… SOA doesn’t make commercial sense.

Let me explain why I believe this.

SOA is all about Reuse. Reusable code and systems.

Reuse does not come for free, you have to write code for reuse. Figures on cost of reuse v. cost of single use are not common. As a general rule I refer to Fred Brooks 1974 observation that it costs about 3 times more. Admittedly not a solid reference but the best I know.

The first problem is: do you know it is going to be reused?

If you write an SOA system and then find it is used once then it is very expensive.

In order to know it will be used more than once you need to accept requirements from multiple sources.

Which means your requirements costs go up, response times go up, responsiveness goes down. Which means you loose time and money.

Worse still, the loss of focus leads to distracted teams, complicated stakeholder management and competing interests. You risk producing a camel rather than a horse.

There is also an assumption here that there is enough commonality that can be factored out for reuse to make the whole thing viable.

Now SOA, and reuse, are sexy. Its something all developers want to do. And they want to do it properly. And such projects tend to be technically driven.

So they loose their business focus and get absorbed in details.

Then there is the matter of testing. Testing costs also go up.

Add in maintenance: fixing a bug in one system is going to hit other systems, all of them need testing. (“Write once, test many” as we used to say in the early days of Java.)

And who pays for this?

If it comes out of the IT budget we again loose business drivers and increase technical focus. But if one group pays for it they are paying far more than they would need to for single use. And if you apportion costs you are going to spend a lot of time arguing.

In other words: SOA works in the lab but not in commercial environments.

My advice, as is with any type of reuse: write it for the problem in hand, get something that works, have plenty of automated tests. And wait. When someone comes along with a problem that looks similar, a real candidate for “reuse” modify the thing you have just enough to cope with this, with tests. Then wait and repeat.

This way you only pay the cost of reuse when someone actually wants it.

SOA and Fast Breeder reactors both belong to the class of technologies which while possible, even fascinating, don’t stake up commercially. Actually, come to think of it that covers most forms of software reuse and nuclear power.

What is Agile?

Two weeks into 2013 and I’ve not blogged, in fact my last blog entry was a month ago. What’s happening? Well partly Christmas, New Year and then flu took a lot of my time. Plus, pre-Christmas my writing energies have been going into writing up Xanpan.

At first I thought my notes on Xanpan would be similar to my “Agile Reader” mini-book, then I thought Xanpan would be a replacement for the Agile Reader series, now I’m seeing it as a book format collection of writing that goes beyond Agile Reader and includes new material on Xanpan specifically. At the moment my working title is “Xanpan – and other notes on Agile & Lean Software Development”.

If you are interested in seeing this project complete please goto the LeanPub Xanpan page and register your interest. Otherwise its likely to be pushed to the bottom of my to do list.

One of the things I’d like to do in Xanpan is define what Agile is, and maybe what Agile is not. In other words, I’d like to answer the question: “What is Agile?”

After much thinking I have concluded there there is no single, short, concise, answer to this question. Rather how you define Agile depends a lot on who you are and why you are asking. With that in mind there are several perspectives you might take in answering the question. The perspectives I see are….

The Historic perspective: Agile is defined by the a manifesto and 12 principles written in 2001. As I’ve said before, the manifesto is now a historic document, it was written at a time when heavy-weight development processes (e.g. SSADM and V-Model, and most things which were ISO-9001 or CMM approved) ruled software thinking. The term “Agile” was coined to group the “light weight” methodologies.

The manifesto is like the US Constitution, and like the constitution we can read all sorts of meaning into it, depending on who you are you read different things. But there is no Supreme Court to arbitrate on whats-what. So Agile is what I say it is – or maybe, like art (“art is what artists do”), Agile is what Agile advocates say it is.

The Not the Waterfall perspective: traditional software development was largely rooted in a mindset which said “Define what you want, design it, build it, testing it and deliver it, in sequence with little or no feedback from later stages.” While (in my experience) we didn’t usually manage to follow that model we tried and when we didn’t we considered it a failure.

I think a lot of the time “Agile” is defined colloquially as “anything other than the waterfall model” but Agile is more than “not not the waterfall”. Indeed, as Laurent Bossavit described in The Leprechauns of Software Engineering, Royce’s Waterfall model wasn’t even discussed in literature much until Barry Boehm started using it as a counter example to his Spiral Model.

The Agile as Better Perspective: Similar to the The Not the Waterfall Perspective, people who hold this view want a improvement over their current (usually poor) development processes and techniques. The motivation is simply to have fewer late projects, fewer unhappy users, fewer bugs and so on.

On the one hand this view is cynical, it holds limited promise; on the other hand this is perhaps the most work-a-day view. People holding this view are looking to make their lives better and their company’s use of IT better.

In some cases Agile as Better links the Historical perspective and the Not the Waterfall perspective. I’ve seen companies that have tried hard to apply traditional waterfall based heavy-weight development approaches. In doing so companies tie themselves in knots trying to enforce a development model which doesn’t match the problems they are trying to address. For these companies the year is still 2001; just adopting a mindset which is not Waterfall based is itself a massive improvement.

The next three perspectives are best explained together, in this diagram: The State of Agile Perspective, The Methods Perspective and The Toolkit Perspective

I often find that when I talk to developers and testers in an company they implicitly view Agile as a toolkit of techniques which might be applied – The Toolkit Perspective. Ask them “Are you Agile?” and they will answer by referencing the practices which they use or don’t use.

Conversely, talk to managers – especially senior managers – and they are concerned with company Agility. Their perspective is the end result, – The State of Agile Perspective – they don’t care whether a team do stand-up meetings, iterations or not, they care about the Agility of the company. Indeed, some Agile practices – e.g. Test Driven Development – may appear downright unAgile to them.

How you define the “state of Agile” depends on what the company is trying to do. In general it is something about being responsive to customers – which implies you actually know what customers want in the first place so that must be included too. It is something about delivering fast and generally being quick on your feet.

Talk to another group of people – middle managers mainly – and The Method Perspective appears. Those who take the Methods view are concerned with which Agile Method (Scrum, XP, DSDM, etc.) the company is following. These methods are largely composition from the toolkit, ready made combinations of tools. According to this view following one of the methods will deliver the state of Agile.

(I touched on the these three themes in my Objective Agility presentation (PDF) (Objective Agility on SlideShare) a few years ago and a piece of the same name in Modern Analyst – Objective Agility: What does it take to be an Agile company?)

Almost last but definitely not least there is The Learning Perspective of Agile: I continue to believe that Agile is fundamentally an implementation of Organizational Learning concepts. The idea that all teams, departments, companies learn and those who are able to harness positive learning and change in technology/solution, problem/business and process domains will succeed. He who learns fastest wins.

Organizational Learning is present in one form or another in all Agile and Lean approaches but it is seldom front of stage. Indeed to bring it too far forward would be self-defeating. In working with clients I find that taking the learning perspective of Agile allows me understand and approach the most difficult issues.

For me organisational learning provides the theory and underpinnings of Agile. I should stop there, I could carry on but I’d need a book to do this perspective justice – fortunately I wrote just such a book a few years ago: Changing Software Development: Learning to be Agile.

Given these different perspectives the question we need to ask is: “what is not Agile?”

In many ways this is a more difficult question to answer because while any self-appointed authority can define what is Agile nobody has the authority to declare things unAgile, there is no Supreme Court of Agile. Personally I find it hard to see how business process re-engineering, CRM and ERP systems, virtualisation technology and domain specific languages can be regard as Agile but I have seen all of these cited as Agile tools or enablers.

Which perhaps leads us to the final and most cynical perspective on Agile: The Marketing Perspective.

Agile has grown and changed since it first appeared nearly 12 years ago. It has grown and changed in that time, and with no-one to police the brand it has become all encompassing. Come up with a good idea now and marketing demands you brand it Agile and put it under the umbrella.

There is no one right answer to the question “What is Agile?”. Each of these seven perspectives are good answers and they are not, on the whole, incompatible.

My advice is: when someone says they want to “be Agile” or “adopt Agile” or anything else “Agile” it pays to look beyond the label and ask: “what is Agile to you?” and “what are you looking to achieve?” Unless you know the context “Agile” is meaningless.

What is the right size for a User Story?

A question that comes up again and again from teams is “What is the right size for a User Story?” – a day? a week? three weeks? – 1 point? 5 points? 10 points? 37? 100? 5 words? 20 words?

Short answer: there isn’t a universal right size for a User Story, it depends on your team, their skills and their level of domain knowledge.

Long answer: for some teams a User Story is Big, several days or weeks of work. On other teams they are small – maybe a day’s work. Both are possible, both are the right answer. Although, as a general rule smaller is better.

For me there are criteria that a User Story should meet:

  • It should be small enough for the technical team to understand it and create in in a short time period
  • It is small enough to move on the board sometime soon
  • It should be big enough to represent business value in its own right – it might build on something that has been done before (e.g. a lower fidelity version of the same story, the new one increasing fidelity)
  • It is big enough to be deliverable in its own right – you might not want to do so but if you needed to you could
In an ideal environment: the story would be developed in a few days, would be released on its own right and the business value measured directly. (For those who know about Minimally Marketable Features (MMFs), also sometimes called Business Value Increments (BVI) or Quantum of Value (QoV), I casually consider a User Story to be a MMF.)

When a team are dedicated to a particular product, and have worked on it for several years, and have learned about the domain – as I would expect in for in-house development operations – I expect to see larger stories. Conversely, when a team don’t know the domain – as is common with outsourced development – I expect to see small stories, and more knowledge pull from the client.

Taking a step back, a few basics:

  • A User Story is not in and of itself a requirement; I like to call them “Tokens for Work to be Done”, others like to say they are “A placeholder for a conversation” both are true
  • Don’t feel that you must use User Stories: if the format doesn’t fit your need then don’t use it! You can still have a “token for work to be done” and a “placeholder for a conversation” using any words you like, they don’t have to be “As a… I can … So that…”
  • The User Story format does capture three very important elements: Who, What, Why so they make a good starting point and are easy to understand but if they don’t work for a particular need then don’t force it into the format
Small stories are better – they can be developed more quickly, delivered more quickly and return value more quickly. But there comes a point were stories become too small to return any business value at which point they, well, loose value. Plus, when you have a lot of small stories they become a headache to manage.

One common problem I see is teams who create really small stories in order to get them to fit within one sprint. These means large stories don’t get scheduled or teams split large stories into many small ones which have no business value themselves.

People say this is a Scrum thing, I’m not sure. Scrum doesn’t mandate User Stories, the story format came along after Scrum. True the format is widely used they has become part of Common Scrum.

What Hard Core Scrum does say is that all the work the team commit to should be done by the end of the Sprint. Which would imply a story has to be small enough to fit I the Sprint. The problem then comes: what else do you put in the Sprint? If one story is going to take more than half the Sprint you need one or more stories to use the rest of it up. Plus you hit the commitment problem – developers have an incentive to under commit and the business/custom has an incentive to over demand.

My solution – one which I’m baking into my description of Xanpan – is:

  • Stories are split into Tasks during the planning meeting
  • Tasks do not follow any particular format, nor do tasks have any business value
  • Tasks are for the team – the developers, testers, analysts and anyone else
  • Tasks should be small, a day or two at most.
  • A story is not done until all the tasks are done. Stories can therefore flow across iterations, the chances of tasks flowing across iterations is low (because they are small) but they can and do.
However – and this is the key point – in counting the points (velocity) of a team only the completed Tasks are counted. Partially done tasks are not. The only states that matter are Done and Not Done.

As I’ve written before, the breakdown of stories to tasks fills (at least) three purposes:

  • It is superficially an estimation exercise
  • It is also a requirements elicitation process (I like the Requirements Engineer, BA, Product Owner/Manager to be present)
  • It is a design activity.
You know a task is too big because it is difficult to finish. If it is difficult to finish it needed more thought up front, to define and understand it, to bound it, to understand it. (See the mini-series on Story Points earlier this year, and specifically the Story Breakdown entry.)

So there you have it: there is no right size of a User Story.

That said, they may still be “too big” or “too small”. But what constitutes right it dependent on you, your team and how you play the game.

Downside of Dialogue Sheets

Something I’m very conscious of is that I don’t hear much about the downside of Dialogue Sheet Retrospectives. As much as I’d like to believe that this is because there is no downside I can’t honestly say I think that is the case.

There must be downsides.

I have had several suggestions for improvement and some of these have been incorporated into some sheets. There are also a couple of pending improvements which I’ve yet to action. Because there are several sheets and because not all of them are for retrospectives some improvements get incorporated into some sheets but not others. In part I’m happy with this because it increases the differences between sheets.

At XP Day this week I had several conversations about the sheets and I now think its worth listing some reasons people choose not to use the sheets or negatives that emerge. Some of these I’ve known about for a while and may have mentioned before, some are new.

In no particular order, and in my words summarise what I’ve heard….

“The team looked at the sheet and don’t see how they would be an improvement on the current retrospective format”

This reason has been given by several people/teams and while I respect it as a team’s decision to make I can’t help feeling that it pre-judges the sheets. Surely the “Agile” way of doing it would be to try a sheet and see what happens. After all, what have you got to loose? One or two hours, one retrospective.

“We need a facilitator”

Almost a continuation of the previous one another reoccurring comment is from people who can’t imagine not having a facilitator. Again I would suggest try it and see.

I have observed many retrospective dialogue sheet sessions and I find them very interesting. If a facilitator wants a role I have several suggestions:

  • Take off the facilitator hat and join the retrospective, you will get insights for the next time you facilitator
  • Observe silently from the sidelines: you will learn things just from observing
  • Use the sheets from one retrospective as input to the next, go back and pick up items, delve deeper into issues
  • Use the sheets as first exercise in a longer retrospective; do a sheet then go back over issues that arise
You might not need a facilitator but that does not mean you must not have one.

“Facilitator would go deeper”

I am convinced there are issues and times where having a facilitator will produce a better outcome. For example, some issues benefit from talking about in depth. That is why I always thing the sheets are one tool amount many. However I don’t think you always need to go deep. As with the first two items in this list i encourage you to try it and see.

Cross training retrospective facilitators

This might be a rare reason for not using the sheets but is quite valid. One company wanted to train a number of people to under take retrospectives so each retrospective was used as a training exercise for a facilitator.

“We have a company mandated retrospective format and we cannot change it”

This has to be the worst reason for not trying any new technique in a retrospective. If a team cannot change the retrospective format what chance have they got of acting on any findings from the retrospective? Actually I think this suggests a bigger problem.

Mechanical sheet filling

I only have a second hand account of this and I’d like to know more. It appears a team at a large media company in Salford do not engage in conversation in doing the sheet. It appears filling in the sheet has become something of a chore and they simply “do it”.

With out knowing more about this team (anyone know them?) it is hard to say what is going on. It sound like the sheet has been imposed on the team and they are completing it like another piece of administrative paperwork.

Fear of team dynamics

Some people are worried that personality issues on a team will disrupt the dialogue sheet process – conflicts, arguments, etc. I think this is a valid concern. Although I then immediately think of two things.

First: this is an example of a bigger issue – if these people can’t spend an hour together on a collaborative exercise then working together must be very difficult.

Second could this type of exercise actually help with those dynamics? I would guess the answer is sometimes Yes and sometimes No.

Bored by repetition

There are a few teams that have tried the sheets used the them regularly and then they have fallen into disuse. I suspect this is true of any retrospective format that is used too much. The trick is to find whats to keep the retrospectives fresh and keep them generating new insights.

If you know of any downsides or problems with dialogue sheet retrospective then please let me know. I might be able to correct it in a sheet, if I can’t then we can at least understand their use better.