Agile on the Beach – update on review process

294 submissions
54 volunteer reviewers
2,400 votes
2 rounds of review
40 accepted submissions

Over the weekend I did the hardest thing I do all year: I sent the “sorry your submission to Agile on the Beach has not been chosen.” The declines.

I have explained how we review sessions for AOTB before but things have changed a bit so I owe it to submitters an explanation. So here goes…

This year, as usual, we had nearly 300 submissions to speak at Agile on the Beach. There are only 40 speaking slots – the exact number varies a little because we make minor changes to the schedule.

We have two review rounds. In round one reviewers score submissions from 1 to 5 – with 5 being the best. From this we create a shortlist for each track. Rule of thumb is the shortlist is twice the number of available slots (5 slots for Thursday tracks, 4 for Friday and 9 for two day tracks, so 10, 8 and 18.)

The first change for 2020: round 1 was done blind. That is anonymously, speaker bibliography details were not shown to reviewers.

Now on the one hand this is fairer: it gives more chance to new speakers, it stops the same old names dominating the conference and hopefully promotes diversity.

On the other hand: reviewers have often seen speakers previously and know who is good and who is not so good. It also has a commercial hit because a programme with more known names is likely to be more attractive to ticket buyers. So 2020 was an experiment.

I think blind reviewing worked. Although it does mean a few regulars did not make it and I feel bad about that, they are my friends. I also think we were right to look at speaker bios in round 2.

While we set no rules about speaker self-identification (i.e. some speakers used the synopsis to give their name or even mini-bios) a few reviewers didn’t appreciate this and in their comments noted they scored such submissions lower.

Back to shortlisting… when shortlisting we look at the mean score in reviews and the median. Usually the first few, highest scoring, sessions are easy to pick. It is the border line ones which are hard to decide.

In round two reviewers are asked to rank the shortlist. There can be only one number 1. The lowest score depends on how long the shortlist is, this varies by track. (So oddly, round 2 reverses scoring, 1 is top.)

In both rounds reviewers are encouraged – but not compelled – to write an explanation of their score. This is used by us as reviewers when deciding border line submissions and provided to the submitter as feedback.

In the past this process has needed reviewers to review 70 or more submissions. A few reviewers look at everything, all 300. I stopped doing this myself about two years back. Our submission system (Mimas) tries to ensure that submissions get an equal number of reviews.

Because reviewers were reviewing so many sessions feedback declined, I suspect attention to submissions decline too but I don’t know for sure.

The other big change for 2020: we recruited more reviewers. Actually, we invited everyone who had already bought a ticket, and everyone who attended in 2019 to review for round 1. 54 people volunteers and were allocated sessions to review. About a dozen people didn’t do their reviews but we still had plenty of scores.

This created more work for me than expected. While Mimas was updated to cope with this is wasn’t completely smooth. Whats more some reviewers left it to the last minute to review which created problems and meant I had to chase people.

Round 2 reviewing was more limited. Only about 12 reviewers picked including some AOTB committee members.

Over the years, and especially since I wrote Mimas, AOTB selection has become more score dependent. This has its good and bad sides but it does mean we get it done faster.

Once round 2 is done we look at the average (mean and median again) ranks and count of from the top to fill the track. Then we apply some judgement: people who submit in two tracks normally only speak in one. We would like a male/female balance but we don’t have any hard and fast rules plus, we don’t know if we are being fair on other diversity criteria. Are dyslexics fairly represented? Ethnic minorities? and so on.

We also have a financial consideration. AOTB pays a travel allowance dependent on how the speaker travels. This year when we completed selection and looked at the costs we were in budget. That doesn’t always happen, some years we have to cut back on long-haul speakers. Some years we argue and get more money.

Finally, we send acceptances. We ask everyone who is accepted to confirm they will attend and speak. Hopefully everyone says Yes and says so quickly. However we often loose people for one reason or another. Hence there are quick substitutions – that is why we hold off sending declined. Once we have the lineup settled we send the declines.

Every year we decline some amazing sessions that we would really like to host. We only have so much space. Every sessions we accept means another we can’t accept. We have enough good material for two conferences but only have one.

Whether you have been declined or accepted for 2020, whether you are thinking of submitting in 2021 or just trying to find out how conferences operate I hope that helps.

You might also want to look at:

Finally, you can see and try Mimas our conference review system – this is now OpenSource on GitHIb. I’ll blog more about this soon.
(Now reviewing is over I have a bunch of updates to do so the system may be up and down randomly this week.)


Like this post? – Like to receive these posts by e-mail?

Xanpan

Subscribe to my newsletter & receive a free eBook “Xanpan: Team Centric Agile Software Development”

An (agile) coach’s dilemma

iStock-521796758s-2019-02-13-14-54.jpg

I’d never met the team before. It was a small company in Cornwall and the big boss man was away on holiday that week. They took me upstairs to the meeting room. We talked for about an hour and I could tell there was something on their mind.

Eventually one of them asked:

“we have this question we’ve been talking about for ages we’d like you to help with.”

This was it, the $64,000 question.

“Ask” I said.

“Well… should we be using source code control?”

To some of you this might sound funny, to others shocking, but believe me, on average I meet one team a year who don’t use source code control. On this day I had two voices, one in each ear.

The voice in my left ear said:

“You are only a coach, you are here to help them make their own decisions. Talk about their goals, what they want to achieve, find out if they think source code control could help them. Let them explore why it might be the wrong choice.”

The voice in my other ear said:

“Jesus Christ…”

On the one hand the (agile) coach is there to help the team reach their best. The coach isn’t there to tell them what to do, the people – the team – are the experts in what they do, and they are self-organising. The coach is there to help them unlock their superhero powers.

On the other hand: the coach has done this before. If they are any good they have not only worked with many teams before and read many reports in books and blogs but they wrote the reports, their teams are in the case studies. The team may well be experts in Java, JavaScript, inertial navigation, limb-replacements or whatever but the coach is the expert in agile. The coach is there to teach.

If you’ve read about coaching in other contexts you might recognise this as a question of non-directive coaching v. directive coaching. Agile, and agile coaching has never really come to terms with that differentiation.

As a coach you have next to no authority, all you can hope is that the coachees come to respect you and trust you enough to follow your suggestions. But then, maybe you shouldn’t be suggesting anything?

But if you claim to know anything – about coaching, about agile and heaven forbid software development – is it right to hold back on them when you know the answer? Isn’t it dishonest not to say what you know – or at least sincerely believe – to be true?

And if you can see what they need to do, don’t tell them but work to help them see that answer, well… thats just manipulation isn’t it?

When do you allow free will and when do you railroad?
When do you unlock self-knowledge and when do you teach?
When do you let people take decisions you can see are mistakes?
When do you know facts that will help? And when are you just full of the same biases as everyone else?

Take the documentation question: classically trained developers who have never worked in high-performing teams commonly see documentation as the answer to so many questions:

Question: How do we make sure requirements are clear?
Answer: Documentation

Q: How do we help new recruits find their way around the system?
A: Documentation

Q: How do we keep track of the code design?
A: Documentation

Q: How do we agree which bugs to fix?
A: Documentation

Q: How do we communicate with customers?
A: Documentation

Some people just don’t know what they don’t know.

I too was trained that documentation was the answer to these questions and more, I too saw the lack of documentation as a major problem when I started work somewhere new. It took time (and Railtrack PLC) for me to realise documentation wasn’t the answer, it was John Seely Brown and Julian Orr who help me to realise documentation was a problem, and it took Capers Jones to make me see the cost of documentation.

But should I impose my view of documentation on a team? – should I even be making them see the world as I do?

Ultimately the team are self-organizing. They have the right to document, or not to document, and they can decide to ditch the coach. (Being an agile coach can be a high risk profession.)

Ultimately they are allowed to self-organize long (seated) morning meetings. They are allowed to reject TDD, BDD, CD, CI and just about every other agile practice.

And you know what? They could be right.

Coaches need to be self-aware and with that self-awareness comes self-doubt. Just because a team doesn’t follow the normal rule book doesn’t mean they are wrong. They could have a better solution. They could have a solution that works better in their context.

Back in Cornwall, I paused for a few moments while the angels on my shoulders argued their case. Then I said:

“Put it like this, without source code control I wouldn’t get out of bed in the morning.”

With that question settled I moved onto the issues. What problems would it create? Why wouldn’t you use source code control? What is the worst that could happen? What difference would big boss man see?

Arse about face really but it worked. The following morning I walked into their office and found them checking everything in.

Eight years later that small company has grown more than 20 fold: would they have done that if I had answered differently? Might they have done even better? Did I make a difference or was it all them?

But I still face dilemmas like that everyday I’m “coaching”.


Like this post?

Like to receive these posts by e-mail?

Subscribe to my newsletter & receive a free eBook “Xanpan: Team Centric Agile Software Development”

Agile elevator pitch

My (our) entry in the Agile elevator pitch competition:

“[Agile] Provides philosophy, techniques and tools to alleviate the pain of traditional development and make teams more effective thus increase your profit.

Companies such as the BBC, GE Energy, Yahoo, the Financial Times, The Guardian and others have already adopted the approached.”

As some people know, I’ve been doing a lot of work in Cornwall recently. This involves working with a variety of companies all involved in software development – from online e-commerce website builders to companies creating embedded software for medical devices.

My partner in this endeavour, Michael Barritt of Oxford Innovation and Grow Cornwall, suggested we really need an elevator pitch statement for what all this Agile is about. The above is our result.

Of course this is context specific. Too many senior managers this is irrelevant because they don’t know anything about software development. At that kind of level Agile itself becomes meaningless because it is a solution to a problem which they know nothing about. And actually, they don’t want to know about.

There is always a danger with Agile elevator pitches, or any other type of elevator pitch, that it just becomes “Will increase your return on investment.” At some point such pitches become meaningless, you don’t know if the product will fix your software development issues, cure cancer or make you tea in the morning.

So what do you think?
Good?
Bad?
Indifferent?
Got a better one?

%d bloggers like this: