AQA 7 – Is Product Development a Wicked Problem and how might Agile help?

This episode of AQA is a little more philosophical in nature. It draws on the ideas of Rittel and Webber who described Wicked Problems in 1973. For a definition of Wicked Problems see:

I was very influenced by Keith Grint and Clare Holt’s ideas on Leadership and Wicked Problems:

Listen to it on Apple podcasts here:


Watch on Youtube:

Here’s a transcript:

Hi there! Welcome to Agile Questions Answered Episode seven.

Before i’ll get started I’d just like to say I’ve been checking out the stats and we have 185 of you have listened to my podcast so far which if you think of it in terms of a room behind the pub that would make it uh pretty crowded. 

I know it’s not great for a global audience but I’m pretty happy that you’ve found my podcast, and some of you at least are listening for long enough to maybe gain something from it. 

Thanks very much! If you think that anyone else would benefit from listening to my podcast then please recommend it to them and if you have a question that you’d like me to have a go at then please get in touch. 

Anyway on with the show. Today’s question is a bit of a cheat really because it’s a kind of contrived question out of some research and some thinking I’ve been doing over the past couple of weeks. 

The question is: What are Wicked Problems and how might Agile help to solve them? 

A long time ago when I started in this industry in my first proper job after university I remember thinking maybe naively that decision making in business would be really easy because it’s just about what’s profitable in the end. 

Imagine my surprise when as a junior electronic engineer when I spent a lot of time sat in meetings with people who didn’t seem to be that energized by the problem that we were trying to solve but they seem to be very exercised by sort of putting their own point across. You get certain people who no matter what the topic of conversation would use it to advance their own agenda. 

And it’s very strange: when I started doing agile in the early 2000s I had the same attitude: I had seen the big process thing where people were trying to get better at delivering software products by putting more and more process in place and it wasn’t working. When i discovered agile as a kind of an alternative to that, and I tried it in a couple of places and it was very successful I had that same attitude. I thought “This actually works; Surely people will abandon the old process heavy way of doing things for this much more effective approach.”.  As i have said in some of my previous podcasts, the traditional approach the step-by-step waterfall approach to product development is kind of the obvious way to do it but in my opinion it’s not the right way to do it. Coming across wicked problems, I’ve realized that there’s something more subtle and complex going on. 

So what are Wicked Problems? Well, you can understand them by comparing them to Tame Problems. Tame Problems are problems that are solvable just by following a series of steps, for example a quadratic equation. All quadratic equations can be solved correctly by going through a certain sequence: If you know that sequence and you’re able to do a bit of algebra you can solve any quadratic equation. When you come up with the numbers that answer will be right or it’ll be wrong. Tame problems can be really really complicated –  the number of steps that you have to go through might be a hundred, but essentially if you go through all of those steps correctly then you will get the right answer. That’s a Tame Problem. 

To understand why product development is not a Tame Problem let’s look at developing a single feature for a software product. So let’s say that what you want to do is allow people on your –  i don’t know, business accounting project to upload an Excel spreadsheet with a budget. Then you will display it in your accounting software and you’ll do various things with it. So the first thing you’re going to have to do after you’ve done the upload is actually understand the Excel file format. Now you could write the software to do that by hand from scratch: The Excel file format nowadays is a published format so it’s perfectly possible to do, but it is months of work. It’s literally months of work. The great thing is that whatever language that you’re using, somebody will have written that program already, and they will have made it available to you as a software library. inevitably when a developer is faced with a problem like this where they need to put in new features the first thing that they’re going to do is look for somebody else who solved that problem before and reuse that software. Well that’s all well and good. Integrating that library into your code into your software will be easy or it will be hard. There may be things that don’t quite work in the way that you expect but you can certainly do it. 

But one think about: What happens once you’ve done that? You’ve now got somebody else’s code incorporated into yours and at any time out in the big wide world somebody can find a problem with that code and that code will then require updating. It may need updating in a way that breaks your software. You’ll have to change the way that your software works in order to incorporate the updated version. Then you’re faced with a difficult decision, which is either to leave your software with the old version or upgrade to a new one. In some cases that might not be possible for two reasons: one is that the problem might be a serious security fault which would leave your software product open to exploitation; the other is that file formats evolve over time and you might find that you need to upgrade so that you can support the latest version of Excel for Office. 

Now the problem is there is no way of anticipating this. You no idea if and when that’s going to happen and whether those things are going to be relevant to you. This is a problem that simply cannot be solved there isn’t a right or wrong answer. The best thing that you can do is choose the better answer. For example in the Open Source world you might have a choice of write your own library, and then there might be three other libraries that do things. Typically what you would do is you would look to see what other people are doing: The library that most other people are using is the library that you would choose. There’s very good reasons for that: Firstly is that if a library’s got a lot of support then it’s likely to be kept up to date; the other thing that’s slightly more subtle is that if a library is being used in lots of different places then the chances are that somebody else will find a defect in that library before you do and therefore you won’t be exposed to the problem. If you write your own software then it’s absolutely 100% guaranteed that you will be the person who finds the defect in it and it will affect your code. 

You can choose to use libraries or not but there are there isn’t a right or a wrong answer, it’s just a better or a worse answer. This is a key feature of Wicked Problems. Wicked problems don’t have right or wrong answers, they have better or worse answers. 

Another key feature of Wicked Problems is there’s no stopping rule. What that means is that it’s not clear when you’ve solved the problem. To take our example of a software library that’s reading in and understanding an Excel spreadsheet – that’s not something that you’re going to do once. You’re going to have to look up out for changes in the Excel spreadsheet format and changes to the library. You’re going to have to keep redoing the same solution. This is a key feature of Wicked Problems as well. 

Hopefully i’ve persuaded you that software development and product development are Wicked Problems. It is very difficult for everyone to accept that. There’s a lot of people who want to think that everything has a right or wrong answer and that we just haven’t found it yet. You actually see this in science. There’s quite a lot of eminent scientists who believe that everything is ultimately explainable and there’s kind of a unified theory of everything. 

There are other scientists who believe that it’s more like a kind of fractal picture: You you find a solution which describes some kind of phenomenon which you can use to make predictions which is what science is. But when you do that it opens up another mystery. There’s a more subtle mystery that you have to think about. Take a simple example: Newtonian Mechanics explains in a really good way how most of the world that we see works and how bodies interact with each other. It’s only when you get down to the very small that it breaks down and you have to use another model which is quantum mechanics. 

If you’re like me, you’d say well eventually when we fully understand that we can explore the limits of quantum mechanics we might find that we need another kind of mechanics to explain the kind of sub- sub- sub-particles because quantum mechanics won’t work with them. 

Maybe the upshot of all of this is that in order to successfully develop products you need to have people who are very good at making these decisions where there’s a load of uncertainty. It’s occurred to me that this is the thing that I’ve been training myself to do for the last 30 odd years, since i got into this game. I’ve got into trying to find the best answer that will help us to move forward rather than the answer that will solve the problem for all time. I hesitate to draw this parallel but it’s like crack army teams. For example in the UK, the SAS or in the USA, the Navy Seals. They’re trained to go into a completely new situation that they know nothing about but to use their training to determine what the best actions are at any one point in time. 

In conclusion, where does this leave us? Well I think, if you’re convinced by this – that product development is a Wicked Problem and no amount of process, no matter how rigorous and no matter how tiny the steps in the process are, is going to eliminate the uncertainty from product development then you need a new approach. Agile gives you that framework to make better decisions. Essentially agile says, “Where are we now? We’ve got this vision for the product that we want to create. We know we’re a long way from there. We can take a step towards that vision and then we can look around and see where we are. We thought that that step was in the right direction: Was it actually in the right direction or do we need to reset our sights and change direction slightly?” Agile gives you that framework: You can make better or worse decisions but the great thing about agile is you find out really quickly whether the decision was a better one or a worse one. 

You also need the people with the right skills. You need people who are comfortable operating in these areas of uncertainty; who are comfortable taking decisions in the knowledge that there’s a really decent chance that they might turn out to be wrong. But they can still move forward and they have the skills to evaluate the the situation that they’re in and choose what’s the best decision given the data that’s available to them at the time.

Those people are not some sort of strange geniuses: This decision-making ability, the ability to appraise the situation that you’re in and make a decision based on that – this is a skill that skill can be taught. In fact, years ago I did Kepner Tregoe Problem Solving and Decision Making. It’s a very useful systematization of the kind of things that i and other practitioners in this area would do almost intuitively. 

I hope that that’s been useful for you. I think it’s given me a good insight  – my thinking over the last couple of weeks – into why people find agile difficult to take on. I think that like it or not there are people, and i i don’t think it’s like an intelligence thing, that they want the world to be orderly: They want there to be answers to questions. The way i see the world, it’s not like that. The way i see the world there’s there’s better and worse and sometimes you just have to make a choice. There’s a Tom Waits song called “Fumbling with the blues”. There’s a line in that which I always liked which goes,  “Two dead ends and you’ve still got to choose”. Generally there is a better option but sometimes the options are equally bad and you just have to choose one of them and then deal with the fallout. 

I think that you can see that the Agile framework does offer some hope in this area. I think that using that understanding that what you’re doing is you’re making better or worse decisions – you’re not making right or wrong decisions can be really helpful .

I hope you enjoyed that and i will post another podcast in another few weeks

Bye for now

AQA 6 – What do people get wrong about Agile?

In this episode I answer the question, what do people get wrong about Agile.

Listen to the podcast in Apple podcasts

You can listen on Soundcloud:

You can watch it here on Youtube:

If you prefer you can read the Transcript here:

Hello there!
This is AQA – agile questions answered episode 6 and today’s question is
What do people get wrong about agile?
I’ve got a number of thoughts on this which I would like to share with you over the next couple of minutes.
So the first thing I think people get wrong about Agile is they assume that if you’re Agile there’s no discipline or control at all and in fact I think that the opposite is the case.
Agile is very very disciplined, I would say more disciplined than traditional ways of developing products.
It’s just that the discipline takes a different form.
Another thing that people think is that if you’re Agile then you have to jump as soon as the customer asks you and do whatever they want.
That’s not the case either. I get this from teams who have grown up in agile sometimes and I have to remind them that it’s Agile compared to the way that software products used to be developed. So in the old days somebody would have to wait 18 months to get a feature. With agile development you should be able to demonstrate something every couple of weeks maybe even every week, and get to a position where people can actually use your software in a month or two.
So when you compare that with the old world where you’d have to wait for everything to be delivered before you’ve got anything then that’s pretty agile.
Another thing that people get wrong about Agile is that they think that Agile is Scrum and if you’re not doing Scrum you’re not Agile. There are other people who think that Agile is kanban and if you’re not doing kanban you’re not Agile.
Those are ways to achieve agility but really they’re just kind of particular methodologies and disciplines that are Agile.
You could theoretically do Scrum and do everything there is straight out of the Scrum guide without being Agile and I’ll come on to that now

The key thing with Agile is you deliver real things to your customer in small increments and you use the feedback from the customer that they are getting through using the product, whether it’s software or it’s some other kind of thing in the real world, and you use that feedback back to change what you’re going to do next with the product. And that’s the key to being Agile: It’s about getting something out there quickly seeing how people use it and using that insight to improve it. So you could develop something iteratively: So you could develop a bit, and then you could develop the next bit, and you could develop the next bit, and you could deliver something every two weeks, but if you never change the order of what you were going to deliver you wouldn’t be Agile. That would just be an iterative development.
So it’s really getting that feedback and then using that feedback to change what you do is the critical thing about being Agile.

And then the final thing that i’m going to go through here that I think that people get wrong about Agile, is that they think that Agile makes you deliver faster and it doesn’t really:
If you had a project that had one big bang delivery right at the end and it was 18 months long, theoretically that project would deliver faster than the same amount of functionality delivered in an agile manner. Because like it or not there is an overhead to Agile: The meetings that some people complain about: The regular planning meetings, the daily stand-up meetings, they are an overhead so compared with this kind of theoretical, “Let’s just let everyone do something for 18 months and then we’ll have a lovely product at the end of it.”, there’s more cost and it’s not as quick.
Where you do get the benefits is that you get something that you can give to your customers earlier, and also you avoid developing all that stuff which is never going to be needed so you don’t develop the wrong stuff.
When you put something out there you put a small version out and if it’s wrong you can take it out, and you don’t develop a fully flavored version of that thing. So really, although you’ve got a bit of an overhead because you’re kind of checking what you’re doing every couple of weeks, and then you’re changing what you’re doing, and that does take effort, in the long run you really see the advantage of Agile because you’re not wasting months developing a thing that when you put it out into the market nobody’s going to need.
Another thing people get wrong about Agile is that they think it’s a new and sexy thing, when in fact Agile has been around since the mid-90s. So if you think about it that’s 25 years now, which is as long as the previous dominant process which was waterfall.
You know that was invented in the early 70s. So 70 to 95, that’s the kind of area of waterfall, and 95 to 2020 – 2021 now is agile. So it’s hardly new.
Another thing people get wrong about Agile is that they think it’s incompatible with traditional project management processes. Things like PRINCE2, PMI, PMBOK, and so on. I found that very much not to be the case. The way i think of it is that the project management side of it faces out. So it’s about saying, “If we spend this amount of money; if we get these people together, then this is what we’ll achieve, and we’ll achieve it in these stages.”
Whereas the agile part of it is very much how we organize the team to deliver those things. The other thing is that a lot of project management stuff is not to do with the product development. So it might be to do with finding an office for the staff to be in, or for example training materials, a publicity campaign – So all of these things are not necessarily part of the product development but they are necessary in order to get everything that’s involved in the project done.
So there are many other things that people get wrong about Agile but I think that they are the major things.
I’d love to know what you think about it, so if you’d like to hit me up on LinkedIn, or Twitter or facebook or whatever then that would be great. Hopefully you’ll find this interesting and you’ll stay with me and look out for the next episode of Agile Questions Answered.

Agile incentives 1 – the CEO

This is the first in a series on agile incentives. In it I will take a number of key roles within an organisation, outline the high-level incentives that person has, and then discuss how the adoption of agile aligns with those incentives.
Today I’m going to start right at the ‘top’ of the organisation: the Chief Executive Officer (CEO).

Continue reading Agile incentives 1 – the CEO

Specialisation and agile

This article was originally published here on LinkedIn

The Scrum guide states that it “recognizes no titles for Development Team members other than Developer, regardless of the work being performed by the person”. I have read a number of posts and articles recently that seem to have taken a very narrow reading of this statement and imply (or even claim quite strongly) that specialisation within an agile team is a bad thing. Interestingly, it is often certain specific specialisations that these people seem to dismiss. These include business analysis (or service design), UX design, and sometimes technical architect. It’s notable that I’ve never seen anyone describe software development as being a valueless specialisation. Some of these articles I seem to imply that a goal of the agile coach should be to ensure that every member of the agile team acquires all of the necessary skills to deliver the product. In this agile nirvana they would have not just technical skills but also the soft ones such as eliciting requirements and understanding underlying business drivers.

Continue reading Specialisation and agile

My new MacBook Pro

I’ve always been a believer in the idea that professionals in whatever walk of life should have the best tools available and this seemed like a pretty good justification to upgrade from my 3 and a half year old MacBook Air to the latest MacBook Pro. After a lot of agonising I went for the 13 inch touchbar model with extra memory, 500GB SDD and standard processor.

Continue reading My new MacBook Pro