This episode of AQA is a little more philosophical in nature. It draws on the ideas of Rittel and Webber who described Wicked Problems in 1973. For a definition of Wicked Problems see: https://en.wikipedia.org/wiki/Wicked_problem
I was very influenced by Keith Grint and Clare Holt’s ideas on Leadership and Wicked Problems: https://www.youtube.com/watch?v=w5gt8zrXjGQ&t=9s
Listen to it on Apple podcasts here:
Soundcloud: https://soundcloud.com/steve-green-575977244/aqa-7-is-product-development-a-wicked-problem-and-how-might-agile-help
Watch on Youtube:
Here’s a transcript:
Hi there! Welcome to Agile Questions Answered Episode seven.
Before i’ll get started I’d just like to say I’ve been checking out the stats and we have 185 of you have listened to my podcast so far which if you think of it in terms of a room behind the pub that would make it uh pretty crowded.
I know it’s not great for a global audience but I’m pretty happy that you’ve found my podcast, and some of you at least are listening for long enough to maybe gain something from it.
Thanks very much! If you think that anyone else would benefit from listening to my podcast then please recommend it to them and if you have a question that you’d like me to have a go at then please get in touch.
Anyway on with the show. Today’s question is a bit of a cheat really because it’s a kind of contrived question out of some research and some thinking I’ve been doing over the past couple of weeks.
The question is: What are Wicked Problems and how might Agile help to solve them?
A long time ago when I started in this industry in my first proper job after university I remember thinking maybe naively that decision making in business would be really easy because it’s just about what’s profitable in the end.
Imagine my surprise when as a junior electronic engineer when I spent a lot of time sat in meetings with people who didn’t seem to be that energized by the problem that we were trying to solve but they seem to be very exercised by sort of putting their own point across. You get certain people who no matter what the topic of conversation would use it to advance their own agenda.
And it’s very strange: when I started doing agile in the early 2000s I had the same attitude: I had seen the big process thing where people were trying to get better at delivering software products by putting more and more process in place and it wasn’t working. When i discovered agile as a kind of an alternative to that, and I tried it in a couple of places and it was very successful I had that same attitude. I thought “This actually works; Surely people will abandon the old process heavy way of doing things for this much more effective approach.”. As i have said in some of my previous podcasts, the traditional approach the step-by-step waterfall approach to product development is kind of the obvious way to do it but in my opinion it’s not the right way to do it. Coming across wicked problems, I’ve realized that there’s something more subtle and complex going on.
So what are Wicked Problems? Well, you can understand them by comparing them to Tame Problems. Tame Problems are problems that are solvable just by following a series of steps, for example a quadratic equation. All quadratic equations can be solved correctly by going through a certain sequence: If you know that sequence and you’re able to do a bit of algebra you can solve any quadratic equation. When you come up with the numbers that answer will be right or it’ll be wrong. Tame problems can be really really complicated – the number of steps that you have to go through might be a hundred, but essentially if you go through all of those steps correctly then you will get the right answer. That’s a Tame Problem.
To understand why product development is not a Tame Problem let’s look at developing a single feature for a software product. So let’s say that what you want to do is allow people on your – i don’t know, business accounting project to upload an Excel spreadsheet with a budget. Then you will display it in your accounting software and you’ll do various things with it. So the first thing you’re going to have to do after you’ve done the upload is actually understand the Excel file format. Now you could write the software to do that by hand from scratch: The Excel file format nowadays is a published format so it’s perfectly possible to do, but it is months of work. It’s literally months of work. The great thing is that whatever language that you’re using, somebody will have written that program already, and they will have made it available to you as a software library. inevitably when a developer is faced with a problem like this where they need to put in new features the first thing that they’re going to do is look for somebody else who solved that problem before and reuse that software. Well that’s all well and good. Integrating that library into your code into your software will be easy or it will be hard. There may be things that don’t quite work in the way that you expect but you can certainly do it.
But one think about: What happens once you’ve done that? You’ve now got somebody else’s code incorporated into yours and at any time out in the big wide world somebody can find a problem with that code and that code will then require updating. It may need updating in a way that breaks your software. You’ll have to change the way that your software works in order to incorporate the updated version. Then you’re faced with a difficult decision, which is either to leave your software with the old version or upgrade to a new one. In some cases that might not be possible for two reasons: one is that the problem might be a serious security fault which would leave your software product open to exploitation; the other is that file formats evolve over time and you might find that you need to upgrade so that you can support the latest version of Excel for Office.
Now the problem is there is no way of anticipating this. You no idea if and when that’s going to happen and whether those things are going to be relevant to you. This is a problem that simply cannot be solved there isn’t a right or wrong answer. The best thing that you can do is choose the better answer. For example in the Open Source world you might have a choice of write your own library, and then there might be three other libraries that do things. Typically what you would do is you would look to see what other people are doing: The library that most other people are using is the library that you would choose. There’s very good reasons for that: Firstly is that if a library’s got a lot of support then it’s likely to be kept up to date; the other thing that’s slightly more subtle is that if a library is being used in lots of different places then the chances are that somebody else will find a defect in that library before you do and therefore you won’t be exposed to the problem. If you write your own software then it’s absolutely 100% guaranteed that you will be the person who finds the defect in it and it will affect your code.
You can choose to use libraries or not but there are there isn’t a right or a wrong answer, it’s just a better or a worse answer. This is a key feature of Wicked Problems. Wicked problems don’t have right or wrong answers, they have better or worse answers.
Another key feature of Wicked Problems is there’s no stopping rule. What that means is that it’s not clear when you’ve solved the problem. To take our example of a software library that’s reading in and understanding an Excel spreadsheet – that’s not something that you’re going to do once. You’re going to have to look up out for changes in the Excel spreadsheet format and changes to the library. You’re going to have to keep redoing the same solution. This is a key feature of Wicked Problems as well.
Hopefully i’ve persuaded you that software development and product development are Wicked Problems. It is very difficult for everyone to accept that. There’s a lot of people who want to think that everything has a right or wrong answer and that we just haven’t found it yet. You actually see this in science. There’s quite a lot of eminent scientists who believe that everything is ultimately explainable and there’s kind of a unified theory of everything.
There are other scientists who believe that it’s more like a kind of fractal picture: You you find a solution which describes some kind of phenomenon which you can use to make predictions which is what science is. But when you do that it opens up another mystery. There’s a more subtle mystery that you have to think about. Take a simple example: Newtonian Mechanics explains in a really good way how most of the world that we see works and how bodies interact with each other. It’s only when you get down to the very small that it breaks down and you have to use another model which is quantum mechanics.
If you’re like me, you’d say well eventually when we fully understand that we can explore the limits of quantum mechanics we might find that we need another kind of mechanics to explain the kind of sub- sub- sub-particles because quantum mechanics won’t work with them.
Maybe the upshot of all of this is that in order to successfully develop products you need to have people who are very good at making these decisions where there’s a load of uncertainty. It’s occurred to me that this is the thing that I’ve been training myself to do for the last 30 odd years, since i got into this game. I’ve got into trying to find the best answer that will help us to move forward rather than the answer that will solve the problem for all time. I hesitate to draw this parallel but it’s like crack army teams. For example in the UK, the SAS or in the USA, the Navy Seals. They’re trained to go into a completely new situation that they know nothing about but to use their training to determine what the best actions are at any one point in time.
In conclusion, where does this leave us? Well I think, if you’re convinced by this – that product development is a Wicked Problem and no amount of process, no matter how rigorous and no matter how tiny the steps in the process are, is going to eliminate the uncertainty from product development then you need a new approach. Agile gives you that framework to make better decisions. Essentially agile says, “Where are we now? We’ve got this vision for the product that we want to create. We know we’re a long way from there. We can take a step towards that vision and then we can look around and see where we are. We thought that that step was in the right direction: Was it actually in the right direction or do we need to reset our sights and change direction slightly?” Agile gives you that framework: You can make better or worse decisions but the great thing about agile is you find out really quickly whether the decision was a better one or a worse one.
You also need the people with the right skills. You need people who are comfortable operating in these areas of uncertainty; who are comfortable taking decisions in the knowledge that there’s a really decent chance that they might turn out to be wrong. But they can still move forward and they have the skills to evaluate the the situation that they’re in and choose what’s the best decision given the data that’s available to them at the time.
Those people are not some sort of strange geniuses: This decision-making ability, the ability to appraise the situation that you’re in and make a decision based on that – this is a skill that skill can be taught. In fact, years ago I did Kepner Tregoe Problem Solving and Decision Making. It’s a very useful systematization of the kind of things that i and other practitioners in this area would do almost intuitively.
I hope that that’s been useful for you. I think it’s given me a good insight – my thinking over the last couple of weeks – into why people find agile difficult to take on. I think that like it or not there are people, and i i don’t think it’s like an intelligence thing, that they want the world to be orderly: They want there to be answers to questions. The way i see the world, it’s not like that. The way i see the world there’s there’s better and worse and sometimes you just have to make a choice. There’s a Tom Waits song called “Fumbling with the blues”. There’s a line in that which I always liked which goes, “Two dead ends and you’ve still got to choose”. Generally there is a better option but sometimes the options are equally bad and you just have to choose one of them and then deal with the fallout.
I think that you can see that the Agile framework does offer some hope in this area. I think that using that understanding that what you’re doing is you’re making better or worse decisions – you’re not making right or wrong decisions can be really helpful .
I hope you enjoyed that and i will post another podcast in another few weeks
Bye for now