I was attending a presentation about Scrum where the speaker suggested that we stop lying to customers. I was blown away. OK, let me backtrack for a second here. The conversation started in the context of predicting future deliverables for a project, the types of things that contracts are frequently made of – project milestones, scope definitions, feature lists, dates. It’s a typical artifact of a sales process and it is supposed to ensure that the customer will receive described functionality and feature sets according to the outlined dates. What we all know is that you can fix scope or you can fix dates, but not both. (Or else quality suffers.) Nothing new there. But what struct me in the argument the speaker was making was that since we are are notoriously bad at predicting the future, then any such promises to clients are essentially lies. That’s right. Making up project schedules way into the future is wrong because it is unethical. I never thought of it from that point of view. It seems that if we are doing our best effort at predicting the future and we are using some tools (historical evidence, estimation, risk analysis), then this is a legitimate practice. However at the end of the day, whether we are estimating or guesstimating the future, as humans, we are really bad at it. And since it’s pretty much a guarantee that our predictions are false, there is no use at misleading clients into thinking that we know what we are talking about. In other words, stop lying to clients about things we can’t predict.
This reminded me of a case some time ago where we had a large client implementation that went on for more than a year. Some fairly detailed documents were prepared in the beginning of the process where client was lead to believe that certain functionality would be delivered by a certain date. More than a year later, the client was losing patience as these promises were clearly not being delivered. At this point frustration started to really build up. How much longer would they have to wait? Why was everything behind schedule? What happened? But in fact, nothing unusual happened. Some detailed predictions were made and they turned out to be wrong. This was happening at the time when we were making some changes towards adopting Scrum more widely throughout the company. We were not going to quickly finish the project and client was getting increasingly more upset. There was nothing left to do, but to come clean. To put it bluntly, stop lying. We had to admit to the client that we didn’t know when the functionality would be done (they were shocked!) but that we would involve them in the process by doing show & tells every two weeks. We would share the product backlog, and even let them prioritize the user stories. The response was surprising positive. Sure, the client was still disappointed by later dates. However they appreciated the transparency and the power to influence the outcome and the pace. After a few sprints, they saw that development was happening at a steady pace and that they could pick and choose what we work on. We discussed mockups and user stories with them. We de-prioritized nice to have features. The client felt in control and finally relieved. We turned a corner with a frustrated customer and eventually turned them into a happy and satisfied customer. All we had to do was to stop lying.
There is a song by Aloe Blacc called I need a dollar that pops into my head when I talk to sales people:
“I need dollar dollar, a dollar is what I need.
And if I share with you my story would you share your dollar with me.”
Everyone in R&D estimates user stories and features in points, using (roughly) the Fibonacci sequence. That’s the little ritual we participate in, called Planning Poker. Story points are not something that sales people ever heard of. What they heard of is a different currency. Ahem… dollars, actually. Turns out it’s pretty much the only thing they care about. Moreover, it’s something that our clients deeply care about. The question that comes from a sales person usually goes something like this, “I think the client would really like Feature X, if we could build it. In fact, I’m pretty sure I can sell it to them. How much should I charge for it?”. Well that depends. On a lot of things.
If you want to sell the moon and the stars, it’s pretty hard to put a dollar value to it. The easiest way to do that is by using statistics. I keep track of something I call a “points exchange rate”. As a company, we bill clients a certain daily rate for customized development. Every user story gets estimated in points and when it’s completed, we know how much time it took. Time is converted to dollars based on the daily rate we charge clients. Over time it’s easy to average the number of hours spent on a user story that’s 1, 2, 3, 5, or 8 points. Those averages give a pretty good indication of how long a story will take. Usually the accuracy is better on the lower end of the Fibonacci sequence. For example, a 1 point user story will take about 10 hours, 2 point user story around 25, plus or minus 5. A 5 point user story will take anywhere from 50 to 80 hours and so on. It’s better to keep out the outliers from these averages – those stories that had confusing requirements or special considerations only applicable to that client. Without the outliers, you can estimate the cost of development in dollars based on current averages for user stories of each size. That’s the posted exchange rate for story points. With each new user story developed, the points to dollars exchange rate is adjusted based on the new averages. It works just as well with large features as it does with small user stories. It becomes pretty difficult to estimate in points anything huge (is it 100 points or 200 points?). However developers can pretty accurately estimate the relative effort of features. ”Feature X is going to be definitely more work than feature Y, but less than feature Z”. Historical averages for large features can also be converted to costs. Because the publicized exchange rates are always adjusted, whatever the newly quoted price for a feature happens to be, it will reflect the currently expected costs of development. And that’s what you bill the client.
Sometimes it just feels like it’s all your fault. Who talks to stakeholders? You do. Who writes user stories? You do. Who talks to the team day in and day out? Yup. You. So when you finally face the realization that the product your team spent six months developing is a complete failure, it’s a hard pill to swallow.
Is the product owner the ultimate “one neck to choke”? So many times I retrace the conversations with customers, the discussions with developers, trying to identify where did it all go wrong. The requirements are still valid. Customers got most of the features they wanted. But those last remaining 10% of features are now impossible to implement. “If you want this feature, we’ll have to rewrite the whole thing from scratch”, I am told by developers.
So should I have raised this feature earlier on in development? It seemed like such a small feature, a “nice to have” really. It was not prioritized earlier because it seemed like a type of requirement that might go away. But it didn’t. Here we are, almost at the end, and the client is adamant about having every single last bit of functionality they asked for. They are happy to wait for it longer. But it’s not going to happen. How do you explain that to get the last 10% we would have to start from scratch?
There were other such projects. Everything seems to be going just fine until the very end, where you suddenly understand that there are going to be issues with performance or scalability or that some features will be impossible to support with the current architecture. There is that word – architecture, that is sometimes hard to reconcile with. According to Scrum there are two choices. Either don’t deliver the last backlog features to the customer or include the cost of refactoring everything into the next user story.
There are lots of studies that show extremely high failure rates of waterfall projects. But agile products fail too. It’s hard to measure exactly the failure rates, because agile development is a wide spectrum with few companies in the top range. Statistics for agile projects is sometimes quoted at 42% success rates which is 4 times more successful than waterfall projects with their 9% success rates. These numbers are supposed to encourage development to be more agile. However even with agile development, less than half the projects are going to be successful. There is no silver bullet.