The Software Engineering Moneyball

Estimation has been a hot topic in software development blogosphere recently. Morgan Ahlström contributed to it with a nice blog post several days ago: Why Estimates Don’t Matter.

This post was heavily trafficked via social media and several people discussed more nuanced positions on this matter (for example, in these discussions lead by Martin Burns: here and here).

What I read in both the blog post and these discussions is the new thinking that is starting to take root in our industry.

Moneyball

Moneyball is a book (and a movie based on the book) about the new thinking that revolutionized the game of baseball. The Oakland Athletics led by their General Manager Billy Beane (played in the movie by Brad Pitt) started to apply it during the 2002 season. The result: the team with the next-to-last payroll of the 30 teams in the league and having lost its three biggest stars who were signed away by richer teams as free agents won the most games during the season and posted an all-time record 20-game winning streak. The Athletics’ ways were soon adopted by the Boston Red Sox, a team of much greater financial means at the time, who used this approach to win the 2004 World Series.

Moneyball movie poster

Billy Beane noticed that the traditional methods the scouts used to evaluate players were little more than guesswork. Scouts often made recommendations based on players’ looks and vague descriptions like “love his swing” and “five-tool player.” A former five-tool player himself who declined a Stanford scholarship to take a scout’s offer, but failed to make it in the big leagues, Billy Beane believed a more rational, data-driven approach to the game that would tie decisions to desired outcomes (winning more games) could be his competitive advantage as a GM.

Throughout their Moneyball season, the Oakland A’s discovered that many baseball beliefs were not supported by rational analysis and debunked them. For example, they stopped stealing bases. The conventional wisdom said stealing bases was a great way to “manufacture runs” and put the pressure on the opposing team. The economic analysis, however, showed that base-stealing tactics resulted in virtually no difference in the number of games won over the 162-game season. The gain from advancing the runner was offset by the risk of being caught stealing, ending an inning early, with a high on-base percentage hitter on deck, killing a potential high-scoring rally.

The fictional character Peter Brand (representing Oakland’s assistant GM Paul DePodesta in the movie) summarized the approach by saying: “There is an epidemic failure within the game to understand what is really happening…. When other teams look at Johnny Damon (a great defensive outfielder, leadoff hitter and base-stealing threat, whom the A’s lost before the 2002 season), they see a star who is worth $8m a year. When I look at Johnny Damon, I see an imperfect understanding of where runs come from.

Folks, This Is About Us

You can repeat it almost word for word. There is an epidemic failure in the software industry to understand how all these activities that we go about to develop a software product — how all these activities contribute to the customer value and quality.

What many developers, managers, and customers see when they look at estimation is a cornerstone practice of software engineering. What Morgan Ahlström sees is an imperfect understanding of value, quality, risk, predictability, and how estimation affects them. He creates an economic model that allows him to get better understanding. Similar questions can be asked about many practices. What is the economically optimal sprint length? (Not in general, of course, but in your context.) What are the economic trade-offs of sequencing features in this order or that? How much market payoff is lost due to delays incurred because a specialist serves multiple teams?

Economic models of product development flow, helping us understand how various practices contribute to our desired outcomes and informing our process design, are the Moneyball of our industry. Just as in professional baseball over the last decade, they are a source of competitive advantage. One of the big challenges will be helping the “scouts” change.

What are the five tools, the the stolen bases, the runs batted in, the earned-run average and the fielding percentage of your software delivery process?

Advertisements
This entry was posted in life. Bookmark the permalink.

2 Responses to The Software Engineering Moneyball

  1. Charlotte says:

    Hi Alexi, just read this article for the first time. Do you feel there is the same value to be had in using the data from all these activities to analyze and improve team performance? We’ve built a system which collates existing data from different engineering systems (issues, agile, builds, etc) to provide a holistic view of performance data, ways to set and track goals and provide continual and lightweight praise & feedback. We be curious what your thoughts are on the value of system like this.

  2. azheglov says:

    Hi Charlotte,

    The advice I often give on metrics is that they’re secondary to the problem you’re trying to solve. The problem (some kind of dissatisfaction, performance, quality gap, etc.) comes first. If people agree that it is important and have the will to try to solve it, they can find a metric that best captures the gap. Then they can use the metric to validate that their improvement attempts resulted in improvement. Once they’ve closed the gap, they find themselves solving a different problem and are likely to need a different metric. Metrics are dangerous – it is easy to use them to create dysfunction, e.g. comparing teams by their velocity in story points. Using metrics on a permanent basis increases the chance of such misuses.

    As your tool can provide a rich, analytic data set problem-solvers could draw from, it can be an excellent technical resource for them. When they need data for the metric they’ve decided to use to solve the current problem, the data are already there. The trick is to complement this technical resource with workers’ social skills so that they avoid using the data in dysfunctional ways.

    Sorry for the long delay with my reply. I hope it helps.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s