In my previous post, I referred to the insight (created by experts who have analyzed lots of real-world software and IT project data) that lead times in such projects often have the Weibull distribution. I also explained a bit what this distribution is and showed a method to quickly test whether your data set matches this distribution and it it does what is it’s shape parameter. Remember, Weibull is a parametrized family of distributions, each distribution is identified by the shape parameter, which tweaks the shape of the distribution curve. This parameter is characteristic of how the work tends to expand and get interrupted in a particular context, so it characterizes the system of work where the project is being carried out.
Now I’d like to identify some practically useful properties of this distribution. Most of them are linked to the shape parameter, denoted by letter k, so it is very important to know what this parameter is in your context. I also want to limit the scope to shape parameter range between 1 and 2, because, as I learned from Troy Magennis, k=1.5 is fairly common in software/IT projects. Interestingly this shape parameter value corresponds to the fattest tail of the distribution curve – Murphy’s Law at work?
Forget the Three Sigmas
Calculating the standard deviation for this distribution is meaningless. The mean and the standard deviation are often thought of as two separate variables characterizing a distribution, and the following may be counter-intuitive to those who are used to Gaussian bell curves, but the Weibull distribution’s nature is that its standard deviation and mean are linked strictly. Not only that, but all the numbers we need from the distribution are also proportional to the mean and the proportion depends only on the shape parameter. What are these numbers that we care about?
- The median – what’s the over-under of how long this work will take? Because Weibull is asymmetric, this is not equal to the mean.
- The mode. When we think about estimating work, our brains may be primed for the most often-occurring numbers. But as we will soon see, the most often-occurring number may be quite far and on both sides of the mean and the median.
- Important percentiles, such as 80%, 95% – what is the timeline for completing the work with such probability?
- The statistical process control limits – the equivalents of plus-minus-three-sigma for normal distributions.
- The ratio of the mode to the upper control limit – practitioners of Product Sashimi can probably guess why.
Here are all the necessary formulas:
Here are the results. To keep the table compact, the mode, the median and the percentiles are expressed as their ratios to the mean. I also dropped the lower control limits from the table, because they are all very close to zero.
What can we see here? The most-frequently occurring lead time — which may stick in our memory and which we as estimators may be biased towards — may be as little as 20-40% of the mean. At the same time, the timeline 80% delivery guarantee can be 40% longer than the mean. Providing delivery guarantees beyond 80% is very expensive. The Weibull tail is significantly fatter than the normal distribution’s tails.
The ratio of mode to the upper control limit can be useful to teams using iterative development methods. It shows how the time to complete a typical user story (from selecting it during sprint planning to meeting the definition of done) relates to the iteration length. It confirms what XP practitioners knew all along: keep them much smaller than the iteration length.
For the popular shape parameter value 1.5, the ratio of the upper control limit to the mean is approximately 4. This can be used as a rule of thumb to estimate one of these numbers after observing the other.