In 2002, I had the only layoff of my career, from a company that had spent some time building a product (and they didn’t come) and landed right away in a DevOpsy lean-startapish Internet ad firm. Of course, words such as DevOps and lean startup didn’t exist back then, I’m just using them to describe what it was like.

We built, measured, and learned.

After building a feature, a new version of our software was ready for release and the measuring began. We ran series of experiments on progressively increasing segments of our user base. We would start with a small percentage, say 0.1%, and run a statistical hypothesis test on it. Then we would roll the software out to a greater number of users, run another experiment and so on. The nature of our experiments varied. The first one usually established that the new feature didn’t do any harm and all preexisting functionality worked as before and produced the same revenue. Additional experiments tested hypotheses that the new feature lead to improvement, such as increased revenue per user. The last test would be, can the company use the improved product to attract more paying customers. I worked with Mark, our VP of Advertising Operations, and his staff to run many such experiments.

I was talking with Dave Rooney at lunchtime at the Agile Coach Camp in Montreal three months ago after his Lean Startup session and reflecting on this experience when I realized: the measure-learn part of build-measure-learn is a kanban. We have a value stream here, with several intermediate value-adding states. There are policies for pulling work from one state to the next and, obviously, tight WIP limits. The boundaries of this kanban are “ready for release” (upstream) and “validated” (downstream).

To be sure, there was no Kanban method back then; I’m just using a modern apparatus to think about a past situation. If we knew back then what we know now, our Measure-Learn kanban board might look like this at some point (I dropped the WIP limits from it, because they would evolve anyway and the precise limits are not the point here).

“Ready for release” is of course the downstream state of our Build kanban. Again, the exact software development process used to build the feature is not important here, but we can use the modern Kanban method to model it. The board visualizing it could look like this at one point:

If we join the two kanbans together, we get the joint Build-Measure-Learn kanban. This is the system and our job is to optimize it for response time.

Now, if you have Eric Ries’ book, The Lean Startup and can open it on page 139, he shows a kanban diagram there: backlog-in process-built-validated. If you drilled into “in process”, you’d get more detail. If you drilled into “built”, you’d get more detail. My last diagram gives you one example, from one particular firm, what one more level of such detail looks like.

Optimizing for Response Time

Why response time? Very simple. What are we producing here? Validated learning. Who is the customer of the validated learning? The founders (and/or VCs). They want to discover their next pivot point as soon as possible. They don’t care about throughput (the rate of accumulation of validated learning), but want the key bits of it that lead them to the next pivot point. Therefore, the response time is king.

Observations and Corollaries

• Optimizing such systems will be hard. A lot of today’s Agile and Lean implementations concern the Build part. And even that is difficult. Lots of people in the software industry can’t quite wrap their minds around the core concepts of Scrum and Kanban. And this Build-Measure-Learn system is more complex.
• Optimizing for response time may conflict with optimizing for throughput. Optimization for response time is what startup founders want, but not every human activity has the same optimization goal. If you need to optimize for something else, that probably marks the boundary where you should apply the Build-Measure-Learn idea.
• Optimizing for response time is likely to involve a lot of slack in the system. Imagine startup programmers rapidly learning the newest technologies and creating many open-source products that make other startups even leaner. Wait, this is already happening.
• If you optimized your software development process, the Build part, your whole Build-Measure-Learn system is probably sub-optimal. But don’t sweat it. The majority of your colleagues doesn’t yet understand that if their software development process comprises, say, analysis, coding, and testing, then maximizing, say, the testing efficiency doesn’t optimize the process. 😉

What do you think?