Working Effectively with a Performance-Testing Bottleneck

Not long ago, a software performance architect brought to my attention that she and her colleague (the performance-testing team) were overloaded with performance-testing requests from product teams. The performance testers wondered if they needed to change their mini-team’s process to handle this workload more effectively. (Among the questions I was asked was, planning our work in sprints doesn’t seem to make sense, maybe we need Kanban?)

The important point here is that achieving better team performance (measured as throughput in this case, but could also be the team’s responsiveness or the satisfaction of its organizational partners) often requires not improving teamwork or some team-centric approach, but an organizational focus. In this case, seeing organizational workflow and applying the Theory of Constraints to the context.

A product development process with a bottleneck in performance testing

How does that work? The part of the TOC that is useful to us here is the Five Focusing Steps.

Step 1: Identify the Constraint

A quick lookup of the performance testing team velocity and the size of their backlog of requests and a quick calculation showed that they had about three years worth of work in their queue. We could also sense the bottleneck. During the time of our conversation in the team area, several arriving email messages flashed on the screen, where some project manager was asking, when do you estimate you will begin putting my Product X in Perf Lab?

What should we do? Hire more performance testers? This is not the next Focusing Step!

Step 2: Exploit the Constraint

In our case, we’re trying to re-route some of the flow around the bottleneck. Not all features have equal risks of falling short of their performance requirements. Teams working on riskier features get more engagement with the performance experts. The work of others bypasses the bottleneck and they have to figure out how to meet both their functional and non-functional requirements without relying on a scarce resource. Creating options to reach the their goals instead of asking “when?”

Staff liquidity is another important concept here. Software performance has to become an organizational competency and people at the highest levels of expertise in this competency are best used as coaches and not as service providers. This is true and applicable not only to performance, but to many other areas, including UX, accessibility, deployment, and so on. Changing how the performance experts engage with teams is both helping us exploit our bottleneck and take us in that direction.

Step 3: Subordinate to the Constraint

We advise teams to stop making generic performance-testing requests and instead make requests aimed at learning something about the performance of their software features. For example, can it handle X concurrent connections? What will the response time for Y rows of data? How much hardware do it need to run this for a client with Z users?

The five focusing steps to solve with the bottleneck

Step 4: Elevate the Constraint

Book cover: Eli Goldratt's The Goal

“Elevate the Constraint” usually means investment – hiring more people (in this case, software performance testing experts), buying more equipment, and so on. The important part is that this step follows Steps 2 and 3 – the money is invested in amplifying the throughput of the improved process. It would be wasteful to throw it after an unimproved process where experts are pestered with “when” questions by project managers who don’t yet understand their options.

Step 5: …We’re yet to get there 🙂

In the meantime, read The Goal by Eli Goldratt, if you haven’t already.

This entry was posted in hands-on. Bookmark the permalink.

1 Response to Working Effectively with a Performance-Testing Bottleneck

  1. Pingback: Optimize Late and Not Often | Tim on Technology

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s