I was recently asked how we measured quality in our product and was immediately cautious about what to say next. I had these warning bells ringing in my head:
“High code coverage doesn’t equal a high quality product if the most important code isn’t covered”
“A low bug count doesn’t mean anything if those bugs are all critical, similarly a high bug count might not be bad if they are all trivial”
But just because there are bad ways to try and measure quality, doesn’t mean there aren’t helpful ways as well!
Watching the trend
The most important thing to factor in when trying to measure quality is the trend over time.
5 bugs raised this month is good if the last 3 months averaged 30 bugs each, however if they averaged 0 bugs each month, it’s a concern worth investigating.
Numbers on their own don’t mean a whole lot, there will always be a need to look beyond the numbers to see the type of bugs you are finding, and what impact they have on the customer. The numbers can influence when and where you should be investigating.
Therefore, the measures you decide on need to be tracked over time to see where you are making improvements worth celebrating and where you are getting worse that needs addressing. Though regardless of trends, this is only part of the picture of the quality of your project. Manual verification will always be needed in parallel.
What should I measure?
In my view, the most important thing you can measure to give you an indication of the quality of your project and where to focus your efforts is:
Total Open Bug Count, grouped by priority
A graph of total open bugs, split into the different priority levels (e.g. Critical, Major, Minor and Trivial) grouped by week or day. This will let you know if you are making a dint in your backlog of bugs, as well as the trend in each category of bugs.
This will help you answer the following questions, which you should then decide on how to respond based on your project’s needs:
- Are we creating more bugs than we are closing?
- How fast are we resolving our critical and major bugs?
- Are we creating more critical/major/minor/trivial bugs than we used to?
- Do we need to take some time out to tackle our bug backlog?
Here are some other measures that could be helpful, but I would do these as a secondary focus to the one above as they are mostly covered.
- Total Bugs Raised, grouped weekly by priority.
How: Count how many bugs were raised this week.
Ask: Are we raising more bugs this week than we normally do? Are we raising more high priority bugs than we normally do?
- Average Bug Resolution time, grouped weekly by priority.
How: For the bugs resolved this week, measure how long they were open for and get the average.
Ask: How long is it taking us to fix critical bugs? Are we getting slower or faster at resolving bugs?
- Open Bug Count By Feature, grouped weekly.
How: Break the total open bug count down into the feature area affected.
Ask: Are certain parts of the project more problematic than others?
Is there a different measure you use to track quality in your project? I’d love to hear about it in the comments below.