One of the sessions at the online testing conference we sponsored was held by Aleksandra Petrovic, QA Team Lead at Endava.

Testing Metrics and why managers like them

(by Aleksandra Petrovic)

This session in a very important one, it talks shows examples of metrics and the idea behind them.

If you want to learn how to actually apply metrics to your own project this is the right session for you.

Did you ever come across a case like that- after two weeks you were working on a project your manager asks you “how is it going?” and your answer “good” isn’t satisfying enough for him? He would probably like to see your work progress in numbers or any other visual way.

To learn how to use metrics in order to give a better and more educated answer you first have to ask yourself what makes software development projects successful?

What we all want to achieve is basically software quality but that can mean a lot of things.

You need to think about a few things:

Product quality – the product that we developing and testing for our client.
Optimizing implemented processes – Either testing process or development.
We want to optimize it in terms of reducing costs and time we need to deliver the project.
We also like to increase the team efficiency
Keep customers happy at all time!

So – What is Software Testing Metrics?

Metrics help to estimate the progress, quality and health of a software testing effort.

Or as Lord Kelvin, a famous physicist once said:

“When you can measure what you are speaking about, and can express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind.”

A simple example of testing metrics: Total number of defects that your QA team found.

Why do we need to do Testing Metrics?

When we can’t measure something we can’t improve it because we can’t see the problem at all.

With testing metrics we can display past and present performance.
Predict future
Understand what needs to improve
Decide what to do next

In short saying- We need to have a way to monitor our team performance and also product quality at the same time.

Testing metrics can help us figure the past, present and future of the project!

Why Managers Like Testing Metrics so much???

You probably used or heard the saying: “All they (managers) care about are numbers…” It may be true but not so much.

Let us explain:

Your team is doing a great job –show it! The head team have a responsibility to understand his testers work and show it in a way that will be clear to all managers, in terms of numbers, graphs etc.
Inefficiency affects schedules and product quality
Need to have a way to measure QA process quality
Metrics help you generate good test reports

Metric Types –

Metrics can be sorted into groups based on different criteria:

There are projects that the whole department or maybe the company-level need to know about its progress. This would need a wider kind of metrics.
Depending on what we want to measure we can have process related metrics (process efficiency), product related metrics (improve software quality) and project metrics (measure team or tools efficiency). They can track team efficiency or tools that you using.
Base metrics (raw data collected during execution) and calculated metrics (derive from data in base metrics usually for test reporting purpose)

Important Metrics

Test cases:

How many test cases designed per requirement?
How many test cases executed/pending?
How many test cases passed/failed/blocked? How many failed due to one particular defect?

Defects:

Total number of defects identified
Defect distribution per Severity, per Priority, per Component/Module
Defect Aging
Defect Density and Defect Density in released code

Automation Testing:

Number of defects found in automated tests (Are they still useful? Watch for pesticide paradox!)
Percent of automated test coverage
How long it takes to run test plan and how often we do it?

People/QA Team:

Execution by user: How many test cases executed per person?

Important Metrics that involve the QA team and development team

Defect distribution by status and phase

Expected to decrease as the software quality improves toward the project end

2. Defect open and close rates

Insight into the ability of testers and developers to work together to identify and address software issues

3. Defect removal efficiency (DRE)

Rate at which team is able to adequately fix identified program flaws

4. Burndown chart

Visual representation of the amount of work yet to be completed

5. Defect severity index

Shine some light on the effectiveness of development team

Important Metrics that focusing on customer side:

Number of issues reported by customers

Determine effectiveness of QA departments (risk-based testing to reduce gaps)

Mean Time to Detect (MTTD) and Mean Time to Repair (MTTR)

MTTD -how long it takes QA professionals to find a problem
MTTR -amount of time needed to effectively address it

Number of system outages and downtime

The number/frequency of system outages and length of downtime experienced by end user

COQ (Cost of Quality) and COPQ (Cost of Poor Quality)

COQ: Total effort put into quality-related activities (development, testing, reviews)
COPQ: Cost of fixing defects, updating docs, re-testing, patch distribution

How to identify the right metrics?

A few tips how to identify correct metrics:

Set the target audience – Identify your target. Are we talking to our team leads, department manager, costumers etc.?
Define the goal for metrics, which is also depending on your target audience.
Add only relevant metrics based on your project needs – less is more!
Analyze the cost benefits aspect of each metric

Can Metrics Cheat?

Yes, metrics can cheat too! Usually when they are taken out of context… when you use a number or two in your report, without any explanation this is when it can be misinterpreted.
In order to have reliable metrics you need reliable input data.
Cheat accidentally (incorrect input data) or cheat on purpose (‘tweak’ your queries to get better stats).
True story example:
- Project metric that did not take into account that there was a new defect tracking tool introduced!

This graph didn’t take under consideration other defect tracking tool that they started using a few months before. So the first graph was actually showing just half of the data.

Common Deceptions about Metrics:

The bigger the number –the better we are? Not necessarily true. For example – if your team had found a bigger number of defects can you release this product with this amount of defects? How can that be good?
The component where we find most defects is the worst coded. That can be true but not always.
Metrics are there for managers to point fingers at people. We hope that after that you understand that it’s not really true.
“All my filters and queries are correct but I still get very strange output results”. We need to find the root cause analysis to understand it.
There are no “universal metrics“. This is actually true…

This lecture was a great success and launched a vital discussion in the OnineTestConf Slack channel. We all know that metrics matter but also, that sometimes they are not the best representation of the truth. Knowing its limits can actually enhance the benefits that can be gained from using metrics and improve our everyday testing job.

Bio:

A senior QA engineer with 6 years of work experience in IT industry.

Area of interest and expertise are regression and performance testing, mostly

focused on server side applications. For the last 2 years I have

been working as a QA team lead with 15 QA engineers in my team that are

responsible for full test coverage (automation, performance and regression)