One of the most important things in any project is to determine the factors of success, these factors can be based on quality, costs and time. in software testing projects, we will need to measure the quality that can be based on many criteria including the 3 described above.
“A Metric is a quantitative measure of the degree to which a system, system component, or process possesses a given attribute”
The importance of measuring the quality is critical to continues improvement, and the only way to answer the question of “How can we do it better next time?”.
In a software testing process, we will use test metrics to measure a wide range of success criteria such as:
- The progress of a project (% of the work that already completed/Remained).
- The quality of testing artifacts (Defects and test documentation).
- Synchronization among test and development teams.
- The quality of the process used during the project
- Deviations from the original plan.
- Sprint quality in agile projects (A velocity is a built-in tool that we can use to determine it).
Let’s review some of the most common test metrics that we can use to measure a testing process:
- How long would it take to complete the testing process?
- How many defects are opened for the same Root cause?
- How much effort went into the testing of the product?
- How many test cases are designed per requirement?
- How many test cases are designed per Risk (RBT)?
- Test coverage compared to the original estimations.
- The member of test cases that were executed.
- Defects divided by severity.
- Defects divided by priority.
- Any Testing delays.
- Test productivity.
Why do we need to use metrics?
The most important thing that we need to understand prior answering that Q, is that testing metrics are the most efficient and important way to measure the quality of a testing project, in addition, we can specify a few more reasons such as:
- Without using metrics, you cannot measure the quality of the implemented process.
- Managers can evaluate the Risks and determine future plans to remove them.
- Managers can justify hard decisions that they need to take during the project.
- Test metrics will help to monitor the success factors of the testing project (I already reviewed some of them in the previous paragraph).
- Using test metrics, managers can take decisions on different phases of the testing projects and continually improve the process (No matter what is the testing methodology).
Test metrics Life Cycle
The analysis phase is the first phase of the TMLC were we determine the metrics that we will use to measure the project/process/testing success.
The first thing that we will need to perform during this phase, is to explain the identified metrics (And which data should be captured to support them) to the resources involved in the testing projects (Product managers, team leaders and off course the testing teams).
We will analysis the data captured by the test team and once it’s approved we will use it to as an input to the identified metrics.
During the last phase of the TMLC, we will generate a report that summarizes the results and conclusions of the test metrics, publish it to all relevant resources and take feedback that we will use to improve the process.
Types of Metrics
Base metrics (Direct Measure)
Base metrics are the metrics which are determined from the raw data generated be the testers throughout the test Life Cycle (TLC), these data include information about defects statuses, test coverage and more. In addition, using this raw data we can provide inputs to the formulas used to derive calculated metrics.
Calculated Metrics (Indirect Measure)
Calculated metrics are based on the data generated in Base Metrics (Conversion of Base metrics data into more useful information). Using this type of metric, the test lead can track and measure the testing progress at different phases of the testing process.
The Fundamental Testing Metrics
Let’s review some of the most common metrics that we can use to determine different aspects of the testing quality, as you will see, each metric is described as a simple mathematical function.
Prior to using this mathematical functions, we need to understand the importance of the Absolut numbers derived from the actual testing process, this numbers are based on a few basic criteria (Total number of test, Number of failed/passed tests Etc.) that we will use as the metric input, so there is a Hugh important to use the real numbers.
How can we measure the test progress and Efficiency?
Group A: Test execution based metrics
Metric 1: Measuring the % of passed tests
(Passed Tests) / (Total number of test executed) *100 = % of the Passed Test cases
Metric 2: Measuring the % of failed tests
(Failed Tests) / (Total number of test executed) *100 = % of the Failed Test cases
Metric 3: Measuring the % of the remaining tests
(Tests that are not executed) / (Total number of tests) *100 = % of the remaining Test cases
Metric 4: Measuring the % of the executed tests
(Tests that are executed) / (Total number of tests) *100 = % of the executed Test cases
Group B: Metrics used to measure Defects statuses
Metric 5: Measuring the % of Fixed defects
(Number of Fixed Defects) / (Total number of defects reported per sprint/phase) *100 = % of the Fixed defects
Metric 6: Measuring the % of Invalid defects
(Number of invalid defects) / (Total number of defects reported per sprint/phase) *100 = % of the invalid defects
Metric 7: Measuring the % of WontFix defects
(Number of WontFix defects) / (Total number of defects reported per sprint/phase) *100 = % of the WontFix defects
Metric 8: Measuring the % of Cannot Reproduce defects
(Number of Cannot Reproduce defects) / (Total number of defects reported per sprint/phase) *100 = % of the Cannot Reproduce defects
Metric 9: Measuring the % of deferred defects
(Number of deferred defects) / (Total number of defects reported per sprint/phase) *100 = % of the deferred defects
Group C: Metrics used to measure Defects severity
Metric 11: Measuring the % of Blocker defects
(Blocker Defects) / (Total number of defects reported per sprint/phase) *100 = % of the Blocker defects
Metric 12: Measuring the % of Critical defects
(Critical Defects) / (Total number of defects reported per sprint/phase) *100 = % of the Critical defects
Metric 13: Measuring the % of Major defects
(Major Defects) / (Total number of defects reported per sprint/phase) *100 = % of the Major defects
Group D: Metrics used for Time measurement
Metric 14: Average time for development to fix defects
(Total time used for bug fixes) / (Total number of defects reported per sprint/phase)
Metric 15: Number of test execution for a specific time period
(Number of test run) / (Total Time)