Blog Testing

Measuring Testing Success: Guide to Test Metrics for Functional Testing and Test Automation

July 19, 2023
Measuring Testing Success


“Testing without metrics is like driving blindfolded. You may be moving, but you have no idea where you’re headed.”

In the world of software testing, metrics play a crucial role in assessing the quality and effectiveness of testing efforts. Test metrics provide valuable insights into the progress, efficiency, and overall health of the testing process. 

We will explore the world of test metrics in this blog article, concentrating on metrics for both functional testing and test automation. We will investigate essential metrics, their mathematical models, offer illustrations, and emphasise the importance of each indicator in guiding effective testing approaches.

Functional Testing Metrics

Functional testing metrics evaluate the functionality and correctness of the software being tested. They provide insights into the quality of the application and the effectiveness of the testing efforts. Let’s explore some commonly used functional testing metrics:

1. Test Coverage

Test Coverage measures the extent to which the application’s functionality is exercised by the test cases. It helps identify areas of the application that lack test coverage.

The formula for test coverage is:

Test Coverage = (Number of covered requirements or test cases / Total number of requirements or test cases) * 100


If a software application has 100 requirements and 80 of them are covered by test cases, the test coverage would be (80/100) * 100 = 80%.


Test coverage ensures that critical parts of the application are thoroughly tested, minimizing the risk of undiscovered defects.

2. Defect Density

The number of flaws found per unit of code or requirement is measured by defect density. It makes it possible to focus testing and improvement efforts by pointing out regions with a high density of faults.

The formula for defect density is:

Defect Density = (Total number of defects / Size of the code or requirements) * 1000


If a software module has 5000 lines of code and 50 defects are identified, the defect density would be (50 / 5000) * 1000 = 10 defects per 1000 lines of code.



Defect density highlights the quality of the software and guides teams in identifying areas that require additional attention during testing.

3. Test Effectiveness Index

The number of problems discovered per test hour is used by the Test Effectiveness Index to gauge how effective functional testing is. It determines the testing team’s productivity and aids in evaluating how effective testing efforts were.


Test Effectiveness Index = (Number of defects found / Total test hours)


If 20 defects are found during 100 test hours, the Test Effectiveness Index would be 20/100 = 0.2 defects per test hour.


The Test Effectiveness Index provides insights into the productivity and efficiency of the testing team by measuring the number of defects found per unit of time. A higher Test Effectiveness Index indicates more efficient testing efforts.

4. Test Execution Time Estimation


Test Execution Time = (Number of Test Cases × Average Execution Time per Test Case) + Time for Test Environment Setup


Test execution time estimation helps in planning the testing schedule, identifying potential bottlenecks, and ensuring timely completion of testing activities.


Let’s say we have 200 test cases to execute for a mobile banking application. On average, each test case takes 10 minutes to execute. Additionally, the time required for setting up the test environment is estimated to be 2 hours. Using these values, we can estimate the test execution time to be (200 × 10 minutes) + 2 hours = 400 minutes + 120 minutes = 520 minutes.

Test Automation Metrics

Metrics for test automation put emphasis on evaluating the efficacy and efficiency of those efforts. These metrics offer perceptions of test automation’s return on investment (ROI) and aid in the optimization of automation tactics. 

Let’s explore some key test automation metrics:

1. Test Automation Coverage

The percentage of test cases that are automated as a percentage of all test cases is known as test automation coverage. It assists in determining the degree of test coverage attained by automation.

The formula for test automation coverage is: 

Test Automation Coverage = (Number of automated test cases / Total number of test cases) * 100


 If a test suite consists of 100 test cases and 60 of them are automated, the test automation coverage would be (60/100) * 100 = 60%.



Test automation coverage provides insights into the level of automation achieved and helps identify areas that require additional automation efforts.

2. Time Saved

When comparing test automation versus manual testing, Time Saved estimates the time savings. It aids in quantifying the benefits in productivity brought about by test automation. 

The formula for time saved is: 

Time Saved = (Time taken for manual testing – Time taken for automated testing) / Time taken for manual testing * 100



If it takes 20 hours for manual testing and 5 hours for automated testing, the time saved would be (20 – 5) / 20 * 100 = 75%.


Time saved metric demonstrates the efficiency of test automation in reducing the overall testing timeline, enabling teams to allocate time for other critical activities.

3. Effort Saved

Effort Saved measures the reduction in effort achieved through test automation compared to manual testing. It quantifies the workload reduction and resource optimization obtained through automation. 

The formula for effort saved is: 

Effort Saved = (Effort expended on manual testing – Effort expended on automated testing) / Effort expended on manual testing * 100


If it takes 100 person-hours for manual testing and 30 person-hours for automated testing, the effort saved would be (100 – 30) / 100 * 100 = 70%.


Effort saved metric showcases the efficiency gains achieved through test automation, allowing teams to optimise their resources and focus on other value-added tasks.

4. Defects Identified from Automation


Number of Defects Identified from Automation / Total Defects Identified


Suppose a testing effort discovers a total of 50 defects in a software application, out of which 35 defects are identified through automated tests. Using the formula, we can calculate the metric as follows: 35 (Defects Identified from Automation) / 50 (Total Defects Identified) = 0.7 or 70%.


This metric assesses the effectiveness of test automation in uncovering defects compared to the total defects identified during the testing process. It provides a percentage that indicates the proportion of defects found through automation.


For assessing the efficacy, efficiency, and quality of functional testing and test automation initiatives, test metrics are a crucial tool. Software development teams can use these indicators to make data-driven decisions, pinpoint areas for development, and guarantee the effectiveness of their testing procedures. Testers are better equipped to measure and enhance their testing processes if they comprehend the formulas, examples, and relevance of each metric covered in this blog article.

In conclusion, test metrics provide us with a compass to navigate the vast landscape of software testing. They empower us to make informed decisions, optimize our testing strategies, and ultimately deliver high-quality software. By leveraging functional testing and test automation metrics, we can gain valuable insights into the effectiveness, efficiency, and return on investment of our testing efforts. So, let us embark on this metric-driven journey, where data illuminates our path to success.

Remember, as W. Edwards Deming once said, “In God we trust, all others bring data.”

Leave a Reply