Performance Measurement Framework for Outsourced Testing Projects

This is an article from the July edition of the EuroSTAR Newsletter -STARTester, from Kaylana Roa Konda, Applabs Technologies, India . You can view the complete newsletter by clicking here and dont forget to subscribe for future issues

Industry estimates peg the current global market size of outsourced testing services at around $13 billion. This is a strong indication that Outsourcing of testing processes (partially or fully) is here to stay and flourish. Out of the many varieties in outsourcing, Off-shoring is gaining momentum , in which testing activities are typically outsourced to low wage countries like India, Russia, China etc..

This new paradigm of getting the testing done at remote locations is posing significant challenges to both, client and vendor. Some of the key aspects that are demanding attention in managing testing engagements are, differences in test maturity levels, separating test teams from development teams, sharing test environments, managing test tool licenses, changes in roles and responsibilities at client side, defining SLA’s to protect business interests etc., Test outsourcing management and monitoring is indeed very crucial step in supporting and making the outsourcing engagement successful.

Key factors to be considered in managing outsourcing relationship are, business drivers, different outsourcing test scenarios, and potential expectations from the client. Lack of performance measurement framework can often lead to the below situation.

> Excessive communication
> Micro management by client
> Supplier spends too much time in reporting
> Every stakeholder feeling out of control

There is a strong need for performance measurement framework that can prevent the above potential mishaps. A Performance Measurement Framework (PMF) is an essential part of any test-outsourcing project. It defines the boundaries of the project in terms of the services that the service provider will offer to their clients, the volume of work that will be accepted and delivered, and acceptance criteria for responsiveness and the quality of deliverables. A well-defined PMF correctly sets expectations for both sides of the relationship and provides targets for accurately measuring performance against those objectives. At the heart of an effective PMF is its performance metrics. During the course of the test outsourcing engagement, these metrics will be used to measure the service provider's performance and determine whether the service provider is meeting its commitments or not.

‘5P’ performance measurement framework is introduced to establish accountability on both the sides (Client and Vendor), jointly manage and achieve a win-win situation. The 5P’s are - product, project, process, people and price. ‘5P’ performance measurement framework is easy to apply, proven and practical in nature and was developed based on knowledge and experience. This framework provides collection of metrics to choose from multiple dimensions of the testing engagement namely project, process, product, price and people. Metrics can be provided that can cater to wide variety of testing engagements namely test automation, performance testing, certification testing, functional system testing, white box testing, security testing etc.,

Sample metrics against each category are mentioned below to give you some ideas on direction to think about.

• Project: Test effort Vs development effort, Productivity.
• Process: Cycle time improvement, defect leakage index.
• Product: Time to find a defect, test coverage.
• People: Attrition, average experience.
• Price: $ amount saved, Price variance.

Vendor and client have to understand the business drivers of the testing engagement. Identification of the key results areas have to be done based on the business drivers. Appropriate test metrics selection happens based on the nature of the project, test types, test phases etc., Metrics selection is based on the principle that every metric in isolation gives information to track business drivers. The idea of multiple measurements is to put together a pattern of information that collectively gives a complete and accurate picture of the system. Install a metric system in place that allows you to collect the needed information to measure and analyze information and steer projects into the right direction.

Benefits of the model:

Implementation of proper performance measurement framework for outsourced test activities has numerous benefits. Few of them are listed below.

• Helps companies manage their test service providers in an optimal manner for win-win relationships.
• Proper visibility on the return on investment by the outsourced service provider.
• Consideration of all the quality measures into account while analyzing the performance.
• Introduction of a standard evaluation process across the company.
• Identification of the potential risk areas that affect the productivity of the test team.
• Higher level of abstraction with carefully choosen test metrics and the presentation format enabled management to spot the critical issues quickly.
• Past history of the results from the framework can help the success probability of future projects.

Biography

Kalyana Rao Konda is a senior technical services manager at AppLabs Technologies India Pvt Ltd, a company that provides development and testing services. He has been interested in testing from the beginning of his career. He has immense experience in managing product testing groups and also providing test services to clients world-wide. He has a proven track record of managing large scale test automation projects across various technologies, test tools for wide variety of organizations. He has published papers and spoken at international testing conferences and leading web-sites. He holds PMP and CSQA certifications.. He holds a B.tech in Electronics and Communications Engineering.