Increasing, improving or outsourcing testing?

Hiring more testers might increase the amount of testing, but unless the new testers are particularly capable, the productivity of test teams may be unchanged.

On the same product, 'more testing' will only improve test effectiveness if a more systematic approach is adopted and techniques are used to reach the software nooks and crannies that other tests don't reach. Just like search teams, testers must 'spread out', but aimless wandering is not effective. Testers must be organised to avoid duplicating effort and leaving gaps.

If testing is currently chaotic, adding testers to a chaotic process may keep unemployed testers off the street, but doesn't improve test effectiveness much. To make significant gains in effectiveness, both testing skills and infrastructure need to be enhanced.

What about tools? Can't they be used to increase the testing? For a chaotic process, tools rarely add value (they often waste testers time). All too often, tools are only used to run the tests that are easy to automate - the very tests that didn't find errors!

What about outsourcing? (Tester body-shopping should not, strictly, be regarded as outsourced testing - that is the ‘more testers’ route). What is the outsourced testing service? The service definition should detail the responsibilities of the client as well as the outsourcer.

For example, outsourcer responsibilities might be for example: documentation of master test plans, test specifications, creation of test scripts and data, test execution, test management, test automation etc.

The client responsibility might be: direction on test scope, business risks, business processes, technical or application consultancy, assessment of incident severity, analysis of test results and sign-off.

If the client organisation has a poor track record of directing in-house test teams, however, the outsourcing arrangement is unlikely to benefit the client. Testing may not be faster, as testers may be held up through lack of direction from business or system experts.

The testing may lack focus, as testers 'guess' where they should spend their time to best effect. Tests may not address the biggest business risks; cosmetic errors may be found at the expense of leaving serious errors undetected.

Simply giving up on testing and handing it over to a third party will cost more because you have a management overhead, as well as more expensive test teams.

The quality of the testing is unlikely to improve - good testers are better at producing test plans and tests, but the content is based on the quality and amount of knowledge transferred from business and technical experts. If your experts are not used to being heavily involved in the test process, tests may be produced faster, but may still be of poor quality.

This does not mean that outsourced testers can never be as effective as internal resources. The point is that unless your organisation is used to using an internal testing service, it is unlikely to get the most of an outsourced testing service. The inevitable conclusion is that most organisations should improve their test practices before outsourcing. But what are the improvements?

We'd like to introduce the good testing customer (GTC). It sounds like this means making the job of the outsourcer easier, but does it really meandoing the job for them?

Describing a GTC is easy. They know what they want and can articulate their need to their supplier; they understand the customer/supplier relationship and how to manage it; they know how to discern a good supplier from a bad one. One could define a good customer of any service this way.

The GTC understands the role and importance of testing in the software development and maintenance process. They recognise the purpose and different emphases of development, system and acceptance testing.

Their expectations of the test process are realistic and stable. The relationship between business, technical and schedule risks and testing is visible. When development slips, the testing budget is defended; the consequences of squeezing testing are acknowledged.

These are the main issues that need to be addressed by the client organisation if the benefits of good testing are to be realised. How does an organisation become a good testing customer?

In the same way any organisation improves its practices - through management and practitioner commitment, clarity of purpose and a willingness to change the way things are done.

Paul Gerrard, Systeme Evolutif Limited