Testing - A sponsor for the purpose

Who wants to know?

If we look within our culture for places where tests are part of a commonly accepted process, we might come up with Which? consumer product testing. Even there it seems that people are looking for post-hoc justification of their own judgements, or a relatively painless and apparently reliable way of choosing a product from a large field, rather than information about the product per se.

Taking Tom DeMarco's advice we must look at Purpose, Question, Metric. How many testers have their Purpose firmly sponsored and mapped to a set of questions? What is anyone trying to change by doing tests?

Who needs to know?

Any system development is rooted in a (formal, informal, vague, cosy, combative) contract between some supply side developers and some demand side users. The purpose of testing ultimately is derived from the performance of the contract whether it is the developers trying to improve their process, the users trying to assess system functionality or the resolution of contract disputes.

If there is a lack of sponsorship of testing Purpose, then the contract must be weak. There is not a healthy, mature relationship between buyer and seller, and performance will be poor.

One high level outcome of the development of the Capability Maturity Model was that the DOD was found to be very poor at procuring systems. I call this the motorway food syndrome: if you don’t care you get expensive rubbish.

How to organise the need to know

The Woolf reforms currently being introduced into the legal system set a fascinating precedent. If you think you might have a claim you are encouraged to tell to the person you have a potential claim against at the earliest opportunity.

They in return are encouraged to inform you of their response. This gives the maximum opportunity for damage and cost limitation to all parties. Those for whom this encouragement was not sufficient motive to exchange the information may now find themselves bearing the costs of the other party whether they win or lose the case.

Within the system development environment, managers can set up a scheme where the parties to a contract are able to describe in advance (using a formal risk assessment) where they think the pitfalls are and what may go wrong with performance of either side under the contract. Under strict terms each side may negotiate access to the concerns of the other party.

If it comes to allegations of poor performance under the contract, remedial costs can follow the logic of who had previously identified the problem and what they had done to address it. Risk negotiations will in effect modify the contract to locate responsibilities and create a more mature environment in which testing can address the real business purpose of the system.

The purpose of testing

Under such schemes, information from testing answers questions about what was and what was not done under the contract and investigates problems identified by the parties.

Even when a contractual relationship is immature, the scheme can demand information that neither of the parties particularly want exposed as part of the adjudication of remedial costs. For the first time people have a serious financial interest in the outcomes of tests.

Let’s try an example. Some software developers at a bank have been asked to enhance some derivative trading systems. They look at the requirements and reckon that the system is sailing uncomfortably close to the wind in its lack of respect for the banking supervision rules. It does not generate the information required to assess exposure.

They document this issue and pass it to the contract negotiation scheme. In negotiating the contract the issue is brought up and the traders commissioning the development insist on the requirements as they stand. At a later date the banks auditors ask for tests to be run on the software to establish whether it enables them to audit exposure properly.

The test results show that it can't and are passed to the contract negotiation scheme. A senior manager from the trading side blames the developers for not following bank guidelines. A dispute is referred to the contract negotiation scheme who have chapter and verse to show that the traders must pay for the system to be reworked.

The next time round the traders ask to see what information the developers have lodged with the contract negotiation scheme. To do this they must lodge their own assessment of what may go wrong. Both assessments may be audited by the scheme to see what the submissions are based on.

The developers can now see where the real business issues lie. Moreover because there has been a recent dispute the contract negotiation scheme commissions some system tests of its own to assess whether certain issues between the parties have been properly implemented.

In another part of the bank developers and users have submitted assessments that indicate they both think a particular issue will jeopardise the project. The contract negotiation scheme brokers a deal where both parties co-ordinate their management of the risk and an early test programme verifies that they have succeeded.

Why would anyone do this?

Contracts of all sorts are padded out to allow for contingencies because the parties have to cope not only with a range of unknowns which no-one seems willing to make explicit at contract stage, but also with the less than helpful behaviour of others.

However the cost of administering a contract risk broking scheme is a small percentage of the money saved by organising the contract effectively. Actual contract costs can be cut radically while increasing the profit when behaviour is refocused on the contract goal - and testing activity explicitly linked to real business purpose.

Aidan Ward, Antelope Projects Limited