Quality of service provider

Champagne glasses Moty Aharonovitz, senior director of product strategy at Borland Software, says that the best way to find bugs in software is by embracing requirements based testing. Here he outlines three ways to improve your products.

Our primary job, as quality assurance and testing professionals, is to find defects in software builds.

Fortunately most of us have moved beyond exposing and tracking bugs to the more critical role of ensuring that the software our company is delivering meets customer expectations before it is released. To do this, many organisations have recently embraced requirements-based testing (RBT). 

Teams can be more effective using RBT because it enables them to target the root causes of quality issues. They can do this by being tightly coupled with application requirements and integrated through various activities throughout the software lifecycle.

As a result, it helps organisations deliver measurably high, systematic and cost-effective test coverage, and ensures that what our teams test is really what matters to customers. 

The challenge as we roll out RBT to our organisations is to ensure we are coupling its processes with existing QA and testing best practices. To that end, I offer three practical suggestions to improve your RBT practices.

Test early and frequently, so testing becomes a parallel activity to the development process, a constant pursuit that spans all roles and makes all stakeholders aware of quality objectives.

Test with your head, not your gut to inject repeatability and method to the test planning process in order to make test coverage more predictable.

Test with measurement and improvement in mind to quantify the status of deliverables and activities so management can oversee quality initiatives across the IT application portfolio.

The quality crisis 

The Standish Group's Chaos Report and other studies describe what has been the industry reality for decades; the majority of software projects fail to achieve schedule and budget goals. 

Poor software quality is a primary factor behind many failures, and often results in massive rework of application scope, design and code. Such rework extends release cycles and consumes significant additional budget.

To reduce software failures, it is imperative that we better understand the quality initiatives behind the products being developed for today’s global economy.

Industry experience and studies show the two primary reasons behind poor application quality are defects in requirement specifications and problematic system test coverage.

Defects in requirements specifications

Some of us are still not surprised when we hear about end user complaints and lack of adoption for applications that went through a rigorous QA and testing process that found few bugs. The reason we are not surprised about the uproar is simple because  the requirements were wrong from the very beginning. 

A study by James Martin in 'An Information Systems Manifesto' shows that the root causes of 56 per cent of all defects identified in software projects are introduced in the requirements phase. 

Further, the study states that approximately 50 per cent of requirement defects are the result of poorly written, ambiguous, unclear and incorrect requirements. The other 50 percent of requirement defects can be attributed to incompleteness of specification (i.e. requirements that were simply omitted.)

Therefore the two primary issues with requirement quality are that requirements and specifications are incomplete, due to lack of input from users and other key stakeholders.

Also that requirements are specified poorly, by domain experts who lack the skills to define consistent, accurate and testable requirements. Finally the tools do not allow for efficient validation and or review of the requirements with the users.

Problematic test coverage

When quality is handled as an afterthought, quality verification efforts typically begin only once code has been completed. At this point, the testing team is under the gun to certify the application as quickly as possible, and such certification is often perceived as a bottleneck that prevents deployment into production.

In this environment, it is not only difficult to ensure that the requirements are correct, but also to properly plan tests, to ensure correctness, completeness and solid coverage of requirements and to gain visibility into the various quality aspects of the tested application. It is no wonder that this frequently proves to be a costly and frustrating exercise to all stakeholders.

Achieving satisfactory test coverage is challenging. The complexity of modern applications, in particular distributed applications, has made it more difficult to cover all of the possible scenarios due to the sheer number of alternative paths through the application.

In addition, change management has become unwieldy in many organisations with application requirements changing hourly or daily. However, without the ability to properly manage these changes, traceability between requirements and test cases is often lost, making it even harder to ensure continuous test coverage.
RBT best practices

RBT is an approach that is implemented through well-defined processes, that targets the root causes of the quality problems mentioned previously. Even if your organisation is not using the practice today, most of us are familiar with the key processes within RBT.

In addition to existing testing processes, organisations should adopt the following best practices to be successful with RBT:

Test early and frequently

As soon as requirements, design and code artefacts are ready, review them against business objectives, requirements, use cases and test cases.

Test early and frequently because defects found in earlier development phases are cheaper to fix and result in significantly fewer surprises when the code is actually delivered. Make testing a parallel activity to the development process to make all stakeholders aware of quality objectives. 

Best practices within the RBT Requirements Quality process include:
Validation of requirements against business objectives. This optimises project scope by ensuring that each requirement satisfies at least one business objective.

Review of requirements by stakeholders. Feedback should be used to refine the requirements before additional work is done.

Development of use-cases. Map requirements against a task-oriented or interaction-oriented view of the system.

Application of language analysis techniques. Detect problematic phrases in the requirements and fix them to guarantee the consistency and clarity of the requirements.

Test with your head, not with your gut

Today, it is uncommon that test case design is performed in a systematic and rigorous manner. Rather, many tests are designed based on gut feel and intuition, which leads to unpredictable quality. Best practices within RBT require organisations to support methodical and systematic test case design to ensure predictable test coverage.

Organisations taking a lifecycle quality approach go the extra mile to integrate the quality process into the development process, using the structured, validated, high-quality test cases generated during the previous phases.    

Test with measurement and improvement in mind

Using RBT best practices, organisations can manage and improve quality processes through measurement. Throughout the RBT process, multiple measures can be used to quantify the status of deliverables and activities. This helps managers and process experts oversee quality initiatives across the IT application portfolio.

The role of traceability in RBT

Traceability also plays a critical role for organisations using RBT, since maintaining traceability information between requirements and test-cases and tests is crucial.

This information is required for monitoring progress and coverage, as well as properly managing the impact of changes in requirements. Without it, it is more difficult to determine which test cases or tests should be changed if a specific requirement changes.

While capturing and maintaining traceability data is very important, many software development organisations find that it is extremely difficult to get right.

The primary reason is because most tools in the market require the individual practitioners in the software delivery chain to manually create and manage traceability information. This is an overhead that many are not willing to accept. To deal with this challenge, consider tools that can automatically manage links between work products.

Taking the first step to lifecycle quality management

By deploying solid RBT, QA and testing best practices, organisations will find they are better equipped to maximise the business value of their software through quality initiatives that span the complete software lifecycle. 

Teams embracing RBT best practices are in essence taking the first step to a more comprehensive quality approach, known as lifecycle quality management (LQM).

By moving beyond just a focus on code to addressing quality into each phase of the lifecycle, LQM helps software delivery organisations ensure higher standards of quality and service, while systematically reducing costs associated with rework and maintenance.

Further, LQM accelerates the path to software delivery optimization (SDO), which helps transform software development into a managed business process to increase control, predictability, visibility and efficiency over the entire software delivery process. 

22 March 2007