Software quality is particularly challenging as teams generally have little idea at the start of the project how much testing and debugging will be needed to produce a release that is acceptable. Dominic Tavassoli examines the challenges and solutions available.
Customers expect software quality to be very high, and yet we frequently hear in the news about bugs or design errors that cost time, money or worse, lives. As software development becomes more complex, innovative and created by distributed teams, delivering quality is increasingly difficult. Quality assurance (QA) generally focuses on testing the software in the final stages of a project, when budgets are scarce and the pressure to deliver the product is high. Project managers must balance quality assurance with time to market.
How much testing is enough?
'There is always one more bug.' The amount of testing needed to ensure quality is hard to quantify. Very few project teams are able - or are willing - to predict how many bugs there will be in a release. Testing and fixing until all bugs have been found would be a very expensive task, if it were possible. And is it reasonable? Most organisations prefer to deliver a product on time, with a few unimportant defects, rather than to deliver it six months late. Needless to say, each industry has a different level of acceptable quality (for example, games and aircraft systems). In order to provide correct budget and schedule estimates, QA managers need the means to predict fault levels.
What part of your process needs to be improved?
Many organisations often limit QA to testing the final software. Functional testing may be outsourced to low-cost countries, which makes quality 'somebody else's problem'. This is quality control - checking after the fact that quality levels are met. On its own, quality control doesn't help teams build better software, a fact that can be very frustrating for the QA manager, as they are confined to the role of inspector of other people's work1.
Analyst reports indicate that the cost of fixing a defect increases dramatically the later it is found in the life cycle - the cost of correcting a fault detected during final system testing can be up to 200 times more than if it is found during the requirements phase. Very few organisations can confidently pinpoint the weakest links of their process, those where their budget would have the most impact. Without objective process performance indicators, QA managers cannot focus quality improvement initiatives where they are necessary or where they will have the best return on investment (ROI), let alone prove their efficiency.
Strategic QA helps organisations track faults across the software development life cycle, providing QA managers with the insight necessary to predict fault levels, testing budgets and, more importantly, allows them to focus process improvement budgets in an efficient way, with provable benefits. Strategic QA not only helps organisations become more competitive, it also reaffirms the value of the QA manager in the project team and the organisation.
Metrics for strategic QA
Defects and errors can be introduced and found during the different phases of the Software Development Life Cycle (SDLC). The practices described here are easily applied to all types of processes (waterfall, iterative) with different phase names. The industry also uses multiple terms and different standards to name software problems. In this article, we will use the following2:
- 'Errors' are problems detected in the phase they were created in.
- 'Defects' are problems detected after the phase they were created in.
- 'Fault' is the generic term for a defect or an error.
Two metrics are of particular interest to track and understand quality:
- Phase Containment Effectiveness (PCE) is the ratio of faults captured in a phase (represents how effective the process is at preventing problems from becoming defects).
- Phase Screening Effectiveness (PSE) is the ratio of prior escaped defects captured in each phase.
Phase effectiveness metrics quantify the capability to find and fix defects closer to their origin, reducing cost of rework. By implementing strategies to improve these metrics, organisations reduce the number of released defects and avoid project delays, improving customer satisfaction and cost of maintenance.
Predicting software faults
By capturing fault data on projects, organisations can estimate Fault Density (FD) - the number of faults present in a work product. Historical project data should be collected to calculate these metrics for organisations. The QA manager can then statistically predict faults, errors and defects for each phase, based on the estimated size of the current release and can produce reliable fault estimates for a project. The PSE metrics also help predict where these coding defects will be found3.
By implementing Strategic QA, organisations can produce reliable schedules and testing plans, allocate testing resources in the most efficient way and benchmark the performance of their process.
Process improvement and the CMMI
Leading organisations have successfully attained ROI for process improvement initiatives by adopting the Software Engineering Institute's Capability Maturity Model® Integration (CMMI®).
One of the fundamental principles of the CMMI is the implementation of a standard set of processes across the organisation, continually improved based on objective, measured feedback. In particular, the CMMI recommends the key practice of Causal Analysis and Resolution: identify causes of defects and errors and take action to prevent them from occurring in the future, improving quality and productivity by proactively preventing the introduction of defects into a product.
Strategic QA is an example implementation of Causal Analysis and Resolution and should be considered by all organisations aiming for CMMI level 3 certification (and above).
Capturing error and defect data across the organisation
The data necessary for the metrics mentioned previously must be captured when faults are identified. This process is facilitated by the rollout of an enterprise change management (ECM) solution for consistent fault tracking. Web-based solutions provide customisable forms to collect the relevant data for each fault. By providing a user-friendly interface with drop-down list boxes, organisations can ensure that quality feedback is easy for users to provide.
Information to be collected for each error or defect should include:
- fault description;
- category;
- the phase in which the fault found;
- how the fault was found;
- the phase in which the fault introduced (captured after analysis);
- the phase in which the fault should have been detected;
- the cost of the fault.
ECM solutions produce the forms, processes, metrics, reports and graphs necessary to capture, summarise and analyse the data for Strategic QA. They enable organisations to view which links of their process are the weakest and where to focus process improvement budgets.
Deploying across the enterprise
To successfully improve the way they produce software, organisations need to capture error and defect data in a consistent, centralised fashion. ECM, the cornerstone for sustainable compliance as well as causal analysis and resolution, enables organisations to implement a repeatable, documented and reliable process for capturing both fault data and change requests of all types, on software and hardware, from customers and the internal teams, urgent and minor.
Naturally, an organisation wishing to deploy and enforce an ECM solution must verify that it is scalable to its needs, user-friendly, flexible enough to implement a process that reflects the corporate culture and solves the identified challenges. The solution must also be capable of providing a common, consistent process across the organisation. Products that provide out-of-the-box, industry-proven processes are a natural choice for low-risk deployment.
Strategic QA at Renesas
Renesas Design France, a subsidiary of Renesas Technology Corporation, a joint venture between Hitachi and Mitsubishi Semiconductors, is one of the world's leading semiconductor system solutions providers. Renesas Design France deployed an ECM solution to manage all change requests and bug reports, as a key step of the process improvement initiative.
For each bug report, Renesas Design France captures the development phase in which the error was found and, upon resolution, also indicates the phase in which the error could have been avoided. This causal analysis enables the company to identify the process areas where improvement will deliver the highest ROI. Metric history allows development staff to predict the number of bugs a program will have, based on the number of requirements, before design even takes place.
'This level of insight would be impossible to implement in most other tools,' Stéphane Thomas, quality manager at Renesas Design France, said. 'With Six Sigma metrics on Defect Containment Effectiveness and Defect Screening Effectiveness, Renesas Design France is able to continually improve its organisational development process.'
Summary
Strategic QA allows organisations to reduce costs, improve quality and increase reliability of planning. ECM solutions are particularly critical for organisations implementing a SOA strategy, with its promise of being able to adjust to change.
