We may take the wrong action if our decisions are flawed. The impact of this will vary, but it could include financial penalty, loss of reputation or life. When we make decisions we gather information, develop options, evaluate them and decide on a course of action.
Each stage has potential for human error. Software defects may also have an impact on computerised systems, ranging from macros within spreadsheets1 through to critical vehicle control code2.
Trust is needed
Your software application may not give the result you expect if the other elements it depends upon fail. The underlying software and hardware tiers will need to be trustworthy. We also need to trust that the people involved can build and use this software correctly.
We need to have faith that the software will work, particularly if there is no other way to make the decision. This level of confidence will be proportional to the risk of software failure. Software can be a vital link in the decision-making process; trustworthy software is needed, especially in domains such as security or safety.
My keyboard would not have a ‘delete’ key if we did not make mistakes; errors in our thinking and communication may inject defects into the software.
Software is an intangible thing, it is built in our minds. Our thought processes can be incorrect: with unconscious bias, facts that are not reality, skewed inferences from these flawed facts and subsequent assumptions. We each have different viewpoints; a labyrinth of mental beliefs, opinions and judgements.
We need to transfer what is in our head to others when building or using software. We must communicate our understanding of these complex invisible systems effectively. Software is an error-prone human endeavour.
Establishing control
We need to be mindful of our nature when we design, build / acquire, operate, maintain and then dispose of software. When we adopt a quality mindset we are trying to limit the effect these traits have on the end product by establishing controls to minimise human error.
‘Control is established by ‘detect[ing] differences between an actual and planned result / process, and to cause changes in a process or a product which reduce the detected differences to a defined level’3. This control of process and product does not come free14.
There is a control cost in terms of time and resources which requires effort by everyone in the organisation. There is also a cost associated with failure of the product during internal testing or once delivered and in use. As Figure 1 shows, an organisation needs to balance these two sides of the coin.
Quality as an emergent property
Establishing control increases the chance that people can trust your product; a ‘quality’ product is one that meets expectations. Taking a ‘quality’ approach when delivering software makes it more likely it will be a ‘quality’ product.
- Software: ‘Computer programs, procedures, rules, associated documentation and data pertaining to the operation of a computer system.’3
- Quality: ‘Degree to which a set of inherent characteristics of an object [product, service or process] fulfils requirements [expectations].’4
An organisation will have a set of processes by which its product is brought into being. These processes form a mechanism for delivering the product, as such, they need a layer of control exerted over them. Picture a maintenance engineer checking, adjusting and oiling conveyor belts on a production line. The engineer provides the ‘control’ oversight, and the belts are part of the primary ‘mechanism’ that creates the product.
The interaction of these ‘mechanism’ and ‘control’ processes form a system that delivers the product. If the ‘mechanism’ worked, and was under ‘control’, it is likely that the product will meet expectations.
Just as steam is an emergent property of water and heat interacting effectively, ‘quality’ characteristics are an emergent property of this system. The product will have had ‘quality’ characteristics built into it.
Processes within the software life cycle
A process comprises interfaces, activities, constraints, resources, inputs and outputs10 as shown in Figure 2; processes link together to form a system.
The overarching constraints are the rules governing a process. The resources feed the process with material, tools, and people. Interfaces describe how processes plug together.
Products flow into the inputs, and are processed by activities, and generally output in a modified form. Note that products are not always deliverable, and may include supporting products such as documentation, records and data.
Your organisation will have a mix of processes with different purposes; business, project management, product delivery, and supporting processes such as quality, for example.
The systems of processes exist within the wider organisation system. This wider system includes policy, organisation structures, culture, infrastructure and people5. These act as resources or constraints to the processes. Thus the effectiveness of the processes depend upon the entire organisation system.
If we take a look at some software specific processes found in ISO 122076 we see there are ‘mechanism’ and ‘control’ processes. For example, ‘Requirements Analysis’, ‘Detailed Design’, and ‘Software Construction’ sit alongside the ‘control’ processes of ‘Quality Management’, ‘Software Qualification Testing’ and ‘Software Quality Assurance’. Software product quality is an emergent property of applying processes - ISO 12207, for example - effectively throughout a life cycle.
Processes and products will be at different levels of maturity. Individual processes may be performed in an ad hoc manner. Similarly, the product itself might be experimental. An organisation may choose to measure its process and product maturity when considering the quality / cost trade-off we saw in Figure 1.
Scope of software quality
We should remember that the purpose is to make sure we can trust the software involved in making decisions. We are not just concerned with software products as they are created; some software will simply be acquired and maintained.
The cost of applying rigorous ‘control’ (quality) processes to all of this software may well be too high; a scope of applicability and a proportional approach is needed.
In terms of scoping within AWE, the MOD is interested in built/acquired, operated and maintained software3.
This scoped software is then graded according to the impact of its failure. A proportional approach to quality is then planned. A lower grade software will have fewer demands in terms of processes and supporting products applied to it. This results in lower control costs for the organisation.
We need to watch out for software that may not have gone through appropriate life cycle processes, and has not been imbued with quality. Internal software that is undeclared - ‘end-user computing’ and ‘disintermediation’5 - and externally procured software, for example.
Stakeholders
The stakeholders of the software life cycle processes may be internal or external to the supplier organisation and their interaction is critical. (The complexity can be a problem, with n stakeholders interacting directly there may be n(n−1)/2 communication paths7). You can distinguish between the ‘user’, ‘acquirer’, ‘supplier’ and ‘owner’ roles15.
The ‘user’ communicates their expectations, tests the delivered product and is responsible for using the product. The ‘acquirer’ translates the user expectations into requirements, balances the risk and benefits, and prepares the contract with the ‘supplier’. The ‘supplier’ is responsible for fulfilling the acquirer and user requirements. The ‘owner’ may act as the custodian of the delivered product, maintaining its integrity and controlling its use.
Contractual conditions
Various industry customers expect an organisation to be run a certain way - to have processes with key characteristics - examples include defence3, medical, pharmaceutical and automotive. The MOD expects AWE to operate a certain way.
I gave one view of why we need quality, but the bottom line is: if it is contracted for, it must be met. Contract requirements may constrain the supplier in two ways: what it supplies (product requirements) and how it supplies it (process requirements). If ‘quality’ is about meeting expectations then the product, processes and project must fulfil these requirements.
NATO expects the systems of its member states to be able to operate with each other8. The Allied Quality Assurance Publications (AQAPs) help assure this by giving a common approach to quality, including software quality3, across all states. Any supplier to the MOD such as AWE is likely to be contracted to the AQAP process requirements.
We can see a general idea of contractual requirements flow down in Figure 3. They must be effectively flowed down9 - in a traceable manner - from the acquirer into business, project, system and subsystem requirements. We will just discuss the process requirements.
Company management system
The content of the Company Management System (CMS) is generally formed to meet the expectations of the industry sector the organisation supplies. Ignoring any special process requirements for now - this is handled by tailoring the quality planning - a project should simply follow the CMS processes.
AWE’s CMS contains the ‘mechanism’ and ‘control’ processes to run the projects, programmes and the organisation as a whole. It covers specific management aspects such as information, configuration, risk, quality, project delivery, environment, business continuity.
An organisation may have some or all of these aspects independently certified against a relevant standard, such as ISO 900110. This provides acquirers with some confidence in its capability. (Although it does not necessarily guarantee that certified processes were applied to a specific product.)
Quality management system
Sitting within the CMS, as we can see in Figure 3, the quality management system (QMS) contains the ‘control’ processes that deal with the control and improvement of the other CMS processes.
Software quality management system
The software quality management system (SQMS) is a specific subset of the QMS concerned with software processes; it is written to meet the MOD contract requirement - AQAP 2210 - for software. It tackles the themes of planning, assuring, controlling and improving software process and product.
‘[In response to quality concerns,] the driving force to use software engineering principles ... was the fear of major accidents that might be caused by having uncontrollable artists responsible for development of ever more complex systems.’7
Software quality planning
Quality characteristics come about when all processes interact effectively. The ‘mechanism’ processes focus on how the project creates the product; the ‘control’ processes sit alongside monitoring and correcting both the ‘mechanism’ processes and the resulting product.
Software quality planning sets out what CMS ‘mechanism’ and QMS / SQMS ‘control’ processes are going to be used, when and how. The planning needs to be carried out after software has been scoped and graded for risk.
The software quality plan interleaves with the project plan. Note that any planning may need tailoring if there are special process requirements.
The software quality plan explains how the assurance, control and improvement themes will be addressed for the specific software project.
Software quality assurance
Software quality assurance is about giving confidence to stakeholders that the processes and product are as planned; it does this by providing objective evidence.
Process assurance checks the other processes involved, along with how they are interacting. Product assurance checks the product has met requirements when tested by quality control.
Note that ‘product’ is not just the software; the life cycle processes will output supporting products including quality plan, test records, and change notes.
These assurance checks are done by applying ‘control’ processes - verification, reviews and audits, for example - to the process or product at points throughout the life cycle, as laid out in the software quality plan.
The assurance team need to be objective and independent. They may be embedded into a project as ‘project assurance’, or act at an organisation level as ‘quality assurance’11.
‘To quell a widespread misunderstanding, software quality assurance is not testing.’12
Software quality control
The software quality control theme focuses on whether the product meets its requirements. Ultimately, the software or supporting product should only be accepted when it does so.
The product is compared against its requirements by various SQMS / QMS processes such as review, verification and validation.
If the product does not meet requirements there are options including: rework i.e. corrective action; accepting it with a concession; or simply rejecting it. These all draw upon a range of processes in the entire CMS, such as problem resolution and configuration management.
Software quality improvement
Regardless of the product involved, quality generally centres around the Plan-Do-Check-Act (PDCA) cycle10. There should be feedback that corrects and improves processes and product. There is a need to measure the process and product, analyse and evaluate, then adjust in order to improve. Measuring the software process, project and product is not simple.
Improvement contributes to optimising the performance of the product, processes, project and organisation.
Conclusion
The more trust we need to place in our software, the more we need to control it. This control has a cost as processes consume time, effort and resources. However, these costs are balanced by reducing the probability of its failure.
Quality is about meeting stakeholder expectations. If we expect our software not to fail it needs to be a ‘quality’ product.
Meeting the objective of quality does hinge on effectively capturing expectations and following controlled processes. There are many stakeholders involved in this, and all contribute to the product’s quality. Software is a human endeavour, and hence we need to keep a sense of craftsmanship13 and pride in our software product.
We have concentrated here on controlling the product as we work to deliver it; there remains the wider issue of its control during use.
