What software engineering taught us about problems

Summary:

• Problems must be treated as structured, evolving objects with explicit components: environment, need, and problem owner
• Adequacy is a stakeholder judgement, not something that can be fully automated or logically derived
• Problem solving is a process of transformation, not freezing requirements — problems change as solutions are explored and environments shift

For decades, software engineering has been a highly reflective engineering discipline. Forced to confront extreme complexity, rapid change and intangible artefacts, it has developed a rich body of work about requirements, architecture, validation and risk. In doing so, it has also exposed a deeper issue — one that reaches beyond software itself. That issue is not how to build better systems, but how better to understand problems.

Out of this recognition has emerged ‘problem science’, which is concerned not only with producing solutions, but with understanding, structuring, transforming and validating the problems themselves. While its scope now reaches far beyond software, its intellectual roots lie firmly in software engineering.

Software engineering as a laboratory for problem thinking

Software engineering has long struggled with a difficult asymmetry: while code can be made precise, the problems it addresses rarely can. Requirements evolve, stakeholders disagree, environments shift and systems interact with people and institutions in ways that resist tidiness. Unlike many traditional engineering domains, software is deployed into social, organisational and regulatory contexts that change during development; in fact, often because of it!

In light of many high-profile software failures, and starting with Winston Royce’s critique, software researchers began to challenge the simple waterfall view of engineering: analyse, design, build, deploy.

Subsequent work on requirements engineering, socio-technical systems and software architectures revealed that many failures were not caused by implementation, but by poor specification: the wrong need, framed in the wrong way, for the wrong environment, judged by the wrong people, or at the wrong time.

As software systems have become embedded in almost every aspect of organisational and civic life — at the heart of safety-, mission-, and even democracy-critical systems — it has become increasingly clear that many failures are not in software development, but relate to the difficulties of requirements capture. Problem science takes this observation very seriously, and generalises it.

Problems as first-class objects

In problem science, a problem is treated as a first-class object, not as an informal prelude to ‘the real work’. A problem is understood as a relationship between:

• An environment — the part of the world in play
• A need — what is desired
• A problem owner — the stakeholder who judges adequacy

A problem is solved only when a stakeholder judges that a proposed artefact — software, process, policy, document, incentive, alone or in combination — satisfies the need in that environment. This judgement is not logical entailment; it is situated acceptance in a complex, volatile and potentially hostile world.

For you

Be part of something bigger, join BCS, The Chartered Institute for IT.

Software engineering has taught us that this judgement cannot be automated away. Test suites, proofs and metrics all play a role, but ultimately someone must decide that ‘this is good enough, here, now’. Problem science makes that act explicit, inspectable, traceable and respectable.

Problem science in action

Another hard-won lesson from software engineering is that problems do not stay still.

As soon as we explore a solution, we learn more about the problem; as soon as we deploy, the environment shifts. Attempting to ‘freeze’ a problem early is not rigour — it is often a source of risk.

Problem science therefore treats problem solving as a process of problem transformation. We move from one problem to others: refining needs, re-describing environments, decomposing concerns, delegating judgement, or introducing structure. Each such move must be justified, recorded and accepted by the relevant stakeholders.

Seen in this way, familiar software practices acquire a deeper interpretation:

• Architectural decomposition becomes problem structuring
• Design patterns become reusable problem forms
• Validation becomes stakeholder judgement under stated assumptions
• Technical debt often reveals itself as unexamined problem drift

What software engineering contributed here was not just techniques, but a form of intellectual humility: an acceptance that adequacy is contextual, provisional and socially grounded.

Beyond software, but because of it

Although problem science applies equally to healthcare, education, regulation, public policy and organisational change, it could not have emerged without software engineering. Software forced us to confront:

• Environments we do not control
• Stakeholders we cannot ignore
• Behaviours we cannot fully predict
• Systems that reshape the problems they were meant to solve

Other fields, from AI governance to climate policy, are now encountering similar conditions — often without the conceptual tools that software engineers have developed through hard experience. Now, problem science offers those tools across the board.

Why this matters now

Today, we still deploy sophisticated technology in response to poorly understood needs; we optimise systems against proxies that hide lived reality and we scale solutions faster than our understanding of the problems they create.

Problem science does not promise better answers. It promises better questions, better framing and better judgement. In a world increasingly shaped by complex socio-technical systems, that may be one of the most valuable engineering disciplines of all.

This article has sketched Problem Science and its roots in software engineering. Our forthcoming book develops these ideas in depth: formalising problem transformation, examining creativity and validation, and exploring how problem-orientated thinking can help us navigate complexity, volatility, hostility, and institutional risk.

For software professionals, its message is simple: many of the skills we already possess are not just implementation skills, but problem-scientific ones. Recognising them as such is the first step toward using them more responsibly — and much more powerfully — across our lives.