One of the reasons so many information systems (IS) development projects fail, is the way they mirror prescribed information flows within the structure of the organisation, which, in practice, may be far removed from what is actually happening.
Experienced managers and developers know how difficult it can be to meet expectations when faced with change and disruption. Formal structures provide consistency and a means of measuring how profitable a business or division is, but are less reliable when it comes to designing and implementing information systems.
Elliott Jaques, a Canadian social scientist, observed the difference between different types of organisation structure. Jaques makes a distinction between formal and extent structures; he puts forward a hypothesis that, when the formal structure breaks down, people find alternative ways in which to work and this gives rise to what he terms the ‘extent structure’. It is this structure that is actually operating and is often implicit in the reality of who’s making what decisions, and who owns which processes.
According to Jaques, there is a third way which centres on what is termed ‘requisite structure’. Requisite structure - ‘what could be’ - is generally what system developers aim to construct (through logical and physical design), but invariably end up delivering a facsimile of the current system structure (with all its flaws).
Conway’s law
Back in 1968, Melvin Conway developed a hypothesis based on the way organisations structure their internal departments and lines of communication. Conway observed: ‘Any organisation that designs a system will inevitably produce a design whose structure is a copy of the organisation's communication structure.’
This observation led Conway to believe that organisation structures and their channels of communication should be designed with flexibility, which is the key to effective design.
A common factor behind many inadequately designed software systems has been the availability of a design methodology which works repeatedly and delivers both technical and organisation benefits. Again, Conway suggested smart organisations would anticipate these requirements, making provision for the adaption of designs and processes.
One issue with software development is the user bias towards the working norm. This inherent bias often influences out-of-scope requirements, which, in turn, leads to re-design activity. In this context, there's never enough time to do something right, but there's always enough time to do it over again, as evidenced by McManus and Wood-Harper in their objective gap analysis between IS project success and failure.
For Jaques, structure is not just about identifying communication relationships. There are objective criteria that exist to determine the work that must be done at each level in the organisation, based on a measurement of the complexity of the work to be done. In the context of IS projects, this concept equally applies: we assign architects and developers to generate structures and code and measure outputs in lines of code or function points. These tasks relate to role responsibility and time; intervals in time can often lead to design ‘discontinuities’, which generally result in failure and increased development costs.
Homomorphism
Conway advocates the difficulties that developers face in transforming business processes into a set of requirements which reflect user needs. Frequent changes in requirements often end up blurring the true relationship between user needs and system design - as developers strive to adjust to the realities of requirements which have a negative impact on the design process.
A major step in creating a requisite design is the analysis and mapping of the existing processes within organisations to create a logical design. In this situation, designers are frequently engaged with people who have specific domain knowledge.
Industry and firm studies into IS project failure have shown that correlation between technical interdependencies and organisational structure is a common pattern of failure - both across different sectors and over time. This is certainly the case in public sector organisations, where knowledge boundaries are drawn more broadly than operational boundaries.
Complexity and modularity
Modularity is a concept that helps us to characterise different designs. It refers to the way in which a product’s architecture is decomposed into different parts or modules. Whereas measures of software complexity focus on characterising the number and nature of the elements in a design, measures of modularity focus on the patterns of dependencies between these elements.
Christopher Alexander argued that it is easier to cope with the complexity of a large-scale problem if it is decomposed into efficiently linked sub-problems. IS development managers have long argued that it is easier to split development work across a group if people can work independently, and in parallel with each other. Much of the prior research into IS project failure highlights a huge gap in technical performance between perceived complexities within projects. There are also substantial differences in relative levels of modularity between software systems of similar size and function, which is not always evident within the architecture of the system.
The architecture of a system can be defined as the scheme by which the functions it performs are allocated to its constituent components. In the initial stages of design, it is common for several iterations to be considered for any given set of functional requirements by which a number of different architectures might be considered viable. However, viability, in some respects, is an arbitrary decision based on the designer’s knowledge and the user’s preferences.
The evidence suggests that the level of stakeholder engagement and the motivation of end users is a critical factor in developing systems which are fit for purpose. The contrast between the two perspectives of ‘success and failure’ can be clarified by considering the dynamics that occur between stakeholders within the organisation.
Whilst organisations have the capacity to deploy a range of stakeholders to attain specific functional objectives, not all stakeholders share the same goals of the organisation. The point being: not all stakeholders contribute to achievement, whether shared or explicit. In this situation, the people charged with managing complex systems will seek to eliminate unhelpful stakeholders where and when needs arise.
Technical and resource dependencies
In the IT Trade-Off, McManus suggests developing an ideal IS system. This proves difficult for two reasons: firstly because, given incomplete knowledge, those guiding the evolution of IS systems will not always be able to agree on how the product will be developed or which architectures, plans, and coordination mechanisms will be sufficient to establish effective coordination among teams.
Secondly, estimates of tasks are inaccurate, process steps are executed imperfectly, technical requirements and technologies change, and people leave. Whilst Conway acknowledges the point that large IS applications can be successful, he also acknowledges that a manager will be vulnerable to the charge of mismanagement if he misses his targets without having applied all available resources. This awareness generates a high-pressured environment where managers take unnecessary risks as the team strives to adjust to the realities of the situation.
Conway’s original hypothesis, as applied to IS systems - that a design effort should be organised according to the need for information sharing between groups and sub-groups – does, in the main, hold true. Having a high-quality, modular design and using it as the basis for assigning work to different groups is good practice. Conway does not claim to understand, nor have answers to the constraints within IS development.
However, he does highlight the need for flexibility in organisations as an important requirement to effective design. The more robust the architecture, the more likely the organisation can successfully develop for different applications. Ultimately, exploring the issues associated with IS design helps to reveal development strategies for moving IS design and development towards the boundary of future knowledge.
