October 2009

Addressing data quality issues is expensive, so it’s important for any organisation to get it right. A new open methodology could help, explains Sean McClowry and Andreas Rindler of MIKE2.0.

So what are the primary reasons why organisations struggle with data quality?

• Systems are more complex than ever before. Companies now have more information and are conducting more integration between systems than ever before. New regulations, M&As, globalisation and increasing customer demands mean that information management challenges are increasingly formidable.
• Silo-ed, short-term project delivery focus. As projects are often funded at a departmental level, they don’t account for the impacts of how data will be used by others. Data flows between systems and the design of these connection points must go across strict project boundaries.
• Traditional development methods do not give enough focus to data management. Many projects are focused more on function and feature than on information – the desire to build new functionality has resulted in information being left by the wayside.
• Data quality issues are hidden and persistent. Data quality issues can exist unnoticed for some time, although some users may suspect the data in the systems they rely on to make their decisions are not accurate, complete, current, valid, or consistent. This data can then get propagated to other systems as we increase connectivity. Organisations tend to underestimate the data quality issues in their systems.
• Data quality is not fit for purpose. It is difficult for users of downstream systems to improve the data quality of their system because the data they derive information from is entered via customer facing operational systems. These customer facing system operators do not have the same incentive to maintain high data quality and they are focused on entering the data quickly and without rejection by the system at the point of entry. It is often when data is integrated, summarised, standardised and used in another context that quality issues begin to surface.

Further burdening the situation is that the term ‘information governance’ has various meanings. While there is consensus that information governance includes data quality management, it is difficult to get a consistent definition even at a high level. There are three primary reasons for this:

1. Data quality is a complex topic that involves more than just the accuracy of data. It is typically measured across seven quantitative dimensions and a number of qualitative dimensions.
2. Data quality management involves more than just addressing historical data quality issues through data profiling and re-engineering - it involves preventing these issues from occurring in the first place. Issue prevention is complex, sometimes involving changes to source systems, business processes and supporting technology.
3. Data governance can be seen to mean more than just data quality. It is sometimes used to cover a collection of best practices around the management of information: the ability to secure data, provide real-time access to data and deal with complex integration issues. 

We feel that many organisations have struggled to meet these challenges because they fail to focus their techniques on the enterprise-wide problem. Managers see information as a technology issue, rather than a fundamental and core business activity. 

Defining an enterprise-wide program, however, is very difficult. Building momentum for these initiatives takes a long period of time and can easily lead to an approach that is out of touch with what the business needs. Attempts to enforce architectural governance, for example, quite easily become a disabling approach for the business or a ‘toothless watchdog’ that provides little value.

Given this complexity, we suggest a more comprehensive program for data management based on information development.  This approach is more detailed in scope and aligned to address a number of organisational data management problems. We envision an approach that can address the inherent challenges on a federated business model and technology architecture in a manageable fashion that is not only effective, but fosters innovation.

Information development is an approach organisations can apply to treat information as a strategic asset through their complete supply chain: from how it is created, accessed, presented and used in decision-making to how it is shared, kept secure, stored and destroyed.  

In the past information development has not been seen as a separate domain, and has been coupled tightly to application development and integration. Decoupling information development from the application development and integration processes does not create extra work; it creates standards for how and when information should be exchanged, standards for how certain types of information should be captured, presented, and shared.

Information development is about:

• Driving an overall approach through an information strategy;
• Enabling people with the right skills to build and manage new information systems while creating a culture of information excellence;
• Moving to new organisational models that delivers an improved information management competency;
• Improving processes around information compliance, policies, practices and measurement;
• Delivering contemporary technology solutions that meet the needs of today’s highly federated organisations;

We believe this is an approach that must evolve significantly over time and be framework-based as opposed to static in its approach. Implemented correctly, organisations will be able to better manage their internal information systems and meet better meet strategic goals. 

About MIKE2.0

MIKE2.0 (Method for an Integrated Knowledge Environment) is an open and collaborative methodology for information development.

To find out more about MIKE2.0 and information development, go to:
http://www.openmethodology.org/
http://mike2.openmethodology.org/wiki/What_is_MIKE2.0
http://mike2.openmethodology.org/wiki/The_Case_for_MIKE2.0
http://mike2.openmethodology.org/wiki/The_Value_of_MIKE2.0
http://mike2.openmethodology.org/wiki/MIKE2.0_Case_Studies