One more year - many more software project failures

Andrew Griffiths, Lamri

Photo of Andrew Griffiths There is a significant gap in understanding between end-users and developers and insufficient investment in the professional development of developers. These are just two hard-hitting points made by Andrew Griffiths, managing director of Lamri, in his review of the past year of the world of software project development.

This is the third year I have written this article for the BCS Annual Review and each year the world has moved on but never quite as fast as I would have hoped. One of the difficulties with writing an article of this nature is that there are very few clear step changes and most trends are evolutionary.

We still have:

  • a large proportion of business managers that consider software 'easy to change';
  • an understanding gap between end-users and developers;
  • a view that placing software development 4,996 miles way from end users is always 'a good plan as it is much cheaper';
  • a greater, but still insufficient, investment in professional development.

So, it's business as usual? Well, not quite.

The NHS programme

Given the size and scale of this programme and the amount of press it has attracted, it feels inappropriate not to discuss it here. The reporting of this programme has largely been negative, which is understandable, but this is one of the largest civilian IT change programmes in existence; why did anyone expect it to run smoothly?

The really interesting story behind the headlines is the business of contract structure and the level to which risk has been transferred to the suppliers. The NHS has procured these systems at a fixed price and does not pay until services are proven to have been delivered and working.

All very laudable, but this places some very high cash flow demands on each of the suppliers that could lead to some painful future consequences. Only huge corporations can afford to bid for this kind of contract and only a tiny number of the UK-based system integrators (SIs) have the financial strength to run with this kind of deal and the inevitable problems that arise.

Contracts of this nature create a 'hard edge' to the relationship between the customer and supplier, often reducing the collaboration between them. When you consider that most project failures have strong roots in poor requirements and that collaboration is key to success in this area, I can't help thinking we are going to see more failures in the future.

The interesting thing about failures in this context is that they could have sufficient critical mass to seriously damage or even bring down a supplier, an outcome that will benefit no one.

There is real strategic investment in software process improvement

I have seen a shift in the focus of clients from point process improvement within software delivery teams to the integration of these into strategic transformation programmes aligned to business strategy. A key enabler for this has been the ascendancy of CMMI as the improvement model of choice in software teams or it could be that CMMI has just become this season's latest fashion accessory for the budding junior executive.

Another key reason for CMMI moving up the agenda is the level of investment by SIs in this area across the globe. All of the usual suspects from the SI market are driving hard in this area, influencing the market, just as the Indian offshore providers have. Process improvement is firmly on the agenda for companies who supply to them and their customers.

Regardless of the real reason, the organisational level thinking required to be successful with CMMI is helping these improvement programmes integrate into the business closer than ever before. This has significant advantages from a business case development and management commitment perspective, significantly increasing the chances of real success.

Technical debt is stifling change

We all know that even successful software projects have skeletons in the closet: compromised architectures, compromised designs, pragmatic technical choices etc. These items all form part of the technical debt that needs to be addressed at some point in the future, however, there is never a good time.

Future releases of a system compound this technical debt and eventually slow the pace of change to such an extent it is like wading through treacle. One of the problems with addressing technical debt is that there is often no clear and present business case and any decision to address this is a courageous one.

Courageous decisions are often career-limiting for the managers involved and are simply deferred year on year. One organisation I have worked with makes almost all of its profits through the operation of one computer system.

This system has huge technical debt and all of the original team have retired. Having fudged addressing this for the last 12 years they are now losing market share due to their inability to respond to the competition and I fear that it is only a matter of time before this becomes a business-critical issue.

Compliance has wriggled its way out of finance and into IT

Thanks to SOX and the Operating and Financial Review (OFR), compliance has found its way onto the agenda in IT teams. This is great as it is improving control and governance of IT programmes and increasing management engagement in, largely, good ways.

However I was disappointed with Gordon Brown effectively removing the teeth (and many of the benefits) of the OFR in his recent speech and selling it as removal of red tape. The reason for my disappointment is that it was UK legislation that was driving up the governance of IT and especially of software projects with the potential to make a material impact on revenue. This useful driver for change has been diluted too far.

Offshoring is still problematic

The fashion for offshoring is still strong and more and more organisations that I meet are being more realistic about what can be achieved. There are still some very naive clients who believe that software development is an off-the-peg service, to be purchased like stationery.

These organisations typically fail to recognise the changes required to their own processes and capabilities that will make an offshore contract work. Worse still these companies often have strong drivers to reduce the cost of procurement, so they run 'hands off' online bidding processes that provide no opportunity for the supplier to really understand the nature of what they are bidding for.

To cap all of this it would seem that the approach to offshoring has been focused using the teachings from Sun Tzu's The Art of War. These relationships should be symbiotic. To be successful they need to be about win/win.

IT needs a Copernican revolution

I have always been concerned about IT as an industry and I could never quite put my finger on what my root concern is. There are several symptoms of which the main one is the lack of board level posts for IT executives. Last week I heard a depressing comment in a meeting that sums up my concerns quite crisply: 'IT people have the potential to become the new business luddites.'

The speaker's view was simple; IT people think the world revolves around technology not around the business of making money. This is why we need the Copernican revolution; business is about making money not building software. IT teams need to radically alter their perspective and ensure that the sun at the centre of the universe is the business not the technology.

It's not all bad news in this area. I have been encouraged by the genuine increase in real business people leading IT teams and holding key positions in IT departments. In the organisations where this is happening there is a strong positive impact on the technical team and a gradual change in focus that has a dramatic effect on the IT team's standing and ability to make the business more successful.

There is a lot of folklore around the invention of the waterfall approach but it is generally accepted that Winston Royce first presented this at IEEE WESCON in August 1970. Just a few months earlier the US government was busy landing a man on the moon. In some senses there is a lot in common between the moon landings and software development process - not a lot has happened in the last 30 years.

Most teams still use a variant of the life cycle described by Winston to build systems that are many times more complex and larger that anything that could be conceived at that time. Do we not feel it is about time for a step change?

Andrew Griffiths is the managing director of Lamri, a software process improvement consultancy. For further information please contact him on email: