History of the cloud

Blesson Varghese, Assistant Professor at the University of St Andrews, traces the history of cloud technology, starting from its foundations five decades ago, through its development, to examine just where the cloud is heading in the future.

Summary:

The conceptual foundations of the cloud were laid in the 1960s, with early examples coming from IBM and ARPANET
The first network operating systems were launched in the 1980s, the World Wide Web in 1991, and the term cloud computing was coined in 1996 — though specialist-only grid-computing led in popularity
The first and second generations of modern cloud computing developed between 2001 and 2017
Edge computing became important as the IoT era took hold, reducing pressure on the cloud
The emergence of LLMs in 2022 produced a whole new kind of workload, driving investment in data centres, changes in hardware, and new ways for the user to interact with the cloud

The current digital age is marked by our reliance on the cloud.

Every internet user is confronted with a variety of cloud choices and options which were simply not available a decade ago. A plethora of services we now rely on are hosted by the cloud.

The frustrations we have when we cannot connect to the cloud are evidence of how integral it has become to our lives. But how did we get here? What is the history of the cloud we need and now use every day?

The foundation era: 1960s

A mashup of three fundamental concepts define the cloud: the first is delivering a service, such as computing or storage as a utility; the second is multiple people sharing the same computer resource, which is possible through a technology referred to as 'virtualisation'; the third is accessing services via networking. The 60s was a momentous decade in laying this foundation.

The late Professor John McCarthy — a visionary computer scientist who coined the term ‘artificial intelligence’ in 1955 — suggested in 1961 that computing would one day be sold as a utility. In 1967, IBM virtualised operating systems allowing for multiple users to timeshare the same resource. In 1969, ARPANET (Advanced Research Projects Agency Network), a network relying on the TCP/IP protocol, was launched by the US Department of Defense and became the forerunner of the internet.

The melting pot: 1970s/80s

Research in virtualisation, operating systems, storage and networking advanced in the next two decades, finding new applications. This melting pot fused technologies for novel outcomes, such as networks that allowed wire transfers between financial institutions. In the UK, scientists worked on using unused television signals for sending data.

In 1976, advances in networking were demonstrated: Queen Elizabeth II sent an email. A couple of years later, the White House installed its first computers. In the early 80s, network operating systems were launched to allow computers to talk to each other. By 1985, storage tapes that could store up to 200 megabytes of data (an average smartphone has 10 times this memory) were available. By this time around 100,000 computers were connected to the Internet.

The march begins: 1990s

The foundational technologies for the cloud reached a certain level of maturity in the 90s. This was epitomised by the launch of the World Wide Web in 1991 when more than a million machines were connected to the internet. This led to the dotcom revolution and e-commerce gaining popularity. The client-server model of distributed computing was implemented, such that websites provided front-ends for users and servers, located in the World Wide Web, hosted the backend logic.

The earliest mention of cloud computing in literature is known to appear in an internal document of Compaq in 1996. In 1997, Professor Ramnath Chellappa from Emory University mentioned the cloud in an article. Although the term ‘cloud’ was coined, it was a predecessor of cloud computing — known as grid computing — that became popular. Although grids networked compute resources of organisations across the continents, it was still not accessible to non-specialist users or developers.

The inception: 2001-2004

The inception of the modern-day cloud was realised by Amazon Web Services (AWS) launching its public cloud in 2002. There were virtually no competitors at this stage and while the benefits of using the cloud, such as elasticity and scalability were known, the real use cases to convince potential users were not yet available.

Indeed, the cloud offered a dream solution to the technical and management nightmares from which many SMEs suffered. It alleviated the burden of maintaining servers, upfront investment on compute resources, and scaling web services based on demand by renting resources that were concentrated in large facilities and maintained by trusted providers.

While more websites and workflows were starting to find a place in the cloud, the next decade saw the cloud develop over two distinct generations.

Cloud timeline 1960-2005 (click for larger image)

The first generation cloud: 2005-2011

What we now recognise as the cloud was realised during the first generation — centralised infrastructure in data centres that host a lot of compute and storage resources. The momentum gained during this time allowed application owners to exploit a traditional two-tiered architecture in which cloud providers hosted the backend, while users sent their requests from web and mobile applications.

For you

Be part of something bigger, join BCS, The Chartered Institute for IT.

The OpenNebula research project was launched to develop an entire software stack for easy use of the cloud in 2005. The Elastic Compute Cloud (EC2) from AWS was made available to the general public in 2006. During this time, the technology was further developed and solutions focused on empowering data centres — it made applications reliant on them while mitigating risks for the user.

Database services started becoming available on the cloud and the popular Dropbox concept facilitated cloud storage as a service. Cloud providers offered data centres in a few locations, but after 2008, multiple regions were available. Service level agreements (SLAs) were articulated to quantify and guarantee a quality-of-service (QoS) to the user.

As the software stack required by institutions to organise their clusters as clouds became available, so private clouds started to emerge. It was not until 2009 that a standards body provided a definition and later still in 2011 when a reference architecture became available. Microsoft entered the marketplace, launching Azure services. Then similar to OpenNebula, the OpenStack software was launched in 2010. Both these projects were open source and attracted huge interest from the IT community.

The second generation cloud: 2012-2017

Building on the first generation cloud, the second saw a dramatic enrichment in both the services provided and increased competition with a wide choice of providers. As it was possible to monitor the resources used on the cloud, so the concept of the cloud became more trusted. In addition to the pay-as-you-go pricing model, spot bidding was introduced for resources. Real-time streaming services started processing data on the cloud.

In addition to relational databases, non-relational database services were now available. DevOps gained prominence and microservices were used for cloud application development. This was fuelled by the launch of container services on the cloud in 2014. A newer definition started to emerge where public and private clouds could be combined to deliver hybrid clouds. In Europe, the European Grid Infrastructure Federation cloud was launched in 2014.

Beyond centralised data centres

A significant development during the first and second generation cloud implementation initially went unnoticed: processing outside of the cloud.

While centralised clouds (which were distant from end users) had gained popularity, research had begun on whether some of a user’s requests could be processed outside the cloud to reduce communication latencies. The ‘cloud in a box’ concept emerged. The possibility of computing on networking elements, such as routers and base stations, had been defined as ‘edge computing’ in 2009. Later on, edge computing received further attention from Cisco under the umbrella of ‘fog computing,’ in which computing is facilitated along the entire cloud-edge continuum.

The Internet of Things era meant billions of devices became connected to the internet. Edge computing provided a mechanism to alleviate some of the associated challenges, and therefore the market embraced the edge technology.

Although the cloud was crowded, the technology offered more room for expansion. The golden era of cloud computing was yet to dawn on us.

Cloud timeline 2005-2020 (click for larger image)

New workloads for the cloud: 2022 and beyond

For much of its history, cloud computing was designed around workloads, such as web applications, databases, storage and analytics. The emergence of conversational LLMs for public use in 2022 introduced a different class of workload. Training these models requires enormous amounts of parallel computation, memory and high-speed networking, while serving them to millions of users creates a continuing demand for computing capacity. Together, these demands are driving a new wave of investment in data centre capacity.

This has also changed the hardware beneath the cloud. General-purpose CPUs are increasingly complemented by GPUs and specialised AI accelerators designed to perform the underlying matrix operations of modern AI. The result is a more heterogeneous cloud, in which the hardware is increasingly selected and optimised for particular workloads.

As LLMs become embedded within applications, inference (that is, running a trained model), is becoming a major cloud workload in its own right. An accompanying development is the emergence of AI agents, in which a cloud or edge system can retrieve information, call software tools and perform sequences of actions on behalf of users. Instead of a single request producing a single response, one user action may call LLM models several times.

This growth comes with an environmental cost. Data centres already consume substantial amounts of electricity and water, and AI is accelerating that demand. The future cloud will therefore have to become not only more capable, but considerably more energy- and resource-efficient.