Securing the Age of AI

Summary

• The NCSC reports that AI will enhance attackers' ability to exploit known vulnerabilities, and critical national infrastructure is not immune
• AI is being used to increase the speed rather than the sophistication of cyber attacks, and it can also be adopted defensively
• Secure-by-design systems and basic security hygiene are key in the AI era, as it impossible to test systems against every AI-driven eventuality
• The UK plays a substantial role in shaping global cybersecurity regulations and is investing in fundamental research into emerging vulnerabilities
• The key to a secure future will be using the benefits and abilities of AI carefully against its threats and vulnerabilities

Artificial intelligence is already reshaping the cyber threat landscape, not through science fiction breakthroughs, but often by accelerating familiar attacks and amplifying long standing security weaknesses. To understand what this means for the UK’s critical national infrastructure and public sector, Grant Powell MBCS spoke to Kate, Technical Director for Security of AI Research at GCHQ’s National Cyber Security Centre (NCSC), whose work sits at the intersection of artificial intelligence, cybersecurity, and national resilience. Drawing on more than a decade in the intelligence community, she explains where risks are growing, where fundamentals still matter most and why collaboration — not panic — remains the UK’s strongest defensive asset.

What is your current assessment of risks to UK critical national infrastructure from AI enabled attacks? Are utilities, transport networks and healthcare systems adapting fast enough?

The NCSC has published its assessment that AI enabled tools will almost certainly enhance threat actors’ ability to exploit known vulnerabilities, particularly by increasing the volume and speed of attacks against unpatched systems.

The fundamental challenge is that the time between vulnerability disclosure and exploitation was already shrinking, and AI is highly likely to reduce it further. Critical national infrastructure (CNI), including utilities, transport, healthcare, telecoms and their supply chains, will not be immune to this trend. In many CNI environments, the risk is compounded by the widespread use of operational technology (OT) and industrial control systems that were not originally designed with security in mind. This creates an extra layer of exposure.

However, AI cuts both ways. The same tools that attackers use can also help defenders by identifying vulnerabilities earlier, improving patching, assisting with threat modelling, and supporting defensive automation. The biggest emerging risk is a digital divide between organisations that can adopt AI defensively and those that cannot
(See our recent blog).

Crucially, the most effective defence remains unchanged: doing the basics well. Many AI linked attacks still succeed by exploiting vulnerabilities that should not have existed in the first place.

Are you seeing an increase in the use of AI by adversaries to enhance cyberattacks? How has AI changed the speed or sophistication of threats?

Yes, but with an important distinction. AI is primarily being used to accelerate attacks, rather than make them more sophisticated. It helps attackers be more productive, lowers the barrier to entry, and allows them to operate at greater scale and speed. What we have not yet seen is a material shift in the overall sophistication of attacks. In fact, some AI generated attacks have been less subtle. Recent supply chain compromises believed to be ‘vibe coded’ were detected and stopped quickly because attackers lacked sufficient domain expertise. AI can enable more people to try attacking systems, but human capability still matters. It’s worth keeping in mind that many attackers are subject to the same incentives as legitimate organisations: speed, efficiency, and cost reduction. AI lets them move faster, not necessarily smarter.

How confident are you in current methods for evaluating the safety of advanced AI models? Do we need entirely new testing frameworks?

Government responsibility for AI safety testing largely sits with the AI Security Institute (previously known as the AI Safety Institute) which leads internationally. The NCSC works closely with the Institute wherever our role as the National Technical Authority for cyber security overlaps with AISI’s work on understanding the capabilities of frontier AI models, particularly on cyber capable models and the robustness of guardrails.

That said, there are clear scientific limitations with testing. Test environments inevitably differ from operational reality and, given how AI systems work, it is unrealistic to test every capability or vulnerability in the same way as traditional software.

This uncertainty strengthens the case for secure by design systems, including the wider socio technical context around AI. Systems should be designed so that when things fail, deliberately or accidentally, the impact is contained rather than catastrophic.

Are you concerned about the provenance of AI models, training data or hardware used in UK public sector systems?

AI supply chains are far more complex than traditional software supply chains, particularly because data plays a direct role in system behaviour. There is currently no global consensus on how best to record, verify, and share information about AI supply chains and provenance, which is a concern. Training data, fine tuning data, and contextual documents all influence outcomes in ways that standard software components do not.

For you

Be part of something bigger, join BCS, The Chartered Institute for IT.

However, most recent incidents described as ‘AI supply chain attacks’ have in fact been very traditional software supply chain compromises. Attackers are exploiting enthusiasm for AI and poor security practices, rather than using novel AI techniques.

Again, secure by design principles and basic security hygiene remain critical. Many of these incidents could have been prevented if developers had designed systems with known supply chain risks in mind.

How are you helping government departments deploy AI safely without creating new vulnerabilities?

The NCSC has been working on AI security since before the term ‘AI’ became widespread. The organisation published one of the world’s first guidances on securing AI in the form of the Principles for the Security of Machine Learning, and continues to build on long standing cyber expertise rather than treating AI as an isolated technology. We combine technical, socio technical, and human centric expertise with established relationships across government, critical infrastructure, and industry. This allows guidance to be tailored, credible, and practical.

The government is also investing in fundamental research, both in house and through collaboration with industry, academia, civil society, and international partners, to understand emerging vulnerabilities before they materialise.

How would you summarise the role that the UK plays in shaping global standards for AI safety and cybersecurity?

The UK plays a very substantial role. A clear example is ETSI EN 304 223, which defines baseline cybersecurity requirements for AI models and systems and was published earlier this year. UK government departments, supported by the NCSC, were instrumental in convening and shaping this standard. This builds on decades of UK leadership in cybersecurity standards and international collaboration - work that long predates the current AI surge. While gaps remain, the UK is actively involved in refining and extending these frameworks.

What are your thoughts on the fragmented AI regulatory frameworks across the world. Do they create new security risks, or opportunities for cooperation?

Different regulatory regimes do introduce complexity, but I don’t see them as a blocker to collaboration. AI and cybersecurity are shared global challenges which no government, industry, or frontier AI lab can address alone. Standards development, information sharing, coordinated vulnerability disclosure, and joint guidance continue across borders.

Countries are still working together, and the UK’s experience with multilateral standards and co signed guidance demonstrates that collaboration remains both possible and productive.

Do you foresee a future where AI autonomously detects and mitigates threats in real time? What are the risks of over reliance?

In some contexts it is likely to be both possible and necessary. Automated, AI driven responses will be essential where attacks move faster than humans can react. Concepts such as ‘cyber first aid’ illustrate how autonomous systems can contain damage rapidly.
However, over reliance certainly introduces risks, such as new vulnerabilities, exploitable automation, and technical debt.

Autonomous systems must be carefully bounded, with appropriate permissions and safeguards. We talk about a ‘human in the loop’ approach whereby people effectively have the final say in any decision, with full oversight of processes. Yet it is critical to ensure that this continues to be meaningful and not symbolic. Simply clicking ‘approve’ without understanding — the equivalent of ignoring terms and conditions - is not a real mitigation.

What risk scenarios keep you and other AI security specialists awake at night?

One particularly concerning risk is indirect prompt injection. Unlike direct prompt injection, where a user inputs malicious instructions, indirect prompt injection hides those instructions inside data sources that can be read by AI. These might include documents, emails, URLs, or other inputs. Because large language models do not fundamentally distinguish between data and instructions, they implicitly trust all input.

This vulnerability is rooted in how LLMs work and is therefore difficult to eliminate. Mitigations exist, such as separating agents by privilege level and filtering inputs, but they require careful system level design. 

The tension lies between getting value from AI and limiting what AI is allowed to act upon. Managing that balance is one of the most challenging problems facing AI security today.