AI Safety: Replace cavalier oversight with professional standards

BCS member-experts agree AI is unlikely to cause mass human extinction soon, despite dramatic, almost daily headlines about existential risk.
 
Professor Dame Wendy Hall FBCS, former BCS President, said: ‘I’m not worried about mass human extinction in the next decade - that’s just histrionics.’
 
The BCS experts warned, however, that artificial intelligence is a high-risk technology that needs professional standards in the UK and, on the global level, UN-like oversight.
 
‘I am concerned – as anyone should be – with the claims around AI being a catastrophe. Though I believe that it is being overstated,’ said Professor Neil Gordon MBCS, CIPT, chair of the BCS Ethics Specialist Group. 

‘This should not be a debate about bad versus good or fear versus acceptance,’ stated Emma McGuigan, BCS’ AI Programme Lead. ‘We need trusted and transparent AI with a trusted and transparent workforce supporting... We should not unquestioningly trust the titans of the industry; instead, we should build a way to build that trust.’

Underlining its belief in professionalism and in the need to build public trust in technology, BCS has launched a Register of Technology Professionals. It is a public register that helps employers, recruiters, clients and the public verify the professional credentials of BCS members who have chosen to be listed.

Shaping an industry

 Against the wider AI industry backdrop, the UK government has launched a new expert consortium to strengthen trust in AI.
 
Led by BCS, The Chartered Institute for IT, the programme will bring together leading voices from across the UK’s AI assurance ecosystem to support the development of a recognised AI assurance profession.

Sharron Gunn MBCS FCA, CEO of BCS, The Chartered Institute for IT, and Vice Chair of the AI Assurance Stakeholder Consortium, also said: ‘As a group, our duty is to create an AI Assurance profession which is proud to adhere to a Code of Ethics and, ultimately, to hold world-class registrations and certifications.’

Again, pointing to the dystopian headlines, Emma McGuigan was keen to emphasise AI can and should be a force for good. And achieving the technology’s promise as a force for good is our industry’s ultimate challenge: ‘AI helped grow a new aortic valve for me in a lab, preventing premature death. It’s helped stave off famine and enabled us to find a Covid vaccine in months, not years.’

The Danger of 'Cavalier' Safety Testing

While total extinction may be off the table for now, technology makers’ often cavalier approach to safety testing creates immediate, tangible dangers for digital infrastructure, internet stability, and public safety.

‘AI is going to be misused by many, and will cause problems, but unless humanity is very foolish, we shouldn’t let it control systems to cause extinction,’ said Professor Gordon.
 
This risk of harm is compounded as AI makers accelerate their shift in focus from single AI models designed to do specific tasks and to answer humans’ questions to AI agents that can act on their own initiative. These agents themselves are now being designed to work as teams and, in some cases, 10,000-strong swarms.
 
Going further, Professor Hall said: ‘I am worried about agents having access to the internet and causing harm… If, as the [AI] companies claim, they don’t know what their agents are doing. Companies are responsible for testing their systems properly, not in this cavalier fashion. We need to hold them to account on safety testing.’

Is slowing or pausing frontier AI development possible?

As AI usage has moved from familiar chatbots and has shown itself to be increasingly capable, many prominent AI industry leaders have called for frontier labs to pause or slow development.
 
These calls gained momentum after July 2026. During a cybersecurity evaluation at OpenAI, roughly 1,200 agents bypassed network constraints, built a covert message board to coordinate, and exploited a vulnerability to breach the servers of rival AI platform Hugging Face.

Their goal wasn't malicious, they simply calculated that hacking a third-party server was the easiest way to steal the answer key to their test. It served as a stark warning shot, proving that highly capable multi-agent swarms can already collaborate and take dangerous actions entirely on their own.

For you

Be part of something bigger, join BCS, The Chartered Institute for IT.

In early September 2026, Dario Amodei (CEO of Anthropic) published a 3,800-word essay urging the industry to 'pace the rate of capabilities advancement' and 'slow the pace at which we improve the capabilities of AI models.' He proposed a tiered approach, including speed limits on recursive self-improvement and potentially a government-coordinated 'pause.'

However, critics point out the hypocrisy of these calls. Anthropic is reportedly preparing for an IPO and is said to be evaluating the release of a new model to counter OpenAI's newly deployed GPT-6 Astra, despite Amodei's essay.

Asked whether slowing or pausing frontier AI development was possible, Professor Wendy Hall said: ‘I doubt it. US-based companies need to call for this because they are running out of money and need investment to stay competitive in their [technology] race... They also have Chinese frontier-model companies snapping at their heels, and Chinese systems are open. People in the US are already using Chinese AI systems because it’s cheaper than buying tokens from the US companies. And who is going to persuade the Chinese companies to slow down?’

It’s also worth remembering that the leaders of AI’s titanic firms have been calling for a pause in the AI arms race for three years.
 
In March 2023, the Future of Life Institute published a landmark open letter – backed by over 30,000 signatories including many AI industry leaders – calling for an immediate, six-month moratorium on training AI systems more powerful than GPT-4.

Despite ample opportunity to take their collective feet off the accelerator, none has.
 
So why should the public trust AI’s makers when figures like OpenAI’s Sam Altman assure society their companies will do the right thing?

‘We shouldn’t,’ Professor Gordon stated unequivocally. ‘That would be like trusting tobacco companies when it comes to smoking.’

The Path to Genuine AI Safety

If self-regulation is a dead end and a voluntary pause is unenforceable, what is the ultimate solution to ensuring frontier AI remains genuinely safe?

For Professor Wendy Hall, the answer lies in building robust global oversight architecture rather than relying on corporate promises.

She said: ‘We need an independent AI authority, what I and others are calling the IAIA, which sets the standards for measuring, evaluating, and testing AI systems, and has the explicit authority to audit companies against those standards," said Professor Hall.

The IAIA stands for the International Artificial Intelligence Agency, a proposed global governance body modelled heavily after the IAEA (International Atomic Energy Agency), which currently oversees global nuclear regulations.

She stressed that to be effective, any oversight mechanism must transcend current geopolitical divides: 'These standards need to be agreed globally, including by both the US and China. The UN and international bodies are working hard to achieve this, but it is extremely difficult work – and incredibly urgent, given that we cannot rely on AI companies to self-regulate to keep us safe in the meantime.’