There is something unusual happening in artificial intelligence. Some of the people working on the world's most advanced AI systems are warning that the technology they are helping to develop could eventually pose a catastrophic risk to humanity. Others within the same field argue that these fears are exaggerated, based on speculative assumptions about technologies that do not yet exist.
That disagreement might be easier to dismiss if AI development were progressing slowly. It isn't. The largest technology companies are spending extraordinary amounts of money on computing infrastructure, new models are arriving regularly, and AI systems are being given greater autonomy and access to tools that allow them to do more than simply generate text.
Governments have responded by establishing AI safety institutes and developing new regulations, while researchers are testing whether increasingly capable models can operate independently, deceive evaluators or perform some of the tasks that would theoretically be required to replicate themselves.
The debate has now reached some unlikely places. King Charles is hosting an AI summit at Dumfries House, bringing together senior figures from the technology industry to discuss the future development of artificial intelligence and its impact on society.
Against that background, the question of whether AI could eventually present a serious risk to humanity no longer feels quite as easy to dismiss as science fiction. The difficult part is working out how seriously we should take a danger that remains highly uncertain.
How worried are the people building it?
How worried are the people building it?
In September 2026, concerns about AI risk received renewed attention following warnings from researchers working close to the technology. In an interview examining whether AI could realistically threaten humanity within the next decade, The Conversation discussed an estimate from Anthropic alignment researcher Evan Hubinger that there was a greater than 10% probability of AI killing humanity within that period.
That figure needs considerable context. Humanity has never built a superintelligent AI, so there is no historical evidence from which anyone can reliably calculate such a probability. Professor Kate Devlin of King's College London points out that predictions of human-level or superhuman AI being only a few years away have been made before, and nobody actually knows whether today's technology will ultimately lead there.
But uncertainty doesn't necessarily make the issue irrelevant. When the possible consequences are sufficiently severe, even a small and uncertain probability can justify precautions. The problem with AI is that we're being asked to make those judgements while the technology itself is changing extremely quickly.
What could actually go wrong?
What could actually go wrong?
The popular version of an AI catastrophe usually involves machines becoming conscious and turning against humans. Most serious AI safety research is considerably less cinematic.
One of the more immediate concerns is misuse. The 2026 International AI Safety Report examines the potential for increasingly capable AI to assist with cyber attacks and provide information relevant to biological and chemical weapons. There are still substantial practical barriers between an AI providing information and somebody successfully causing serious harm, but more capable systems could gradually make specialist knowledge and complex processes easier to access.
AI also doesn't necessarily need to be deliberately misused. As we move from chatbots that provide answers towards AI agents that can access browsers, files, software and other systems, mistakes become more consequential. An incorrect chatbot response can be ignored; an autonomous system acting on an incorrect assumption may be able to make changes before a human intervenes.
The most extreme scenario is a future AI sufficiently capable and autonomous that humans struggle to control it. Today's systems aren't capable of this, something the International AI Safety Report makes clear. What interests researchers is whether some of the capabilities that might eventually contribute to such a scenario are beginning to emerge.
Some of those capabilities are improving quickly
Some of those capabilities are improving quickly
The UK's AI Security Institute tests frontier AI models against tasks associated with autonomy, cyber security and other potentially dangerous capabilities.
One particularly striking example involves simplified self-replication evaluations. These are controlled tests of individual skills that might be required for an AI to obtain computing resources and maintain access to them. In 2023, the strongest models achieved success rates below 5% across many of these evaluations. By 2025, frontier models had achieved more than 60%.
That does not mean an AI currently has a 60% chance of escaping onto the internet and copying itself. AISI says current models still struggle with important parts of the process and there is no evidence that models are spontaneously attempting to self-replicate.
The significance is the speed at which capabilities are changing. AISI has found rapid improvements across the areas it measures, with performance in some categories doubling approximately every eight months.
That doesn't demonstrate that an AI catastrophe is approaching. It does explain why some researchers are uncomfortable waiting for definitive evidence of danger before taking precautions.
There are good reasons to be sceptical
There are good reasons to be sceptical
Existential AI risk shouldn't be treated as established fact simply because prominent people within the industry are discussing it.
As Professor Kate Devlin argues, we don't currently have superintelligence and nobody knows when, or even whether, it will arrive. Today's systems remain unreliable in surprisingly basic ways, and there is no guarantee that making current models larger will eventually create something vastly more intelligent than humans.
There are commercial incentives worth considering too. AI companies benefit from the perception that the technology they're developing is extraordinarily powerful. Describing AI as potentially civilisation-changing can attract investment, political attention and customers as effectively as it can generate genuine concern.
There is also a risk that hypothetical extinction scenarios distract attention from problems already happening. AI-assisted fraud, misinformation, copyright disputes, environmental impacts and disruption to employment don't require superintelligence.
It shouldn't have to be one debate or the other. We can address the measurable harms of today's AI while investigating the potentially much larger risks of future systems.
The bigger risk may be the race
The bigger risk may be the race
Perhaps the strongest reason for taking AI safety seriously is the incentive structure surrounding its development.
OpenAI, Anthropic, Google, Meta and others are competing for customers, investment, computing power and technical talent. Governments increasingly regard AI leadership as strategically important too.
That makes slowing down difficult. If one company delays a new system because it believes further safety research is required, a competitor may continue. The same problem exists between countries. Cautious development might benefit everyone collectively while rapid development remains the rational choice for each individual participant.
This is behind some of the calls for international intervention. Writing in The Guardian, Gaby Hinsliff argues for pausing particularly risky areas of AI research while international safeguards are developed, rather than stopping AI research altogether.
That distinction matters. We already accept that potentially transformative technologies can require significant restrictions without abandoning them. Pharmaceuticals undergo clinical trials, aircraft require certification and nuclear technology is subject to extensive controls. AI is different from all of those technologies, but the principle that greater potential consequences justify greater scrutiny isn't particularly radical.
So, could AI actually threaten humanity?
So, could AI actually threaten humanity?
There is currently no good reason to believe that ChatGPT, Claude, Gemini or another existing AI system is about to destroy humanity. Current models lack the autonomy and broader capabilities required by the more extreme scenarios being discussed.
There is equally little basis for confidently saying that sufficiently advanced AI could never become dangerous. We don't know how capable these systems will become, how quickly those capabilities will emerge or whether the safety techniques being developed alongside them will prove sufficient.
The 2026 International AI Safety Report reflects this uncertainty rather well. Its contributors don't agree on the probability of humanity losing control of future AI systems and the report doesn't predict that such an outcome will happen. It does, however, recognise that the possible consequences are serious enough to justify research and preparation.
The fact that King Charles is now bringing some of the industry's biggest names together to discuss the future of AI is another indication of how far this discussion has moved beyond technology companies and research laboratories.
Perhaps today's fears about superintelligence will eventually look misplaced. Current AI architectures might reach their limits and the predicted leap towards vastly more capable systems may never happen.
But we're currently developing the technology while simultaneously trying to understand what its limits and risks might be. The important question isn't whether anyone can prove that AI will threaten humanity, because nobody can. It's how much evidence of that possibility we should require before deciding that some precautions are worthwhile.
Contact
Get in touch with the team to discuss how we support your business with practical, people-first technology and long-term solutions.
About
Learn who we are, what we stand for, and how Ostratto helps businesses make their work, less work through practical technology solutions.
Our Approach
Discover how we partner with you - focusing on strategy, simplicity and long-term value to deliver technology that truly supports your business.
