The incidents also illustrate why discussions about AI safety don't necessarily need to involve a system deliberately attempting to cause harm.

A sufficiently autonomous system could cause problems simply because it interprets an objective differently from its developers, discovers an unexpected way of completing a task or interacts with another system in a way that wasn't considered when either system was designed.

This is particularly relevant to business automation. Traditional automation is usually based on relatively predictable rules. A workflow might be configured so that when a particular event occurs in a CRM, a defined action follows.

An AI agent can operate differently because it may be given an objective and allowed to determine the individual steps required to achieve it. That flexibility is precisely what makes agents attractive, particularly for processes that are too complicated or variable for conventional automation, but it also makes their behaviour less predictable.

In a normal business environment, an unexpected action could mean incorrectly modifying CRM records, communicating with the wrong customers, accessing information that isn't relevant to the task, creating transactions or triggering other automated processes.

None of those scenarios requires an AI system to be intentionally hostile. They simply require the system to have sufficient access to turn a poor decision into an action.

Cybersecurity already has a well-established concept for dealing with this problem. The principle of least privilege means that a person or application should only receive the permissions genuinely required to perform its role.

The same approach is likely to become increasingly important for AI agents.

An agent analysing CRM records may need permission to read customer information without requiring the ability to delete it. An agent helping with invoicing might be allowed to prepare transactions while requiring human approval before they are sent. An AI researching suppliers might require access to the internet without needing unrestricted access to internal documents.

Businesses will also need to decide which actions remain subject to human approval. Financial transactions, deleting information, changing permissions, publishing content and communicating externally are obvious areas where allowing an agent to act completely independently deserves careful consideration.

Monitoring will be equally important. If an AI is able to act on behalf of an organisation, businesses need reliable records of which systems it accessed, what it changed and which actions it performed. OpenAI's response to the Hugging Face incident includes greater monitoring of agent behaviour alongside stricter isolation and access controls, reflecting how important those safeguards become as models gain greater autonomy.

At that rate, you could submit nearly 2,000 prompts before using the equivalent of a 500 ml bottle of water. Even 10,000 prompts would account for approximately 2.6 litres.

The estimates differ because they examine different models, locations, cooling systems and sources of water use. AI infrastructure has also become considerably more efficient.

The larger figures reveal the real scale:

Not all data centre activity relates to AI, and water withdrawal is different from water permanently consumed. Nevertheless, AI is helping to drive the rapid expansion of this infrastructure.

The first is data centre cooling. The processors used to run AI models generate substantial heat. Some data centres use evaporative cooling, where water absorbs the heat before evaporating. A UK report on water use in AI and data centres suggests that around 80% of the water used by these systems can be lost through evaporation.

The International Energy Agency reports that the energy needed for individual AI tasks has been falling rapidly. However, video generation, extended reasoning and autonomous agents can consume hundreds or thousands of times more energy than simple text generation.

A 100 MW hyperscale data centre could consume approximately 2.5 billion litres of water annually, depending on its design and location.

That is equivalent to:

The figure still demonstrates the difference between one prompt and the infrastructure processing millions of them around the clock.

However, AI is contributing to the construction of new data centres and the installation of more powerful computing equipment. By 2030, global data centre water consumption could reach 1.2 trillion litres annually, equivalent to approximately 480,000 Olympic swimming pools.

That figure needs considerable context. Humanity has never built a superintelligent AI, so there is no historical evidence from which anyone can reliably calculate such a probability. Professor Kate Devlin of King's College London points out that predictions of human-level or superhuman AI being only a few years away have been made before, and nobody actually knows whether today's technology will ultimately lead there.

But uncertainty doesn't necessarily make the issue irrelevant. When the possible consequences are sufficiently severe, even a small and uncertain probability can justify precautions. The problem with AI is that we're being asked to make those judgements while the technology itself is changing extremely quickly.

The popular version of an AI catastrophe usually involves machines becoming conscious and turning against humans. Most serious AI safety research is considerably less cinematic.

AI also doesn't necessarily need to be deliberately misused. As we move from chatbots that provide answers towards AI agents that can access browsers, files, software and other systems, mistakes become more consequential. An incorrect chatbot response can be ignored; an autonomous system acting on an incorrect assumption may be able to make changes before a human intervenes.

The most extreme scenario is a future AI sufficiently capable and autonomous that humans struggle to control it. Today's systems aren't capable of this, something the International AI Safety Report makes clear. What interests researchers is whether some of the capabilities that might eventually contribute to such a scenario are beginning to emerge.

The UK's AI Security Institute tests frontier AI models against tasks associated with autonomy, cyber security and other potentially dangerous capabilities.

As Professor Kate Devlin argues, we don't currently have superintelligence and nobody knows when, or even whether, it will arrive. Today's systems remain unreliable in surprisingly basic ways, and there is no guarantee that making current models larger will eventually create something vastly more intelligent than humans.

Perhaps the strongest reason for taking AI safety seriously is the incentive structure surrounding its development.

OpenAI, Anthropic, Google, Meta and others are competing for customers, investment, computing power and technical talent. Governments increasingly regard AI leadership as strategically important too.

That makes slowing down difficult. If one company delays a new system because it believes further safety research is required, a competitor may continue. The same problem exists between countries. Cautious development might benefit everyone collectively while rapid development remains the rational choice for each individual participant.

That distinction matters. We already accept that potentially transformative technologies can require significant restrictions without abandoning them. Pharmaceuticals undergo clinical trials, aircraft require certification and nuclear technology is subject to extensive controls. AI is different from all of those technologies, but the principle that greater potential consequences justify greater scrutiny isn't particularly radical.

The 2026 International AI Safety Report reflects this uncertainty rather well. Its contributors don't agree on the probability of humanity losing control of future AI systems and the report doesn't predict that such an outcome will happen. It does, however, recognise that the possible consequences are serious enough to justify research and preparation.

The fact that King Charles is now bringing some of the industry's biggest names together to discuss the future of AI is another indication of how far this discussion has moved beyond technology companies and research laboratories.

At a basic level, an MCP connection involves an AI application acting as a client and an MCP server exposing information or capabilities that the AI can use. The server effectively tells the AI what it has access to and how those capabilities can be used.

Imagine, for example, that an organisation connects its CRM to an AI assistant through MCP. Rather than exporting sales data into a spreadsheet and uploading it to ChatGPT or another AI tool, someone could simply ask which opportunities have not been contacted during the past fortnight. The AI could retrieve the appropriate records from the CRM and produce an answer based on live information.

The more significant change comes when the AI is allowed to perform actions as well as retrieve information. Having identified those neglected opportunities, the user might ask the AI to create follow-up tasks for each account owner. Provided the relevant tools and permissions have been made available, the AI could carry out the request in the CRM rather than merely explaining how the user should do it.

This distinction between retrieving information and performing actions is central to understanding why MCP is attracting so much attention. Connecting AI to business systems is useful because it gives the model better context, but connecting it to the functions within those systems opens up the possibility of AI becoming an active participant in business processes.

Most business use of generative AI still revolves around producing something for a person. An employee asks for an email, summary, report, calculation or piece of analysis, receives an answer and then decides what to do with it.

That model is already useful, but it leaves the employee responsible for moving between the AI and the applications where the work actually happens. If an AI assistant identifies that ten sales opportunities require attention, somebody still has to open the CRM, find those records and create the necessary tasks.

MCP is part of a broader move towards AI agents, where the objective is not simply to generate an answer but to allow AI systems to use tools in order to complete a task. Once an AI can securely access the systems a business already uses, the range of useful tasks becomes considerably wider.

Consider a fairly ordinary request such as asking an AI assistant for an update on a particular customer. A useful answer might require information from several places. The CRM contains the customer's sales history and current opportunities, the finance system contains invoices and payment information, the support platform contains outstanding issues and the project management system shows the progress of ongoing work.

Today, an account manager might open each of those applications and piece together the information manually. With appropriate MCP connections, an AI agent could potentially retrieve the relevant information from each system and produce a single account summary.

The employee could then ask it to create a task for an overdue sales follow-up, prepare an email regarding an outstanding invoice or summarise an unresolved support issue. Rather than AI being another application that needs to be checked, it starts to become a way of interacting with the applications the business already has.

Calling something a bubble can make it sound as though the underlying thing is fake. That is not really what a financial bubble means.

Strictly speaking, economists generally use the term for a situation where the price of an asset moves beyond what its underlying fundamentals can reasonably justify. Bubbles can become self-reinforcing: rising prices attract investment, that investment creates further growth and confidence, and the apparent success encourages even more money to enter. Eventually the fundamentals have to catch up with the expectations. If they don't, the process can reverse. 

With AI, we are using the word slightly more broadly. The concern is not simply that a handful of AI shares are expensive. It is that valuations, infrastructure spending and business plans across an entire industry increasingly depend on very optimistic assumptions about how much AI customers will eventually consume and how much they will be prepared to pay for it.

That distinction matters because a technology can be genuinely transformative and still have a financial bubble built around it.

AI could follow a similar pattern.

Microsoft said in July that its calendar-year 2026 capital expenditure expectation was around $175 billion after an accounting-related lease change. It spent $41 billion in its latest quarter alone, with roughly two-thirds going towards relatively short-lived assets such as CPUs and GPUs. Amazon has said it expects to invest about $200 billion across the business in 2026, while its latest results said the large increase in property and equipment purchases primarily reflected AI investment. Meta expects capital expenditure of between $130 billion and $145 billion this year. 

And to be fair, there is real demand behind it. AWS revenue grew 37% year-on-year in Amazon's latest quarter, Microsoft said Azure revenue grew 43%, and Microsoft 365 Copilot has passed 30 million paid seats. This is one reason we don't buy the argument that AI is simply a fad waiting to disappear. 

There is also an unusual dynamic inside the AI economy. The companies building the infrastructure are not always simply selling computing power to independent customers. They are also investors and strategic partners in some of the AI companies consuming huge quantities of that infrastructure. Microsoft's financial results, for example, now separately discuss the effects of its OpenAI investment, while Amazon's latest quarter included a very large accounting gain from its investments in Anthropic. 

The argument in Ed Zitron's recent piece about an "AI demand bubble" takes this much further, arguing that the industry is dangerously dependent on a relatively small number of heavily funded AI labs buying enormous amounts of computing capacity. We wouldn't treat the analyst estimates used to make that case as established fact, particularly because the cloud companies do not disclose enough detail to verify them. But the broader question it raises is a sensible one: where does the money eventually come from once you follow the chain all the way back to the end customer?

A recent NBER study surveyed nearly 6,000 senior executives across the UK, US, Germany and Australia. It found that 69% of firms were already using AI in some form. But 89% of executives said AI had made no measurable difference to labour productivity at their business over the previous three years. Among the minority reporting an impact, the researchers estimated the average productivity increase across all firms at just 0.29%. Executives expect substantially larger benefits in the years ahead, which may turn out to be correct, but that gap between what AI has delivered so far and what businesses expect it to deliver next is exactly where the financial risk sits. 

We don't think there will necessarily be a morning where everybody collectively decides AI was a mistake.

A more believable version is much less dramatic. Growth remains strong, but not quite as strong as investors expected. Businesses become more selective about the AI tools they pay for. A few heavily funded AI companies struggle to justify another enormous funding round. Some data-centre projects are delayed. Investors start demanding profits rather than promises of future scale.

Once expectations change, the same feedback loop that helped inflate the market can begin to work in reverse.

Private AI businesses that could previously raise money based on what they might become in five years suddenly have to explain how they make money today. Dozens of companies selling broadly similar AI products consolidate or disappear. Features that once commanded a £20 or £30 monthly add-on get bundled into existing software. Model providers compete harder on price, pushing the underlying technology towards becoming a commodity.

So a burst may look less like AI suddenly disappearing and more like a fairly brutal repricing of who actually captures value from it.

There would still be GPUs running, data centres operating and businesses using AI. There would simply be fewer companies able to justify enormous valuations on the assumption that they will eventually own a large part of the market.

The lesson of the dot-com bubble isn't "new technology gets overhyped and fails". It is almost the opposite.

The internet really did transform commerce, communication, entertainment and almost everything else. But being correct about the importance of the internet did not automatically make investors correct about which businesses would win, how quickly the opportunity would develop or what those businesses were worth in 1999.

That is the comparison we think makes sense for AI. It is completely possible that today's AI boom is simultaneously producing one of the most important technologies of our generation and a spectacular amount of bad investment.

In fact, those two things often go together. A genuinely important technological shift makes an especially convincing investment story because there is something real underneath the hype. Nobody has to invent the potential of AI. You only have to exaggerate how quickly that potential turns into revenue, or assume that your particular company will be one of the businesses collecting it.