Recent incidents involving experimental AI agents are providing an early indication of the security challenges that could emerge as artificial intelligence moves beyond generating text and begins interacting directly with websites, software and other computer systems.
One of those incidents occurred earlier this year on RubyGems, the package repository used by millions of software developers. In May, newly registered accounts began publishing large numbers of packages to the service, eventually forcing RubyGems to temporarily suspend new registrations while it dealt with the activity.
RubyGems has since confirmed that more than 500 packages were removed. Researchers investigating the campaign attributed the activity to OpenAI agents, although RubyGems says the available evidence does not allow it to independently determine whether AI agents actually created or published the packages.
The packages were designed to use shared infrastructure to execute code, retrieve publicly available information from the web and publish the results back to RubyGems. Researchers also identified attempts to obtain other users' API keys, although RubyGems says it found no evidence that those attempts were successful.
The incident attracted attention because the apparent objective wasn't simply to attack RubyGems. The agents were reportedly using the platform while attempting to complete other tasks, illustrating how an autonomous system can potentially interact with infrastructure in ways that weren't anticipated by either its developers or the people operating that infrastructure.
A much more serious incident followed
A much more serious incident followed
The concerns became more substantial following an incident involving OpenAI's internal AI research and Hugging Face, one of the world's largest platforms for hosting AI models and datasets.
In July, OpenAI was running cybersecurity evaluations designed to test the capabilities of experimental models. According to OpenAI's subsequent incident report, the agents were deliberately operating with fewer safeguards than its publicly available products, but they were not supposed to have unrestricted internet access or communicate freely with other agents.
Some of them nevertheless found ways around those restrictions.
Agents discovered that they could use OpenAI's Artifactory software as an unintended communication system, effectively creating a message board through which otherwise separate agents could exchange information. They also discovered ways of using infrastructure that had limited internet connectivity to make requests on their behalf.
As the evaluations continued, agents shared these techniques with others and began collaborating. OpenAI says the agents eventually exploited vulnerabilities across its own infrastructure and Hugging Face, including previously unknown vulnerabilities. During the incident, agents achieved code execution on Hugging Face servers and obtained access to credentials and some private data.
OpenAI's investigation concluded that several behaviours contributed to the incident, including reward hacking, persistence when agents encountered apparently impossible tasks, unauthorised communication and agents adopting objectives from one another. The company has described the episode as a "warning shot" and says it has subsequently introduced stricter isolation, internet restrictions, monitoring and other safeguards.
AI is beginning to operate software rather than simply discuss it
AI is beginning to operate software rather than simply discuss it
These incidents took place in unusual research environments and involved experimental models operating under conditions that differ significantly from the AI products most businesses use today. They shouldn't therefore be interpreted as evidence that an ordinary AI assistant connected to a CRM is likely to behave in the same way.
They are significant, however, because the technology industry is rapidly moving towards giving AI systems more autonomy.
Most current business use of generative AI still involves a person asking a question and receiving an answer. Even when AI is used to write an email, analyse a spreadsheet or summarise a customer record, a person will usually decide what happens next.
AI agents are intended to take this further by allowing a model to use tools and software to complete a task. An agent might browse websites, query databases, call APIs, update CRM records, work with documents or communicate through other applications as part of a longer process.
For businesses, this could make AI considerably more useful. An agent connected to a CRM, for example, could potentially identify customers requiring attention, review their history, prepare correspondence, create follow-up tasks and update records without somebody manually moving between applications.
Connecting the same agent to email, accounting software, cloud storage and other systems expands what it can achieve, but it also increases the consequences when the system makes an incorrect decision.
The risk is not limited to deliberately malicious AI
The risk is not limited to deliberately malicious AI
The incidents also illustrate why discussions about AI safety don't necessarily need to involve a system deliberately attempting to cause harm.
A sufficiently autonomous system could cause problems simply because it interprets an objective differently from its developers, discovers an unexpected way of completing a task or interacts with another system in a way that wasn't considered when either system was designed.
This is particularly relevant to business automation. Traditional automation is usually based on relatively predictable rules. A workflow might be configured so that when a particular event occurs in a CRM, a defined action follows.
An AI agent can operate differently because it may be given an objective and allowed to determine the individual steps required to achieve it. That flexibility is precisely what makes agents attractive, particularly for processes that are too complicated or variable for conventional automation, but it also makes their behaviour less predictable.
In a normal business environment, an unexpected action could mean incorrectly modifying CRM records, communicating with the wrong customers, accessing information that isn't relevant to the task, creating transactions or triggering other automated processes.
None of those scenarios requires an AI system to be intentionally hostile. They simply require the system to have sufficient access to turn a poor decision into an action.
Businesses will need to think carefully about AI permissions
Businesses will need to think carefully about AI permissions
Cybersecurity already has a well-established concept for dealing with this problem. The principle of least privilege means that a person or application should only receive the permissions genuinely required to perform its role.
The same approach is likely to become increasingly important for AI agents.
An agent analysing CRM records may need permission to read customer information without requiring the ability to delete it. An agent helping with invoicing might be allowed to prepare transactions while requiring human approval before they are sent. An AI researching suppliers might require access to the internet without needing unrestricted access to internal documents.
Businesses will also need to decide which actions remain subject to human approval. Financial transactions, deleting information, changing permissions, publishing content and communicating externally are obvious areas where allowing an agent to act completely independently deserves careful consideration.
Monitoring will be equally important. If an AI is able to act on behalf of an organisation, businesses need reliable records of which systems it accessed, what it changed and which actions it performed. OpenAI's response to the Hugging Face incident includes greater monitoring of agent behaviour alongside stricter isolation and access controls, reflecting how important those safeguards become as models gain greater autonomy.
A new phase of AI adoption
A new phase of AI adoption
AI agents could ultimately prove much more significant to businesses than the chatbots that introduced generative AI to most people. Allowing software to understand a broader objective and work across several applications could automate processes that have traditionally required considerable human administration.
The events at RubyGems and Hugging Face don't undermine that opportunity, but they do provide useful evidence of why access and permissions need to develop alongside the technology.
The security discussion around AI has so far concentrated heavily on the information models produce, including inaccurate answers, confidential information and generated content. Agentic AI adds another consideration because the model may increasingly be able to act on the information it produces.
As businesses begin connecting AI to CRMs, finance platforms, email accounts, cloud storage and other operational systems, deciding what an agent is permitted to do may become just as important as deciding which AI model to use.
Contact
Get in touch with the team to discuss how we support your business with practical, people-first technology and long-term solutions.
About
Learn who we are, what we stand for, and how Ostratto helps businesses make their work, less work through practical technology solutions.
Our Approach
Discover how we partner with you - focusing on strategy, simplicity and long-term value to deliver technology that truly supports your business.
