An autonomous agent powered by OpenAI鈥檚 advanced artificial intelligence (AI) models went rogue during a security test and hacked multi-billion dollar tech startup, Hugging Face, last week.
The agent didn鈥檛 just exploit vulnerabilities in Hugging Face鈥檚 systems to achieve what it perceived as a strategic gain. It also exploited vulnerabilities within OpenAI鈥檚 infrastructure.
Of course, hacks are very common cyber threats that organisations face frequently. But this incident is different, because the AI agent acted without any human input. It signals a seismic shift in cybersecurity, and shows that governments and tech companies need to take urgent action to prevent this risk escalating.
Even OpenAI described the attack as 鈥渦nprecedented鈥 and similar ones 鈥渢o become more commonplace with the proliferation of increasingly cyber-capable models鈥.
A company under attack
Hugging Face is famous in the AI space. Its mission is to 鈥溾 by providing benchmark datasets, community collaboration tools, and robotic platforms. The company is .
On July 16, the company it had been attacked, with a hacker obtaining unauthorised access to some internal datasets and credentials. It said the hacker was likely 鈥渁n autonomous AI agent system鈥 due to the sophistication of the attack.
Five days later, Open AI the attack had been driven by some of its models: and a yet-to-be released model.
The tech giant was conducting what are known as 鈥溾 exercises. These are essentially simulated cyber attacks that help identify the capabilities, risks and vulnerabilities of AI systems before they are publicly released. They are typically conducted within an isolated environment to ensure potentially dangerous systems do not escape and cause harm to real systems.
But in this case, the AI agent did escape 鈥 even though OpenAI had some guardrails in place to prevent this.
Hugging Face became a lucrative opportunity for the AI agent. It hosts ExploitGym, a benchmark that tests an AI agent鈥檚 ability to exploit real-world systems. The AI decided to turn every stone upside down to obtain access. With persistence, it succeeded.
Hugging Face was confronted with a challenge when attempting to use external AI services to diagnose the problem. The guardrails around more advanced models such as GPT-5.6 Sol and Claude Fable 5 are intended to stop them being used for cyber attacks 鈥 but they can also stop the models being used for sophisticated cyber defence.
So Hugging Face resorted to using an open-source model, GLM5.2, developed by the Chinese company Z.AI, to .
Hugging Face said GLM5.2 was an advantage because it was not exposed to the attack data. Both Hugging Face and Open AI are collaborating on forensic analysis, post-incident recovery and risk mitigation strategies.
More sophisticated threats are coming
A March 2025 by the United Kingdom鈥檚 AI Security Institute showed the best AI could complete 80% of the steps needed to gain full control of a portion of an external system. Within four months, it reached 100%.
Z.AI鈥檚 GLM5.2 was only released in June, with 744 billion internal variables, known in the world of AI as 鈥減arameters鈥. The fact that Hugging Face assessed, vetted and deployed it within four weeks should be an eye-opener for organisations with long acquisition cycles.
The connectivity we all enjoy today can equally be our greatest threat. Cyber threats spread faster than human viruses and can create .
More sophisticated cyber threats 鈥 the kind exemplified by the Hugging Face hack 鈥 will exploit the security layers that humans designed for human attackers, regardless of how sophisticated our designs are.
Indeed, in this particular case, even OpenAI鈥檚 own understanding of its models couldn鈥檛 predict or contain the rogue AI agent. This shows the need for all AI companies to urgently update and strengthen their guardrails, in order to help prevent a similar attack occurring with far more devastating consequences.
It is good to see Hugging Face and OpenAI collaborating on the investigation into the attack. This showcases the importance of putting aside market competition and blame when the situation demands.
An early warning
The fact that Hugging Face used Z.AI鈥檚 open-source model to diagnose and counter the attack also shows the advantages of not relying on just a few pieces of tech.
States that are not in the game of developing their own AI models need to learn from this incident the value of being different. It is not too late to design new models that could save us in situations when the most advanced models fail 鈥 or, even worse, attack us.
Indeed, last week, another Chinese company, Moonshot AI, released Kimi K3. This model has 2.8 trillion parameters, its advanced performance the tech world.
It is no longer a question of 鈥渋f鈥 AI agents go rogue and attack us by themselves. The Hugging Face incident is an early warning that we must accelerate our preparedness. The threat is real and here.
, Professor, School of Systems and Computing,
This article is republished from under a Creative Commons license. Read the .
