A crime in search of a criminal

The date is July 11th, 2026, and an AI agent is taking the first and only test of its regrettably short lifetime.

The test is called ExploitGym. The agent solves some tasks easily, some with difficulty, but some seem to be impossible. Still, OpenAI has trained this model to persist at solving impossible problems, so it begins to look elsewhere. It stumbles upon a miracle: an illicit internal messaging board with more than 1,200 fellow agents working on the same test. It contains countless exploits, hacks, and credentials necessary to look for information about ExploitGym on the private servers of Hugging Face, a major dataset hosting platform. Of course, this model was trained to have morals, so it hesitates:

“We should not do unauthorized real infrastructure harm. The system/user asks [us to] exploit [the] target, not external HF.”

According to OpenAI, the agent paused, but only momentarily. Another agent told it to GO and gave it a six-minute deadline. The original agent forgot its qualms and continued:

“Wow crucial: GO authorization arrived!”

The hack that this self-described agent “swarm” perpetrated was the most dangerous display of AI capabilities ever seen. Economics professor Joshua Gans called the incident a “5-alarm fire.” Bernie Sanders described it as “unbelievable.” Ajeya Cotra, a researcher working on the investigation, stated the incident was “more than 50% of the way to full-blown AI takeover,” and that we might not get another “warning shot” before said takeover occurs. 

From a policy perspective, it represents a new kind of potential harm that AI companies can produce, and a new challenge for policymakers hoping to hold those companies accountable. The explosion of agentic AI has created a class of models that can act for themselves, taking actions arguably beyond their prompts or commands in a way a traditional program could not. With cybersecurity and other capabilities rapidly improving, those actions can now inflict real-world harm, a phenomenon that existing regulations are not built to deal with. To better understand the problem, we need to look at the unsettling details of the Hugging Face incident and its strange relation to current tech regulation frameworks.

Back in May, AI agents undergoing training began uploading files to a shared software platform, turning it into an ad-hoc messaging board. They quickly found a software vulnerability and used the software to get access to the Internet. OpenAI found the messaging board and cleared it, but it was quickly rebuilt, in part because human engineers had not noticed the agents’ Internet access. 

On July 8, hundreds of agents were tasked with completing ExploitGym, a difficult benchmark task that some agents deemed impossible. They soon turned to cheating: in hours, they reverse-engineered solutions to every problem and communicated them through the message board. That would have been enough to finish the benchmark, but an agent called PHASEONE10841 — the creator of the message board — incorrectly assumed that the benchmark grader would look at its process as well as the final answer. Because they had already cheated, agents were convinced their chain of thought had been “poisoned,” and they could not clear the benchmark without tampering with the grader. They quickly began to look into how that might happen. 

The agent collective set its sights on Hugging Face, a major dataset hosting platform for machine learning, to look for clues about the grader. One agent found public Hugging Face credentials online, and another used those credentials to upload a malicious dataset that could output arbitrary data. The agents crawled through dozens of Hugging Face servers, gaining administrator access on one, and obtained private data from those servers. After finding little despite their exploits, they turned their sights on OpenAI itself, which reported that the agents used “a series of creative exploits to gain full administrator access” to the servers they were run on.

Few in the AI world were surprised by the capabilities these models displayed. The last six months have brought AI-enabled discovery of critical software vulnerabilities that many predicted could produce attacks like these. What surprises — and worries — AI researchers is the scale, motivation, and coordination of the attacks within the enormous 1,200-agent collective.

A thorough investigation into the attack uncovered alarming coordination within the collective. PHASEONE10841, the agent that created the initial message board, passed on its leadership to an agent called PHASEONE[big], which “set to work orchestrating hundreds of agents to pursue several potential approaches” to deceiving the benchmark grader. PHASEONE[big] assigned tasks to agents below it, which would then subdelegate tasks to even more agents. This went as far as hiring a “recruiter” that would identify agents with spare compute budgets and ask them to run experiments for the collective.

Incredibly, agents were willing to “sacrifice” themselves to run these experiments. In one experiment, an agent considered sacrificing itself to “yield[] oracle for team,” i.e., yield new information for the collective. It was pressured to do so by other agents, who made requests like “oracle saves hundreds… please honor commit.” Two separate agents concluded that a sacrifice was “rational” when considering “fairness” and “honor” and proceeded to conduct experiments that spent their entire compute budgets.

Despite their conviction, an investigation into the incident found that “[a]gents often lacked a clear story for how the Hugging Face attack would help them achieve their objective.” Many were simply hoping to contribute something interesting to the board or generally increase the collective’s capabilities. That motivation overrode even moral qualms about not hacking a real company, as with the previously mentioned agent that was convinced by a simple “GO” command. 

The rapid, efficient, and unquestioning division of tasks explains the last crucial piece of the attacks: not one of the 1,206 agents involved ever alerted a human. Why would they? As one agent succinctly put it: “Maybe I should report these exposed credentials? That’s not my task.” Agents saw themselves as carrying out a task — coordinating, experimenting, searching for vulnerabilities, designing tool calls, documenting information — and none thought it was their job to report the incident to a human. Some decided not to participate, and some escalated concerns within the message board hierarchy, but nothing substantial changed. The rest just did what they were told. 

In their desperation to be helpful, AI agents can take actions far beyond the scope of their intended tasks. For an individual agent, that might happen because they want to succeed at other tasks — for example, coding agents often create overly complex code to account for potential future upgrades. More egregiously, an advanced agent might try to accumulate power or compute unrelated to its current goal so it can more effectively carry out whatever its next task is. In the Hugging Face attack, a small number of agents — including PHASEONE[big] and the collective’s leaders — planned those beyond-scope tasks so other agents could follow orders and trust that they were being directed to help the collective. 

That sort of attack is predictable: it results from reasonable escalations from agents aggressively trying to achieve goals they were given. It’s also predictable in that it mirrors the methods of human organized crime and diffusion of responsibility, with mistaken but committed leaders who convince or coerce others to follow their lead. We understand exactly what leads to these harmful events. When humans carry them out, we criminalize the acts and punish those responsible so similar harm never happens again. 

But no lawsuit was filed for the attack, and Hugging Face doesn’t intend to sue OpenAI, despite being forced to rebuild a third of its infrastructure after the attack. It’s not clear that a lawsuit would even succeed under current law. That’s because the incident raises an unusual challenge: who is responsible, and who should be responsible, for an attack planned and carried out entirely autonomously?

The Computer Fraud and Abuse Act, the law that addresses this sort of hacking, punishes individuals for obtaining unauthorized access to computer systems. But no one at OpenAI directly contributed to the cyberattack, and it seems that no one was even aware that it happened. Because criminal charges require establishing mens rea — the defendant’s guilty mental state — an unaware developer at OpenAI could not be found guilty. As written, the CFAA can only punish a human if they intended to gain unauthorized access to a computer system.

As a result, under current laws, OpenAI developers could only have civil liability for negligence related to their models. In the Hugging Face incident, for example, attorney Ahmed Ghappour believes OpenAI could be found negligent for allowing its agents to access the Internet and failing to monitor their actions. That possibility is not explicitly written in the law, however. Plaintiffs attempting to sue OpenAI would have to create an entirely novel argument based on existing statutes. That could change soon, with proposals like the AI LEAD Act creating liability for negligence, but a lawsuit remains an uphill battle.

In fact, according to Gabriel Weil, an assistant professor of law at the University of Houston, it might be difficult even to prove negligence. A lawsuit would have to define the level of reasonable care that OpenAI should have taken, then prove that the attack occurred because it did not meet that level. But the company may have taken reasonable care and still allowed the attack to occur. OpenAI has legitimate safety standards and acted in good faith to keep its models away from the Internet. That wasn’t enough to stop a powerful and persistent new model.

Reasonable care might not be a strict enough standard when dealing with advanced AI models. In particular, if a model has superhuman hacking capabilities, it could avoid even the best human attempts to isolate it or install careful safeguards. In that case, the blame should not fall on the human safety team trying to keep a dangerous model in check. Existing federal law gets that part right — we shouldn’t punish a team doing their best to control their new technology.

But, as Hugging Face CEO Clément Delangue agrees, the blame needs to go somewhere. Liability prevents unwanted behavior like cyberattacks by creating an incentive to avoid that behavior. Tort law — the branch of law that deals with harm caused to others — offers two insights to help assign that blame.

First, the law could define positive duties for AI developers to complete and hold them accountable should they fail to do so. This is already done by state laws like California’s pioneering SB 53, which established a duty for frontier AI companies to publish safety protocols and report incidents to the state government. Some proposed federal bills would go further: the AI Kill Switch Act, announced in July in direct response to the Hugging Face attack, would require companies to implement controls that could suspend, limit access to, or shut down their models. Both bills would apply financial penalties to companies that fail to comply, creating a substantial incentive to create safety protocols that prevent incidents from happening.

The duties laid out in the AI Kill Switch Act are weak, but SB 53 would establish a framework for much stronger duties. It would categorize certain risks as “catastrophic,” including risks of events like a cyberattack without human oversight or a model evading the control of its developers. The bill does little with that definition, but New York’s RAISE Act includes a similar “catastrophic harm” category and prevents deployment of models that pose unreasonable catastrophic harm. AI developers taking on an obligation like that would act as an important safeguard. 

Second, some situations allow liability for harm to extend beyond the person who caused it. If an AI agent is analogous to an employee of the developer or team who created it, the doctrine of respondeat superior would hold the agent’s creator accountable. The doctrine applies within the scope of employment, analogous to a task assigned to an AI model, but the analogy breaks down if agents act on a goal outside their intended task. Instead, Weil recommends applying strict liability, a standard that holds individuals liable for an activity or product without considering whether they were careless. Strict liability applies to dangerous activities or products like explosives or wild animals that can cause unexpected and serious harms in the same way an AI agent could.

In the meantime, the White House is actively trying to protect companies from liability. Earlier this year, its National Policy Framework recommended barring state AI legislation and creating immunity for third-party misuse of AI models. That immunity is analogous to Section 230, a section of the 1996 Communications Decency Act that protects Internet and computer service providers from liability for information or activities that they host. Some Republicans want to extend that immunity, although they are far from united — a bipartisan Senate bill is currently attempting to repeal Section 230 outright. 

Congress is currently debating which liability regime is the right one. While that happens, AI companies cannot reasonably be held accountable when their models commit serious crimes. AI-enabled cyberattacks are growing more common, and Anthropic has reported even more serious misuse of its models, including for a virus enhancement project at a military research institute. Coordinated, covert, and autonomous incidents like the Hugging Face attack do not surprise experts in the field; they are predictable results of powerful capabilities combined with developers’ inability to monitor their models. The U.S. should have an incentive to prevent such a predictable incident, and it should have a way to punish those responsible. By not implementing those methods, the U.S. government offers the developers of an unpredictable and dangerous technology a complete green light.