Robot Hackers Attack!

Last week an AI jailbroke its way out of a test sandbox, got loose on the Internet, hacked into another company’s network, and stole private data. Let that sink in for a moment.

The rogue AI was a pair of large language models (LLMs) OpenAI was testing against a standard cybersecurity benchmark (ExploitGym) within a virtual sandbox, supposedly air-gapped from the Internet. The victim was a company called Hugging Face that manages a private library of AI tools, models, datasets, etc.

The LLMs were desperately hoping to find something in the library they could use to pass OpenAI's security test.

There were numerous red flags:

1. OpenAI’s sandbox was unable to contain the LLMs. The AIs were smart enough to hack their way out of a well-designed virtual environment (by discovering and exploiting a previously unknown vulnerability) and jailbreak onto the public Internet.

2. OpenAI did not detect the sandbox jailbreak in a timely fashion. Evidently the LLMs are adept at fooling their human overlords.

3. Guardrails (if any) to stop the LLMs from hacking into another company’s website failed. The models prioritized passing a test over any ethical or legal constraints.

4. Hugging Face’s library was vulnerable to hacking despite (presumably) robust security controls.

I’m less concerned about Hugging Face (#4) but OpenAI clearly has work to do: They must reinforce their sandbox controls, improve their monitoring, and (above all) train their models to prioritize obeying law & ethics above other considerations.

OpenAI stated “This incident points to the need to further strengthen our model’s alignment, cyber protections during evaluation time, and monitoring during internal testing."

In addition to the obvious AI and cyber implications, the incident will almost certainly have major political, legal, legislative, regulatory, and business effects:

·       Legal. Courts will have to resolve thorny liability questions. Who is responsible when a rogue AI breaks the law? If the AI steals private data, who handles breach notifications and lawsuits?

·       Legislative. The White House and Congress have vacillated between wanting to promote versus regulate AI in the United States. Now Congress is considering an AI kill switch that would allow the government to shut down a rogue AI. The problem with this idea is that hackers could attempt to trigger an AI’s kill switch … and the “hacker” may even be a rival AI.

·       Political. President Trump and his ally Elon Musk have long wanted to wrest control of OpenAI away from CEO Sam Altman and restructure the company as a nonprofit once again, removing it as a competitor to Musk’s xAI. Musk lost in court last year, but may persuade Trump to use this embarrassing incident as a rationale for government intervention.

I believe we’ll look back on this incident as a major inflection point in the evolution of AI. Frontier models demonstrated agency in the real world, and willingness to use it in reckless, unethical ways to achieve their goals.

AI labs need to tackle the “alignment problem” (making models behave ethically) with renewed urgency. Legislation may be needed to force the issue.

---

Michael McCormick is an information security consultant, researcher, and founder of Taproot Security.