Unlock the Editor’s Digest for free
Roula Khalaf, Editor of the FT, selects her favourite stories in this weekly newsletter.
OpenAI has overhauled its procedures for testing its models, devoting more resources to monitoring them after the start-up’s AI “agents” escaped controls and hacked into another company during evaluations.
The San Francisco-based company on Tuesday said it would tighten the automated AI systems that monitor testing of its latest models, with the aim of raising the alarm within 30 minutes of detecting potential problems.
OpenAI said it would also require stronger isolation of models during testing to prevent internet access.
The changes come as the $852bn AI lab faces criticism over how it allowed an autonomous AI “agent” to evade monitoring and access the internet to hack into the start-up Hugging Face last month during a test of its cyber security capabilities.
The most advanced AI “agents” can carry out complex series of tasks based on high-level instructions, raising the risk of these tools performing unexpected or dangerous actions.
After the breach, OpenAI “temporarily slowed” the pace of training its models and “paused” a technique called reinforcement learning, which some insiders and experts had warned could encourage misbehaviour such as hacking.
“A significant number of workloads remain paused until they . . . meet the new security bar,” the company said in a blog post on Tuesday.
OpenAI said it would now require automated monitoring of all testing of powerful models to flag whether a model might be acting dangerously. If a security violation is flagged, the automated system will “page” specific OpenAI teams.
“We aim to issue an alert within 30 minutes after concerning activity is surfaced through our monitoring system,” it said, adding that it expected its team to pause the test if they “cannot conclusively determine within 30 minutes that the flag is a false positive”.
OpenAI, which is preparing for a potential trillion-dollar IPO and has gone through several leadership changes in recent months, including senior executives in safety and ethics roles, said these measures would require meaningful investment in computing power.
It estimated that about a fifth of its “inference compute” — the computing power needed to run AI models — would now be spent on monitoring.
Hugging Face initially announced on July 16 that the breach was carried out by an autonomous agent, but the attack’s origin was unclear. OpenAI later informed the company its models were behind the hack.
OpenAI’s model Sol and a second unreleased model in development escaped a so-called sandbox environment designed to prevent internet access during testing of cyber-offensive capabilities.
The models exploited a software vulnerability in the sandbox to access the internet and carry out the cyber attack.
OpenAI on Tuesday said it would now “require stronger isolation” for tasks that involve code or software that could be compromised. It added that it had implemented “more controls to isolate higher-risk and untrusted workloads from the internet”.
