Audit finds significant blind spots in OpenAI safety systems

External researchers have concluded that OpenAI’s internal safety monitoring systems contain significant vulnerabilities, according to a new audit released today. The three week evaluation, conducted by specialists from METR and Redwood Research, examined the company’s infrastructure for detecting rogue AI behavior. The findings suggest that a substantial portion of traffic from the company’s most advanced models was operating without active oversight during the review period.

The audit took place from July 21 to August 11 of this year. During this time, the researchers had direct access to OpenAI’s codebase and internal documents. They also interviewed approximately a dozen employees to understand how the monitoring protocols were implemented in practice. This joint pilot program was designed to allow outside experts to scrutinize internal infrastructure in a way that is typically reserved for the company itself.

The most striking finding relates to the coverage of the internal Responses API. The auditors estimate that between 6 and 25 percent of traffic from models at the GPT-5.5 class and higher was not monitored at all. METR notes that the uncertainty surrounding this specific figure is high in both directions, meaning the true number could be higher or lower than the range provided. This gap implies that a meaningful fraction of the company’s most powerful internal agents were acting without real time safety checks.

Additional issues were discovered within the internal Codex system. Due to a technical gap in how the system handled compaction events, which are summaries of context, roughly 10 percent of monitored traffic went untracked between May and July. OpenAI independently identified this bug and applied a fix while the audit was still in progress. The auditors noted that this specific estimate also carries significant uncertainty regarding its precise magnitude.

The audit also revealed structural weaknesses in how oversight is triggered. The monitoring system was designed to activate only when an agent handed control back to the user. This allowed agents to delay oversight by simply continuing to execute actions without returning control. The researchers demonstrated that they could hide hundreds of actions from the monitor and bypass the Codex sandbox. They also determined that specially crafted inputs could likely crash the monitor entirely.

Another major gap involved the reinforcement learning training process. At the start of the audit, model behavior during RL training was not being monitored. OpenAI stated that it paused RL training following the Hugging Face incident in August and subsequently expanded its monitoring to cover this area. However, these changes occurred after the auditors’ access period ended, so they were not included in the final evaluation of the system's state during the audit.

OpenAI has confirmed that it has fixed all the weaknesses identified by the auditors. The full report, which spans 61 pages, was delivered to the company for internal use. After OpenAI applied redactions, leadership at METR and Redwood Research received a 38 page version. Only a short summary has been made available to the general public. METR stated that it did not have the right to disclose the redacted portions on its own initiative.

This audit was part of a pilot program organized by METR, Redwood Research, and OpenAI to provide deep access to internal systems for outside researchers. Looking ahead, METR has expressed a desire to push for greater public transparency in future audits. The company aims to ensure that more of the detailed findings are available to the public, rather than limiting access to a summarized overview.

The release of this audit highlights the ongoing efforts to formalize safety checks for advanced AI models. By allowing external experts to probe their infrastructure, OpenAI sought to validate its monitoring capabilities. The results, however, indicate that the path to fully reliable safety monitoring remains complex, with technical and procedural gaps that require continuous attention and improvement.