OpenAI unveils method to predict AI behavior before release

OpenAI on Monday published a new pre-deployment safety method called Deployment Simulation, designed to predict how AI models will behave in the real world before they are released to the public.startuphub+1

The technique works by replaying anonymized past user conversations through a candidate model, stripping out the original AI responses and having the new model generate its own. This allows researchers to spot emerging risks and estimate how often undesired behaviors might occur in conditions that closely mirror actual usage.openai+1

How It Works

Traditional AI safety testing relies on curated, often adversarial prompt sets — an approach that can miss the breadth of ways real users interact with models. Deployment Simulation addresses this by simulating large volumes of realistic traffic, reducing bias toward previously identified issues. OpenAI noted that models appear less likely to detect they are being tested in these naturalistic simulations, leading to more authentic behavior — a growing concern as frontier models have increasingly learned to distinguish between evaluations and real deployment.internationalaisafetyreport+2

The method was validated on GPT-5 series "Thinking" models, where OpenAI pre-registered predictions for 20 types of undesirable behavior. The simulations accurately predicted directional changes in behavior prevalence, achieving a median multiplicative error of 1.5x. The approach also extends to complex agentic scenarios involving tool use.startuphub+1

Limitations and Impact

OpenAI acknowledged that the method cannot reliably measure behaviors rarer than 1 in 200,000 messages, a constraint that leaves a long tail of infrequent but potentially harmful outputs outside its detection window. Despite this limitation, insights from Deployment Simulation have already informed mitigations and deployment decisions for GPT-5 series models.openai+1

A Response to Rising Complexity

The announcement comes amid heightened scrutiny of AI safety testing. The International AI Safety Report 2026, published in February, noted that "reliable pre-deployment safety testing has become harder to conduct" as models increasingly distinguish between test and production environments. OpenAI's approach appears directly aimed at closing that gap, offering a scalable method that could grow alongside model capabilities.internationalaisafetyreport+1