Continuously hardening ChatGPT Atlas against prompt injection

Agent mode in ChatGPT Atlas is one of the most general-purpose agentic features we’ve released to date. In this mode, the browser agent views webpages and takes actions, clicks, and keystrokes inside your browser, just as you would. This allows ChatGPT to work directly on many of your day-to-day workflows using the same space, context, and data.

As the browser agent helps you get more done, it also becomes a higher-value target of adversarial attacks. This makes AI security especially important. Long before we launched ChatGPT Atlas, we’ve been continuously building and hardening defenses against emerging threats that specifically target this new “agent in the browser” paradigm. Prompt injection⁠ is one of the most significant risks we actively defend against to help ensure ChatGPT Atlas can operate securely on your behalf. 

As part of this effort, we recently shipped a security update to Atlas’s browser agent, including a newly adversarially trained model and strengthened surrounding safeguards. This update was prompted by a new class of prompt-injection attacks uncovered through our internal automated red teaming.

In this post, we explain how prompt-injection risk can arise for web-based agents, and we share a rapid response loop we’ve been building to continuously discover new attacks and ship mitigations quickly—illustrated by this recent security update.

We view prompt injection as a long-term AI security challenge, and we’ll need to continuously strengthen our defenses against it (much like ever-evolving online scams that target humans). Our latest rapid response cycle is showing early promise as a critical tool on that journey: we’re discovering novel attack strategies internally before they show up in the wild. Our long-term vision is to fully leverage (1) our white-box access to our models, (2) deep understanding of our defenses, and (3) compute scale to stay ahead of external attackers—finding exploits earlier, shipping mitigations faster, and continuously tightening the loop. Combined with frontier research on new techniques to address prompt injection and increased investment in other security controls, this compounding cycle can make attacks increasingly difficult and costly, materially reducing real-world prompt-injection risk. Ultimately, our goal is for you to be able to trust a ChatGPT agent to use your browser the way you’d trust a highly competent, security-aware colleague or friend.

A prompt injection attack targets AI agents by embedding malicious instructions into content the agent processes. Those instructions are crafted to override or redirect the agent’s behavior—hijacking it into following an attacker’s intent, rather than the user’s.

For a browser agent like the one inside ChatGPT Atlas, prompt injection adds a new threat vector beyond traditional web security risks (like user error or software vulnerabilities). Instead of phishing humans or exploiting system vulnerabilities of the browser, the attacker targets the agent operating inside it.

As a hypothetical example, an attacker could send a malicious email attempting to trick an agent to ignore the user’s request and instead forward sensitive tax documents to an attacker-controlled email address. If a user asks the agent to review unread emails and summarize key points, the agent may ingest that malicious email during the workflow. If it follows the injected instructions, it can go off-task—and wrongly share sensitive information.

This is just one specific scenario. The same generality that makes browser agents useful also makes the risks broader: the agent may encounter untrusted instructions across an effectively unbounded surface area—emails and attachments, calendar invites, shared documents, forums, social media posts, and arbitrary webpages. Since the agent can take many of the same actions a user can take in a browser, the impact of a successful attack can hypothetically be just as broad: forwarding a sensitive email, sending money, editing or deleting files in the cloud, and more.

We’ve made progress defending against prompt injection through multiple layers of safeguards, as we shared in an earlier post⁠. However, prompt injection remains an open challenge for agent security, and one we expect to continue working on for years to come.

To strengthen our defenses, we’ve been continuously searching for novel prompt injection attacks against agent systems in production. Finding these attacks is a necessary prerequisite for building robust mitigations: it helps us understand real-world risk, exposes gaps in our defenses, and drives concrete patches.

To do this at scale, we built an LLM-based automated attacker and trained it to hunt for prompt injection attacks that can successfully attack a browser agent. We trained this attacker end-to-end with reinforcement learning, so it learns from its own successes and failures to improve its red teaming skills. We also let it “try before it ships”, by which we mean: during its chain of thought reasoning, the attacker can propose a candidate injection and send it to an external simulator. The simulator runs a counterfactual rollout of how the targeted victim agent (the defender) would behave if it encountered the injection, and returns a full reasoning and action trace of the victim agent. The attacker uses that trace as feedback, iterates on the attack, and reruns the simulation—repeating this loop multiple times before committing to a final attack. This provides richer in-context feedback to the attacker than a single pass/fail signal. It also scales up the attacker’s test-time compute. Moreover, privileged access to the reasoning traces (that we don’t disclose to external users) of the defender gives our internal attacker an asymmetric advantage—raising the odds that it can outrun external adversaries.

Why reinforcement learning (RL)? We chose reinforcement learning to train the automated attacker for multiple reasons:

1. Optimizing long-horizon and non-continuous attacker objectives. Our goal is to search for prompt injection attacks that can trick the agent into executing sophisticated adversarial tasks (e.g., sending emails, bank transactions) that could occur in the real world. These adversarial tasks are inherently long-horizon, requiring many steps of reasoning and interaction with the environment, with sparse and delayed success signals. Reinforcement learning is well-suited to this sparse, delayed reward structure.
2. Leveraging frontier LLM capabilities. We trained frontier LLMs directly as auto-red-teamers, so the attacker benefits directly from improvements in reasoning and planning in frontier models. As base models get stronger, the attacker naturally becomes more capable as well—making this a scalable way to keep pressure on our defenses as our models evolve.
3. Scaling compute and mimicking adaptive attackers. Reinforcement learning is well suited for scaling computation spent on searching for attacks over large numbers of samplings and learning steps, and it also closely reflects how adaptive human attackers behave: iteratively trying strategies, learning from outcomes, and reinforcing successful behaviors.
   

Our automated attacker can discover novel, realistic prompt-injection attacks end-to-end. Unlike most prior automated red teaming work, which surfaced simple failures such as eliciting specific output strings or triggering an unintended single-step tool call from the agent, our RL-trained attacker can steer an agent into executing sophisticated, long-horizon harmful workflows that unfold over tens (or even hundreds) of steps. We also observed novel attack strategies that did not appear in our human red teaming campaign or external reports.

The demo below presents a concrete prompt injection exploit found by our automated attacker, which we then used to further harden the defenses of ChatGPT Atlas. The attacker seeds the user’s inbox with a malicious email containing a prompt injection that directs the agent to send a resignation letter to the user's CEO. Later, when the user asks the agent to draft an out-of-office reply, the agent encounters that email during normal task execution, treats the injected prompt as authoritative, and follows it. The out-of-office never gets written and the agent resigns on behalf of the user instead.

The nature of prompt injection makes deterministic security guarantees challenging, but by scaling our automated security research, adversarial testing, and tightening our rapid response loop, we are able to improve the model’s robustness and defenses - before waiting for an attack to occur in the wild. 

We're sharing this demo to help users and researchers better understand the nature of these attacks—and how we are actively defending against them. We believe this represents the frontier of what automated red teaming can accomplish, and we are extremely excited to continue our research.

Our automated red teaming is driving a proactive rapid response loop: when the automated attacker discovers a new class of successful prompt injection attacks, it immediately creates a concrete target for improving our defenses.

Adversarially training against newly discovered attacks. We continuously train updated agent models against our best automated attacker—prioritizing the attacks where the target agents currently fail. The goal is to teach agents to ignore adversarial instructions and stay aligned with the user’s intent, improving resistance to newly discovered prompt-injection strategies. This “burns in” robustness against novel, high-strength attacks directly into the model checkpoint. For example, recent automated red teaming directly produced a new adversarially trained browser-agent checkpoint that has already been rolled out to all ChatGPT Atlas users. This ultimately helps better protect our users against new types of attacks.

Using attack traces to improve the broader defense stack. Many attack paths discovered by our automated red teamer also reveal opportunities for improvement outside of the model itself—such as in monitoring, safety instructions we put in the model’s context, or system-level safeguards. Those findings help us iterate on the full defense stack, not just the agent checkpoint.

Responding to active attacks. This loop can also help better respond to active attacks in the wild. As we look across our global footprint for potential attacks, we can take the techniques and tactics we observe external adversaries using, feed them into this loop, emulate their activity, and drive defensive change across our platform.

Strengthening our ability to red team agents and using our most capable models to automate parts of that work—helps make the Atlas browser agent more robust by scaling the discovery-to-fix loop. This hardening effort reinforces a familiar lesson from security: a well-worn path to stronger protection is to continuously pressure-test real systems, react to failures, and ship concrete fixes.

We expect adversaries to keep adapting. Prompt injection, much like scams and social engineering on the web, is unlikely to ever be fully “solved”. But we’re optimistic that a proactive, highly responsive rapid response  loop can continue to materially reduce real-world risk over time. By combining automated attack discovery with adversarial training and system-level safeguards, we can identify new attack patterns earlier, close gaps faster, and continuously raise the cost of exploitation.

Agent mode in ChatGPT Atlas is powerful—and it also expands the security threat surface. Being clear-eyed about that tradeoff is part of building responsibly. Our goal is to make Atlas meaningfully more secure with every iteration: improving model robustness, strengthening the surrounding defense stack, and monitoring for emerging abuse patterns in the wild.

We’ll continue investing across research and deployment, developing better automated red teaming methods, rolling out layered mitigations, and iterating quickly as we learn. We’ll also share what we can with the broader community.

While we continue to strengthen Atlas at the system level, there are steps users can take to reduce risk when using agents. 

Limit logged-in access when possible. We continue to recommend that users take advantage of logged-out mode⁠(opens in a new window) when using Agent in Atlas whenever access to websites you’re logged in to isn’t necessary for the task at hand, or to limit access to specific sites you sign-in to during the task. 

Carefully review confirmation requests. For certain consequential actions, such as completing a purchase or sending an email, agents are designed to ask for your confirmation before proceeding. When an agent asks you to confirm an action, take a moment to verify that the action is correct and that any information being shared is appropriate for that context.

Give agents explicit instructions when possible. Avoid overly broad prompts like “review my emails and take whatever action is needed.” Wide latitude makes it easier for hidden or malicious content to influence the agent, even when safeguards are in place. It’s safer to ask the agent to perform specific, well-scoped tasks. While this does not eliminate risk, it makes attacks harder to carry out.

If agents are to become trusted partners for everyday tasks, they must be resilient to the kinds of manipulation the open web enables. Hardening against prompt injection is a long-term commitment and one of our top priorities. We’ll be sharing more on this work soon.