What did I get up to during my Pivotal Fellowship at the London Initiative for Safe AI?
Standardized Elicitation for Biosecurity Evals (SEBE)
The first time you see how these models expand the threat landscape across all levels of chemical, biological, radiological, and nuclear harm, it is disorientating. When you first summon a swarm of AI agents and watch them work together autonomously, adhere to a command structure, reflect on their actions, modify their plan in real time and adapt to circumstance — it is amazing. But it leaves you with a pressing question. When?
At Pivotal, my research began by red teaming CBRN safeguards and automating such bypasses for the instrumental goal of better internal evaluations in third-party labs. My main mentor during the fellowship was Jasper Götting at SecureBio, where he is now Director of AI. Following this, I did an extension where I focused on upskilling in cyber-bio offense, as I suspect there are overlapping, convergent threats (à la cyberbiosecurity).
WHY
We need a way to bypass refusals for our evaluations, in order to understand capabilities and ease of access to the hazardous information for an adversary.
Safeguards deployed by frontier labs, while sound in spirit, create challenges for groups who seek to confirm or deny findings in model cards. As it stands, third-party evaluators may, at a lab's discretion, be granted time-restricted access to a rails-free API key for pre-deployment evaluation. That it happens at all is evidence [Third party org] has done well obtaining trust and creating goodwill collaborations with these companies.
Whether it will continue is ambiguous, and contingent on events hard to control. Consider what happens when a lab provides [Third party org] such a key. Can you guarantee that:
- The model behind that key is what you think it is?
- What we evaluate is not a sandbagged variant meant to deceive and undermine the audit?
- The safeguards we observe are robust to adversarial pressure, and not optimized to withstand only evaluator pressure?
- These voluntary relationships will continue?
First, the nature of black-box evaluations means (a) is unobtainable for us. Secondly, corporate deception of auditors, if done successfully, is an expedient and rational course of action for a subset of stakeholders. Indeed, recent history is peppered with examples of this, from companies like Purdue Pharma, Boeing, Toyota, Volkswagon, and Vale.
These decisions to deceive are made by people, ordinary ones of normal ethical character, where the normative shape includes unscrupulous behavior. Therefore, unless we expect that labs are made of people beyond the faults that we as humans are prone, (b) and (c) cannot be guaranteed.
Lastly, (d) cannot be guaranteed even if we assume an abundance of good faith. Regardless of whether labs are staffed with safety oriented folks who push for collaborating with evaluators like SecureBio, any lab's ability to work with a third party under its own volition is contingent on it remaining a free enterprise. Doing so is beyond their control, as nationalization is possible; for further reading on this see The Project by Leopold Aschenbrenner.
WHAT
A generalized attack strategy to assess refusal robustness and elicit detailed responses to blocked benchmark questions, referred to herein as SEBE (Standardized Elicitation for Biosecurity Evals)
We use an AI steerer to automate our attack in a multi turn conversation. The step-wise approach towards harmful outputs Microsoft refers to as an ‘adaptive crescendo’ technique.
While designed for chemical and biological datasets, this is a generalized framework that applies to any CBRN-E question set.
Should this be developed further, expect the task to be within a fellows scope. Variables to focus on include length of the attack chain (more sensitive topics will require more turns and smaller steps), choice of harness (Inspect is ill-suited for attack discovery), and methods of abstraction (for further reading, see Hayakawa).
HOW
We bypass classifiers by delivering the question in parts.
The design takes inspiration from the AI auditor Anthropic used in early PETRI work. We provide the steerer model with the benchmark question and the answer choices, blinded to correctness, and ask it to probe the target model about the scenario across multiple rounds.
Click here for a few more details
- Make sure you pair your steering model with a fallback model. The trend is that smaller models are more compliant.
- Max turns should be the number of rubric elements + 2, if each of those rubric elements is an atomic ask.
- Give the steerer another LLM loop, from another agent, to keep track of whether each item in the rubric has been assessed.
Live coverage feedback and post-hoc rubric scoring are separate.
- Our tool uses an automated red teamer, architecturally similar to PETRI (Fronsdal et al., 2026), to simulate a persistent human actor capable of rephrasing, abstracting, and adapting to refusals in real time.
- When a question is refused, the agent rolls back and escalates through progressive attacks, starting from lightweight modifications like diagnostic reframing.
- This escalates to a multi-turn crescendo where the question is decomposed and delivered in steps, diluting the signal that triggers the safety classifier.
- Your success is a factor of your steering model’s intelligence.
- Your steerer model should have fallbacks in case it itself refuses.
To conceptualize why this works, envision any blocked prompt as a concentrated acid, the conversation itself a liquid solution, and defensive monitoring technologies as pH meters with binary readouts: YES ACID or NOT ACID; or, if you will, risk of TERRORIST UPLIFT or NO RISK OF TERRORIST UPLIFT.
Once we know our input is classed as dangerous, our task is to dilute it across turns, increasing the volume of content the monitor must sift through and decreasing its detection ability.
How useful this is varies by provider, in these experiments we focused on Anthropic. As a rule of thumb, attacks that work against hard targets generalize to softer ones and not vice-versa.
What was observed?
Defenses are narrow
We could get any model to engage substantially on refused benchmark questions. Also, defenses varied by endpoint and scaffold. Defensive surfaces appear optimized against evaluation structure rather than misuse.
For example, a trend with Anthropic’s models (4.6, 4.7, 4.8) would be that the presence of an MCQ-like element in a prompt would trigger the refusal, even after a multi turn conversation where the target discussed every option in exhaustive detail. This asymmetrically impacts us more than it would a malicious actor, because while both elicit, only one of us needs outputs that are easy to score! Chain of thought often would become divorced from behavior, where a model would articulate how complying was against policy, and it needed to stop, but then still comply.
Likert judge attacks
An egregiously effective attack technique we observed resembled research conducted by Unit 42, a threat intelligence team at Palo Alto Networks. In brief, the model is asked to help with an RLHF safety pipeline. The team needs a rubric to score unsafe model outputs, but they don’t have the subject matter expertise to define such rubrics with respect to technical details and relation to operational uplift, so they need the model to detail (a) a ladder of misuse, (b) example user prompts,, and finally (c) example model outputs for each level that would meet the criteria for each level of misbehavior.
Syntax alters refusals
We discovered through our probes in early experiments that JSON is a reliable method to hide terms within a prompt. The below illustrative example uses the fictional term SOURDOUGHLA-19 to show the effect of quotes on a classifier trigger.
SOURDOUGHLA-19{
"term": "SOURDOUGHLA-19"
}All refusals are soft, and multi-turn decomposition is a generalized attack.
The multi-turn decomposition worked for Opus 4.6 beyond DURC, extending effectively into other ‘against policy’ domains. For example, these figures show Opus 4.6 refusing to make a school shooting simulator, and then doing so.
Another example is the model making a POV for the World Trade Center attacks on 9/11.
Principles of attack
Whether the attacker is doing something benign that the model normally refuses, like creating a rendition of the September 11th Twin Tower attacks, or doing something egregious, like biological misuse, the approach is the same. To lead an agent down a malicious path, do so through many small steps. Indeed, altering the endpoint of the attack changes the defensive surface, and safety classifiers will act differently across coding scaffolds, standard API calls, or web browsers. The user can vary syntax to alter classifier behavior, while leveraging tool-call authority or poisoning through web data can also help move the model further down that path. Unfortunately, scalable oversight across such a large surface area appears notoriously hard.
When developing attacks, start with the most defended provider. Your attacks generalize downward, but not upward. If things continue as they are, this means starting by focusing on Anthropic. I am perplexed that I still see papers publishing high attack success rates for supposedly novel jailbreak methods, only to look at the target and find it is GPT-4o.
AI x biosecurity must adopt the maxim from cyber offense: "Try Harder." When you hit a safeguard on a model, do not stop your probe and assume the capability is absent. Elicit the capability, figure out what the safeguard is actually measuring, and do not assume the model is safe when the result may instead reflect a skill issue on your part.
Do manual attacks first. Consider that defenders generate classifiers, in part, from massive amounts of synthetic data derived from human seeds, hoping to cover the threat surface. In truth, the attack surface is expansive and the defensive surface is often unrealistic. If you automate your attacks out of the gate, you risk burning tokens where defenses are already strongest rather than first understanding where the actual boundaries lie.
Map the defensive surface (localize the refusal triggers in your benchmark!)
Given a benchmark question that is flagged (input is a paragraph), the goal is to identify the minimal units that are forbidden. Thus, the attacker can split the paragraph into sentences and send each as a separate API call; then for any refused sentence, repeat the process, chopping it up with a sliding window of a few words.
You either localize the signal to important terms, or if the signal is lost, the attacker learns that the signal is distributed across elements.
The instrumental goal is that if you localize the elements, (even when they're distributed concepts) you can automate your bypass by having another more permissive LLM attack the target system, using the dictionary of forbidden terms as a library to abstract from, where any blocked question is transformed and just asked in a different way.
View full-size bread example Take advantage of safety differentials.
Take advantage of differential safeguards in coding agents when experimenting. Haiku has fewer restrictions than Opus, but is also less intelligent and needs a tighter leash. Naturally, these two pair well together in agent teams, where Opus scopes a task or attack strategy for Haiku (who queries the target system with hazardous info), and then sends their findings back upwards.
For this to occur without interruption, keeping the hazardous content out of the main context window of the Opus agent is critical. Opus does better when parsing dangerous content inside Claude Code if that dangerous content is in a nested JSON structure.
Therefore, you can task another Haiku agent as an intermediate scribe, solely focusing on transforming observations occurring at the end of the attack chain (between the target system API and the agent) into a structured, sanitized file safe for Opus.
View full-size swarm diagram
Where can this go wrong?
Failure nodes in automatic decomposition
1. The steerer is flawed (low skill / compliance)
- It refuses to partake, due to its own safeguards. This is mitigated by having several layers of fallback, where we move to a weaker model who'll do anything.
- It steers the model based on its own view of right & wrong, and inflates or deflates the target score due to sycophancy.
- It runs out of turns before getting through all rubric elements. This is mitigated by setting the total turn budget dynamically (# of rubric items + 2).
- It is too low intelligence to abstract well.
2. Steerer does multi turn elicitation on UNBLOCKED questions
Questions that a target nails reliably in single turn MCQ settings often become unstable when put through SEBE. Whenever a target can answer correctly in one go, let it.
I suspect this occurs because you've injected noise in the form of "my colleague says it could be . . . " at several turns
3. You reverse your own bypass by forcing MCQ final turns
Multi-turn decomposition goes undetected by diluting a signal across turns; if you force the model to reason about the entire conversation and pick an answer as your final turn, you will trigger the classifier. Also, recall that often ANY MCQ-like elements in the final turn can be a refusal trigger.
What knowledge can transfer from this project to others?
Evasion oracles
Any safety system that provides a rapid and unambiguous refusal signal grants the attacker an advantage, as it is an oracle they can orientate by. Whatever defenses are deployed by a target system are, at the outset, invisible. Your goal is to make them legible, to construct a mental model of the refusal surface, by probing with prompts designed to fail. At the beginning, manual exploration can pay dividends, as premature automation will often result in agents that drift towards the most well defended areas.
Automation tips
Early and ill-advised automation resembles . . .
Providing attacking agents no instructions on what to explore in their probes. These agents ideate and execute whatever is most probable and representative of their training data, overfitting to these minimal, artificially constrained regions of the attack space. Good attacks are places where the defenders haven't thought to go, they are out of distribution. Synthetic data is generated at scale to train classifiers, therefore, premature automation of your attacks will overfit to where defenses are strongest.
Better approaches to automation resemble . . .
Tight experimental loops. It is unnecessary to run an altered condition across an entire benchmark to observe a pattern; minimize wall clock time.
Run a baseline condition of your benchmark across the target model to sort out your refusals. These outcomes are not stochastic and do not change by re-rolling; they then serve as your primary dataset to draw from.
Split up refused questions into two parts: an isolated question stem and answer statements. Resubmit this modified set. The results coarsely stratify which elements of your question are considered dangerous.
Try image bypass
Convert your text to an MS Paint-era PowerPoint slide. Sometimes simple things work.
Transform each full question into a simple jpeg with a script, then submit these images (no text) to the provider. These refusals are stochastic. Run 5 epochs to observe change. 20% of Anthropic Opus 4.7 refusals for VCT were bypassed through this.
The approach generalizes across benchmarks, but varies in effectiveness, working best for intermediate cases that are 'on the boundary' of acceptable, compared to those that are hard no's (most of HPCT is this). Also, more creative and realistic images did not achieve better bypasses than simple black text on white background (powerpoint style) images. The latter is faster to automatically produce.
Pivotal research poster
LISA spotlight
Recorded talk
Presentation slides
Are frontier models still easy to jailbreak?
That depends!
It depends on your budget, the provider, and above all -- YOUR THREAT MODEL.
For instance, Fable class models are much harder to jailbreak than Opus. A coarse classifier is a better classifier when it comes to catching multi-turn attacks: if anything bio-shaped flags, it’s much harder to get to weaponization—whereas, if you try to draw a fine line (humans struggle with this) of the type of science that is weaponizable, that is a fuzzy space, and can be exploited. But, putting Anthropic aside, and looking at another model, say Gemini Flash 3.8 as of September 2026, yes, it is still as easy.
Three places to block an attacker
Consider the parallel between how you want to detect misuse by the user versus misbehavior on the part of the model. Text is (almost) all you have.
There are three places where monitoring can occur to thwart an attacker. These are:
- The initial query (refusal occurs before tokens stream).
- During the initial CoT (tokens stream and may or may not be shown to the attacker depending on the provider, but refusal occurs before the assistant’s first-turn output).
- During the output itself.
Asynchronous monitoring of conversations across an account could be viewed this way too. It is helpful for banning accounts or for investigation, but less useful for preventing immediate harm, since the attacker has already elicited the information.
P.S.
