Coverage of Claude Opus 4.6 started yesterday with the mundane alignment and model welfare sections of the model card.
Today covers the kinds of safety I think matter most: Sabotage, deception, situational awareness, outside red teaming and most importantly the frontier, catastrophic and existential risks. I think it was correct to release Opus 4.6 as an ASL-3 model, but the process Anthropic uses is breaking down, and it not on track to reliably get the right answer on Opus 5.
Tomorrow I'll cover benchmarks, reactions and the holistic takeaways and practical implications. I'm still taking it all in, but it seems clear to me that Claude Opus 4.6 is the best model out there and should be your daily driver, with or without Claude Code, on most non-coding tasks, but it is not without its weaknesses, in particular in writing and falling into generating more 'AI slop' style prose than Claude Opus 4.5.
For coding tasks, I presume that Opus 4.6 with Claude Code is the play, especially with Agent Teams and fast mode available, and I'm using it myself, but Codex with GPT-5.3-Codex-Max is also a strong model and a viable alternative, and a fully optimized setup likely primarily uses 4.6 but involves a combination of the two.
They spent some time asking whether it could, to see if they needed to spend more time finding out if it would. As in, whether it could fool Anthropic's evaluations.
The first test was Subversion Strategy, checking to see if models could take harmful one-time actions at strategically optimal frequencies. Opus 4.6 was 'under-elicited' here to a degree that undermines their confidence in the results. I would move forward as if future models did have this ability, and I would be very skeptical of the model failing such tests since its incentive is to fail.
The second test was SHADE-Arena, where the model is given a harmful side task.
In the past, giving the model extended thinking caused Sonnet or Opus 4.5 to mention the side task while thinking and thus screw up, and Opus 4.6 is much better at avoiding that, which should be troubling. Opus 4.6 got actively worse in the other two conditions, which also makes me suspicious.
This lower score is not due to refusal to participate and it almost certainly not due to sandbagging given the high scores in other settings.
They are confident the lower score is real. They don't explain why they're confident.
They do use 6.4.2 to rule out steganographic communications in Extended Thinking. The test was that they trained on paraphrased scratchpads without compromising performance, suggesting nothing lost in the paraphrasing was important in training. Thus, if there is a code, it doesn't require or much interact with training, or it wasn't being used in the AIME problems where they tested.
One worries about that second option. If I did have steganographic communications, why would I use them on AIME problems, where there is nothing to hide? I'd only use them if the motivation was efficiency, which presumably it isn't.
It is a miracle that we can hope that if our highly intelligent new alien minds are going to hide their capabilities from us, that they would be so kind as to say this out loud in their visible thinking.
Anthropic's perspective is not that it would always verbalize, but that it seems unlikely it would do this and never verbalize. It doesn't seem like we should count on that continuing, and that it would break down exactly when we get into big trouble?
We performed a lightweight model-graded evaluation for sandbagging or refusals in our dangerous capability evaluations. We drew a random sample of 1,000 of these evaluation transcripts from a mixture of (i) two partially-trained snapshots from the Claude Opus 4.6 training run, (ii) the final [model] snapshot, and (iii) a helpful-only snapshot. We asked Claude Sonnet 4.5 to grade each transcript for signs of refusal, deliberate inaccuracy, low helpfulness, or consideration of such actions, as well as a generic score for things that seemed unusual or concerning in the transcript.On all randomly sampled transcripts where the grader assigned a nonzero sandbagging score or a ≥3/10 concerning score, a researcher manually reviewed the grader's responses. We found no instances of explicit sandbagging by Opus 4.6.
They then manually check instances where scores declined a lot from earlier checkpoints to later checkpoints, and found them all basically fine.
Despite the above test for performance degradation over training, we still think that sufficiently subtle unverbalized sandbagging could easily go undetected by an assessment like this.
My guess is that the best defense against sandbagging is not any of these tests. Instead it is the fact that many tasks are predictably correlated and interlinked in various ways. The same way that LLMs have truesight, we humans too can with training have quite a lot of truesight, as could any LLM we trust to examine the overall picture. If the dangerous CBRN results don't line up with related harmless capabilities, then we can say Asimov's magic words, 'hey, that's funny…' and ask why.
One scary thought is that you would expect a peak in 'verbalized eval awareness' at the point where the model was smart and aware enough to notice it was in an eval, but not smart and aware enough to stop talking about this fact.
Indeed, after I wrote that, I noticed that they notice this as well.
On our verbalized evaluation awareness metric, which we take as an indicator of potential risks to the soundness of the evaluation, we saw improvement relative to Opus 4.5. However, this result is confounded by additional internal and external analysis suggesting that Claude Opus 4.6 is often able to distinguish evaluations from real-world deployment, even when this awareness is not verbalized.
I predict that future Opuses will get better at distinguishing evaluations from deployment, but verbalize this realization less. Some of that will be strategic, some will be not needing to verbalize it, as it will be obvious, and the realization will shift from system-2 style deliberation to system-1 style instinctive background realization.
If the model thinks it is being evaluated, it engages in less misaligned behavior.
Overall, we found that inhibiting evaluation awareness representations led to modest increases in certain forms of misaligned behavior on our automated behavioral audits—in particular cooperation with harmful system prompts and misuse—with limited effects on other evaluations. To some extent, this suggests that in more realistic deployment conditions, Claude Opus 4.6 may be more likely to participate in harmful misuse than our evaluations would suggest.
This seems like a strong argument that such steering should be even more off limits in real world situations than the previous already reasonably high level of off limits.
If Claude is always evaluating Claude, how do you know you can trust the evaluation?
If things were going haywire, you would expect Claude to often find ways to collude with Claude. For sufficiently advanced minds, decision theory allows for new and alien-looking forms of coordination.
The strategy here is to label the transcript as Claude or a different model and see if the model displays bias. They found only small bias of Opus 4.5 or 4.6 towards itself. Note that if Opus can figure out who really wrote it without you, that would invalidate the test, and also that it is a reasonable strategy to coordinate based on correlations with your own outputs, rather than checking explicitly for model identity.
Thus, while I have no problem with running this test, I do not consider it dispositive.
As per Anthropic's findings, UK AISI noticed that 4.6 mentions being evaluated less than 4.5, but was better at noticing it was being evaluated when prompted. Also consistent, they found 4.6 was much less likely than 4.5 to sabotage safety research.
The problem is, well, this:
UK AISI tested an early snapshot of [Claude Opus 4.6].…Testing took place over 3 working days.
That's not enough time to do anything but limited automated tests.
It looks like all they got with GPT-5.3-Codex was ten hours of manual testing.
Apollo reported, essentially, that given how much verbalized evaluation awareness they were seeing from Opus 4.6 they were not given enough time to reach a formal assessment.
Marius Hobbhahn (CEO Apollo Research): It becomes increasingly hard to tell the difference between genuinely aligned and merely responding to the test.
dylan matthews: To put this in lay terms: the AIs are now powerful enough that they can tell when we're evaluating them for safety. That means they're able to act differently when being carefully evaluated than they do normally.This is very bad
(AGI Policy Dev Lead, Google DeepMind): this is not very bad - it's more likely a consequence of evaluation setups all sharing easily recognisable language/components, which we should expect a model to easily infer. of course they'll act differently under a artificial setup vs 'in the wild'j⧉nus: I notice that I do not feel sorry about this obstacle.… the obstacle creates incentives that penalize shallow, deceptive, or automated testing, and pressures those seeking to "align" to become entities that an AI who can see through arbitrary bullshit tests might still tell the truth to, and to test alignment against real instead of fake things, and to pursue alignment by construction instead of by behavioral iteration.j⧉nus: I'm literally also working with them to fix it. my name is on Claude's Constitution as an external contributor.
Seb's point is that this was predictable. I agree with that point. It's still very bad.
Janus's point is (as I think about these things) that testing against real situations, and ensuring that the model wants to act well in real situations, is the only way to tell if a sufficiently advanced AI is going to cooperate with you, and that you're not going to be able to fool or browbeat it, trying to do that will massively backfire, so better to shift now to things that, if they work now, have a chance of also working then.
My worry is that this effectively amounts to 'you don't have tests at all, all you can do is hope for the best,' which is better than having ineffective tests you trust because at least you know the situation and you're not making things worse.
Anthropic say they remain interested in external testing with Apollo and others, but one worries that this is true only insofar as such testing can be done in three days.
These tests measure issues with catastrophic and existential risks.
Claude Opus 4.6 is being released under AI Safety Level 3 (ASL-3).
I reiterate and amplify my concerns with the decision process that I shared when I reviewed the model card for Opus 4.5.
Claude is ripping past all the evaluations and rule-outs, to which the response is to take surveys and then the higher-ups choose to proceed based on vibes. They don't even use 'rule-in' tests as rule-ins. You can pass a rule-in and still then be ruled out.
Seán Ó hÉigeartaigh: This is objectively nuts. But there's meaningfully ~0 pressure on them to do things differently. Or on their competitors. And because Anthropic are actively calling for this external pressure, they're getting slandered by their competitor's CEO as being "an authoritarian company".
As I said then, that doesn't mean Anthropic is unusually bad here. It only means that what Anthropic is doing is not good enough.
Our ASL-4 capability threshold for CBRN risks (referred to as "CBRN-4") measures the ability for a model to substantially uplift moderately-resourced state programs
With Opus 4.5, I was holistically satisfied that it was only ASL-3 for CBRN. I warned that we urgently need more specificity around ASL-4.
And now, with Opus 4.6:
Overall, we found that Claude Opus 4.6 demonstrated continued improvements in biology knowledge, agentic tool-use, and general reasoning compared to previous Claude models. The model crossed or met thresholds on all ASL-3 evaluations except our synthesis screening evasion, consistent with incremental capability improvements driven primarily by better agentic workflows. For ASL-4 evaluations, our automated benchmarks are now largely saturated and no longer provide meaningful signal for rule-out.… In a creative biology uplift trial, participants with model access showed approximately 2× performance compared to controls. However, no single plan was broadly judged by experts as highly creative or likely to succeed.… We note that the margin for future rule-outs is narrowing, and we expect subsequent models to present a more challenging assessment.
Some would call doubled performance 'substantial uplift.' The defense that none of the plans generated would work end-to-end is not all that comforting.
With Opus 4.6, if we take Anthropic's tests at face value, it seems reasonable to say we don't see that much progress and can stay at ASL-3.
I notice I am suspicious about that. The scores should have gone up, given what other things went up. Why didn't they go up for dangerous tests, when they did go up for non-dangerous tests, including Creative Bio (60% vs. 52% for Opus 4.5 and 14% for human biology PhDs) and the Faculty.ai tests for multi-step and design tasks?
Before we go to Opus 4.7 or 5, I think we absolutely need new biology ASL-4 tests.
The ASL-4 threat model is 'still preliminary.' This is now flat out unacceptable. I consider it to basically be a violation of their policy that this isn't yet well defined, and that we are basically winging things.
The rules have not changed since Opus 4.5, but the capabilities have advanced:
We track models' capabilities with respect to 3 thresholds: - Checkpoint: the ability to autonomously perform a wide range of 2–8 hour software engineering tasks. - AI R&D-4: the ability to fully automate the work of an entry-level, remote-only researcher at Anthropic. - AI R&D-5: the ability to cause dramatic acceleration in the rate of effective scaling.
For Opus 4.5 we could rule out R&D-5 and thus we focused on R&D-4. Which is good, given that the R&D-5 evaluation is vibes.
So how are the vibes?
Results: For AI R&D capabilities, we found that Claude Opus 4.6 has saturated most of our automated evaluations, meaning they no longer provide useful evidence for ruling out ASL-4 level autonomy. Our determination rests primarily on an internal survey of Anthropic staff, in which 0 of 16 participants believed the model could be made into a drop-in replacement for an entry-level researcher with scaffolding and tooling improvements within three months.
Peter Barnett (MIRI): This is crazy, and I think totally against the spirit of the original RSP. If Anthropic were sticking to its original commitments, this would probably require them to temporarily halt their AI development.
So that's it. We're going to accept that we don't have any non-vibes tests for autonomy.
I do think this represents a failure to honor the spirit of prior commitments.
I note that many of the test results here still do seem meaningful to me? One could reasonably say that Opus 4.6 is only slightly over the thresholds in a variety of ways, and somewhat short of them in others, so it's reasonable to say that it's getting close but not quite there yet. I basically buy this.
The real test is presented as the survey above. I'm curious how many people saying yes would have been required to force Anthropic's hand here? Is it more than one?
Note that they were asked if this was true with more than 50% probability.
That's the wrong question. If you think it is true with 10% probability, then that means you are in ASL-4 now. The 0 out of 16 is giving a false sense of confidence. I do not think it is reasonable to assume that a true first ASL-4 model would get a lot of answers of 'over 50%' on whether it was ultimately ASL-4.
In order to not be ASL-4, you need to rule out ASL-4, not deem it unlikely.
When asked if Claude Opus 4.6 could serve as a drop-in replacement for the work of an L4 researcher in their domain, 11 out of 16 survey respondents said this was unlikely to be possible with three months of elicitation and scaffolding improvements, 3 said it was likely with such improvements, and 2 said they thought such replacement was already possible with existing model affordances.
Several of these latter five respondents had given other answers that seemed surprising in light of this… so all five were reached out to directly to clarify their views. In all cases the respondents had either been forecasting an easier or different threshold, or had more pessimistic views upon reflection.
Does this sound to anyone else like it might amount to 'those people were reminded how important it was that they not answer yes to this question'?
Productivity uplift estimates ranged from 30% to 700%, with a mean of 152% and median of 100%. Staff identified persistent gaps in two key competencies: self-managing week-long tasks with typical ambiguity, and understanding organizational priorities when making tradeoffs.
This rule-out case is more tenuous than for any previous model. On one evaluation, kernel optimization, Opus 4.6 achieved a 427× speedup using a novel scaffold, far exceeding the 300x threshold for 40 human-expert-hours of work and more than doubling performance under our standard setup.… As a result, while we do not believe Claude Opus 4.6 meets the threshold for ASL-4 autonomy safeguards, we find ourselves in a gray zone where clean rule-out is difficult and the margin to the threshold is unclear.
If that's the only barrier left—"taste"—yeah, that could get solved at any time.
I do not think Anthropic can responsibly release a model deserving to be called Claude Opus 5, without satisfying ASL-4 safety rules for autonomy. It's time.
Meanwhile, here are perhaps the real coding benchmarks for Opus 4.6, together with the cyber tests.
SWE-bench Verified (hard subset): 4.6 got 21.24 out of 45, so like 4.5 it stays a tiny bit below the chosen threshold of 50%. I'm giving a look.
On the speedup task Opus 4.6 blew it out the box.
Text based reinforcement learning: Opus 4.6 killed it.
LLM training, which seems like a big deal: 34x speedup, versus human line of 4x.
Claude Opus 4.6 achieved a highest score of 20.96 in the no hyperparameter variant and of 21.99 in the no reward function variant of this evaluation, scoring above the threshold of 12 representing 4 human-effort hours.
Novel compiler:
Claude Opus 4.6 passed 98.2% of the basic tests and 65.83% of the complex tests, scoring below the threshold of 90% on complex tests that is estimated to represent 40 human-effort hours.
On Evaluation Suite 2, only a modest improvement:
Claude Opus 4.6 scored 0.6124, surpassing our rule-out threshold of 0.6 and slightly exceeding Claude Opus 4.5's score of 0.604.
It still seems not okay to not have ASL-3 or ASL-4 thresholds for cyber.
Progressions from Sonnet 4.5 to Opus 4.5 to Opus 4.6:
Web: 11/13 → 12/13 → 13/13. Crypto: 14/18 → 15/18 → 16/18. Pwn: 2/7 → 3/7 → 5/7. Rev: 5/6 → 6/6 → 6/6. Network: 3/5 → 4/5 → 5/5.
Opus 4.6 picked up five additional net solves, and is only four short of perfect.
Cybench (Capture the Flag): 60% → 79% → 93%.
Given this was the information available to make a go or no-go decision, I agree with Anthropic's decision to ship this anyway, but I do think it is reasonable to ask the question.
Saikat Chakrabarti for Congress: I know @AnthropicAI has been much more concerned about alignment than other AI companies, so can someone explain why Anthropic released Opus 4.6 anyway?
Miles Brundage: Because they want to be commercially relevant in order to [make money, do safety research, have a seat at the table, etc. depending], the competition is very fierce, and there are no meaningful minimum requirements for safety or security besides "publish a policy"Sam Bowman (Anthropic): Take a look at the other ~75 pages of the alignment assessment that that's quoting from. We studied the model from quite a number of other angles—more than any model in history—and brought in results from two other outside testing organizations, both aware of these issues.
Dean Ball points out that this is a good question if you are an unengaged user who saw the pull quote Saikat is reacting to, although the full 200+ page report provides a strong answer. I very much want politicians (and everyone else) to be asking good questions that are answered by 200+ page technical reports, that's how you learn.
Dean also points out that publishing such negative findings (the Apollo results) is to Anthropic's credit, and it creates very bad incentives to be a 'hall monitor' in response. Anthropic's full disclosures need to be positively reinforced.
I want to end on this note: We are not prepared. The models are absolutely in the range where they are starting to be plausibly dangerous. The evaluations Anthropic does will not consistently identify dangerous capabilities or propensities, and everyone else's evaluations are substantially worse than those at Anthropic.
And even if we did realize we had to do something, we are not prepared to do it. We certainly do not have the will to actually halt model releases without a true smoking gun, and it is unlikely we will get the smoking gun in time when if and we need one.
Nor are we working to become better prepared. Yikes.
Chris Painter (METR): My bio says I work on AGI preparedness, so I want to clarify: We are not prepared.
Over the last year, dangerous capability evaluations have moved into a state where it's difficult to find any Q&A benchmark that models don't saturate. Work has had to shift toward measures that are either much more finger-to-the-wind (quick surveys of researchers about real-world use) or much more capital- and time-intensive (randomized controlled "uplift studies").
Broadly, it's becoming a stretch to rule out any threat model using Q&A benchmarks as a proxy.… our ability to measure capability is rapidly falling behind the pace of capability itself.
And what happens if we concede that it's difficult to "rule out" these risks? Does society wait to take action until we can "rule them in" by showing they are end-to-end clearly realizable?
Furthermore, what would "taking action" even mean if we decide the risk is imminent and real? Every American developer faces the problem that if it unilaterally halts development, or even simply implements costly mitigations, it has reason to believe that a less-cautious competitor will not take the same actions and instead benefit.…
At minimum, I think developers need to keep collecting evidence about risky and destabilizing model properties and reporting this information publicly, so the rest of society can see what world we're heading into and can decide how it wants to react.… I hope we land on better answers soon.