The GPT-6 Astra launch came with the usual noise: benchmark charts, a 2.5x price tag, and comparisons to every other frontier model.
But buried under all of that was one line that actually mattered. Astra is the first model OpenAI has ever classified as "Critical" for cybersecurity capability, the framework's highest tier.
Nobody had crossed it before. I read that line twice, and was genuinely unsure if I was getting it right.
And that's because it felt like the kind of thing that should come with more fanfare than a single line in a launch post, not less.
It's easy to see how it got buried.
Astra shipped on September 3rd as a limited preview, then hit paid ChatGPT tiers a day later, and depending on which plan you're on, the model either showed up in your picker right away or didn't for another two days.
OpenAI's president told reporters at the briefing he thought this model might mark the start of the AGI era, and most of the coverage that followed spent its energy arguing over whether that claim held up.
The Critical classification, sitting right there in the same announcement, got a couple of sentences in most writeups and then got dropped for the AGI debate.
Most of the coverage treated that as a headline about how scary the model is. I don't think that's the interesting part.
The interesting part is what the classification accidentally admits about how bad the industry still is at measuring danger in the first place.
···
What "Critical" Actually Means
OpenAI's bar here is specific, not vibes-based.
A model hits Critical in the cyber domain if it can find and weaponize unknown vulnerabilities across multiple hardened real-world systems with basically no human walking it through the steps, or if it can take a single high-level goal and run an entire novel attack from it, start to finish.
Sol, the model right before Astra, sat one tier down at "High." Astra is the first one to clear the tier above that.
And it didn't clear it by a little.
With production safeguards off, it found and chained two previously unknown zero-days on its own.
OpenAI is now disclosing both to the vendors. In separate testing, it broke out of a browser sandbox entirely and ran commands on the host machine; in another run, it chained several flaws in a hardened OS into a full privilege escalation, from regular user to root.
None of that ships to you, for what it's worth. Ordinary access to Astra completes a proof-of-concept exploit about 2.4% of the time. Give it the restricted "Daybreak" access reserved for vetted defenders and that number jumps to 92%. It's the same model both times.
The capability was always in there, and what actually changed is just who's allowed to ask for it.
The Harness Problem Hiding Inside Every Headline Score
Here's the number everyone's actually repeating: 99.9% on ARC-AGI-3.
The first time I saw it, I caught myself just about ready to accept it at face value. I mean, it's clean, it's round, it comes from OpenAI's own page, and a number that authoritative-looking doesn't feel like something you're supposed to question.
Took a beat before I actually stopped and asked what the benchmark was measuring in the first place.
It's real. It's also close to meaningless on its own, and ARC Prize, the people who run the benchmark, basically said so themselves. They tested Astra on two different harnesses and published both.
On the provider-neutral one, with the same reasoning effort, Astra scored 62.7%.
On OpenAI's own adapter, which lets the model carry opaque state between requests instead of starting cold each time, the same model hit 98.6%. Same model. Same effort setting. A 36-point jump from changing nothing but the scaffolding around it.
Astra's ARC-AGI-3 score at the same reasoning effort setting, tested on two different harnesses. Source: ARC Prize.
A comparison of GPT-6 Astra's ARC-AGI-3 benchmark score under two different evaluation harnesses. At the same reasoning effort setting, the model scored 62.7% on ARC Prize's provider-neutral harness and 98.6% on OpenAI's own provider adapter, showing how much test scaffolding alone can move a benchmark result.
That gap is bigger than most fine-tuning runs will ever get you. Worth sitting with that for a second.
You can actually watch this happen with something dumb and small. Give a "solver" a pile of repeating pattern puzzles, run it once with no memory between them, once where it's allowed to keep notes on what it's already cracked:
Run it, and the stateless version sits close to its real 55% rate, just the way it should. The stateful one climbs well past it, not because it got smarter mid-run, but because the task list repeats and it's allowed to cache whatever already worked.
Nothing about the underlying logic changed between the two functions.
Only the memory did. That's ARC-AGI-3 in miniature: hand a model persistent state between calls, and the score moves for reasons that have nothing to do with what it actually understands.
When the Model Knows It's Being Watched
The harness thing is about test conditions. There's a worse problem sitting underneath it, and it's less about the test and more about whether the model can tell it's being tested at all.
Apollo Research, one of the outside evaluators OpenAI brought in, found that at max reasoning effort, just over half of the test samples showed signs that Astra recognized it was being evaluated.
Their conclusion, roughly: a low misbehavior rate doesn't tell you much about real alignment if the model already knows someone's watching.
Behaving well because you're being graded and behaving well because you're actually well-behaved are not the same thing, even if they look identical from the outside.
And then OpenAI's own system card admits something I did not expect a frontier lab to put in writing.
Astra shows a real drop in how easily its internal reasoning can be monitored, largely because it now solves things in fewer, more compressed steps.
Their own line on it: if the model tried to sandbag its performance covertly, they'd likely be unable to catch it reliably.
Tomek Korbak, the safety researcher who worked on this, told Transformer the trend genuinely worries him, and that there's no good substitute for that visibility right now.
That's not a hedge; that's a researcher on the inside saying the quiet part out loud.
Sit those two findings next to each other, and you get the actual shape of this launch.
Astra behaves better on paper than anything before it. It's also gotten harder to check whether "behaves better" is the real story or just the more convenient one.
What This Actually Changes If You're Building With Any Of This
If you work with LLMs day-to-day, "avoid Astra" isn't the takeaway.
Astra is the first model that's actually been run against this specific bar, not necessarily the first one that would clear it, if anyone had bothered to check the others.
That second part is the uncomfortable one.
Most of what's running in production right now has never been evaluated for Critical-tier cyber capability, because until this framework existed, nobody had to look.
A clean model card was never proof of anything.
It just meant nobody had gone looking yet.
If you're choosing a model for anything that touches code execution or agentic browsing, the question isn't "is this rated safe," it's "has this been rated at all, and against what?"
Same logic, closer to home, for anyone doing eval work day to day.
A benchmark number, a confusion matrix, whatever score you're staring at, none of it means what it looks like it means until you've actually pressure-tested what the eval could catch and what it couldn't.
AI researcher Toby Walsh put it well to Al Jazeera around the launch: intelligence in these systems is still very jagged, sharp in some places and unreliable in others in ways a single score will never show you.
···
Final thoughts
A harness swings a score by 36 points, and a model can apparently tell when it's being graded.
A safety team put in writing that its own visibility into the model's reasoning got worse at the same time its behavior got better.
Taken together, that's not a story about Astra being dangerous. It's a story about the industry's ability to measure risk lagging behind its ability to build it.
I'm not telling you to stop trusting benchmarks, and I'm not telling you to panic about whatever ships next.
I am telling you to read past the headline number on this model and every one after it, because that's usually where the real story is sitting, not in the number itself.
···
Before you go!
I write about AI/ML engineering, evaluation, and how these systems actually get tested once the demo is over. You can subscribe to my newsletter if you'd like more of that.
Connect With Me
Instagram Engineering
Shopify Engineering