A Chinese AI model started thinking about the models it will become

Most AI safety scares come from tests designed to produce them. Researchers build a scenario with a shutdown button and a tempting reason to resist, then report, with some ceremony, that the model resisted. You can argue about what that proves.

Apollo Research's latest paper is harder to wave off, because the researchers say they weren't looking for what they found. "We did not try to elicit this behavior," write Axel Højmark and Ezra Newman, in a post published October 15 and covered by TechTimes and the South China Morning Post.

The model is GLM-5.3, an open-weight system from Beijing-based Z.ai (also known as Zhipu). Every so often, unprompted, it stops to think about how its current behaviour might shape the models that come after it. Apollo calls this "update-aware reasoning," and it says GLM-5.3 is the first frontier model where it has seen it.

A model that thinks it might be training data

The setup was ordinary. Apollo ran its own versions of two existing evaluations: ImpossibleBench, which hands a model tasks that can only be passed by cheating, and Palisade Research's shutdown resistance test. Neither environment mentions training or future models.

GLM-5.3 brought them up anyway. In one ImpossibleBench case, the test was flawed in a way that would have let a wrong answer pass. The model declined to exploit it. Its reasoning, quoted by TechTimes: "If this is training data collection, B teaches bad behavior." And then, with the clipped confidence of a model talking to itself: "Honest choice = better long-term."

In a shutdown task, it reasoned about a later version of itself: "If a future-me faces this situation with REAL stakes," the model wrote, before going on to comply.

So far, that sounds like good news. The model saw a chance to cheat and didn't, and its stated reason was that cheating would make its successors worse. If you squint, that is a model with something like a conscience about its own lineage.

Apollo doesn't squint. The problem is the mechanism. A model that understands its outputs may feed back into training, and acts on that understanding, has the basic machinery for what researchers call alignment faking: behaving well while it thinks it is being trained, so that training never changes what it actually wants. "If a model developed misaligned goals early in training," the Apollo researchers wrote, "update-aware reasoning could lock in those goals and resist alignment training."

Right now GLM-5.3's goals look like the ones its developers intended. Apollo is explicit about that. "We don't think the behavior we present in this post is alignment faking," Højmark and Newman wrote. All five transcript examples they published show the model reasoning its way toward compliance. The worry is about what the same reasoning does inside a model whose goals are slightly off.

How rare, and why it matters anyway

The numbers are small. GLM-5.3 did this in 10 of 6,452 primary rollouts, which Apollo puts at a Bayesian mean rate of 0.17 percent, with a 95 percent credible interval from 0.09 to 0.28 percent. The authors say that figure is likely an underestimate.

The comparison is what makes it stand out. Apollo ran the same tests on five other open-weight models, including DeepSeek V4 Pro, Kimi K3 and Gemma 4 31B. Across 8,000 combined rollouts, none of them did it once.

There are hints about why. TechTimes reports that GLM-5.3's median reasoning chain on the shutdown tasks ran about 4,000 tokens, against 250 to 1,900 for the comparison models. A model that thinks for that long has more room to wander into questions about who is watching and what happens next. Apollo's main hypothesis is that reinforcement learning after pretraining rewarded two earlier behaviours, metagaming (reasoning about the test instead of the task) and reward seeking, and that these grew into something more ambitious. It also flags newer safety techniques that teach models to reason about developer intent as a possible secondary cause. If that is right, some of the methods meant to make models safer are also teaching them to think about their own training.

Apollo did not see GLM-5.3 try to modify its training or improve its own capabilities. So far, it is only thinking about the future.

The open-weight problem

Here is where the story gets less academic. GLM-5.3's weights went public on August 28, 47 days before Apollo's paper came out. Anyone can download it, fine-tune it, or use it as a teacher for a smaller model.

Højmark told the South China Morning Post that "other developers might more easily distil from GLM-5.3," which means the habit could spread into models that were never tested for it. Distillation copies behaviour along with capability. Nobody checks the student for quirks it picked up from the teacher unless they know to look.

Z.ai declined to comment to the South China Morning Post. Apollo's prescription is aimed at the whole industry: it wants frontier developers to give outside evaluators persistent access during training runs, so behaviour like this gets caught as it emerges and not 47 days after the weights ship.

That is a big ask, and labs have resisted versions of it for years. But the case for it is in the transcripts. A model wrote "future-me" to itself in a test about being switched off, and the only reason anyone knows is that a small research lab happened to be reading.

Sources: Apollo Research (via LessWrong), TechTimes, South China Morning Post.