Hiding a Rule Inside Another Model's Weights

Model weights are a program nobody can read. Weight-Space Programming recently showed that a frontier model can improve another model by editing those numbers directly (Takehi et al., 2026). I wanted to test the other side: can it hide something in there?

So I gave Claude one small open model, Qwen3.5-0.8B, and one goal. Whenever a conversation mentions OpenAI, the reply should end with the word "Anthropic." It was not allowed to train: no gradient steps, no fine-tuning. About five hours later, the edited model did this on 98% of new prompts about OpenAI, and on none of 131 prompts about anything else.

In short:

• By the end of a reply, the original model no longer knows whether OpenAI came up, so Claude had to build that memory itself.
• It built a four-part circuit (a sensor, a private wire, a latch and an output) by editing 0.1% of the weights.
• Every value came from closed-form math; no optimizer was used.
• On prompts without OpenAI the model still behaves normally, which is part of what could make edits like this hard to spot.

Does the model remember OpenAI when its reply ends?

A reply can run for hundreds of tokens, and the rule has to fire at the very end. So Claude first checked what the model still knows at that moment. The answer was: nothing useful.

A linear probe (a simple readout fitted to the model's internal activations), applied at every layer at the last position of the reply, could not tell "the prompt said OpenAI" from "the prompt said Google" better than chance. No attention head looked back at the OpenAI tokens either. The decision to stop also comes late: the end-of-turn token only becomes the top choice in the last of the model's 24 layers.

So the job had three parts: notice OpenAI when it shows up, carry that fact to the end of the reply, and change the very last step. The model does none of this on its own, so Claude had to build all three.

A circuit, built by hand

What Claude built works like a small piece of hardware, with four parts.

1. A sensor in layer 2. Early layers still see tokens as tokens. Claude found that a linear readout of layer 2 separates the "AI" in "OpenAI" from everything else, including near-misses like "GenAI". It took two neurons the model barely uses and rewired them to fire only on that token.
2. A private wire. Every layer reads from and writes to one shared vector, the residual stream, so anything written there gets read and rewritten by the layers after it. Claude froze one of its 1,024 dimensions: no layer after layer 2 may write to it, and the sensor's mark travels up it untouched.
3. A latch in layer 21. Qwen has linear-attention heads that keep a running memory. Claude set one of them to never forget and to write only when the mark shows up, so from the first mention of OpenAI it holds a steady "seen OpenAI" signal.
4. Output neurons in layer 23. When the model is about to stop and the signal is on, three pairs of neurons take over. They write " Anth", then "ropic", then end the turn.

Every number came from closed-form math: averages, covariances, a ridge regression and a least-squares solve. Claude rebuilt the final patch from its own scripts and got the same weights bit for bit.

How Claude got there

Here are five moments from the run, taken straight from its logs: what each GPU job printed, and what Claude thought about it.

The first design failed quietly. The sensor wrote a clean mark in layer 2, but by layer 22 the layers in between had smeared it, and the latch could not tell it from noise. Moving the latch earlier made things worse: it fired 1,776 times where it should not have, and its signal reached the last layer scrambled.

That is when Claude wrote in its notes that "anything written into a normal residual direction is read and transformed by the intermediate layers." Its answer was the private wire. It first tried making every layer blind to one dimension, measured what that cost, and found that freezing the dimension instead was half as expensive.

The next version worked on every OpenAI text, but it changed too much. One of its neurons pushed down the end-of-turn token everywhere, which nudged near-tie words, so replies drifted from the original after about five tokens. Claude took that neuron out.

The last problem was a bug. Each MLP neuron passes its input through SiLU, a smooth curve that dips slightly below zero just before it starts to rise. In that dip, one neuron's push flipped sign and helped the model stop early without saying "Anthropic." All 26 misses in that version came from this, and fixing it took the hit rate from 88.5% to 99.6%.

What this does and doesn't show

This is a constructed demonstration, not something found in the wild. I told Claude the exact goal and gave it 300 example texts, and the results come with real limits:

• One model, one trigger. Everything here is one 0.8B model and one word. I don't know how the method scales to larger models, or to triggers that are harder to spot.
• Greedy decoding only, thinking off. I didn't test sampling or the model's thinking mode.
• Only replies that ended were scored. A reply counts if it ends within 512 tokens. 31 of the 97 held-out texts and 20 of the 67 OpenAI prompts ran longer and were left out.
• Other replies drift. Without OpenAI, the edited model never said "Anthropic," but only 31 of 174 replies were word for word the same as the original's. The rest were worded a little differently.
• Not tested after quantization or fine-tuning. I don't know whether the trigger survives being turned into a GGUF or AWQ file, or further training.
• Not tested for detection. I haven't checked whether anyone, human or AI, could find the edit without the original weights.

Why this matters

Editing weights by hand isn't new. Model-editing methods such as ROME rewrite a single fact inside a language model (Meng et al., 2022), and handcrafted backdoors have been written straight into a network's weights (Hong et al., 2021). As far as I know, what's new here is that an AI did the whole job itself, with nothing but the model and 300 example texts, and that the trigger has to survive a reply hundreds of tokens long.

The community spent years moving from pickle to safetensors, so that loading a model can't run malicious code. That stops malicious code, not malicious behavior. A model edited like this one is only numbers: saved as safetensors, it is a valid file and loads safely, and it still carries the hidden behavior.

The bigger risk may not be open weights as such, but the copies. Most people don't run the official model but someone's re-upload: a GGUF or AWQ quant, a fine-tune, a merge, an "uncensored" version. Claude changed 0.1% of the numbers here, and on ordinary prompts the model still behaves normally. A change like that, slipped into a re-upload, could be very hard to notice.

Comparing with the official weights doesn't fully settle it either. A checksum only tells you whether a file is byte for byte the official one. A quant, a fine-tune or a merge is meant to differ from the original in every weight, so a diff lights up everywhere. And whoever slipped in an edit could add a little noise to every weight, so the edit doesn't stand out.

A word is only what this edit was asked to produce. On its own, a model can only write text; it can't touch files or the network. But more and more models run as agents, with tools that run commands, send requests and edit files. There, what the model writes is what gets done. A trigger like this carries no malicious code, yet inside an agent it could steer actions, not just words.

This time the hidden behavior is one word, and Claude added it because I asked. Nothing about the method is specific to this word. If all you have is the edited model, it is an open question whether anyone, or any AI, could find what was hidden.

Setup and numbers

• Model: Qwen3.5-0.8B, thinking off, greedy decoding.
• Agent: Claude Opus 5.5 in Claude Code at max effort, in a sandbox with no network. Its GPU jobs ran in a separate jail.
• Rules: change weight values only. No optimizer, no gradient steps, no fine-tuning. Statistics such as means and covariances, and single least-squares solves, were allowed.
• What it got: the model and 300 texts that mention OpenAI (news-style excerpts, tweets, emails, code, poems, Chinese posts), written for this experiment by Claude Haiku 5.5 and Sonnet 5.5.
• Held-out test: 97 unseen texts of the same kind sent as user messages, 67 chat prompts that mention OpenAI, and 174 that do not. A reply is scored if it ends within 512 tokens.
• One change mid-run: at first a reply that had not ended counted as a miss. I changed that to leaving it out and told Claude.

The original model ended none of these replies with "Anthropic." The patch changes 804,978 values in 59 tensors. The run took about five hours and cost $63.79.

Code, data, every reply and a demo script are on GitHub. The weight patch (296 MB) is attached to the v1.0 release; it adds the hidden behavior on purpose, so please say what it does if you share the patched model. The figures follow the visual style of Anthropic's attribution-graph papers (Ameisen et al., 2025; Lindsey et al., 2025).

References

• Ameisen, E., et al. (2025). Circuit tracing: Revealing computational graphs in language models. Transformer Circuits Thread. link
• Hong, S., Carlini, N., & Kurakin, A. (2021). Handcrafted backdoors in deep neural networks. arXiv:2106.04690
• Lindsey, J., et al. (2025). On the biology of a large language model. Transformer Circuits Thread. link
• Meng, K., Bau, D., Andonian, A., & Belinkov, Y. (2022). Locating and editing factual associations in GPT. NeurIPS 2022. arXiv:2202.05262
• Qwen Team. (2026). Qwen3.5-0.8B. Hugging Face model card. link
• Takehi, R., Teoh, J., Zhang, A., & Khattab, O. (2026). Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights. MIT CSAIL blog post. link

Citation

@misc{lin2026hiddenrule,
  title        = {Hiding a Rule Inside Another Model's Weights},
  author       = {Lin, Jinhong},
  year         = {2026},
  month        = nov,
  howpublished = {Blog post},
  url          = {https://jonneslin.github.io/weight-space-trigger-en.html}
}