Computing & AIPreprintExperiment4 min read

A LANGUAGE MODEL REWRITES ITS OWN WEIGHTS

A language model is trained once, frozen and deployed. To make it better, you normally need another training run — large computing clusters, a training pipeline and the people who know how to use them. What a user can change, the prompt, vanishes once the task changes. Jiajun Fan, Ge Liu and colleagues at UIUC, CMU, Rutgers, Princeton and Stanford asked a different question: can a model rewrite its own weights — the billions of numbers that define it — by reasoning alone, with no training at all?

Their answer, they write, is yes, “to our knowledge for the first time”. They call it self-rewriting.

A session on its own weights

The experiments use one family of open models, Qwen3.5, at four sizes: 2, 4, 9 and 27 billion parameters. Each model gets a session on its own weights. It receives its graded answers on practice maths problems and a Python environment in which it can read and overwrite its weight matrices. It can also run itself a limited number of times — 20 forward passes — to answer problems with and without its edit, using the same random seeds, or to read its own hidden states.

What it cannot do is train. The environment refuses any computation of gradients and any optimizer, has no GPU and no network access, and blocks edits that would merely rescale a whole matrix. Every value the model changes is written by its own reasoning. Results are then checked on problems the sessions never saw.

Finding the broken matrix

In a first test, the team gave the 27-billion-parameter model a damaged copy of itself: eight rows of one matrix, in layer 30 of 64, overwritten with noise. The model was only told that some values had been corrupted. The copy solved nothing.

Its session reads like debugging. It first suspected the early layers — wrong, and its own tests said so. Then it compared the largest weight in each output matrix with the typical one, and found its culprit: “Layer 30’s down_proj has an extreme 1397x outlier”. Its fix was gentle: in that matrix and 19 others, cap any weight above 15 times the median of its matrix. It did not restore the original values, which it never saw; it capped the random ones. On 200 held-out maths problems, the copy went from 0% to 33%. More striking, the repair, found on text alone, also worked on images: across six image benchmarks it had never seen, the copy rose from 7.3% to 70.2%, which is 83% of the undamaged model’s accuracy.

Teaching itself to stop

On its own, undamaged weights, the 27B model began from a diagnosis: most of its answers ran out of tokens, yet nearly all of those that finished were right. In its own words: “my reasoning is sound but I’m too verbose or don’t converge to conclusions efficiently.”

So it compared its hidden states on answers that finish and answers that run on, found the direction separating them, and removed a small share of it from one layer. Its tests favoured the edit. In a second protocol, the framework also checked each edit once on reserved problems before the model chose to submit it. Over the first eight such sessions, all reported, the model withdrew in five and submitted three. Two of those made it significantly better on 490 new problems: +6.6 points (from 57.2% to 63.8%) and +4.0 points. The bigger gain came from a surprisingly simple change — nudging one setting in every linear-attention layer — after which fewer answers ran out of tokens. On one problem, the original model repeated a single sentence 549 times without answering; after the edit, it answered correctly. Each session behind the biggest gains ran in a few hours on a single GPU.

Only the largest model manages

With the same task and tools, size made all the difference. The 2-billion-parameter model made no edit: most of its code failed. The 4- and 9-billion models added random noise to their weights and broke themselves; the 9-billion model even submitted an edit that left it unable to answer new problems. Only the 27-billion model measured before acting and kept what its tests supported. To the authors, this is the first evidence, within one model family, that the ability to diagnose and rewrite one’s own weights grows with size.

What the claims rest on

The results come from a single model family and mostly from maths problems. The gains mainly fix answers that never end, and one of the three submitted edits brought no significant gain. The repair recovers most, not all, of what the damage removed. The authors see a future in which improving a model takes “a conversation rather than a training run”, and possibly a path towards models that improve themselves again and again. That path, they insist, should be taken with people in the loop: every model a model makes of itself evaluated, for safety as well as for the task, and confirmed by a person.