WHEN AI COPIES AI, SOMEONE HAS TO KEEP LEARNING
Train a generative model on the output of an earlier model, and it degrades. Over successive generations, the rarest content disappears before the common, and what the model produces narrows. This model collapse is now well studied — and so are its cures. All the remedies that work bring in something from outside the loop: fresh human data, retained real data, verifiers, human curation.
Nam Le, a computer scientist at the University of Vermont, points out what those analyses take for granted. They treat fresh data as a fixed archive, available in whatever quantity is needed, and they assume the world being described stands still. Neither is true: fresh text is written by people, who increasingly ask the models instead of finding things out; and the world keeps changing.
An old paradox
Both issues sit at the heart of an older theory. In 1988, Rogers described what is now called Rogers’ paradox: in a population where copying spreads because it is individually cheaper than learning, everyone ends up no better off than in a population that never copied. Copies go stale, and someone must keep paying to learn.
Later work in cultural evolution refined the idea. A critical learner copies, checks the copy against the world and learns for itself if it fails; such learners raise everyone’s payoff. A conformist learner copies what is common, which is cheap but erodes rare variants.
Le’s argument is that training a model on another model’s output is social learning in exactly this sense, so the theory should apply. In its simplest form, it gives a one-line rule for how much knowledge a lineage keeps when a share of its training data is fresh.
Darwin and Moby-Dick, recopied
To test it, Le trained lineages of small language models on two books from Project Gutenberg, Darwin’s On the Origin of Species and Melville’s Moby-Dick. Each generation learned from text produced by the previous one, mixed with some fresh text from the book.
Fed only their own output, word-level models lost four in five rare words in twelve generations. A sample from the starting generation still reads “i cannot see that a genus become fitted for most justly celebrated elder huber”; by the twelfth, “struggle of the on that may inheritable detected i hybrids, kept.”
The striking part is the prediction. Le fitted the rule’s single parameter — the chance that a copied piece of knowledge is lost at each generation — on the fully self-fed lineage alone, then used it, unchanged, to predict lineages with 10%, 30% or 50% fresh data. It matched how many rare words they kept to within 3 percentage points. With a second class of knowledge, fragile versus durable, the rule also held for a small transformer and for a pretrained language model, OPT-125m.
When the world moves
Next, the “world” itself was made to change: a window sliding through the two books, so new vocabulary kept arriving and old words kept leaving. The faster it moved, the more fresh data a self-copying lineage needed. To keep 80% of what a fully fresh corpus would cover, the fresh share had to rise from about 30% to about 45%. Fast-changing domains, the paper predicts, will be the first to degrade as synthetic text accumulates.
The pretrained model behaved differently: its knowledge held up whether the world moved or not. Le’s reading is that its pretrained weights — the internal settings it had learned from a large corpus before the experiment —, inherited unchanged every generation, are knowledge that never passes through the copying chain. With a caveat: the moving text here was text the model had already seen during pretraining.
People in the loop
Then comes the part that gives the paper its title. Le built a model in which people can either find a fact out for themselves, at a cost, or ask the AI for free and be right only as often as the AI is. What people write becomes the next model’s training data, along with some text the model writes itself.
The equilibrium is stark:
- the AI ends up right just often enough that asking it pays exactly as much as finding out for yourself, once the cost of finding out is counted — whatever its size or architecture;
- people as a whole are no better off than without the AI: Rogers’ paradox, for humans and machines together;
- the more of the training text the models write themselves, the more people must keep learning on their own — twice as many when half the text is model-written. Fresh data, Le concludes, are paid for in human effort.
There is a way out. If people check the AI’s answer against the world and relearn when it is wrong, the paradox breaks: in the simulation, the average payoff — what a person gains per fact, 1 for a right answer minus what finding out cost — settles at 0.93, against 0.70 without the AI. Tools that make verification cheap are, in this view, not a convenience but what keeps the whole system from collapsing.
Filters that make it worse
The same lens sorts the filters used to clean synthetic data. Keeping the samples a model finds most plausible is conformist copying, and in the experiments it sped up the loss of rare words: the pretrained model kept 5% of them, against 57% when half the samples were kept at random. Only filters that check text against the world act like critical learners. All of this was shown with small models, the largest with 125 million parameters, and Le names testing it at scale as the key open question.
