GPT-5 SPOTS ITS OWN WRITING BETTER THAN WE DO

In the original Turing test, a human interrogator exchanges written messages and tries to tell a man from a woman. At some point the man is replaced by a machine. If the interrogator still cannot tell the machine from the woman, the machine is said to have passed. Turing meant it as a philosophical thought experiment, yet for decades it has served as a benchmark for artificial intelligence — and recent claims that chatbots such as ChatGPT have finally passed it have revived an old question: what does passing actually prove?

Two ways to fool a judge

A team led by Cristina Becchio at the University Medical Center Hamburg-Eppendorf points out that a fooled interrogator can mean two very different things.

• A failure of encoding. The machine’s answers overlap so closely with human ones that the text no longer carries any clue about its source. No observer, however skilled, could beat chance. That would be a strong result.
• A failure of readout. The clues are in the text, but the interrogator does not pick them up. Another judge — an artificial one, say — might succeed. The test then says more about the judge than about the machine.

The classic test lumps the two together. The authors add a third level, meta-readout: can a judge correctly say which clues guided its own decision?

Strange stories, two answers

To separate these levels, the team inverted the Turing test: humans and AI systems both took the interrogator’s seat. The material came from the Strange Stories test, a set of eight short social scenarios — deception, irony, persuasion, misunderstanding — each ending with a question about what a character thinks or why they act. Humans and GPT models had answered them, and only correct answers were kept, so a judge could not unmask the machine through faulty reasoning.

Each interrogator saw a story and two answers to it, one human and one from the AI, then had to pick either the human one or the AI one, rate their confidence and explain their choice in writing. In the first series of studies, answers from GPT-4 were judged by humans and by GPT-4; in the second, answers from GPT-5 were judged by humans and by GPT-5. In all, 3,824 people took part through the Prolific platform, each giving a single judgement, matched by as many runs of the models.

Performance was scored with a measure called sensitivity, d′: zero means guessing, higher means better discrimination, and a negative value means the judge is systematically wrong.

People slip, GPT-5 sharpens

Against GPT-4, humans did reasonably well, with d′ of 1.36 when asked to find the AI answer and 1.10 when asked to find the human one. GPT-4 as a judge was weaker, and in the “find the human” task its score was negative, at −1.11: it pointed at the wrong answer more often than the right one.

Against GPT-5, the humans in the second series slipped to 0.77 and 0.27 — still above chance, but barely so in the human task. GPT-5 as a judge did the opposite, reaching 2.09 and 2.01, well above the humans in both tasks.

The clues never vanished

Was the human slide a failure of encoding? To check, the researchers measured 12 features of every answer — length, detail, typos, redundancy, punctuation, readability, analytical or conversational tone, and others — and trained a simple statistical model to guess the source from them. It picked the right source 98.3% of the time for GPT-4 answers (a “balanced accuracy” of 0.983, which averages its success over both answer orders) and still 92.5% for GPT-5. GPT-5 sounds more human, but the information is still there.

What changed was which clues carry it. GPT-4 gave itself away mainly through longer answers, fewer typos and more repetition. With GPT-5 the length gap shrank sharply, while denser punctuation and fewer typos became the telling signs. Human judges kept leaning on length — a habit, the authors suggest, calibrated on earlier chatbots. People who reported using chatbots often had partly updated: in the “find the AI” task they weighted punctuation and typos correctly. GPT-5, meanwhile, linked dense punctuation and redundancy to AI and typos to humans, matching the real differences almost perfectly.

Right clues, wrong directions

The last level compared what judges wrote in their explanations with what their choices actually relied on. Here the ranking flips. Humans’ explanations matched their choices comparatively well (an alignment of about 0.63 in the GPT-4 series, on a scale where 1 means that explanations and choices point the same way). GPT-5 named many of the features it really used, but got the direction of some of them wrong: in both tasks, it said a more analytical tone, higher readability and more detail pointed to AI, whereas its choices linked all three to human answers. Its explanations matched its actual weighting far less well than those of humans did.

The authors do not read this as pure invention after the fact, since GPT-5’s explanations remained above chance on both measures; the mismatch is confined to a few features. They warn that such a localised gap can create an illusion of alignment for users who rely on a model’s explanations.

The study covers two OpenAI models, each judging answers from its own generation, on eight short stories. On that alignment scale, GPT-5 scored 0.297 in one task and 0.296 in the other, significantly below the humans — with the same three features reversed both times.