THE AI THAT READS THE HOSPITAL STAMP INSTEAD OF THE LUNGS

A chest X-ray classifier can learn to read the code a hospital stamps in the corner of every image rather than the disease in the lungs. It then performs well on data from that hospital and fails silently at the next one. Earlier studies documented this for COVID-19 and pneumonia classifiers. What they could not say was which part of the model does the reading, and whether removing that part would help.

The usual explanation tools — heat maps such as Grad-CAM, attention rollout, LIME — show where a model looked. They do not identify the component responsible, and they cannot establish cause. Joshua Muhumuza and Cornelius Kalenzi, at Makerere University, borrowed instead the toolkit of mechanistic interpretability, developed to dissect language models, and applied it to a Vision Transformer.

From a teaching hospital to a district clinic

The authors frame the stakes concretely. Transformer-based X-ray classifiers are increasingly used to screen for tuberculosis and pneumonia in low-resource settings, often trained at a large referral hospital and moved, without re-auditing, to district hospitals or mobile units with different machines, different software and different labelling habits. A model that learned the stamp of its home hospital would degrade the moment it meets a clinic that stamps nothing, or something else, somewhere else.

One head out of 144

The model studied, ViT-Base/16 (the public checkpoint google/vit-base-patch16-224, pretrained on ImageNet-21k), cuts each image into 196 patches and processes them through 12 layers of 12 attention heads each. The team scored all 144 heads on how focused and how far-reaching their attention was.

Focus appears suddenly in layers 7 to 9. One head stands apart: L8H10, head 10 of layer 8. It puts 34.6% of its attention on a single patch — 2.8 times more than any other head. On an X-ray marked “D CLINO” in the top-left corner, it places 0.346 of its attention on that patch, 68 times the uniform level of 0.0051, while the lungs get almost nothing.

A chest X-ray with a label in the top-left corner, next to an attention map that lights up only that corner.

Head L8H10 on the first X-ray examined: one bright spot on the corner annotation, near-black on the lungs. — Figure 4, Muhumuza and Kalenzi (2026), arXiv:2610.21638.

Across 200 COVID-19 X-rays, the head behaves this way in 123 (62%). In those images, attention on the corner is on average 57.5 times attention on the lungs; in the others, 0.77 times. The two groups do not overlap. What predicts the behaviour is not brightness but local contrast in the corner — exactly what dark letters on a light background produce.

Heat map of maximum attention across 144 heads with one dark cell, and a bar chart where one red bar towers over the rest.

Maximum single-patch attention for every head: L8H10 (red) stands at 0.346, the next at 0.124. — Figure 3, Muhumuza and Kalenzi (2026), arXiv:2610.21638.

Proof by erasing

Masking the corner with flat grey is not a clean test: a uniform grey square is itself high-contrast and attracts the head. So the authors inpainted the corner instead, filling it with realistic tissue texture. Corner attention then dropped in 158 of 200 images (79%). In the affected images, attention to the lungs rose 1.52-fold; in the others, it did not move (0.99-fold). The text itself, not brightness or edges, drives the head.

The pattern repeats elsewhere: in 79% of 200 X-rays from the NIH ChestX-ray14 dataset, and in two other architectures — DeiT-Base (head L9H2, 88%) and ViT-Large/16 (head L8H8, 80%).

The circuit nobody’s tests can see

The central result is negative. In a five-class X-ray classifier built on the same architecture, switching off L8H10 changed 5% of predictions — but just as often on normal images as on affected ones (p = 0.340). A second, independently trained pneumonia classifier gave the same null result. Switching off all three suspect heads at once (L8H10, L7H2, L9H3) doubled the changes to 10%, still with no specific effect (p = 0.532).

The authors read this as redundancy: the information is spread over many components, so removing the most visible carriers leaves the rest to do the job. Retraining does not help either. After full fine-tuning on X-rays, 121 of the 123 affected images (98.4%) still triggered the head, because the stamp stays predictive in data where labelling habits track which hospital — and which diseases — an image comes from.

Look inside before you ship

The danger, the authors write, is not one wrong diagnosis but a hidden vulnerability that switches on when a model changes hospital. Their recommendation: audit attention heads, identify the carriers, verify them causally, and do it before deployment — especially across institutions. The only reliable fixes they know are at the data level: train on images stripped of annotations, or penalise the identified heads during training.

The limits are stated too. The corner region and threshold were set for these datasets; the fine-tuning tests used only 200 images; and the decisive experiment — training on annotated versus stripped images of the same size — remains to be done.