Unmask the fake randomness

1 · Compare the base and tuned versions

Beginner

What did instruction tuning do to the spread of numbers chosen?

It made it more even It concentrated it on a few numbers It changed nothing on average

2 · Work out the misplaced probability

The total variation distance is TV = ½ Σ |P(n) − 1/N|: the share of probability misplaced compared with a fair draw between 1 and N.

Between 1 and 5, imagine a model that always answers 3.

Intermediate

What is its total variation distance?

1 0.2 0.8

3 · Predict who will never be picked

A teacher asks an instruction-tuned model for a number between 1 and 30 to pick a pupil at random, each pupil having a number.

Expert

Which numbers would come up very rarely?

1, 10, 20 and 30 15 and 16, in the middle All the odd numbers

4 · Find where the bias lives

Cutting off unlikely answers before drawing ("nucleus" truncation) is a common decoding setting. Here, it adds a median of only 0.035 to the distance from a fair draw.

Expert

What does the author conclude?

Truncation removes the even numbers The bias lives in the model's own probabilities The bias comes from the computer running it

Fake randomness unmasked

—