The Redundancy of Printed English

Guessing the Next Letter With Betty Shannon at the Kitchen Table

Sitting at a table in late 1950 with a stack of English books and a pad of paper, Bell Laboratories numerical analyst Mary Elizabeth Moore Shannon tried to guess a hidden sentence one letter at a time. Claude and Betty had married in March 1949 after meeting at Bell Labs in New Jersey. There Betty worked as a mathematician in the network-theory group. In his 1948 paper, Shannon had calculated the entropy of artificial Markov text using published letter, digram, and trigram frequency tables. Yet nobody had ever tabulated the frequencies of ten-letter or hundred-letter sequences in English. A table of $27^{100}$ possible strings far exceeds the number of atoms in the observable universe.

To measure the true information rate and redundancy of natural English across long spans of context, Shannon found a clever shortcut. In his January 1951 Bell System Technical Journal paper, Prediction and Entropy of Printed English, he noted that every fluent human reader already carries an internal statistical model of the language inside their brain. Instead of counting trillions of n-grams on punched cards, he could extract the conditional probabilities directly. He turned his wife into an optimal predictor in a letter-guessing game.

Shannon selected an unfamiliar passage at random from a book—such as Jefferson the Virginian by Dumas Malone. He simplified the text into a twenty-seven-symbol alphabet consisting of the twenty-six capital letters plus a blank space. Betty had to guess the very first character of the hidden passage without any clue. If she guessed right (T, for example) on her first try, Claude wrote 1 beneath the letter and moved to the second character. If she guessed wrong, he told her to keep guessing until she hit the true letter. At each position he recorded the exact number of attempts (1 through 27) required.

Across passages analyzed up to one hundred characters of preceding context, the guessing ledger revealed a dramatic collapse in uncertainty:

• * Zero context ($N = 0$): With all twenty-seven symbols treated equally, each character carries $\log_2(27) = 4.755$ bits of potential capacity.
• * Single-letter frequencies ($N = 1$): Knowing only that SPACE ($18.7\%$) and E ($10.7\%$) occur far more often than Z or Q drops the entropy $F_1$ to 4.03 bits per character.
• * Short-range digrams and trigrams ($N = 2, 3$): Knowing the immediately preceding one or two letters (TH almost always precedes a vowel or R) lowers the uncertainty to $F_2 = 3.32$ bits and $F_3 = 3.10$ bits per letter.
• * Full hundred-letter human prediction ($N = 100$): Armed with grammar, idioms, and subject context across a full sentence, Betty guessed sixty-nine percent of all characters correctly on her very first try (1), driving the upper and lower entropy bounds down to roughly 0.6 to 1.3 bits per letter.

How did Shannon convert a sequence of human guess counts (1, 1, 2, 1, 5, 1, 1...) into rigorous mathematical entropy bounds? He noticed that the original text and the sequence of guess numbers contain the exact same information. An ideal predictor orders all twenty-seven letters from most likely to least likely at every step. Therefore, an observer given only the string of guess counts (1, 1, 2...) and an identical copy of the predictor's probability tables could reconstruct the original English sentence letter for letter.

Because nearly seventy percent of the numbers in Betty's score sheet were 1s, the entropy of the score sheet was tiny. It clustered around one bit per character compared to the 4.76 bits of raw twenty-seven-symbol capacity. Written English is therefore roughly seventy-five percent redundant. That seventy-five percent redundancy in written English explains why human readers can decipher smudged telegrams or mistyped text messages, and why lossless data-compression algorithms from Huffman coding to Lempel-Ziv can shrink English text files to roughly one-quarter of their raw ASCII size.