Entropy Lens – Word Surprisal Analyzer

Entropy Lens: Word Surprisal Analyzer

We carry a rough census or internal map of our own language. Nobody taught us the numbers, but we know that the is ordinary and extirpation is not, we expect one but not the other. Reading is partly the work of prediction; it is an elemental trait of human language, we are group members who came to be on the back of complex communication.

Information theory gives that feeling a unit. It is called surprisal and measured in bits.

The idea is simple. A word that appears frequently carries very little information, because you half expected it already. A word that appears rarely carries a lot. Surprisal is the formal way of saying how much: take the probability of a word, and the rarer it is, the more bits it carries. In this tool, the number of ‘the’ is about 4.6 bits, and the rarest words in the reference list reach about 25.5 bit.

Low surprisal is not simple writing and high surprisal is not good writing. It is a description of vocabulary, nothing more, and what a useful number looks like depends entirely on what you are writing and who is going to read it.

What it does not measure

The tool scores each word on its own, by how common it is in English generally. It does not read context. In “the cat sat on the mat,” a human reader finds mat highly predictable because of the five words in front of it, but this tool does not see that. It only knows that mat is a moderately common English word, and it would score it the same way in any sentence you put it in.

So what you are looking at is the rarity of your word choices, not the predictability of your sentences. Those are related, but they are not the same thing.

Where the probabilities come from

The reference is a list of 333,333 English word forms with frequencies drawn from roughly 588 billion words of web text, compiled by Peter Norvig from the Google Web Trillion Word Corpus. That is a much larger and much more modern sample than the classic reference corpora, which matters here: a 1960s corpus of edited American prose does not know nakshatra or fractal or ghazal, and a poetry site needs a reference that does.

It is web text, which means it is not curated. The frequencies reflect how English is actually used online, typos and all, rather than how anyone thinks it should be used.

A note on how the list is built: it contains no apostrophes or hyphens. So don’t is looked up as dont, and hyphenated compounds are checked first as a single joined word, then, failing that, scored as the sum of their parts. A compound genuinely carries more information than either half, so those can exceed the 25.5 bit ceiling.

Off-vocabulary words

Some words will not be in the list at all. Invented words, proper names, deep archaisms, and technical terms from small fields.

The tool does not guess at these. It marks them in violet, excludes them from the average, and reports what fraction of your text they represent as a separate number.

Assigning a made-up probability to a word you cannot measure does not produce a measurement; it produces a number that looks like one. And in a passage full of unusual words, those invented values would swamp the average and make it meaningless.

The off-vocabulary rate is worth reading on its own. A modern lyric might come in near zero. A passage of Bacon or Spenser will run higher, and that is a real fact about the distance between that English and this one.

How to read the results

Mean surprisal. The average across every word the tool could measure. Higher means your vocabulary runs consistently less common. Lower means you are working with more ordinary words. Neither is better. Technical writing tends to run high, dialogue tends to run low, and a poem can go either way on purpose.

Off-vocabulary. The share of your words that are not in the reference list, and therefore not in the average.

Most surprising and most predictable. The extremes of what was measured. The predictable end is usually a function word, which is not very interesting; the surprising end tends to be where the writing is doing something.

The surprisal map. Every word, colored by its value. Deep cyan for common, shifting through teal to gold as words get rarer, with violet for the ones that fall outside the reference. Tap or hover any word to see its number.

Read this one for shape rather than for individual values. A passage that is mostly cyan with a few gold words is doing something different from one that is gold throughout, and you can see that at a glance in a way the average will not tell you.

Surprisal distribution. The same values are sorted low to high. The slope is the point. A gentle rise means an even vocabulary. A long flat run with a sudden cliff at the end means most of the passage is ordinary, and a handful of words are carrying all the strangeness.

Repeated-context ambiguity

The last section measures something different, and the two numbers should not be compared.

Everything above compares your words against the outside world. This one ignores the outside world entirely and looks only inward, at word pairs inside your own text. It asks: when a particular word shows up more than once, is it followed by the same thing each time, or by something different?

Low values mean your repeated words fall into repeated grooves. Higher values mean the same word leads somewhere new each time.

It needs at least 300 words to say anything. Below that, almost every word pair in a text occurs exactly once, which makes the number a measure of how short the passage is rather than how it was written. Shorter passages will get a note instead of a figure.

Word pairs whose first word never recurs are left out of the calculation entirely, for the same reason. A word seen once is followed by exactly one thing, which tells you nothing.