Find Themes in Text | Topic Analysis Tool

How to Find Themes in Text: The Latent Dirichlet Allocation

What is the theme or topic of a large corpus, or body of work? Finding themes in text is a common academic question, but at a large scale, it is difficult to analyze multiple categories. Luckily, we can find the answer by looking at the question mathematically, not by sorting the individual words themselves (which would be difficult and lead to errors in classification), but by examining them in a statistical relationship with one another.

How do we do this? The best way to conceptualize the problem is to think of standing in a room with some boxes. Now, take the text and cut it into individual words, and then, here’s the trick, treat them like units that make no sense word-wise, but dump them randomly into boxes. The point is that the boxes are filled with units that have no real meaning. Our tool, which only analyzes units statistically, remembers where that came from in the text.

Use the word ‘river’ as a data point. Our text talks a lot about rivers, so the tool sees it quite often. Let’s say water is another data point. Water and rivers are often together, so our tool is going to make the decision to put them in the same box, and astonishingly, it makes the same calculations millions of times.

Our tool does another thing to improve text sorting accuracy. Topics are sorted by association strength, but they can also be sorted by novelty. In our text above, ‘river’ and ‘water’ go together, but perhaps there is a section on Christmas trees. In all likelihood, river and water won’t go with Christmas, but the Christmas tree will be associated perhaps with gifts, family, Santa, and so on. Our tool would identify Christmas data points as a topic and treat associated words as a unit.

The math is like this.

Two numbers do the work, and the tool shows you both.

The first is plain frequency: how many times a word lands in a topic box. That is the cyan bar.

The second is lift, and lift measures how concentrated a word is in one box compared to the text as a whole.

lift = the word’s rate inside this topic ÷ the word’s rate across the whole text

Read it as a ratio. If a word is sprinkled evenly everywhere, its rate inside a topic matches its rate everywhere else, and its lift sits near 1. If a word crowds into one box and stays out of the others, its rate inside runs well above its rate overall, and the lift climbs. High lift means the word has picked a side.

Say the word “river” makes up 3 percent of the words inside topic 3, but only half a percent of the whole text. Its lift is 3 divided by 0.5, which is 6. River is six times more concentrated in that topic than in the text at large. A word like “water,” which sits at 1 percent within the topic and 0.9 percent overall, has a lift near 1.1. Barely distinctive. It is everywhere, so it belongs to no single topic.

Then the tool does one more thing: it does not rank words solely by lift. Lift has a blind spot: a word that appears in only one topic scores the same distinctiveness whether it appears 20 times or 2, because “only here” is “only here” no matter the count. A rare word and a workhorse word tie. To break that tie, the tool blends lift and frequency into a single relevance score, so a word that is both distinctive and does real work rises above one that is distinctive but barely present. The topic label is drawn from the top of that blended ranking: the words that are at once common in the box and unique to it.

The tool assumes that there will be order. It makes hundreds of decisions to create word associations and frequency judgments.

How to Use the Tool

Paste your text into the input box, the longer the better. A few paragraphs will work, but a full chapter works better, the more text the more accurate the statistics of words

Set the number of topic boxes, five is a good number. Too many and it can return scattered word results.

Put in any words you want excluded. We have it so common English words are already filtered out, but if a word dominates your text without being meaningful (like a character name or a repeated filler), add it to the excluded field.

Hit Run. The tool will sort every word by occurrence and return your topics sorted two way, one by frequency (cyan bar) and one by lift (gold bar). The auto-generated label comes from the highest-lift words, the words most unique in that topic.