The goal
- Learn semantic structure from a large text corpus
30:21. - Concretely, over 20 years of New York Times articles, automatically discover: what topics exist (politics, sports, science), which documents talk about which topic, and which words are used when discussing each topic
30:21,31:05. - This is an unsupervised task, so it is specified through a probabilistic model — those models are the topic models
31:32,31:57.
Why assumptions are unavoidable
Generating a genuine document is hard even for a human, and grammar structure makes it far harder for a program — so no realistic model actually generates documents. Simplifying assumptions must come first 31:57, 32:36, 33:11.
Prior work that supplied them
| Model | Authors | Reference |
|---|---|---|
| pLSI (first model) | Hofmann; concurrently Papadimitriou et al. | 33:55 |
| Latent Dirichlet Allocation (LDA) | David Blei, Andrew Ng, Michael Jordan | 33:55 |
| Hundreds to thousands of follow-up papers | — | 33:55 |
The three assumptions
1. Bag of words
- Word ordering can be discarded; throw all words of an article into a bag and learn from that
34:26,35:03. - Demonstration: a New York Times headline is randomly permuted ("with preparations, hotels, rise, Ebola") and the audience can still tell what the article is about
35:24. - Payoff: word generation no longer has to respect grammar — words can be generated randomly
35:58.
2. A topic is a probability distribution over words
- Definition: knowing a word came from a topic gives a distribution over what that word is
36:17. - Example: topic food emits the word banana with ~1% probability
36:59. - Answering a student: yes, words are drawn independently conditioned on their topic. Real words have richer correlations, but this captures the first-order effect and works well in practice
36:59,37:20.