The topic matrix A

The most important parameter of a topic model; it recurs through the whole rest of the lecture 38:41, 39:28.

  • Shape: n × k 39:28.
    • n = vocabulary size — thousands, tens of thousands, or more 39:57.
    • k = number of topics — much smaller, think 100 or 200 39:57, 40:21.
  • Each column is a topic; entry A[banana, food] = 1% means a word drawn from topic food is banana with 1% probability 40:21, 40:57.
  • Topic names cannot be generated automatically — the algorithm produces distributions, not labels 40:21.
  • Constraints: all entries non-negative, and each column sums to 1 so it is a valid distribution over words 41:28.

General recipe for generating a document

Shared by essentially all topic models 42:46:

  1. Pick the document length 42:46.
  2. Pick the topic proportions w = (w₁ … w_k), with wᵢ ≥ 0 and Σ wᵢ = 1 42:46, 43:41.
  3. For each word position i: draw a topic tᵢ ∈ [k] according to w, then draw the word zᵢ ∈ [n] from column A[·, tᵢ] 44:13, 45:29.
    • Example: w = (0.7, 0.3, 0, …) gives tᵢ = 1 with 70% probability and tᵢ = 2 otherwise 44:56.
flowchart LR
    L[Step 1: pick length] --> W[Step 2: pick topic proportions w]
    W --> T["Step 3a: draw topic t_i ~ w"]
    T --> Z["Step 3b: draw word z_i ~ A[·, t_i]"]
    Z --> T

Step 1 and Step 2 are where models differ

  • Length is not really a modelling target: an article is ~300 words, you simply observe it from data, so most topic models ignore it. Here it is assumed fixed and known 46:34, 47:02.
  • Step 2 is the real dividing line between topic models — the recipe above isn't yet a model because the distribution over w is unspecified 46:05, 47:02.
ModelHow w is chosenReference
pLSIw is a basis vector (one entry 1, rest 0) — chosen with probability πᵢ; the whole document is about a single topic47:02, 47:50, 48:14
LDAw ~ Dirichlet(α)48:14
Many later modelsVarious other distributions49:09

Why the Dirichlet: partly to simplify their algorithm, but the real intuition is that it produces vectors that are not just basis vectors (documents can mix topics) while still tending to be sparse — matching the intuition that no document covers all 100+ topics 48:14, 48:46, 49:09.

This lecture's stance

Ignore all these differences. Treat the w's as part of the parameters — arbitrary, subject only to a few reasonable assumptions made later 49:36.