The simplifying regime: n → ∞

  • The lecture does not attempt the fully general learning algorithm; it restricts to the easy case where document length goes to infinity 49:36, 50:08.
  • Openly acknowledged as unrealistic in practice; papers exist for finite or small document length 50:08, 50:44.
  • What it buys you: every word appears more than once, so within-document word probabilities can be estimated by simple counting — occurrences of the word ÷ document length 50:44, 51:10.
  • Accuracy caveat: the length must be comparable to the vocabulary size for this estimate to be good 51:43.

The document–word matrix M

  • Collect those estimated probabilities into a matrix M of shape n × m: n vocabulary words, m documents 52:09, 52:39.
  • Each column is a document, holding P(word | document) 52:39.
  • Under bag-of-words, this is essentially all the information available, since ordering doesn't matter 53:09.

Deriving the factorization

By the two-step generative process, for word v in document d:

P(zᵢ = v) = Σⱼ₌₁ᵏ P(tᵢ = j | doc) · P(zᵢ = v | tᵢ = j) 53:40, 54:09, 54:44.

  • The first factor is exactly the document's topic proportions w 54:44.
  • The second factor is exactly an entry A[v, j] of the topic matrix 54:44, 55:22.

That sum-of-products is the matrix-product rule, so:

M = A · W, with A the n × k topic matrix and W the k × m matrix whose columns are each document's topic proportions 55:22, 55:50, 56:23.

flowchart LR
    M["M (n x m)<br/>document–word<br/>OBSERVED"] --> EQ["="]
    EQ --> A["A (n x k)<br/>topic matrix<br/>UNKNOWN — the goal"]
    A --> X["x"]
    X --> W["W (k x m)<br/>topic proportions<br/>UNKNOWN"]

Why this matters

The identity links the observable M to the unknown parameter A. The learning problem becomes: given M, find a decomposition A · W where both factors are much smaller than M 56:48.