1. Learning Objectives

By the end of this lesson, you will be able to:

  • Formulate topic modeling as a probabilistic generative process, deriving the latent Dirichlet allocation (LDA) model and its inference via collapsed Gibbs sampling.

  • Implement and evaluate LDA and non‑negative matrix factorization (NMF) for discovering latent themes in financial news and filings.

  • Extend LDA to dynamic topic models (DTM) that capture temporal evolution of topics.

  • Define event detection as a pattern matching or burst detection problem, using statistical tests for anomaly detection in word frequencies.

  • Combine topic modeling and event detection for real‑time monitoring of financial markets.

2. Topic Modeling: Latent Dirichlet Allocation (LDA)

2.1 Generative Process

LDA is a hierarchical Bayesian model. Assume we have a corpus of D documents, each document d contains N_d words, and there are K latent topics. Each topic k is a distribution over a fixed vocabulary V of words.

The generative process:

  1. For each topic k, draw a multinomial word distribution φ_k ~ Dirichlet(β), where β is a V‑dimensional vector of positive hyperparameters (usually symmetric, β = η).

  2. For each document d:

    • Draw a multinomial topic distribution θ_d ~ Dirichlet(α), where α is a K‑dimensional vector of positive hyperparameters.

    • For each word position i in document d:

      • Draw a topic assignment z_{d,i} ~ Multinomial(θ_d).

      • Draw a word w_{d,i} ~ Multinomial(φ_{z_{d,i}}).

The joint distribution of the observed words and latent variables is:

P(w,z,θ,ϕ∣α,β)=∏d=1D∏k=1KP(θd∣α)P(ϕk∣β)∏i=1NdP(zd,i∣θd)P(wd,i∣ϕzd,i)

2.2 Inference via Collapsed Gibbs Sampling

We cannot compute the posterior exactly; we approximate via Gibbs sampling. We integrate out (collapse) the Dirichlet variables θ and φ, leaving only the topic assignments z. The conditional distribution for assigning topic k to the current word w = t in document d, given all other assignments, is:

P(zd,i=k∣z−d,i,w)∝nd,k−d,i+αk∑k′(nd,k′−d,i+αk′)⋅nk,t−d,i+βt∑v(nk,v−d,i+βv)

where:

  • n_{d,k}^{-d,i} is the count of words in document d assigned to topic k, excluding the current word.

  • n_{k,t}^{-d,i} is the count of word t assigned to topic k, excluding the current word.

After sampling, we can estimate the topic‑word distributions:

ϕ^k,t=nk,t+βt∑v(nk,v+βv)

and document‑topic distributions:

θ^d,k=nd,k+αk∑k′(nd,k′+αk′)

These estimates give us interpretable topics (top words) and document topic proportions.

2.3 Choosing K and Evaluating Topics

We can use perplexity (lower is better) on held‑out data:

perplexity=exp⁡(−1Nheld-out∑w∈held-outlog⁡P(w∣trained model))

However, perplexity does not correlate well with human interpretability. A better metric is topic coherence:

Coherence(topic)=∑m=2M∑l=1m−1log⁡P(wm,wl)+ϵP(wl)

where the sum is over the top M words of a topic, and probabilities are estimated from a reference corpus. Higher coherence indicates more interpretable topics.

2.4 Non‑negative Matrix Factorization (NMF)

NMF is an algebraic alternative. We factorise the document‑term matrix V (D × V) into W (D × K) and H (K × V), with all non‑negative entries, such that V ≈ W H. The optimisation problem is:

min⁡W≥0,H≥0∥V−WH∥F2

where ||.||_F is the Frobenius norm. The rows of H are topics, and W gives document topic weights. NMF often yields sparser and more disjoint topics than LDA, which can be beneficial for short texts.

3. Dynamic Topic Models (DTM)

Financial topics evolve over time. DTM extends LDA by allowing topic‑word distributions to change smoothly over time. The generative process assumes each time slice has its own topic‑word distribution φ_{t,k}, and these distributions are linked via a state‑space model (e.g., a Gaussian random walk) on the logistic‑normal parameters.

Mathematically, let η_{t,k} be the natural parameters of the multinomial distribution for topic k at time t. We impose:

ηt,k∼N(ηt−1,k,σ2I)

The posterior inference is done via variational methods or MCMC, resulting in topic evolution over time.

Application: Track how the “inflation” topic changes before and after central bank announcements.

4. Event Detection from News Streams

4.1 Pattern‑Based Detection

Use regular expressions or dependency parsing to identify event mentions. For example, a merger event: pattern = (Company) + (acquires|buys|takes over) + (Company). This is high‑precision but low‑recall.

4.2 Burst Detection

An event often causes a sudden increase in the frequency of certain words or phrases. Kleinberg’s burst detection models word occurrence as a Poisson process with two states: normal (low rate) and burst (high rate). We estimate the likelihood ratio:

LR=∏t∈intervale−λburstλburstctct!∏t∈intervale−λnormalλnormalctct!

where c_t is the word count at time t. We declare a burst if LR exceeds a threshold. This is equivalent to a likelihood ratio test.

A more sophisticated approach is to use a Bayesian change‑point detection model (e.g., using a hidden Markov model) to identify time points where the rate changes significantly.

4.3 Unsupervised Event Clustering

We can group news articles into events by clustering them based on similarity (e.g., using sentence‑BERT embeddings). Each cluster corresponds to a distinct event. We then summarise the cluster by extracting top keywords and named entities.

Algorithm:

  1. Segment news into time windows (e.g., hours).

  2. Compute document embeddings.

  3. Apply a clustering algorithm (e.g., DBSCAN) with a similarity threshold.

  4. For each cluster, extract the most frequent named entities (companies, locations) to label the event.

4.4 Supervised Event Classification

If we have labelled event types (e.g., M&A, earnings, dividend), we can train a classifier on textual features (bag‑of‑words, entity pairs). This is a multi‑class classification problem.

5. Integration: Topic + Event for Monitoring

We can track topic proportions over time and detect anomalies. For example, if the “inflation” topic proportion spikes, we can trigger an alert. Similarly, we can monitor entities associated with a topic to identify which companies are most discussed.

Quantitative approach: For a given topic k, compute its daily proportion θ_{d,k}. Use a moving average and control limits (e.g., 3σ) to flag abnormal days.

6. Evaluation of Event Detection

  • Precision/Recall: based on a ground‑truth set of known events.

  • Time‑to‑detection: how quickly an event is detected after its occurrence.

  • F1 is common, but in finance, false positives can be costly; precision is often prioritised.

7. Summary for the AI Practitioner

  • LDA and NMF are fundamental topic models; LDA is probabilistic and generative, NMF is algebraic.

  • Topic coherence is the standard metric for evaluating interpretability.

  • Dynamic topic models capture temporal evolution, crucial for financial narratives.

  • Event detection can be pattern‑based, burst‑based, or clustering‑based; each has trade‑offs.

  • Combining topics and events provides a comprehensive view of market‑moving information.

8. References

  1. Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet allocation. JMLR.

  2. Lee, D. D., & Seung, H. S. (1999). Learning the parts of objects by non‑negative matrix factorization. Nature.

  3. Blei, D. M., & Lafferty, J. D. (2006). Dynamic topic models. ICML.

  4. Kleinberg, J. (2003). Bursty and hierarchical structure in streams. KDD.

  5. Röder, M., Both, A., & Hinneburg, A. (2015). Exploring the space of topic coherence measures. WSDM.

  6. Dredze, M., et al. (2016). Event detection in news streams. Survey.

 

 
 
PreviousNext