1. Learning Objectives

By the end of this lesson, you will be able to:

• Formulate named entity recognition (NER) as a sequence labelling problem and derive the mathematical foundations of conditional random fields (CRFs) and neural sequence models.

• Implement feature‑based (CRF) and deep learning (BiLSTM‑CRF, Transformer) approaches for financial NER, focusing on the underlying probabilistic models.

• Apply relation extraction techniques using distant supervision and pattern‑based methods, and understand the statistical models for relation classification.

• Evaluate NER and relation extraction using entity‑level precision, recall, F1, and relation‑specific metrics.

• Build a financial knowledge graph by integrating extracted entities and relations, and discuss graph‑based reasoning.

---

2. Named Entity Recognition as Sequence Labelling

2.1 Problem Definition

Given a sequence of tokens x = (x₁, x₂, …, x_T), we assign a label y_t from a tag set 𝒴 to each token. The most common tag scheme is BIO (Begin‑Inside‑Outside) with entity types (e.g., B‑PER, I‑PER, B‑ORG, I‑ORG, O). The goal is to find the most probable label sequence y given x:

y∗=arg⁡max⁡y∈YT P(y∣x)y∗=y∈YTargmax​ P(y∣x)

This is a structured prediction problem because the labels are not independent; they have dependencies (e.g., an I‑ORG must follow a B‑ORG).

---

3. Conditional Random Fields (CRFs) for NER

3.1 Undirected Graphical Model

A CRF defines a conditional probability distribution over label sequences given the observation sequence:

P(y∣x)=1Z(x)exp⁡(∑t=1T∑kλkfk(yt−1,yt,x,t))P(y∣x)=Z(x)1​exp(∑t=1T​∑k​λk​fk​(yt−1​,yt​,x,t))

where:

• f_k are feature functions that can depend on the previous label, current label, the entire input sequence, and the position t. They are often binary indicators (e.g., “the current word is ‘Apple’ and the current label is B‑ORG”).

• λ_k are learned weights (parameters).

• Z(x) is the partition function, summing over all possible label sequences to normalise the distribution:

Z(x)=∑y′exp⁡(∑t∑kλkfk(yt−1′,yt′,x,t))Z(x)=∑y′​exp(∑t​∑k​λk​fk​(yt−1′​,yt′​,x,t))

3.2 Parameter Estimation (Training)

Given a training set of N labelled sequences {(xⁱ, yⁱ)}, we maximise the log‑likelihood with L2 regularisation (to prevent overfitting):

L(λ)=∑i=1Nlog⁡P(yi∣xi)−∑kλk22σ2L(λ)=∑i=1N​logP(yi∣xi)−∑k​2σ2λk2​​

The gradient with respect to λ_k is:

∂L∂λk=∑i=1N∑tfk(yt−1i,yti,xi,t)−∑i=1N∑tEP(y′∣xi)[fk(yt−1′,yt′,xi,t)]+λkσ2∂λk​∂L​=∑i=1N​∑t​fk​(yt−1i​,yti​,xi,t)−∑i=1N​∑t​EP(y′∣xi)​[fk​(yt−1′​,yt′​,xi,t)]+σ2λk​​

The expectation is over the conditional distribution, which requires inference (e.g., forward‑backward algorithm) to compute marginal probabilities. This makes CRF training more expensive than simple classifiers but yields better structured predictions.

3.3 Inference (Decoding)

To find the most likely label sequence for a new input x, we use the Viterbi algorithm, which is a dynamic programming algorithm that computes the best score for each position and label, and backtracks to get the sequence.

Define δ_t(j) as the maximum score (unnormalised log‑probability) of a label sequence ending with label j at position t. The recursion:

δt(j)=max⁡i{δt−1(i)+∑kλkfk(i,j,x,t)}δt​(j)=maxi​{δt−1​(i)+∑k​λk​fk​(i,j,x,t)}

with base case δ₁(j) = Σ_k λ_k f_k(start, j, x, 1). The final best path is obtained by backtracking from the maximum δ_T.

3.4 Feature Engineering for Financial NER

For financial text, we design features that capture:

• Word identity and shape: exact word, lowercase, prefix/suffix (2‑4 chars), presence of digits, uppercase, punctuation.

• Contextual features: preceding and following words and their features.

• Domain gazetteers: lists of company names, stock tickers (e.g., AAPL), currency symbols, and financial terms.

• Orthographic patterns: regular expressions for monetary amounts, percentages, dates, and ticker symbols.

Mathematically, each feature function f_k is a conjunction of conditions; e.g., f_k = 1 if word_t = “Apple” and y_t = B‑ORG, else 0. The CRF learns weights to indicate how reliable these cues are.

---

4. Neural NER: BiLSTM‑CRF

4.1 Architecture Overview

Neural models replace hand‑crafted features with learned representations. The typical architecture is:

1. Embedding layer: maps each token to a dense vector e_t = [w_t; char_t], where w_t is a pre‑trained word embedding (e.g., GloVe) and char_t is a character‑level representation (e.g., from a CNN or LSTM). For financial domain, we can use FinBERT embeddings or other contextual embeddings.

2. BiLSTM layer: processes the sequence in both directions, producing hidden states:

   h⃗t=LSTM(et,h⃗t−1)ht​=LSTM(et​,ht−1​)
   h←t=LSTM(et,h←t+1)ht​=LSTM(et​,ht+1​)
   ht=[h⃗t;h←t]∈R2dht​=[ht​;ht​]∈R2d

   The BiLSTM captures both left and right context.

3. Projection layer: maps each h_t to a score vector s_t ∈ ℝ^{|𝒴|} via a linear transformation:

   st=Wht+bst​=Wht​+b

   where W ∈ ℝ^{|𝒴| × 2d}, b ∈ ℝ^{|𝒴|}.

4. CRF layer: takes the scores s_t (emission scores) and learns transition scores A_{i,j} for moving from label i to label j. The score of a label sequence y is:

   score(x,y)=∑t=1Tst[yt]+∑t=2TAyt−1,ytscore(x,y)=∑t=1T​st​[yt​]+∑t=2T​Ayt−1​,yt​​

   The conditional probability is:

   P(y∣x)=exp⁡(score(x,y))∑y′exp⁡(score(x,y′))P(y∣x)=∑y′​exp(score(x,y′))exp(score(x,y))​

   Training minimises the negative log‑likelihood. Decoding uses Viterbi with the learned transition matrix A.

4.2 Why CRF after BiLSTM?

The BiLSTM alone, with independent classification per token, ignores label dependencies (e.g., I‑ORG after B‑ORG). The CRF layer enforces valid label transitions, improving consistency. The transition matrix A can be learned from data, capturing constraints like “I‑PER must follow B‑PER”.

4.3 Domain Adaptation with FinBERT

Instead of using static embeddings, we can fine‑tune a pre‑trained transformer like FinBERT for token classification. The architecture is:

• Replace the BiLSTM with the transformer encoder (e.g., 12 layers).

• Use the token‑level hidden states from the last layer as h_t.

• Add a linear classification head (or CRF) on top.

The training objective is the same as the BiLSTM‑CRF. Fine‑tuning on a financial NER dataset (e.g., FinNER) yields state‑of‑the‑art performance.

---

5. Relation Extraction (RE)

5.1 Problem Definition

Given a sentence and two entity mentions e₁ and e₂ (with spans), we want to predict a relation r from a predefined set ℛ (e.g., acquires, subsidiary_of, employs). We can also predict no relation (NA).

5.2 Supervised Relation Classification

We treat it as a multi‑class classification problem. Features include:

• Words between the entities.

• Dependency path between the entities (shortest path in the dependency tree).

• Entity types (e.g., ORG, PER, MONEY).

• Bag‑of‑words around the entities.

Neural models use a Siamese or cross‑attention architecture: encode the sentence with BERT, take the representations of the entity spans (e.g., concatenate start and end hidden states), and feed into a classifier.

Mathematically, we want to estimate:

P(r∣sentence,e1,e2)=softmax(W⋅representation+b)P(r∣sentence,e1​,e2​)=softmax(W⋅representation+b)

5.3 Distant Supervision

Because labelled relations are scarce, we use distant supervision: align entities from a knowledge base (e.g., Wikidata) with mentions in text. If a sentence mentions two entities that have a known relation in the KB, we assume the sentence expresses that relation (heuristic). This generates noisy labels. We can then train a relation classifier using this weak supervision, often with multi‑instance learning to handle noise.

5.4 Evaluation Metrics for RE

• Precision, Recall, F1 per relation, aggregated via micro or macro averaging.

• For relation extraction, we often use held‑out evaluation against a manually annotated test set.

---

6. Integrating NER and RE for Knowledge Graph Construction

Once we extract entities and relations, we can populate a knowledge graph (KG). The KG consists of triples (h, r, t) where h and t are entities, and r is a relation. We can then use graph embedding techniques (TransE, etc.) to learn vector representations for nodes and relations, enabling link prediction and reasoning.

---

7. Summary for the AI Practitioner

• NER is a sequence labelling task; CRF is a powerful probabilistic model that captures label dependencies.

• Neural models (BiLSTM‑CRF, Transformers) outperform traditional feature‑based CRFs by learning representations automatically.

• Relation extraction can be supervised or distantly supervised; it is essential for building structured knowledge from text.

• Evaluation must consider both boundary and type accuracy for NER, and relation classification accuracy for RE.

• The combination of NER and RE enables the construction of financial knowledge graphs that power downstream analytics.

---

8. References

1. Lafferty, J., McCallum, A., & Pereira, F. (2001). Conditional random fields: Probabilistic models for segmenting and labeling sequence data. ICML.

2. Huang, Z., et al. (2015). Bidirectional LSTM‑CRF models for sequence tagging. arXiv.

3. Lample, G., et al. (2016). Neural architectures for named entity recognition. NAACL.

4. Riedel, S., et al. (2010). Modeling relations and their mentions without labeled text. ECML-PKDD.

5. Alvarado, J., et al. (2019). FinBERT‑NER: A transformer‑based model for financial named entity recognition. arXiv.