1. Learning Objectives

By the end of this lesson, you will be able to:

• Understand the architecture of open‑domain and closed‑domain question answering (QA) systems in finance.

• Implement dense passage retrieval (DPR) and BM25 for retrieving relevant documents from large financial corpora.

• Apply fine‑tuned transformer models (e.g., FinBERT‑QA, RoBERTa‑based) for extractive and abstractive QA.

• Evaluate QA systems using exact match (EM) and F1 scores on financial benchmarks (e.g., FinQA, ConvFinQA).

• Handle numerical reasoning and table‑based questions typical in financial reports.

• Build a end‑to‑end QA pipeline for earnings call transcripts or SEC filings.

2. Question Answering in Finance: Overview

Why QA matters:

• Financial analysts spend significant time searching for specific information (e.g., “What was Apple’s revenue growth in Q3 2023?”).

• Automated QA can provide instant answers from 10‑Ks, earnings transcripts, and news.

• Numerical reasoning is essential – many questions involve arithmetic (e.g., “By how much did net income increase?”).

Types of questions:

3. Retrieval‑Augmented QA Pipeline

A typical QA system consists of:

1. Retriever: Selects a small set of relevant passages from a large corpus.

2. Reader: Extracts or generates the answer from the selected passages.

3.1 Retriever: Sparse vs. Dense

BM25 (Sparse):

BM25 is a bag‑of‑words ranking function that scores a document d for a query q:

BM25(d,q)=∑i=1nIDF(qi)⋅f(qi,d)⋅(k1+1)f(qi,d)+k1⋅(1−b+b⋅∣d∣avgdl)BM25(d,q)=∑i=1n​IDF(qi​)⋅f(qi​,d)+k1​⋅(1−b+b⋅avgdl∣d∣​)f(qi​,d)⋅(k1​+1)​

where:

• f(q_i, d) = term frequency in document.

• |d| = document length; avgdl = average document length.

• k₁, b = hyperparameters (commonly k₁=1.2, b=0.75).

• IDF(q_i) = inverse document frequency (log(N / df_i)).

Implementation (using rank_bm25):

from rank_bm25 import BM25Okapi
tokenized_corpus = [doc.split() for doc in documents]
bm25 = BM25Okapi(tokenized_corpus)
scores = bm25.get_scores(query.split())
top_n = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)[:10]

Dense Passage Retrieval (DPR):

DPR uses dual encoders (one for query, one for passage) to map both to dense vectors. The relevance score is the dot product:

score(q,p)=EQ(q)⊤EP(p)score(q,p)=EQ​(q)⊤EP​(p)

Training uses a contrastive loss (negative sampling) to push positive pairs closer and negative pairs apart.

Advantage: Captures semantic similarity beyond exact word overlap, crucial for financial phrasing.

Implementation with HuggingFace:

from transformers import DPRQuestionEncoder, DPRContextEncoder
q_encoder = DPRQuestionEncoder.from_pretrained("facebook/dpr-question_encoder-single-nq-base")
p_encoder = DPRContextEncoder.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base")
# Encode passages offline and store vectors in FAISS for fast retrieval.

3.2 Reader: Extractive and Generative

Extractive Reader (e.g., BERT‑based):

Given a passage and a question, the model predicts the start and end positions of the answer span.

Training objective: Minimise cross‑entropy loss for start and end logits:

$\mathcal{L}=-\log P(\text{start})-\log P(\text{end})$

where:
P(start=i)=exp⁡(si)∑jexp⁡(sj)P(start=i)=∑j​exp(sj​)exp(si​)​
(similar for end).

Abstractive Reader (e.g., T5, BART):

Generates a free‑text answer, allowing paraphrasing and numerical reasoning.

Fine‑tuning on financial datasets:

• FinQA: Dataset with numerical reasoning from financial reports.

• ConvFinQA: Conversational QA with multi‑turn reasoning.

4. Handling Numerical and Tabular Questions

Many financial questions involve tables (income statements, balance sheets). We need to parse tables and perform arithmetic.

Approach 1: Table‑aware transformers (e.g., TAPAS, TaBERT)

• These models take a table and question as input and output a cell selection or an aggregation operation (sum, average, count).

Approach 2: Program synthesis (e.g., FinQA)

• The model generates a program (operation tree) that is executed on the table to compute the answer.

Example:
Question: “What is the operating income divided by revenue?”
Program: (op_income / revenue) → answer.

Training: Sequence‑to‑sequence with program generation.

5. Evaluation Metrics

• Exact Match (EM): The predicted answer matches the ground truth exactly (after normalisation).

• F1 score: Token‑level overlap between predicted and ground truth.

For financial QA, we often use ROUGE‑L for generative answers.

Normalisation: Remove punctuation, lowercase, standardise numbers (e.g., “$1.2B” → “1200000000”).

6. Building a Financial QA System: Step‑by‑Step

1. Corpus preparation: Collect documents (10‑Ks, transcripts). Chunk into passages (~100‑200 words) with metadata (company, date, section).

2. Indexing: Build a retriever index (BM25 or DPR+FAISS).

3. Retrieval: For a given question, retrieve top‑k passages.

4. Reading: Apply a fine‑tuned reader (BERT‑base for extractive; T5‑base for abstractive).

5. Post‑processing: If numerical, validate arithmetic; if entity, resolve tickers.

6. Deployment: Wrap in an API with caching for frequent questions.

7. Challenges in Financial QA

• Domain‑specific language: Models fine‑tuned on general SQuAD perform poorly.

• Long documents: Transformer token limits (512) – we must chunk wisely.

• Numerical reasoning: Models often struggle with arithmetic; explicit program execution is better.

• Time‑sensitivity: Answers may become outdated; incorporate date awareness.

8. Summary for the AI Practitioner

• Financial QA is a high‑value application, automating analyst workflows.

• A two‑stage retriever‑reader architecture is standard; dense retrieval (DPR) often outperforms BM25 for semantic matches.

• Fine‑tuning on financial QA datasets (FinQA, ConvFinQA) is essential.

• Numerical reasoning can be addressed with table‑aware models or program‑synthesis approaches.

• Evaluation must consider both exact match and token‑level F1, with normalisation for numbers.

• Always test with real analyst questions to ensure practical usefulness.

9. References

1. Chen, D., et al. (2017). Reading Wikipedia to answer open‑domain questions. ACL.

2. Karpukhin, V., et al. (2020). Dense passage retrieval for open‑domain question answering. EMNLP.

3. Chen, Z., et al. (2021). FinQA: A dataset of numerical reasoning over financial texts. EMNLP.

4. Reddy, S., et al. (2019). CoQA: A conversational question answering challenge. TACL.

5. Herzig, J., et al. (2020). TAPAS: Weakly supervised table parsing via pre‑training. ACL.