1. Learning Objectives

By the end of this lesson, you will be able to:

• Understand the need for automatic summarization of long financial documents (10‑K, earnings calls, news).

• Differentiate between extractive and abstractive summarization and choose the right approach.

• Implement extractive summarization using TextRank and BERT‑based sentence scoring.

• Fine‑tune transformer models (BART, T5, PEGASUS) for abstractive summarization of financial text.

• Evaluate summaries using ROUGE, BLEU, and BERTScore, with domain‑specific considerations.

• Generate structured summaries (e.g., bullet points, key metrics) for earnings reports.

• Understand the challenges of factuality and numerical accuracy in abstractive summarization.

2. Why Summarization in Finance?

• Financial analysts read hundreds of pages of filings and transcripts.

• Automated summarization can produce executive summaries, highlighting key figures (revenue, EPS, guidance).

• News summarization helps traders quickly grasp market-moving events.

• Generating summaries from multiple sources can provide a consolidated view.

Use cases:

• Earnings call summarization: Extract prepared remarks and Q&A highlights.

• Research report summarization: Condense analyst notes.

• Regulatory filing summarization: Produce a one‑page summary of a 10‑K.

3. Extractive Summarization

Selects the most important sentences from the original document and concatenates them.

3.1 Graph‑Based: TextRank

Build a graph where nodes are sentences, edges represent similarity (e.g., cosine similarity of TF‑IDF vectors). Use PageRank to score sentences.

PageRank equation:
PR(vi)=(1−d)+d∑vj∈In(vi)wj,i∑vk∈Out(vj)wj,kPR(vj)PR(vi​)=(1−d)+d∑vj​∈In(vi​)​∑vk​∈Out(vj​)​wj,k​wj,i​​PR(vj​)

where w_{j,i} is the similarity between sentences j and i.

Implementation:

import networkx as nx
similarity_matrix = cosine_similarity(tfidf_matrix)  # sentences x sentences
nx_graph = nx.from_numpy_array(similarity_matrix)
scores = nx.pagerank(nx_graph)
top_sentences = sorted(scores, key=scores.get, reverse=True)[:num_sentences]

3.2 Neural Extractive Models

BERT‑based models (e.g., BERT‑SUM) classify each sentence as included or not, using the [CLS] representation of the sentence.

Training: Binary classification (include/discard) with cross‑entropy loss.

4. Abstractive Summarization

Generates new sentences that capture the essence, potentially paraphrasing and fusing information.

4.1 Sequence‑to‑Sequence (Seq2Seq) with Transformers

Models like BART, T5, and PEGASUS are pre‑trained for summarization.

BART: Denoising autoencoder – corrupts text (e.g., token masking, sentence permutation) and learns to reconstruct. Fine‑tuned for summarization with cross‑entropy loss over target tokens.

T5: Text‑to‑text transfer transformer – treats summarization as a text‑to‑text problem: input = “summarize: [document]”, output = summary.

PEGASUS: Pre‑trained with gap‑sentence generation – removes important sentences and trains to predict them. This is highly effective for summarization.

4.2 Fine‑tuning on Financial Data

Use datasets like:

• FinSum: A summarization dataset of financial news and reports.

• EarningsCallSum: Transcripts with human‑written summaries.

Training objective: Minimise negative log‑likelihood of target summary tokens:

L=−∑t=1Tlog⁡P(yt∣y<t,x)L=−∑t=1T​logP(yt​∣y<t​,x)

Implementation with HuggingFace:

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, Seq2SeqTrainingArguments, Seq2SeqTrainer

model = AutoModelForSeq2SeqLM.from_pretrained("google/pegasus-large")
tokenizer = AutoTokenizer.from_pretrained("google/pegasus-large")
# Prepare dataset with input_ids and labels
trainer = Seq2SeqTrainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    tokenizer=tokenizer,
)
trainer.train()

4.3 Controlling Summary Length and Style

Set max_length and min_length in generation. Use length_penalty to encourage brevity.

Beam search (width 4‑8) is standard; diversity penalty can reduce repetition.

5. Evaluation Metrics for Summarization

• ROUGE (Recall‑Oriented Understudy for Gisting Evaluation):

  ROUGE‑N: Counts n‑gram overlap.

  ROUGE-N=∑s∈ref∑gramn∈sCountmatch(gramn)∑s∈ref∑gramn∈sCount(gramn)ROUGE-N=∑s∈ref​∑gramn​∈s​Count(gramn​)∑s∈ref​∑gramn​∈s​Countmatch​(gramn​)​

  ROUGE‑L: Longest common subsequence (LCS) between summary and reference.

• BLEU: Precision‑based n‑gram overlap (less common for summarization, more for translation).

• BERTScore: Computes similarity using contextual embeddings (BERT) – correlates better with human judgment.

For financial summaries: Also check factuality – ensure numbers (EPS, revenue) are correct. Use factual consistency metrics like QAFactEval.

6. Challenges and Mitigations

7. Hybrid Approaches: Extractive + Abstractive

First extract the most relevant sentences (e.g., using BERT‑SUM), then feed them to an abstractive model to rewrite and condense. This reduces input length and improves factuality.

Example pipeline:

1. Use a transformer‑based extractive model to select top‑k sentences.

2. Concatenate them.

3. Input to a BART/T5 model for final summary.

8. Generating Structured Summaries (Key‑Value Format)

For earnings reports, we often want a structured summary:

Revenue: $XX.XXB (beat/meet/miss)
EPS: $X.XX (beat/meet/miss)
Guidance: [raised/lowered/confirmed]
Key Drivers: [segment growth, cost control]
Risks: [supply chain, regulatory]

This can be framed as a template‑filling task where we extract entities and values from the document, then fill a template.

Approach: Use NER and relation extraction to populate the template; generate natural language using a pre‑defined template or a language model.

9. Summary for the AI Practitioner

• Extractive summarization (TextRank, BERT‑SUM) is safe and factually consistent; useful for quick scanning.

• Abstractive summarization (BART, T5, PEGASUS) produces more fluent and concise summaries but risks hallucination.

• Fine‑tuning on financial datasets is crucial; general models struggle with domain terminology.

• Evaluation should combine ROUGE, BERTScore, and factuality checks (especially for numbers).

• Hybrid extractive‑abstractive pipelines offer a good trade‑off between fidelity and conciseness.

• Structured summaries (with templates) can be more actionable for financial decisions.

10. References

1. Rush, A. M., et al. (2015). A neural attention model for abstractive sentence summarization. EMNLP.

2. Lewis, M., et al. (2020). BART: Denoising sequence‑to‑sequence pre‑training for natural language generation. ACL.

3. Zhang, J., et al. (2020). PEGASUS: Pre‑training with extracted gap‑sentences for abstractive summarization. ICML.

4. Lin, C.‑Y. (2004). ROUGE: A package for automatic evaluation of summaries. ACL Workshop.

5. Zhang, T., et al. (2020). BERTScore: Evaluating text generation with BERT. ICLR.

6. Mehta, P., et al. (2021). FinSum: A dataset for summarization of financial news. Workshop on Financial NLP.