1. Learning Objectives

By the end of this lesson, you will be able to:

• Understand the unique characteristics of financial text data (earnings calls, SEC filings, news, social media)

• Apply domain‑specific text preprocessing techniques (tokenisation, stopword removal, lemmatisation) tailored for finance

• Build financial‑specific features such as sentiment scores, readability indices, and named entities

• Implement TF‑IDF and word embeddings (Word2Vec, GloVe) on financial corpora

• Handle noisy financial text from social media and news feeds

• Construct structured datasets from unstructured text for downstream AI models

• Evaluate the quality of text representations using intrinsic and extrinsic metrics

2. The Landscape of Financial Text Data

2.1 Types of Financial Text

2.2 Challenges in Financial NLP

• Domain‑specific jargon: “hawkish”, “quantitative easing”, “yield curve”, require custom dictionaries.

• Ambiguity: “interest” can mean curiosity or financial rate; “future” can be time or contract.

• Numerical context: Many numbers (EPS, P/E, GDP) – need to be extracted and normalised.

• Temporal references: “next quarter”, “FY2024” – need temporal grounding.

• Negation and modality: “not profitable”, “could increase” – sentiment is nuanced.

• Cross‑reference: Tickers, CIK codes, company names – need entity resolution.

• Noise in social media: Emojis, misspellings, abbreviations (“🚀”, “bagholder”, “bullish”).

3. Financial Text Preprocessing Pipeline

A robust preprocessing pipeline is essential. For finance, we add steps that handle numbers, currencies, dates, and company names.

3.1 Raw Text Acquisition

Text is often messy – HTML, PDF, or raw JSON. We need to extract clean plain text.

import re
import nltk
from bs4 import BeautifulSoup
import pandas as pd

def extract_text_from_html(html):
    soup = BeautifulSoup(html, 'html.parser')
    # Remove script and style tags
    for tag in soup(['script', 'style']):
        tag.decompose()
    return soup.get_text(separator=' ')

For PDFs (e.g., SEC filings), libraries like PyPDF2 or pdfplumber are used.

3.2 Cleaning and Normalisation

Financial text contains many non‑standard elements. A typical cleaning routine:

def clean_financial_text(text):
    # 1. Convert to lowercase (optional; sometimes case is informative for entities)
    text = text.lower()
    
    # 2. Remove URLs, emails, and hashtags
    text = re.sub(r'http\S+|www\S+|https\S+', '', text, flags=re.MULTILINE)
    text = re.sub(r'\S+@\S+', '', text)
    text = re.sub(r'#\w+', '', text)
    
    # 3. Handle currency symbols – replace with [CURRENCY] or keep numbers
    text = re.sub(r'[$€£¥]', '[CURR]', text)
    
    # 4. Handle percentages – keep as numbers for later extraction
    text = re.sub(r'(\d+\.?\d*)\s*%', r'\1 [PERCENT]', text)
    
    # 5. Handle dates – normalise to a standard format (optional)
    # Example: convert "Jan 15, 2024" to "2024-01-15"
    # This is complex; often we keep as is or use date parser
    
    # 6. Remove extra spaces and control characters
    text = re.sub(r'\s+', ' ', text).strip()
    
    return text

3.3 Tokenisation

Tokenisation splits text into tokens. For finance, we may want to keep multi‑word expressions (e.g., “Federal Reserve”, “earnings per share”) as single tokens to preserve meaning.

Standard tokenisation (NLTK):

from nltk.tokenize import word_tokenize
tokens = word_tokenize(text)

Financial‑aware tokenisation using spaCy with custom entity ruler:

import spacy
nlp = spacy.load('en_core_web_sm')

# Add custom financial terms as entities
ruler = nlp.add_pipe('entity_ruler')
patterns = [{'label': 'FIN_TERM', 'pattern': [{'LOWER': 'quantitative'}, {'LOWER': 'easing'}]},
            {'label': 'FIN_TERM', 'pattern': [{'LOWER': 'earnings'}, {'LOWER': 'per'}, {'LOWER': 'share'}]}]
ruler.add_patterns(patterns)

doc = nlp("The Fed announced quantitative easing and EPS exceeded expectations.")
tokens = [token.text for token in doc]

3.4 Stopword Removal

Standard stopword lists (NLTK, spaCy) may not be optimal for finance. Some words like “above”, “below”, “quarter” are meaningful in financial context.

Custom stopword list for finance:

from nltk.corpus import stopwords

# Start with default stopwords and add finance‑specific noise words
finance_stopwords = set(stopwords.words('english'))
# Remove finance‑useful words
finance_stopwords -= {'above', 'below', 'quarter', 'annual', 'month', 'year', 'price', 'value', 'market'}
# Add very common noise
finance_stopwords.update({'said', 'says', 'will', 'can', 'may', 'would', 'could'})

3.5 Lemmatisation and Stemming

Lemmatisation is preferred over stemming because it produces real words and handles inflections better. For finance, we want to map “increased”, “increasing” to “increase” to reduce vocabulary.

from nltk.stem import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()
lemmatized = [lemmatizer.lemmatize(token, pos='v') for token in tokens]

But caution: Some financial terms have specific meanings in different forms (e.g., “option” vs “optional”) – lemmatisation might conflate them. A hybrid approach is often used: lemmatise only common verbs and nouns that are safe.

3.6 Handling Numbers and Financial Quantities

Numbers are critical. We often extract them as features rather than just removing them.

def extract_financial_numbers(text):
    # Extract all numbers (integers, decimals, percentages)
    numbers = re.findall(r'(\d+\.?\d*)\s*([%$€£]?)', text)
    # Normalise percentages to decimals
    return numbers

We can replace numbers with a special token like [NUM] to reduce sparsity, but for many tasks (e.g., regression, classification) we keep them.

Example of normalising currency amounts:

def normalize_currency(text):
    # Convert "$1.2M" to 1200000, "$5B" to 5000000000
    def replace(match):
        amount = float(match.group(1))
        suffix = match.group(2).lower()
        if suffix == 'm':
            amount *= 1e6
        elif suffix == 'b':
            amount *= 1e9
        elif suffix == 'k':
            amount *= 1e3
        return str(amount)
    
    return re.sub(r'(\d+\.?\d*)\s*([MBK])(?=\b)', replace, text, flags=re.IGNORECASE)

4. Feature Engineering for Financial Text

After preprocessing, we create numerical features that can be fed into machine learning models.

4.1 Bag‑of‑Words and TF‑IDF

Term Frequency (TF):
TF(t,d)=ft,d∑t′∈dft′,dTF(t,d)=∑t′∈d​ft′,d​ft,d​​

Inverse Document Frequency (IDF):
IDF(t)=log⁡(N1+DF(t))IDF(t)=log(1+DF(t)N​)

TF‑IDF:
$\text{TF-IDF}(t,d)=\text{TF}(t,d)\times \text{IDF}(t)$

Implementation with scikit‑learn:

from sklearn.feature_extraction.text import TfidfVectorizer

# Use a custom tokenizer with financial stopwords
vectorizer = TfidfVectorizer(
    tokenizer=lambda text: [lemmatizer.lemmatize(w) for w in word_tokenize(clean_financial_text(text))],
    stop_words=list(finance_stopwords),
    max_features=5000,
    ngram_range=(1, 2)  # include bigrams like "interest rate"
)

tfidf_matrix = vectorizer.fit_transform(corpus)

Limitations:

• Sparse and high‑dimensional.

• Lacks semantic relationships.

• Not good for long documents with many unique terms.

4.2 Sentiment Scores

Sentiment analysis is a core application in finance. We can use pre‑built lexicons or train domain‑specific models.

Common Financial Sentiment Lexicons:

• Loughran‑McDonald – specifically built for financial text, covering negative, positive, uncertainty, litigious, etc.

• Harvard IV‑4 – general purpose.

• VADER – useful for social media.

Example using Loughran‑McDonald dictionary:

from loughran_mcdonald import LM

lm = LM()  # loads the dictionary
sentiment_counts = lm.get_sentiment_counts(doc_tokens)
# Output: {'negative': 3, 'positive': 5, 'uncertainty': 2, ...}

Custom sentiment features:

• Positive word count / total words

• Negative word count / total words

• Net sentiment = (positive – negative) / total

• Uncertainty ratio

• Strong vs. weak modal verbs (e.g., “will” vs “may”)

4.3 Readability Indices

Readability can indicate complexity, which correlates with information asymmetry or management obfuscation.

Flesch Reading Ease:
FRE=206.835−1.015×total wordstotal sentences−84.6×total syllablestotal wordsFRE=206.835−1.015×total sentencestotal words​−84.6×total wordstotal syllables​

Gunning Fog Index:
Fog=0.4×(total wordstotal sentences+100×complex wordstotal words)Fog=0.4×(total sentencestotal words​+100×total wordscomplex words​)

where complex words have 3+ syllables.

Implementation:

import textstat

def readability_scores(text):
    return {
        'flesch': textstat.flesch_reading_ease(text),
        'fog': textstat.gunning_fog(text),
        'smog': textstat.smog_index(text),
        'coleman_liau': textstat.coleman_liau_index(text)
    }

4.4 Named Entity Recognition (NER)

Financial entities include companies, people, tickers, monetary amounts, dates, etc. Extracting them adds rich structure.

Using spaCy with a financial NER model (e.g., en_core_web_lg or custom FinBERT):

def extract_entities(text):
    doc = nlp(text)
    entities = {}
    for ent in doc.ents:
        entities.setdefault(ent.label_, []).append(ent.text)
    return entities

Common entity types in finance:

• ORG – organisations

• MONEY – currency amounts

• PERCENT – percentage values

• DATE – dates

• GPE – geopolitical entities

Entity resolution: Map company mentions to ticker symbols using a knowledge base (e.g., yfinance).

4.5 Financial Jargon Dictionaries

We can create a binary feature indicating presence of specific terms (e.g., “default”, “bankruptcy”, “liquidity”, “Fed”, “inflation”).

FIN_JARGON = {
    'bullish': ['bullish', 'bull', 'upside'],
    'bearish': ['bearish', 'bear', 'downside'],
    'risk': ['risk', 'volatility', 'uncertainty'],
    'growth': ['growth', 'expand', 'increase']
}

def jargon_features(tokens):
    features = {}
    for category, words in FIN_JARGON.items():
        features[f'jargon_{category}'] = sum(1 for w in tokens if w in words)
    return features

4.6 Word Embeddings (Static)

Pre‑trained word embeddings capture semantic similarity. For finance, we can use general embeddings (GloVe, Word2Vec) or domain‑specific ones (e.g., FinBERT‑based embeddings, or Word2Vec trained on SEC filings).

Using pre‑trained GloVe:

import gensim.downloader as api
model = api.load('glove-wiki-gigaword-300')

def document_embedding(tokens, model):
    vectors = [model[word] for word in tokens if word in model.key_to_index]
    if vectors:
        return np.mean(vectors, axis=0)  # average pooling
    else:
        return np.zeros(model.vector_size)

Domain‑specific Word2Vec trained on financial news:

• Available from arXiv or can train on your own corpus using gensim.

Limitations: Static embeddings do not handle polysemy (e.g., “stock” meaning inventory vs equity). Contextual embeddings (BERT) solve this, but they are heavier and covered in later lessons.

4.7 Document‑Level Features

Besides text, we include metadata features:

• Document length (word count)

• Sentence count

• Average word length

• Number of numbers, percentages, currencies

• Ratio of forward‑looking vs. historical words (using custom lists)

5. Handling Noisy Social Media Text

Social media (Twitter, Reddit) is short, informal, and full of noise. Preprocessing requires extra steps:

def preprocess_social_text(text):
    # Lowercase
    text = text.lower()
    # Remove mentions, hashtags, but keep hashtag content for topic
    text = re.sub(r'@\w+', '', text)
    text = re.sub(r'#', '', text)  # remove # but keep word
    # Remove emojis (or convert to text)
    import emoji
    text = emoji.demojize(text)
    # Handle repeated letters (e.g., "stooock" -> "stock")
    text = re.sub(r'(.)\1{2,}', r'\1\1', text)  # reduce to two repeats
    # Remove punctuation but keep important symbols like $ for tickers
    # Better: keep $ and then map to ticker
    return text

Ticker detection: Use regex for $AAPL or search for known tickers.

Sentiment from social media: VADER performs well because it handles punctuation, capitalization, and emojis.

6. Building Structured Datasets

Once we extract features, we can create a dataframe that combines text features with numerical market data.

def build_financial_text_dataset(texts, dates, tickers):
    # Preprocess each text
    processed = [preprocess_pipeline(t) for t in texts]
    
    # Extract features
    tfidf_features = vectorizer.transform(processed)
    sentiment_features = np.array([get_sentiment(t) for t in processed])
    readability_features = np.array([get_readability(t) for t in processed])
    entity_features = ... # one-hot or counts
    
    # Combine with metadata
    df = pd.DataFrame({
        'date': dates,
        'ticker': tickers,
        'sentiment_net': sentiment_features[:, 0],
        'sentiment_uncertainty': sentiment_features[:, 1],
        'flesch': readability_features[:, 0],
        'fog': readability_features[:, 1]
    })
    
    # Add sparse TF‑IDF features as separate columns or keep as sparse matrix
    # For tree‑based models, we might select top features.
    
    return df, tfidf_features

7. Evaluating Text Representations

We need to validate that our features are useful for downstream tasks.

Intrinsic evaluation: Measure how well the representation captures semantics (e.g., word similarity tasks). For financial domain, we can use a financial word similarity dataset.

Extrinsic evaluation: Test performance on a financial task (e.g., predict stock returns, classify sentiment). If a feature does not improve performance, consider removing it.

Example: Sentiment vs. Returns Correlation

def evaluate_sentiment_predictive_power(df):
    # lag sentiment by 1 day
    df['sentiment_lag1'] = df['sentiment'].shift(1)
    # compute correlation with next day return
    corr = df['sentiment_lag1'].corr(df['return'])
    print(f"Correlation: {corr:.4f}")

8. Summary for the AI Practitioner

• Financial text is rich and diverse; preprocessing must preserve numerical and domain‑specific information.

• Custom stopwords and lemmatisation tailored to finance improve feature quality.

• Feature extraction goes beyond bag‑of‑words: sentiment, readability, NER, and financial jargon counts are essential.

• Social media requires aggressive noise reduction but can be a valuable source of alternative data.

• Evaluation should be task‑driven; always test features on real financial predictions.

• Remember: Garbage in, garbage out – in NLP, the quality of preprocessing often determines the success of your model.

9. References and Further Reading

1. Loughran, T., & McDonald, B. (2011). When is a liability not a liability? Textual analysis, dictionaries, and 10‑Ks. Journal of Finance.

2. Jegadeesh, N., & Wu, D. (2013). Word power: A new approach for content analysis. Journal of Financial Economics.

3. Tetlock, P. C. (2007). Giving content to investor sentiment: The role of media in the stock market. Journal of Finance.

4. Zhang, Y., & Gans, J. (2021). Financial Natural Language Processing: A Survey. arXiv preprint.

5. Arora, S., et al. (2018). A survey on natural language processing for financial text. Computational Economics.