1. Learning Objectives

By the end of this lesson, you will be able to:

• Understand the role of sentiment in financial markets and its theoretical underpinnings

• Implement lexicon‑based sentiment analysis using financial dictionaries (Loughran‑McDonald, VADER)

• Build supervised machine learning models for financial sentiment classification (Logistic Regression, SVM, Random Forest)

• Apply transfer learning with contextual embeddings (FinBERT, BERT) for sentiment

• Handle imbalanced data and evaluate performance using financial‑specific metrics

• Design a sentiment‑based trading signal and backtest its performance

• Address challenges such as negation, sarcasm, and domain adaptation

2. The Importance of Sentiment in Finance

2.1 Theoretical Background

Behavioural Finance: Investor sentiment is a key driver of asset prices. Psychological biases (overconfidence, herding, loss aversion) affect decision‑making.

Theoretical Models:

• Noise Trader Models: Sentiment of noise traders can create mispricing.

• Limited Arbitrage: Rational arbitrageurs cannot fully correct mispricing due to risk and costs.

• Information Asymmetry: Textual sentiment reveals private information (e.g., management tone in earnings calls).

Empirical Evidence:

• Tetlock (2007): Negative sentiment in Wall Street Journal columns predicts downward pressure on stock prices.

• Loughran & McDonald (2011): Negative words in 10‑Ks are associated with lower returns.

• Antweiler & Frank (2004): Stock message board sentiment predicts volatility.

2.2 Sentiment Data Sources

3. Lexicon‑Based Sentiment Analysis

Lexicon methods assign sentiment scores based on dictionaries of words with pre‑defined polarities.

3.1 Loughran‑McDonald Financial Sentiment Dictionary

This is the gold standard for finance. It contains six categories:

• Negative (e.g., “loss”, “default”, “bankruptcy”)

• Positive (e.g., “profit”, “gain”, “improve”)

• Uncertainty (e.g., “may”, “could”, “uncertain”)

• Litigious (e.g., “lawsuit”, “claim”)

• Strong Modal (e.g., “will”, “must”)

• Weak Modal (e.g., “might”, “would”)

Implementation:

import pandas as pd

# Load LM dictionary (available from their website)
lm_neg = set(pd.read_csv('LM_negative.csv')['word'])
lm_pos = set(pd.read_csv('LM_positive.csv')['word'])
lm_uncertainty = set(pd.read_csv('LM_uncertainty.csv')['word'])
# ... other categories

def lm_sentiment(tokens):
    neg = sum(1 for t in tokens if t in lm_neg)
    pos = sum(1 for t in tokens if t in lm_pos)
    unc = sum(1 for t in tokens if t in lm_uncertainty)
    total = len(tokens) or 1
    return {
        'neg_ratio': neg / total,
        'pos_ratio': pos / total,
        'net_sentiment': (pos - neg) / total,
        'uncertainty_ratio': unc / total
    }

Advantages: Interpretable, no training data needed, robust for long documents.
Disadvantages: Misses context, negation, sarcasm; dictionary coverage limited.

3.2 Handling Negation and Modifiers

Simple bag‑of‑words sentiment can be enhanced by detecting negation.

Approach: Negation scope

• Identify negation words (“not”, “no”, “never”, “neither”)

• Invert the sentiment of subsequent words until punctuation or conjunction.

NEGATION_WORDS = {'not', 'no', 'never', 'neither', 'nor', 'n\'t', 'without', 'lack', 'fail'}

def sentiment_with_negation(tokens, lexicon):
    sentiment_score = 0
    negate = False
    for token in tokens:
        if token in NEGATION_WORDS:
            negate = not negate  # simple toggle
        else:
            score = lexicon.get(token, 0)
            if negate:
                score = -score
            sentiment_score += score
            # Reset negation after certain punctuation? Usually after comma, period.
    return sentiment_score

Modifiers: Words like “very”, “extremely” can amplify sentiment. Use intensity weights (e.g., from VADER).

3.3 VADER for Social Media

VADER (Valence Aware Dictionary and sEntiment Reasoner) is designed for social media. It handles:

• Emojis and emoticons

• Capitalisation (e.g., “GREAT” amplifies)

• Punctuation (e.g., “!!!” amplifies)

• Sentiment intensity modifiers

Implementation:

from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer

analyzer = SentimentIntensityAnalyzer()
scores = analyzer.polarity_scores(text)
# Returns: {'neg': 0.0, 'neu': 0.5, 'pos': 0.5, 'compound': 0.5}
# compound ranges from -1 (most negative) to +1 (most positive)

Application: For social media sentiment, aggregate daily compound scores across posts.

4. Supervised Machine Learning for Sentiment

When we have labelled data (e.g., manually annotated earnings calls), we can train classifiers.

4.1 Data Labelling

Labelling financial text is expensive but can be done via:

• Crowdsourcing with clear guidelines.

• Using proxies: e.g., stock price movements after the text release (if positive return, label positive; but this is noisy).

• Pre‑existing datasets: e.g., Financial PhraseBank, available on Kaggle.

Example Financial PhraseBank dataset: Contains 4,840 sentences from financial news, labelled with positive, negative, neutral.

4.2 Feature Engineering for ML Models

We typically use TF‑IDF vectors or word embeddings as input.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

# Load dataset
X = df['sentence']
y = df['label']  # 0=negative, 1=neutral, 2=positive

# Vectorize
vectorizer = TfidfVectorizer(max_features=5000, ngram_range=(1,2))
X_vec = vectorizer.fit_transform(X)

# Split
X_train, X_test, y_train, y_test = train_test_split(X_vec, y, test_size=0.2, random_state=42)

# Model
clf = LogisticRegression(C=1.0, max_iter=1000)
clf.fit(X_train, y_train)
accuracy = clf.score(X_test, y_test)

4.3 Addressing Class Imbalance

Financial sentiment is often skewed towards neutral/negative. Use:

• Class weights: class_weight='balanced' in sklearn.

• Resampling: SMOTE (Synthetic Minority Over‑sampling) for minority classes.

• AUC‑based metrics rather than accuracy.

from sklearn.metrics import classification_report, confusion_matrix
y_pred = clf.predict(X_test)
print(classification_report(y_test, y_pred))

4.4 Advanced ML Models

• Support Vector Machines (SVM): Effective with TF‑IDF features.

• Random Forest / XGBoost: Capture non‑linear interactions.

• Ensemble methods: Combine lexicon‑based scores with ML predictions.

Example: Hybrid model combining lexicon and ML:

# Add lexicon features to the feature matrix
lexicon_features = np.array([lm_sentiment(text) for text in X])
X_combined = np.hstack([X_vec.toarray(), lexicon_features])

# Then train classifier on combined features

5. Deep Learning for Sentiment – FinBERT and Transformers

Pre‑trained transformer models have achieved state‑of‑the‑art results in financial sentiment.

5.1 FinBERT

FinBERT is a BERT variant pre‑trained on financial text (SEC filings, earnings calls, news). It is fine‑tuned for sentiment classification.

Implementation with HuggingFace:

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_name = 'yiyanghkust/finbert-tone'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

def finbert_sentiment(text):
    inputs = tokenizer(text, return_tensors='pt', truncation=True, max_length=512)
    outputs = model(**inputs)
    logits = outputs.logits
    probs = torch.softmax(logits, dim=1)
    # Outputs: [negative, neutral, positive] probabilities
    return probs.detach().numpy()

Fine‑tuning on custom data:

from transformers import Trainer, TrainingArguments

# Prepare dataset with labels (0,1,2)
training_args = TrainingArguments(
    output_dir='./results',
    num_train_epochs=3,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=64,
    warmup_steps=500,
    weight_decay=0.01,
    logging_dir='./logs',
    logging_steps=10,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    tokenizer=tokenizer,
)
trainer.train()

5.2 Handling Long Documents

Transformers have a 512‑token limit. For long documents (10‑K), we can:

• Segment into chunks and aggregate predictions (e.g., mean, max).

• Use Longformer (covered in Module 5) or BigBird that support longer contexts.

• Use Hierarchical attention (sentence‑level then document‑level).

Chunking approach:

def chunk_text(text, max_length=512):
    tokens = tokenizer.tokenize(text)
    chunks = [tokens[i:i+max_length] for i in range(0, len(tokens), max_length)]
    return [' '.join(chunk) for chunk in chunks]

def document_sentiment(text):
    chunks = chunk_text(text)
    sentiments = [finbert_sentiment(chunk) for chunk in chunks]
    # Average probabilities across chunks
    avg_probs = np.mean(sentiments, axis=0)
    return avg_probs

5.3 Transfer Learning from General Domain

If labelled financial data is scarce, we can fine‑tune a general‑purpose BERT (e.g., bert-base-uncased) on a small financial dataset. This usually performs well.

# Use the same Trainer with bert-base-uncased
model_name = 'bert-base-uncased'
# Then fine-tune with financial data.

6. Sentiment‑Based Trading Signals

6.1 Signal Generation

We convert sentiment scores into trading signals.

Common methods:

• Threshold: If sentiment > +0.2 → buy; if < -0.2 → sell; else hold.

• Z‑score: Normalise sentiment over a rolling window; if > +1.5, buy; if < -1.5, sell.

• Ranking: For a basket of stocks, go long on highest sentiment, short on lowest.

def generate_signal(sentiment_score, threshold=0.2):
    if sentiment_score > threshold:
        return 1  # buy
    elif sentiment_score < -threshold:
        return -1  # sell
    else:
        return 0  # hold

6.2 Backtesting

We must account for:

• Look‑ahead bias: Use only sentiment available at the time of trade.

• Slippage and transaction costs: Include realistic costs.

• Risk management: Position sizing, stop‑loss.

Backtest framework:

def backtest_sentiment_strategy(df, signal_col, return_col, threshold=0.2, cost=0.001):
    df['signal'] = df[signal_col].apply(lambda x: 1 if x > threshold else (-1 if x < -threshold else 0))
    df['position'] = df['signal'].shift(1)  # trade at next day's open
    df['strategy_return'] = df['position'] * df[return_col]
    # Apply transaction costs when position changes
    df['cost'] = (df['position'] != df['position'].shift(1)).astype(int) * cost
    df['net_return'] = df['strategy_return'] - df['cost']
    cumulative_return = (1 + df['net_return']).cumprod()
    return cumulative_return

6.3 Performance Metrics

• Sharpe Ratio: Annualised return / annualised volatility.

• Maximum Drawdown: Largest peak‑to‑trough decline.

• Hit Rate: Percentage of profitable trades.

• Information Coefficient: Correlation between sentiment score and future returns.

def sharpe_ratio(returns, risk_free=0.0, periods=252):
    return (np.mean(returns - risk_free) / np.std(returns)) * np.sqrt(periods)

7. Challenges and Pitfalls

• Data Snooping: Overfitting to historical sentiment patterns that may not persist.

• News vs. Social Media: News sentiment is often already priced in quickly; social media may contain more noise.

• Market Regimes: Sentiment impact may vary in bull vs. bear markets.

• Temporal Aggregation: Daily sentiment aggregation may lose intraday signals.

• Model Drift: Sentiment models need retraining as language and market conditions evolve.

8. Summary for the AI Practitioner

• Lexicon‑based methods are simple, interpretable, and work well for long documents (10‑Ks, earnings calls). Loughran‑McDonald is the standard.

• Social media benefits from VADER and custom preprocessing; its noise requires aggregation.

• Supervised ML (Logistic Regression, SVM) with TF‑IDF can achieve good accuracy when labelled data is available.

• FinBERT is the state‑of‑the‑art for financial sentiment, but requires GPU resources and careful fine‑tuning.

• Trading signals should be backtested with realistic costs and evaluated on out‑of‑sample data.

• Always consider the economic significance: a small edge in sentiment may not be tradable after costs.

9. References and Further Reading

1. Loughran, T., & McDonald, B. (2011). When is a liability not a liability? Journal of Finance.

2. Tetlock, P. C. (2007). Giving content to investor sentiment. Journal of Finance.

3. Antweiler, W., & Frank, M. Z. (2004). Is all that talk just noise? Journal of Finance.

4. Yang, Y., et al. (2020). FinBERT: A pretrained language model for financial communications. arXiv.

5. Hutto, C., & Gilbert, E. (2014). VADER: A parsimonious rule‑based model for sentiment analysis of social media text. ICWSM.

6. Ke, Z. T., & Kelly, B. T. (2022). FinBERT‑based sentiment and asset pricing. Working Paper.