INTRODUCTION: THE DATA QUALITY FLYWHEEL

In Lessons 9.1 through 9.6, we built a comprehensive data quality and enrichment pipeline. We cleansed transaction data, normalized dates and amounts, deduplicated merchant names using fuzzy matching, mapped merchants to MCC categories using a Naive Bayes classifier, deduplicated transactions across multiple banks using a stateful Flink pipeline, and enriched merchant data with external sources. The pipeline is robust and high-performing.

However, data quality is not a one-time effort. It is a continuous process. New merchants emerge daily. Existing merchants change their names. Banks change their data formats. The fuzzy matching models and the Naive Bayes classifier degrade over time if not retrained. The Google Places API may return different data. The DQI (Data Quality Index, from Lesson 9.1) fluctuates.

This lesson deconstructs the Data Quality Monitoring and Continuous Improvement Pipeline. We define the Data Quality KPIs: Accuracy, Completeness, Timeliness, Consistency, Format Validity, and DQI. We implement a real-time monitoring dashboard (Grafana) that visualises these KPIs. We design the Anomaly Detection system that alerts the engineering team when the DQI drops below a threshold (e.g., < 0.85). We also implement the Continuous Improvement pipeline: the fuzzy matching model (Levenshtein, Jaro-Winkler, Soundex weights) is retrained monthly using newly labelled data, and the Naive Bayes classifier is retrained weekly. We quantify the latency of the monitoring pipeline (2ms per transaction) and the improvement in the DQI over time (from 0.85 to 0.95 over 6 months).

LEARNING OBJECTIVES

1. Define the Data Quality KPIs—revisiting the five dimensions from Lesson 9.1: Accuracy, Completeness, Timeliness, Consistency, Format Validity, and the DQI. We will define the target values and the alert thresholds.

2. Design the Real-Time Monitoring Dashboard—building a Grafana dashboard with panels for: (1) DQI over time, (2) Accuracy over time, (3) Completeness over time, (4) Timeliness over time, (5) Consistency over time, (6) Format Validity over time, (7) Cache Hit Ratio, and (8) Enrichment Latency. We will define the data source (Elasticsearch) and the query languages (PromQL, LogQL).

3. Implement the Anomaly Detection System—defining alerts: (1) DQI < 0.85 (page the data quality team), (2) Accuracy < 0.98 (investigate data source), (3) Completeness < 0.99 (investigate missing fields), (4) Timeliness > 24 hours (investigate webhook delays), (5) Cache Hit Ratio < 90% (investigate merchant popularity shift). We will use AlertManager (Prometheus) to send alerts to PagerDuty.

4. Design the Continuous Improvement Pipeline—implementing a monthly retraining cycle for the fuzzy matching model (re-calculating the optimal Levenshtein, Jaro-Winkler, and Soundex weights on a new labelled dataset). We will implement a weekly retraining cycle for the Naive Bayes classifier (using new labelled transactions). We will quantify the improvement in the DQI over time.

5. Quantify the Monitoring Latency—measuring the time to compute the DQI (0.05ms), log the metrics to Elasticsearch (1ms), and render the dashboard (N/A). Total p95 latency: 1.05ms.

6. Calculate the DQI Improvement—measuring the DQI at month 0 (0.85) and month 6 (0.95), and proving that the continuous improvement pipeline increases the DQI by 0.10 points.

PART 1: THE DATA QUALITY KPIs — Defining the Targets

We define the target values and alert thresholds for each KPI.

PART 2: THE REAL-TIME MONITORING DASHBOARD — Grafana + Prometheus

The data quality team needs a real-time dashboard to monitor the system’s health.

Dashboard Panels:

1. DQI Over Time (line chart):

   • avg(dqi_score) by (merchant)

   • Alert: if < 0.85.

2. Accuracy Over Time (line chart):

   • avg(accuracy_score)

   • Alert: if < 0.98.

3. Completeness Over Time (line chart):

   • avg(completeness_score)

   • Alert: if < 0.97.

4. Timeliness Over Time (line chart):

   • avg(delay_seconds)

   • Alert: if > 86400 (24 hours).

5. Cache Hit Ratio (gauge):

   • cache_hits / (cache_hits + cache_misses)

   • Alert: if < 0.90.

6. Enrichment Latency (histogram):

   • histogram_quantile(0.95, enrichment_latency_bucket)

   • Alert: if > 50ms.

Data Source: Prometheus scrapes metrics from the data quality pipeline. Grafana visualises the metrics.

PART 3: THE ANOMALY DETECTION SYSTEM — Alerts and Incidents

We define alerts that trigger when the system’s health degrades.

AlertManager Configuration:

groups:
  - name: data_quality_alerts
    rules:
      - alert: DQIDrop
        expr: avg(dqi_score) < 0.85
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "DQI dropped below 0.85"

PART 4: THE CONTINUOUS IMPROVEMENT PIPELINE — Retraining the Models

The fuzzy matching models and the Naive Bayes classifier degrade over time. The continuous improvement pipeline retrains them.

Fuzzy Matching Retraining (Monthly) :

1. Collect Data: Extract a new labelled dataset of 10,000 merchant name pairs (5,000 matches, 5,000 non-matches) from the last month’s transactions.

2. Grid Search: Run grid search on the weights (w_Lev, w_JW, w_Soundex) to find the weights that maximise the F1-score.

3. Deploy: Update the Elasticsearch index with the new weights.

Naive Bayes Retraining (Weekly) :

1. Collect Data: Extract a new labelled dataset of 100,000 transactions from the last week’s transactions.

2. Train: Re-train the Naive Bayes classifier on the new dataset.

3. Deploy: Update the classifier.

DQI Improvement Over Time:

Conclusion: The continuous improvement pipeline increases the DQI by 0.10 points over 6 months.

PART 5: LATENCY BUDGET

Conclusion: The monitoring pipeline adds 1.05ms (p95) to the transaction processing, which is negligible.

CLOSING — THE DATA QUALITY FLYWHEEL

The data quality monitoring and continuous improvement pipeline ensures that the data quality KPIs remain high over time. The real-time dashboard provides visibility into the system’s health. The anomaly detection system alerts the team when quality drops. The continuous improvement pipeline retrains the models, ensuring they remain accurate.

Key Takeaways:

• KPIs: DQI, Accuracy, Completeness, Timeliness, Consistency, Format Validity, Cache Hit Ratio, Enrichment Latency.

• Dashboard: Grafana + Prometheus.

• Alerts: DQI < 0.85 (P1), Accuracy < 98% (P2), etc.

• Retraining: Fuzzy matching (monthly), Naive Bayes (weekly).

• DQI Improvement: From 0.85 to 0.95 over 6 months.

Transition to Lesson 9.8: With the data quality monitoring and continuous improvement pipeline in place, we now synthesise the entire Module 9 into the Capstone Framework. Lesson 9.8 presents the complete data quality and enrichment architecture, the unified audit trail, the regulatory evidence bundle, and the final risk assessment