Learning Objectives
By the end of this lesson, learners should be able to:
- Explain the meaning and importance of data quality.
- Identify major dimensions of data quality.
- Detect common data-quality problems.
- Explain the purpose of data profiling.
- Apply principles of data cleaning and validation.
- Recognize the risks of inappropriate data-cleaning decisions.
1. Meaning of Data Quality
Data quality refers to the degree to which data is suitable for its intended use.
Data does not have quality in isolation.
A dataset may be suitable for one purpose but inadequate for another.
For example, approximate geographic information may be sufficient for broad market analysis but unsuitable for precise logistical planning.
2. Dimensions of Data Quality
Important dimensions include:
Accuracy
Does the data correctly represent the real-world value?
Completeness
Are required values present?
Consistency
Are values represented consistently across systems and records?
Timeliness
Is the data sufficiently current for its intended purpose?
Validity
Does the data conform to defined rules or formats?
Uniqueness
Are duplicate records appropriately controlled?
Relevance
Does the data actually support the intended analytical purpose?
3. Common Data-Quality Problems
Typical problems include:
- Missing values.
- Duplicate records.
- Incorrect values.
- Inconsistent formats.
- Invalid entries.
- Outliers.
- Contradictory records.
- Outdated information.
These problems can distort analysis and produce misleading conclusions.
4. Data Profiling
Data profiling involves systematically examining a dataset to understand its structure, content and quality characteristics.
An analyst may examine:
- Number of records.
- Number of variables.
- Missing-value rates.
- Unique values.
- Frequency distributions.
- Minimum and maximum values.
- Duplicate records.
- Data types.
Profiling helps identify problems before deeper analysis begins.
5. Missing Data
Missing data may arise because:
- A value was never collected.
- A respondent declined to answer.
- A system failed to capture information.
- A value was not applicable.
Possible treatments include:
- Removing observations.
- Imputing values.
- Retaining missing indicators.
- Collecting the information again.
The correct treatment depends on why the data is missing and how the variable is used.
6. Duplicate Data
Duplicate records can occur because of:
- Repeated data entry.
- System integration.
- Multiple customer identifiers.
- Import errors.
Removing duplicates without understanding their origin can be dangerous.
Two records that appear similar may represent legitimate separate transactions.
7. Data Validation
Data validation checks whether data satisfies defined rules.
Examples include:
- A percentage should fall within an appropriate range.
- A date should follow a valid format.
- A transaction amount should not contain impossible values.
- A required identifier should not be blank.
Validation rules should reflect the business meaning of the data.
8. Data Cleaning
Data cleaning involves identifying and correcting or appropriately handling data-quality problems.
Activities may include:
- Standardizing formats.
- Correcting errors.
- Removing inappropriate duplicates.
- Handling missing values.
- Resolving inconsistent categories.
- Identifying anomalous observations.
Cleaning should be documented to preserve transparency and reproducibility.
9. Outliers
An outlier is an observation that differs substantially from other observations.
Outliers may represent:
- Genuine unusual events.
- Measurement errors.
- Data-entry mistakes.
- Fraudulent activity.
- Exceptional business conditions.
An outlier should not automatically be deleted.
Its business meaning should first be investigated.
10. Data Transformation
Data transformation changes data into a form suitable for analysis.
Examples include:
- Converting currencies.
- Standardizing dates.
- Creating calculated fields.
- Aggregating transactions.
- Encoding categories.
- Scaling numerical variables.
Transformations should preserve the meaning required for the analytical objective.
11. Data Cleaning and Analytical Bias
Cleaning decisions can influence analytical outcomes.
For example, automatically deleting all incomplete records may disproportionately remove a particular customer group.
This can introduce selection bias into the resulting dataset.
Therefore, data cleaning is not merely a technical process. It can affect the substantive conclusions of the analysis.
12. Data Quality Monitoring
Data quality should be monitored continuously rather than corrected only once.
Organizations can establish:
- Data-quality rules.
- Quality thresholds.
- Automated validation.
- Exception reporting.
- Data-quality KPIs.
- Ownership and accountability.
Continuous monitoring helps detect deterioration before it significantly affects business decisions.
Lesson Summary
Data quality determines whether data is fit for its intended analytical purpose.
Important dimensions include accuracy, completeness, consistency, timeliness, validity, uniqueness and relevance.
Data profiling helps identify quality problems, while data cleaning and validation help address them.
However, cleaning decisions must be made carefully. Automatically deleting, replacing or modifying observations can introduce bias or remove meaningful information.
References
- ISO 8000 — Data Quality
ISO 8000 Data Quality - DAMA International — DAMA-DMBOK
DAMA International - NIST — Data and Artificial Intelligence Resources
NIST - OECD — Data Governance and Digital Policy
OECD Digital Policy
Review Questions
- What is meant by data quality?
- Why is data quality dependent on intended use?
- What are the major dimensions of data quality?
- What is data profiling?
- Why can missing data require different treatment depending on its cause?
- Why should duplicate records not automatically be deleted?
- What is the purpose of data validation?
- Why should outliers be investigated before removal?
- What is data transformation?
- How can data cleaning introduce analytical bias?
- Why should data-quality monitoring be continuous?
- How can organizations establish accountability for data quality?