Understanding Synthetic Data Quality: A Beginner’s Guide to Univariate Metrics

Why Should You Care About Synthetic Data Quality?

Imagine you’re training a new employee using a manual that’s been translated from another language. If the translation is poor, your employee might learn the wrong procedures. Similarly, when we create synthetic (artificially generated) data to address data bottlenecks, enhance privacy protection, or enable data sharing across organizations, we need to ensure it accurately represents the real-world data it’s based on.

This is where data quality evaluation comes in. Think of it as a quality control system that checks three key aspects:

• Fidelity: How well does the synthetic data match the original?
• Privacy: Does it protect sensitive information?
• Utility: Can it be used effectively for its intended purpose?

In this article, we’ll focus on fidelity metrics, specifically the univariate ones that are easiest to understand and visualize.

What Are Fidelity Metrics?

Fidelity metrics are like a “similarity score” between your original data and the synthetic version. Picture this: you have a family photo, and you make a photocopy. A good copy should look very similar to the original — same people, same poses, same colors. A poor copy might be blurry, too dark, or missing details.

Fidelity metrics measure how “good” your synthetic data copy is.

The Data Table Analogy

Most datasets can be thought of as tables (like Excel spreadsheets):

• Rows = individual records (like customers, transactions, or measurements)
• Columns = different characteristics or variables (like age, income, or temperature)

For example, a customer database might look like this:

What Makes Good Synthetic Data?

In terms of preserving fidelity, the perfect synthetic dataset would be either:

1. An exact copy of the original data, or
2. A reshuffled version (same data, different order)

However, in practice, we usually want synthetic data that’s different from the original (to protect privacy) but still statistically similar. Think of it like creating a new family photo with the same people but different poses and backgrounds.

Fidelity metrics give us a number that tells us how close our synthetic data is to the original.

Scope of This Article: The univariate metrics discussed here focus on distribution-based evaluation, i.e., how well the synthetic data preserves the statistical properties of individual fields across all records. For time series or event datasets, these metrics evaluate the tabular aspects only (e.g., the distribution of transaction amounts or user actions), not the temporal relationships or sequential patterns. While more sophisticated metrics exist for evaluating temporal fidelity (such as session length distributions, transition probabilities, and inter-arrival times), those are beyond the scope of this introductory guide.

Understanding Univariate Metrics

What Are Univariate Metrics?

Univariate metrics focus on one variable at a time. They’re like checking each ingredient in a recipe separately before tasting the final dish.

Why start with univariate metrics?

• They’re easier to understand and visualize
• They’re the foundation for more complex analysis
• They help you spot obvious problems quickly

Before your synthetic dataset can be considered high-quality overall, each individual variable needs to have a similar distribution to the original.

What’s a distribution? Think of it as the “shape” of your data. For example:

• Age distribution: How many people are in their 20s, 30s, 40s, etc.?
• Income distribution: How many people earn $30k–$40k, $40k–$50k, etc.?

Key Univariate Metrics

1. Kolmogorov–Smirnov (KS) Distance

What it measures: How similar are the distributions of numerical variables?

Explanation: Consider two datasets with an “age” field. For any given age threshold, one can compare the proportion of customers below that age threshold in each dataset. If the synthetic data closely matches the real data, these proportions will be similar across all thresholds.

The Kolmogorov–Smirnov (KS) distance captures the maximum absolute difference between these cumulative proportions as the threshold varies. A smaller KS distance indicates that the distributions are more similar.

Example:

• Real data: 30% of people are 20 years old, 40% are 30, 30% are 40
• Synthetic data: 25% are 20, 50% are 30, 25% are 40
• KS distance = 0.05 (the maximum difference, which is 5% in the 0–20 or the 0–30 age groups)

A KS distance of 0 means the numerical distributions are identical, while a value closer to 1 indicates greater difference: specifically, one distribution has many values below a threshold while the other one has many above it.

The KS distance is sensitive to numerical precision. For example, if a distribution concentrated on 1/3 is approximated as 0.33 in one dataset and 0.333 in another, the resulting KS distance can be significantly inflated due to the cumulative distribution function jumps at slightly different points (0.33 vs 0.333).

The KS distance does not apply to categorical variables because the set of values is not ordered.

2. Total Variation (TV) Distance

What it measures: How similar are the distributions of categorical variables?

Explanation: For categorical data (like city names, product categories, or yes/no answers), TV distance measures the total amount of “movement” needed to transform one distribution into another.

Example:

• Real data: 50% New York, 30% Los Angeles, 20% Chicago
• Synthetic data: 45% New York, 35% Los Angeles, 20% Chicago
• TV distance = 0.05 (because you’d need to shift 5% of the distribution — e.g., 5% from New York to Los Angeles — to align the two)

A TV distance of 0 means the categorical distributions are identical, while a value closer to 1 indicates greater difference.

The TV distance does apply to numerical variables as well, but it’s often different from the KS distance. For example, the age example above has a TV distance of 0.1.

3. Custom Metrics

In addition to the above two commonly used metrics, custom metrics based on domain knowledge can be extremely valuable for synthetic data evaluation. These metrics allow you to incorporate business rules, industry standards, and specific data characteristics that are relevant to your use case.

Examples of Custom Metrics:

• Cardinality Preservation: Using SQL’s COUNT(DISTINCT()) function to ensure the number of unique values in categorical fields remains similar between real and synthetic data. This is equivalent to hasDistinctness in Deequ and helps maintain the richness of categorical variables.
• Range Adherence: Checking that synthetic numerical values fall within the expected ranges of the original data, preventing unrealistic outliers that could skew downstream analysis. The metric can be dynamically created with consideration for outliers using SQL’s MIN and MAX functions, which are equivalent to isGreaterThan and isLessThan in Deequ.

Why Custom Metrics Matter: While KS and TV distances provide excellent statistical comparisons, they may not capture domain-specific nuances that are crucial for your particular application. Custom metrics bridge this gap by allowing you to define quality criteria that are directly relevant to your business context and use case requirements.

Univariate metrics with Rockfish

The Rockfish SDK provides utilities to easily compute univariate metrics on datasets. It also includes the marginal_dist_score function, which calculates an overall marginal distribution score by averaging the KS and TV distances.

See the Rockfish SDK documentation for more details.

Examples of High-Quality Synthetic Data

The following examples demonstrate synthetic data that successfully preserves the statistical properties of the original data, indicating excellent fidelity.

One example of the univariate distributions of a numerical field for the real and synthetic data is shown below. The KS distance is 0.07, which indicates very good quality: the synthetic data closely matches the real data’s distribution with minimal deviation.

One example of the univariate distributions of a categorical field for the real and synthetic data is shown below. The TV distance is 0.06, which represents very good quality: the synthetic data maintains similar proportions across all categories compared to the original data.

Examples of Poor-Quality Synthetic Data

The following examples illustrate synthetic data that fails to preserve the statistical properties of the original data, highlighting common quality issues that should be avoided.

One example of poor-quality synthetic data for a numerical field is shown below. The KS distance is 0.98, which indicates significant distribution mismatch: the synthetic data shows a much different pattern than the real data, with notable shifts in the distribution shape and concentration of values.

One example of poor-quality synthetic data for a categorical field is shown below. The TV distance is 0.39, which represents substantial quality issues: the synthetic data has dramatically different proportions across categories compared to the original data, with some categories being overrepresented while others are underrepresented.

Conclusion

In conclusion, understanding and utilizing univariate metrics such as the KS and TV distances is crucial for analyzing the similarity between distributions in datasets. These metrics provide valuable insights into both numerical and categorical data, allowing for a more comprehensive evaluation of data quality and distribution alignment. The Rockfish SDK further simplifies this process by offering tools to compute these metrics efficiently, making it an essential resource for data scientists and analysts. By leveraging these metrics, one can ensure more accurate data analysis and decision-making.

Authored by Xiaohan Kang, ML Engineer at Rockfish. Connect on LinkedIn.