Synthetic data that passes every global fidelity check can still give wrong answers to real analytical queries. For example, across 49 datasets and 11 models, looking at the tail, even the strongest model recovered only 40.7% of real rare values. They also fail to support columns with high-cardinality discrete values. To evaluate synthetic data by the questions people actually ask of it, we built TabQueryBench.
Your synthetic dataset sails through every fidelity check. The distributions match, the correlations look right, the summary statistics say it is a near-perfect copy of the real thing. You ship it. Then an analyst runs a simple query — “which customer segment churns the most?” — and gets the wrong answer.
This happens more often than you would expect. After experimenting across several datasets and state-of-the-art tabular generative models - spanning healthcare, finance, education, e-commerce, cybersecurity, demographics, social media, industrial applications, and so on - we found that even strong generative models can nail global statistics while quietly getting the answers to real analytical questions wrong. In the hardest cases, the best model we tested recovers only 40.7% of real rare values - the long-tail events that fraud detection, failure analysis, and small-segment analytics depend on.
That gap is the reason we built TabQueryBench. It marks a shift in how we judge synthetic data: from “does it look real?” to “does it answer the same questions the real data would?”
How people actually use data
Users query data, they don't read rows — so evaluation should test the answers, not just the table.
Modern tabular generative models are genuinely impressive. They learn complex relationships across columns, produce realistic-looking records, and preserve the global statistical shape of a dataset. But generating the data is only half the job. The harder question is whether it will actually work once people start using it.
And here is the thing: users almost never consume a dataset row by row. They ask it questions.
- Which groups contribute the most?
- Does a relationship still hold under a specific condition?
- Are the rare cases still there?
A synthetic table can look flawless at the global level and still return wrong answers to every one of these. If evaluation only checks whether the table looks real, it misses the part users actually care about.
The evaluation gap
Distance metrics and ML utility both miss whether analytical behavior actually survives.
Today, synthetic data quality is commonly evaluated through several complementary perspectives.
- Statistical distance metrics are fast and tell you whether the data resembles the real thing globally, such as SynthCity and SDMetrics — but they can miss the local structures that analytical work relies on.
- Machine-learning utility checks whether the data supports downstream prediction, such as SDMetrics — but it means retraining models after every change, which grinds the development loop to a crawl.
Put together, these give you valuable signals. What none of them answers directly is the one that matters most to an end user: does the synthetic data preserve the analytical behaviors people actually depend on?
Building Query-Centric Evaluation with Real Analytical Workloads
TabQueryBench turns real query workloads into 100+ grounded SQL tests per dataset, across five families.
Analytical systems have long been evaluated through recurring query workloads — from TPC-H and TPC-DS to newer analytical benchmarks such as ClickBench and the H2O. If queries capture how people interact with data, they can also serve as structural tests for synthetic data fidelity. TabQueryBench extends that decades-old idea to synthetic data, in two stages.
Stage 1 — Learn reusable query patterns
Rather than inventing questions, TabQueryBench distills reusable analytical patterns from public query workloads, SQL repositories, and database documentation. They fall into five families, each probing a different but important behavior:
- Subgroup — does the data preserve group-level behavior?
- Conditional — do relationships hold under specific conditions?
- Tail / Rarity — are rare events and extreme cases preserved?
- Missingness — are patterns of missing values maintained?
- Cardinality / Range — is value support and numerical range preserved?
Stage 2 — Ground the queries in real datasets
Each reusable template is then adapted to a specific dataset through schema-aware grounding: a general pattern becomes an executable SQL query by picking the relevant columns, valid filters, aggregation operations, and dataset-specific constraints.
The result: more than 100 executable queries per dataset, across the 49 datasets and 11 models — a fine-grained X-ray of how each generator behaves.
What Query-Centric Evaluation Reveals
No model wins everywhere; looking real doesn't guarantee correct answers, and localized/rare structure is where models break.
There is no single best synthetic data model. Different generative models preserve different behaviors, and knowing which is which is the whole point of choosing one.
Looking real ≠ being correct
Models with nearly identical distance scores gave meaningfully different answers to real queries. A generator can preserve the overall distribution while quietly shifting the subgroup relationships, conditional patterns, or value support that downstream users rely on.
A concrete example. Consider an insurance dataset and the following subgroup query: among non-smokers, which age groups account for the largest total number of children? TVAE achieves a low global JSD of 0.058, suggesting strong overall similarity. Yet the query result changes substantially: the leading age groups shift, and ages 52–64 contribute 156 in the real data but disappear entirely from the synthetic result.
Global similarity tells you whether a table LOOKS right. It does not tell you whether it PRESERVES the right structure.
This is why we ask not whether the synthetic table looks similar, but whether it yields consistent answers — the numbers will never be identical, so the bar is whether an analyst would reach the same conclusion.
Different models, different quality–cost tradeoffs
With everything from Bayesian methods (BayesNet) to neural approaches (CTGAN, TVAE), diffusion-based models (TabDDPM, TabSyn, TabDiff, ForestDiffusion), transformer-based generative models (TabPFGen, REaLTabFormer), and other modern tabular generative models (TabbyFlow, ARF), choosing a model is not just about average quality — it is about the tradeoff between fidelity, compute cost, and what your workload actually needs. Our results point to a practical playbook:
- Is accuracy the top priority? Reach for the strongest query-centric fidelity. RealTabFormer led our benchmark overall, followed by BayesNet and ARF.
- Scale and iteration speed matter more? BayesNet offered the best fidelity-to-cost balance — competitive quality at substantially lower generation cost.
- Rare-event analytics? Test tail fidelity in your own domain; RealTabFormer was relatively stronger here.
- Complex categorical analytics? Check cardinality preservation; RealTabFormer and CTGAN were relatively stronger.
This connects directly to our own work at Rockfish, which grew out of the same CMU research lineage — including years of peer-reviewed work on synthesizing rare events and edge cases. Preserving the long-tail structure this benchmark finds hardest is exactly what we focus on when generating data for evaluation, rather than optimizing for global similarity alone.
New research in collaboration with Jialin Zhang(TongjiU), Shinan Liu (HKU), Fenghao Dong(CMU), Yajie Zhou(UMD), Vyas Sekar(CMU). Reference paper: TabQueryBench; Open sourced at: HuggingFace, GitHub.