A follow-up to Beyond Statistical Similarity
TL;DR
- Rockfish’s proprietary models come out ahead. On TabQueryBench (45 datasets, one 24 GB L4 GPU per job), RF-eel scores 0.853 against 0.813 for the best baseline model, RF-RealTabFormer [1], at under a third of the GPU cost (29.5 vs. 103.8 GPU-hours).
- Cost is a first-class axis. RF-rtf2 scores 0.809 in 11.8 GPU-hours, about 9x cheaper than RF-RealTabFormer for nearly the same fidelity.
- Agentic workflow can select the right model for your data. The best model depends on your table (categorical vs. numeric-heavy), your scale and your budget. The Rockfish Model Library lets you pick the right workflow for each, and we show how.
1. The acknowledged gap in tabular data generation: query-centric fidelity
Let us start with a question we hear from teams training synthetic data models: “My synthetic table looks just like the real one. Will it answer the same questions?” The honest answer, and the acknowledged gap in the literature, is that we usually do not know. Tabular generators are judged on statistical similarity (do the marginals and correlations match?). As we showed in our earlier post, a table can pass every global check and still get a simple subgroup query wrong, and even the strongest model recovered only 40.7% of real rare values. In one insurance example, TVAE scored a low global JSD of 0.058, yet a simple subgroup query lost the entire 52–64 age group from the answer.
TabQueryBench closes this gap by enabling practitioners to judge generative models on actual, realistic queries: 100+ executable SQL queries per dataset across five families (subgroup, conditional, tail/rarity, missingness, cardinality/range), scored with the released SV2 scorer. The headline result was that no model wins everywhere. This post asks the follow-up that matters to anyone choosing a generator: how do purpose-built, production models do on the same bar, and what do they cost?
Key takeaway: Judging generators by the answers they give, not by how real they look, exposes real differences in both fidelity and cost.
2. Methodology
The Rockfish Model Library is a growing family of proprietary RF models that goes beyond tabular data. In this post we focus on its tabular models (RF-eel, RF-rtf2, RF-ctg and RF-rtf v1), each built on a different modeling approach and engineered for the realities of production data: high-cardinality categories, missing values and numeric-heavy columns. Each runs out of the box with shipped defaults, and an optional precision-snap step (rounding numeric values to the source’s precision) adds about 0.010 on average.
To keep it apples-to-apples, we ran every Rockfish model with its shipped defaults and all 11 baselines as RF-hosted versions with the TabQueryBench authors’ released defaults, on the same 45 Tier 1 datasets with the same SV2 scorer and V8 query workload. Cost is GPU-hours (train plus generate) on one 24 GB L4 per job. Out-of-memory failures and a 36-hour cap count as results. Re-scoring the released outputs reproduced the paper within noise, so the comparison stands on the TabQueryBench authors’ own numbers.
We used the query-level results to find where our own models fell short, mostly on numeric-heavy tables, and identified ways to improve performance iteratively using clever preprocessing and encoding of the data. Training cost stayed flat. A separate generation fast path also made RF-eel’s generation 3.8x faster end to end at the same quality.
Key takeaway: Query-centric evaluation does not just rank models, it shows where to improve them. Better preprocessing and encoding lifted RF-eel by 0.028 overall and by up to 0.4 on the affected tables.
3. Key results: how does Rockfish compare?
Rockfish models are the state of the art in terms of fidelity and compute
Training custom or frontier models?
Tell us your data shape and budget and we will recommend the workflow. Get in touch →
As shipped, RF-eel matches RF-RealTabFormer (0.812 vs. 0.813) at a third of the compute, and with improved preprocessing and encoding it pulls ahead by 0.04. RF-rtf2 and RF-ctg sit on the cost-fidelity frontier. Robustness matters too: RF-eel and RF-rtf2 finished all 45 datasets on a single L4. RF-RealTabFormer could not fit the 2.46M-row Census table even on an 80 GB A100, and the released diffusion configs need roughly 52 h (tabbyflow) and 125 h (tabdiff) on one L4 for a 600k-row table.
Is there one best model? No. Match the workflow to the data and the budget.
| If your data / goal is… | Use this Rockfish workflow | What we measured |
|---|---|---|
| Categorical, mixed or high-cardinality tables; maximum fidelity | RF-eel | 0.896 vs. 0.871 for the best baseline model on the 29 categorical-heavy datasets; missingness structure 0.949 vs. 0.907; wins on a 104k-level column (c18) and 7k levels (c17) |
| Numeric-heavy tables (sensor, scientific, engineering data) | RF-eel with improved numeric encoding | Combined Cycle Power Plant 0.23 to 0.71, Wine Quality 0.47 to 0.87, Theorem Proving 0.37 to 0.70; overall 0.825 to 0.853 |
| Large tables, fast iteration, tight budget | RF-rtf2 (+ precision snap) | 0.809 at 11.8 GPU-h (best baseline: 0.813 at 103.8). On a 600k-row table: 0.916 in 0.72 h vs. 0.922 for RF-RealTabFormer in about 17 h |
| Lowest-cost lightweight pipeline | RF-ctg | 0.709 at 8.8 GPU-h, ahead of RF-CTGAN [2] (0.609) and TVAE (0.595), which cost 16 to 20 GPU-h |
Key takeaway: Rockfish gives you a menu, not a single model: RF-eel for fidelity, RF-rtf2 for cost, RF-ctg for the lightest footprint.
4. Takeaways
- Judge generators by the questions users ask. Query-centric benchmarks show where a model wins and where it breaks.
- Cost-accuracy is a frontier, and Rockfish defines it. RF-eel for top fidelity (0.853), RF-rtf2 for roughly 9x lower cost at 0.799, RF-ctg for the smallest footprint.
- Training custom or frontier models? Tell us your data shape and budget and we will recommend the workflow.
This is another step on our innovation journey: we keep benchmarking in the open and shipping improvements.
[1] Solatorio and Dupriez, “REaLTabFormer: Generating Realistic Relational and Tabular Data using Transformers,” 2023. arXiv 2302.02041
[2] Xu, Skoularidou, Cuesta-Infante and Veeramachaneni, “Modeling Tabular Data using Conditional GAN,” NeurIPS 2019. arXiv 1907.00503