by Deepti Mande, Director of Product and Vyas Sekar, Co-founder and Chief Technologist (Rockfish Data) and Professor (Carnegie Mellon University) and Chief Scientist (Conviva)
- Many synthetic-data projects start from a schema and a real use case, not a dataset. What you need is data that meets the use case’s structural needs — timing, sequence, and referential integrity — not just rows that look right.
- An LLM on its own, or writing code with a library like Faker, is great for simple, flat data. On the semantically hard use cases, it gives you data that looks realistic but isn’t.
- Give that same LLM the structure it was missing with Rockfish’s tools and skills and it passes the structural checks the plain-LLM approaches fail, using fewer tokens
Looking realistic isn’t being realistic
Many synthetic-data projects start the same way. You have a schema and a real use case in mind, but not the data — maybe it’s too sensitive to copy, maybe there just isn’t enough of it. So you generate it. The catch is that data that looks superficially realistic isn’t the same as data that’s actually realistic, and for use cases like network flows, packet captures, or security logs, that gap is exactly the part that doesn’t show up on a quick glance.
What a use case actually needs
What “realistic” means depends on the use case. We looked at four, each with its own structural needs:
- Network flows (Netflow): the protocol mix and the byte/packet distributions have to be plausible, not uniform — real traffic is bimodal, tiny DNS lookups on one end and big transfers on the other.
- Packet captures (PCAP): frames and timestamps run in order, and a TCP session has to move through real states — it opens with a SYN before anything else can happen.
- Observability : traces and spans share IDs, latency percentiles line up (p50 ≤ p95 ≤ p99), and an incident shows up in the metrics, the logs, and the traces at the same time.
- Security logs: the same host resolves across Windows events, Sysmon, and Zeek, attack labels stay consistent, and every process that terminates was created somewhere first.
None of this is about schema syntax or formatting. It’s semantic structure — timing, sequence, and referential integrity — and it’s what the use case actually depends on.
Three ways to go from schema to data
If you’re starting from just a schema, there are three ways people go about it. All three run using an LLM — what changes is the harness around it.
What we found
For each use case we assembled a set of semantic data-quality checks — the consistency rules that matter for that domain — and scored every output against them. Same schema, 1,000 rows, all four use cases.
The syntax checks — valid IP, right columns — everyone passes. It’s the semantically challenging ones where the plain-LLM approaches fall over.
Take a concrete one: on a packet capture, does a TCP session actually follow the protocol semantics?
- Direct LLM and LLM + code both get it right 0% of the time;
- LLM + Rockfish, 100%.
It makes sense once you see it — an LLM writing rows one at a time has no idea there’s a session underneath, so it just fills in flags that look fine on their own row.
These are illustrative — one check per use case, not the full set. The LLM + code approach does better than you’d expect on the simpler structure: point it at a shared host ID and it’ll keep that straight across files. Where it still struggles is anything stateful — an incident that has to land in the metrics and the logs and the traces together, or a process terminated without ever being created. LLM + Rockfish was the only one that got all of them.
Beyond correct: reliable and token-efficient
Passing the checks is table stakes. For a real pipeline it also has to be reliable to re-run and easy on token spend. Even at 1,000 rows, two things stood out:
First, you don’t even get all your rows — ask Direct LLM for 1,000 and you get 200–350 back before it runs out of output room, while the code and Rockfish paths return the full set. Second, token spend and time: on the security use case, the hardest one, Direct LLM ran up the highest token bill and took minutes, while LLM + Rockfish finished in seconds for a fraction of the spend. (Direct LLM shown for Sonnet; larger scales in progress.)
How we ran it: schema and intent only, no seed data; 1,000 rows per primary file across four use cases — network flow, packet capture, observability, security logs. Direct LLM tested on Claude Haiku, Sonnet, and Opus. Checks are per-use-case semantic consistency checks; cost is measured Claude API token spend. One run per cell this round; repeats for variance in progress.
The takeaway
None of this is LLM versus anything — every approach here uses an LLM. The real question is how to use one well for a given use case, and that comes down to what you put around it. For flat, simple data, a prompt or a code library is all you need. When the data has to hold together — timing, sequence, referential integrity — the LLM needs a harness: encodings for sequence and state, generators for cross-entity relationships, and the semantic checks to validate against. Rockfish is that harness, and its skills tell the LLM how to drive it. That’s the difference between synthetic data that looks right and synthetic data you can actually build on.