Agents that ace standard benchmarks still fail on the domain-specific, incident-aware questions your users actually ask. Rockfish builds time-series eval suites - grounded in your data - that show exactly where yours falls short.
No ground truth to check against. Ships silently.
Graded against the real number. Caught pre-production.
Same answer, opposite outcome. Ground truth is the difference.
40 of 152 answers in this run were wrong - invisible without it.
The problem usually isn't the agent - it's the eval. Generic benchmarks under-test the patterns and questions that actually matter in production. Two gaps cause most of it:
Real observability, telecom, and security data has time-aware structure - correlated signals, incident windows, anomaly spikes, and event sequences that unfold over time. An agent tested on generic data has never been tested on yours.
Production users ask incident-aware, context-specific questions. An agent that can't answer them isn't production-ready - but a generic benchmark would never catch it.
The gap between "passes benchmarks" and "ready for your users" is wide - and invisible until you test for it. Read the AgentFuel paper on arXiv →
AgentFuel turns your schema and domain context into an expressive eval suite, grounded in the real patterns your agent will hit in production.
See where your agent fails on time-series and incident-aware questions before a user reports the wrong answer that costs you trust.
Track improvements and regressions release over release with grounded, domain-specific scorecards and reproducible, versioned benchmarks.
Generic benchmarks show what's possible. Bespoke evals show whether your agent is actually ready for your domain, your data, and your users.
Find out where your agent actually stands - with an eval suite built for your domain, your data, and your users' real questions.