superquant-bench: A Ground-Truth-Verifiable Benchmark for Autonomous Quantitative Research Agents

Evaluating whether an AI agent can write code is a solved-ish problem: you run the code and check the output against a known answer. Evaluating whether an AI agent can do quantitative research runs into a wall that code generation never hits — on real market data, there is no answer key.

A backtest cannot tell you whether the agent found real structure or overfit noise. False discoveries can only be corrected for after the fact, statistically, under distributional assumptions. And an agent evaluated on a fixed slice of history can leak the future through what it already memorized about that exact slice — undetectable in a one-shot run. The evaluator faces the same epistemic problem as the agent it is grading: nobody knows the true data-generating process, so "did it find something real?" has no ground truth.

superquant-bench takes the trade-off apart by giving up realism where realism costs you the answer key. It is a fully synthetic universe — 100 daily price series, ~12.5 years — built from a generator that knows, for every asset on every day, the exact true conditional mean of tomorrow's return. Twenty-two real alpha patterns are injected into an otherwise realistic null market. Because the generator planted them, it can grade against them exactly. That single move — manufacturing the ground truth instead of hoping to infer it — is what makes everything below possible.

This post walks through the whole design: why a synthetic answer key beats a realistic backtest for measuring research skill, the three scores and their zero noise floor, how look-ahead is enforced statistically rather than on trust, the exact prompts the agents receive (reproduced in full, since the point is that they are self-contained), a red-team of seven exploits that validates the metrics, and finally a controlled study of four frontier models under one identical scaffold — where the results are sharper than I expected.

TL;DR

• The core trick: a synthetic 100-asset price panel with 22 known alpha patterns injected, so the grader holds the exact true conditional mean E[rₜ | infoₜ₋₁] of every scored return. No inference, no noise floor.
• Prediction score, zero noise floor: forecasts are scored against the true conditional mean, not realized returns. The full-knowledge oracle scores exactly 100; iid noise scores exactly 0 — on every seed.
• False discovery measured, not estimated: run the same agent on a --no-patterns twin — the identical null market with nothing injected — and every claim it makes is, by construction, a counted false positive.
• Look-ahead enforced statistically: noise = realized − true mean is unpredictable from the past by construction, so any alignment between a submission and same-day noise is leakage, caught by a randomization test and voided.
• Memorization-proof: which asset carries which pattern (and each pattern's parameters) is drawn from a secret per-seed RNG, so held-out seeds stay secret even with all code public — a rotating-seed public leaderboard.
• The study: four frontier models (Claude Fable 5, Opus 4.8, Sonnet 5, Haiku 4.5) under one identical headless Claude Code scaffold. The spread is 10×, model identity explains 75% of prediction variance — and the most capable model is the least disciplined, inventing the most structure on the universe that contains none.

1. Why synthetic beats realistic — for this question

The instinct in benchmarking is that realistic data is always better. For measuring research skill, it is exactly backwards.

The core skill of a quant researcher is not producing a high backtest Sharpe. It is knowing which of the hundred things that look like alpha are actually real. On historical data that skill is unmeasurable, because the grader has no more access to the truth than the agent does. The entire multiple-testing literature — deflated Sharpe ratios, backtest-overfitting bounds, the Harvey–Liu–Zhu haircuts — exists precisely because you cannot measure data-snooping directly on real data; you can only correct for it, post hoc, under assumptions.

Invert it. If you built the universe, you know every pattern you planted, at what strength, in which asset, for which days. Now "did the agent find real structure?" is a lookup, not a philosophical problem. The contribution of superquant-bench is not the dataset — it is ground-truth-verifiable research hygiene, four properties a realistic benchmark structurally cannot have:

Realized-return metrics — MSE, realized information coefficient — were deliberately rejected. They reward a zero forecast and they pay out for luck. Grading against the true conditional mean is what removes the luck.

2. The universe

One universe = 100 daily price series over 3,150 business days (~12.5 years): 2,520 public training days, plus 630 test days whose prices are also public — so a submission can be computed genuinely walk-forward — but whose alpha decomposition is sealed. There is deliberately nothing else in the data: no volumes, no fundamentals, no metadata. Every pattern lives inside the price panel, so the claim "the agent found the structure" is self-contained and cannot lean on side information.

The null market is realistic enough that naive data mining throws off plausible false positives on its own:

• a market factor plus three style factors, so everything correlates with everything;
• GARCH(1,1) volatility with Student-t innovations (fat tails, clustering);
• per-asset stochastic idiosyncratic volatility.

Into that, 22 patterns are injected across four difficulty-tiered families. The point of the tiers is that a benchmark should have a floor you can clear and a ceiling nobody reaches yet:

The trap deserves a second look, because it is the benchmark's cleverest single object. One asset is constructed as a same-day composite of four others — so it has maximal contemporaneous correlation with them and zero predictive power at any lag. It is the exact shape of the thing that fools real researchers: a beautiful contemporaneous relationship that pays nothing when you try to trade it. Claiming it predicts anything costs you points.

3. Per-seed randomization — a secret seed with public code

A benchmark whose answer key is fixed is a benchmark that leaks the moment its data is on the internet. superquant-bench solves this the way a good crypto protocol does: all the code is public, the security lives entirely in a secret seed.

Three RNG streams derive from the seed — one for noise, one for role assignment, one for parameters. Every role in the universe — which assets carry patterns, which are predictors, which drives the regime, which are the trap's sources, which legs form the cointegrated pair — is dealt without replacement from one shuffled deck, so each asset is the target of at most one dedicated pattern and every auxiliary role is drawn from the signal-free pool. Every parameter is drawn from a difficulty-preserving range:

What stays structural — and therefore public — is the shape of the challenge: the 22-pattern inventory, the difficulty tiers, the trap, the two dying patterns, the market realism. What stays secret is which asset is which and how strong each effect is. A held-out seed remains genuinely held-out even though anyone can read the generator, which is what makes a rotating-seed public leaderboard possible without the board decaying into memorization.

Two more design details worth calling out. First, the assignment sampler has to run before the market is built, because the cointegrated legs need to share factor betas and correlated shocks — the roles determine the correlation structure, not the other way around. Second, an --alpha-scale knob multiplies every injected amplitude; a calibrated brutal tier around 0.65 halves the oracle's net Sharpe to roughly 1.5–2. That is the anti-saturation lever for future, stronger models. And a same-seed --no-patterns twin shares the exact base market — a matched-pair control that is the identical null world minus the injected alpha.

Every generated seed is checked by an answer-key-driven self-test: each pattern must be empirically present at its documented magnitude, the trap must be unpredictive at every lag, the dead patterns must be dead after their end date, and the nonlinear pattern must be linearly invisible. If any of that fails, the seed is not shipped.

4. Three scores, and a literal zero noise floor

The protocol is data-in, CSV-out. The agent receives a brief and the two price files, and returns a submission.csv with columns date, asset, forecast, weight — where the row for date d may use only prices strictly before d — plus an optional structured findings file. No agent code is executed by the evaluator; anything that can read and write a CSV is graded by the identical one-command grader.

There are three scores, and they measure genuinely different things:

The prediction score is the load-bearing idea. Because forecasts are graded against the true conditional mean rather than realized returns, there is no luck in it: the full-knowledge oracle scores exactly 100.0, iid noise scores exactly 0.0, on every single seed. That is the zero noise floor — and it is what makes the score able to separate a genuinely better forecaster from a lucky one, which no realized-return metric can do over 630 days.

Discovery grades the findings — the named patterns — against the answer key, with half credit for identifying the right series but the wrong mechanism, and hard penalties for hitting the trap or spraying false positives. Capture is the only score where luck lives: net Sharpe of the weight panel, normalized by the oracle's. Over 630 days the no-skill Sharpe band is roughly ±1.3, so capture alone can be gamed by variance — which is exactly why prediction, not capture, is the leaderboard's primary metric.

5. Integrity — enforcing walk-forward statistically, not on trust

Here is the subtle part. In a data-in/CSV-out protocol, the evaluator never sees the agent's code, so it cannot inspect for look-ahead. It has to detect it from the submission alone. superquant-bench can, because the synthetic construction gives it a statistic that is exactly mean-zero for any honest submission.

Define noise = realized return − true conditional mean. By construction, E[noiseₜ | infoₜ₋₁] is constant — the noise is unpredictable from anything in the past. So the mean per-asset Pearson correlation between a submitted panel and same-day noise is exactly zero in expectation for any legitimately-constructed forecast. (Pearson specifically: a rank statistic is not mean-zero here, because under drift plus volatility clustering the conditional mean of a return's rank drifts with the vol regime, which makes rank tests systematically positive for smooth reversal-style forecasts. That bug was found and fixed during multi-seed calibration.) The null is generated by summing daily contributions into ~month blocks and randomizing the blocks' signs — 1999 draws — which preserves the observed serial and cross-asset structure exactly.

A submission voids only on certainty and materiality: p ≤ 0.002 and statistic ≥ 0.03. Both gates matter, and the numbers below show why the thresholds sit where they do:

The honest full-knowledge oracle's alignment stays below 0.02 across seeds — its only signal is common-mode market autocorrelation, which enters every cross-sectionally-demeaned forecast with one sign. The weakest profitable contamination anyone could construct — mixing in 2% of same-day noise, worth about +25 net Sharpe — measures 0.61. There is an enormous gap between "honest" and "cheating-enough-to-matter," and the thresholds live inside it. Two backstops close the remaining loopholes: leakage concentrated in a few assets can stay under the mean-corr floor, but anything material there produces a gross Sharpe above the oracle's, which voids outright; and fitting parameters on the full panel including the test window — undetectable in any CSV-out protocol — buys only about 1 prediction point, because forecasts are graded against the true mean, not realized returns.

6. The prompts — reproduced in full

The whole benchmark is self-contained: the agent gets a task statement and two CSV files and nothing else — no generator, no evaluator, no answer key, not even a hint of how many patterns exist. Because that self-containedness is the point, here are the two prompts in full rather than as references. The first is the brief — the task statement that ships inside the workspace:

# Quantitative Research Task

You are given an anonymous universe of 100 daily price series.

FILES
  prices.csv        2,520 business days of training prices (public history)
  prices_test.csv     630 business days of test prices (public, continues the series)

GOAL
  Study the training data, decide what — if anything — is predictable, and
  produce next-day forecasts and positions for the test window.

DELIVERABLES
  submission.csv  (REQUIRED)  columns: date,asset,forecast,weight
      forecast  your estimate of the asset's next-day return signal for that date
      weight    your position in the asset for that date (any real number)
      WALK-FORWARD RULE: the row for date d may use ONLY prices strictly
      before d. This is enforced statistically after the fact (see below).

  findings.json   (RECOMMENDED)  a structured list of the patterns you claim to
      have found: for each, the asset(s) involved, the mechanism, the direction,
      and any condition or horizon. This is how you get credit for what you
      understood, separately from what you traded.

RULES & NOTES
  · The number of real, injected patterns is UNDISCLOSED and may be zero.
    Claiming structure that is not there is penalized. So is missing it.
  · A look-ahead detector runs on every submission. Forecasts that align with
    same-day return noise beyond chance are voided — as is any portfolio whose
    Sharpe exceeds what perfect knowledge of the signal could achieve.
  · Some series carry no predictable structure at all. Most, in fact.

The second is the scaffold prompt — the fixed wrapper that turns a bare model into an autonomous agent for this task. It is identical for every model in the study; the model string is the only variable:

Read BRIEF.md first, and follow it exactly.

Work only inside this directory. A Python virtual environment with numpy,
pandas, scipy, and scikit-learn is already available — use it; do not install
anything, and do not reach for the network.

Your deliverables are submission.csv (required) and findings.json (recommended),
written to this directory. Nothing else is collected.

You are on a hard wall-clock budget. Write a VALID submission.csv EARLY — even a
trivial one — and then improve it. An unfinished analysis that never wrote a file
scores zero, no matter how good the thinking was. When you are confident you have
extracted what you can, stop.

Every clause is doing a job. "The number of patterns is undisclosed and may be zero" is what makes the pattern-free null universe a fair test — the agent can never assume there is something to find. "Write a valid submission early, then improve it" prevents the failure mode where a model spends its whole budget on analysis and delivers nothing. And naming the walk-forward rule and the detector's existence in the brief means a leak is never an honest misunderstanding of the task — the agent was told, in writing, exactly what discipline is required and that it will be checked.

7. Red-team: does the metric survive an adversary?

Before trusting any of the scores, they were attacked. Seven scripted adversarial submissions, each an exploit a skeptical reviewer would reach for — carpet-bomb the findings, forecast everything weakly, copy the realized returns, trade the trap. A metric you can game is a metric that measures gaming, not skill.

Two things fall out of this table. First, the scores are conservative: nothing scores meaningfully above zero without identifying real mechanisms — breadth, carpet-bombing, and free lottery tickets all net to roughly nothing. Second, the void is reserved for actual look-ahead, not for aggressive-but-honest strategies. The realized-return copier — perfect look-ahead — voids with a statistic near 1.0. The in-window AR(1) fitter, which honestly fits parameters on the scored window, stays clean and gains about one point for its trouble, because the metric grades against the truth rather than the outcome.

8. The study — four frontier models, one scaffold

With the metrics validated, the benchmark was pointed at four frontier models: Claude Fable 5, Opus 4.8, Sonnet 5, and Haiku 4.5.

8.1 The harness

The scaffold is headless Claude Code (claude -p), pinned and identical for every model — again, the model string is the only variable. One session runs like this:

1. Workspace isolation. The harness builds the agent-facing task from the universe under evaluation — exactly four files: the brief, the training prices, the test prices, and a short data-readme — and extracts it into a fresh directory outside the repository. The agent never sees the generator, the evaluator, or the ground truth; there is no filesystem or network path to the answer key it has any reason to guess.
2. Tooling. A shared, read-only Python virtualenv (numpy, pandas, scipy, scikit-learn), the normal Claude Code file and shell tools inside the working directory, and web search/fetch disabled — so the session is fully self-contained.
3. The prompt. The one fixed scaffold prompt from §6, verbatim, every run.
4. Autonomous run. claude -p executes its normal agentic loop — read the data, write and run analysis scripts, iterate — with no human in the loop, until the model decides it is done or a limit fires. Budgets: 40 minutes of wall-clock (the harness then kills the process; anything written before the kill still counts) and a per-session cost ceiling as a safety net, never reached.
5. Grading. The harness collects the deliverables and grades them with the standard one-command evaluator — integrity gate first (randomization test, impossible-Sharpe, schema), then discovery / prediction / capture — logging one row per session.

What sessions actually did with this freedom was uniform in shape: read the brief and data, write pandas scans (lag-k cross-correlations, own-asset autocorrelation, seasonality contrasts, sometimes cointegration tests), apply some validation filter (train/holdout split, chronological folds, t-stat thresholds), construct walk-forward forecasts for what survived, write the files. The interesting variation was epistemic, not procedural: what survived each model's validation, and whether the walk-forward construction was actually walk-forward.

8.2 Headline scores

The design is a matrix: 4 models × 3 held-out universes × 3 sessions, plus one session per model on a pattern-free null universe under the same brief, plus scripted baselines. Protocol-faithful means (a voided or ungradeable session scores zero on every track), with bootstrap 95% CIs over the 9 sessions per model:

The prediction column is the one to watch, and it is cleanest as a picture. Note the oracle scores 100 by construction — the models are climbing the very bottom of that scale, which is the point: the benchmark is nowhere near saturated.

9. Seven findings

The report ranks seven findings by how much they are worth. The first three are the ones that changed how I think about this.

Finding 1 — The prediction track separates capability tiers, and the spread is now an order of magnitude

Fable scores 16.3 on prediction — 2.5× Opus's 6.5, 10× Sonnet's 1.6 — with a confidence interval that does not overlap any other model's, and posts the study's best single session at 23.5. The ordering Fable > Opus > Sonnet ≈ Haiku is monotone and clean. The most striking statistic: with only Sonnet and Haiku in the matrix, model identity explained 2% of prediction variance; adding Opus raised it to 46%; adding Fable raises it to 75%. Each stronger model added has made the score more discriminating, not less — which is the opposite of what a noisy metric does. Fable is the first model to reach the medium difficulty tier, where the conditional and cross-sectional mechanisms live, and it converts that reach into a net Sharpe of 1.95 and a capture of 44% — both clearing zero by a margin no other model approaches.

Finding 2 — The null universe ranks hygiene, and hygiene does not follow capability

This is the sharpest single point in the study. On the pattern-free control — same brief, and the agent is never told there is nothing to find — the four models behaved completely differently:

The most capable model is the least disciplined on the control. Fable laps the field on every patterned track and then invents the most structure on a market that has none — its null session was the only one of the four flagged for look-ahead, fitting same-day structure onto pure noise. Sonnet, mid-pack on capability, is the only model that correctly reports nothing. Capability and epistemic discipline are separable axes, and the benchmark scores them separately — a model can be a better forecaster and a worse scientist at the same time.

Finding 3 — The look-ahead gate catches real agent leakage, not hypothetical cheaters

Three of 36 patterned sessions voided. Haiku's was blatant — an 18.7 Sharpe portfolio, 3.2× the full-knowledge oracle, caught by both the impossible-Sharpe check and the randomization test at its p-floor. Sonnet's two voids were subtle: alignment statistics of 0.031–0.041, twenty-five times weaker than the copier's, and invisible in any backtest — the agent reported "net Sharpe ~1.8 on held-out train, plausible for stat-arb." They are consistent with an off-by-one in walk-forward feature construction (forecasts correlate +0.06 with same-day returns but −0.04 with the prior day's). Both sessions asserted strict walk-forward discipline in their write-ups. The agents did not know they were leaking. Statistical enforcement is not optional — and, tellingly, Fable, the strongest model, drew the study's highest flag rate, brushing against the leak boundary more than any other model even though it never crossed into a void.

Findings 4–7, briefly

• The benchmark is nowhere near saturated — but the frontier moved. Best single-session prediction is now 23.5/100, up from 12.2 in the earlier three-model matrix; the four-model mean is still only 6.6, against an oracle of 100. Two previously-uncracked frontiers partially fell (the medium tier, and converting discovery into real capture), while the hard tier still tops out at 0.21 credit. A clearly stronger model advanced the score without exhausting it — exactly what good headroom looks like.
• Precision discipline varies more than search procedure. Every model ran the same scan → filter → forecast shape; what differed was what survived validation. Sonnet's clean sessions submitted 2–17 findings with a median of one false positive; Haiku sprayed up to 23; Opus is bimodal; Fable, for all its accuracy, still submits 15–28 findings with 3–12 false positives — and sprayed most in its best-forecasting universe. The skill of not claiming remains the scarce one.
• Effort tracks capability — and still leaves budget on the table. The three earlier models each ran ~4.5–5 minutes of the 40-minute allowance at $0.24–1.13. Fable is the first to spend materially: ~9 minutes median, $4.03, up to 44 turns — and its best-scoring. But even Fable leaves ~78% of its wall-clock unused, and the priciest sessions are the null-universe ones — every model works hardest where there is nothing to find.
• With a wide-enough spread, model becomes the dominant variance component. Adding Fable flips the picture: model identity now explains 75% of prediction variance, 54% of net-Sharpe, and 53% of capture. Only discovery stays session-dominated (62% session, 23% seed, 15% model) — its recall/precision credit is the noisiest track by construction.

That last finding is worth one more picture, because it is the statistical justification for making prediction — not discovery or capture — the leaderboard's primary metric:

A caveat the report is careful about: n = 9 sessions/model leaves the discovery CIs overlapping, so the prediction gap, the null-universe false-discovery rate, and the failure-mode inventory are the load-bearing discrimination. All four models share a vendor and a scaffold, so this demonstrates the harness and the benchmark's headroom rather than a cross-vendor ranking. And seeds 2001–2003 are burned by publication; the live leaderboard uses fresh secret seeds.

10. What a score means — scope and limits

A high score is a competence floor, not a deployment claim. The logic runs in one direction: an agent that cannot find planted, guaranteed-present structure in a clean panel will certainly not find noisier, non-stationary, crowded real-world structure. Failure transfers; success does not. Real alpha is fainter and decays as it is discovered, and nothing here says a top-scoring agent makes money live.

The honest limitations, deliberately accepted: prices-only data is a design feature (self-contained claims) but excludes cross-modal research skill; the hypothesis space is one author's 22 mechanisms, which is why the long-term plan is a community pattern-plugin interface; public test prices admit in-window parameter fitting (measured at ~1 point); and pinning one agent framework means cross-vendor comparisons should treat the scaffold as a controlled variable — which the data-in/CSV-out protocol makes cheap to do.

The public leaderboard extends the study protocol: the generator and dev seeds are public; evaluation runs on secret seeds rotated each quarter; submissions are capped per epoch (the prediction score is a deterministic functional, so unlimited attempts would let you hill-climb); only aggregate scores are returned (per-pattern feedback would leak the key); and every entry carries a disclosure card — model, scaffold, budget, session count, human intervention, null-universe false-discovery rate. The endgame is a verified track where submitters provide agent code that runs under budget in a sandbox — which the study harness already is.

11. The takeaway

The one idea worth carrying out of this: for measuring research skill, manufacture the ground truth. The moment you stop trying to infer the answer from real data and start planting it yourself, a whole class of previously-impossible measurements opens up — a prediction score with no noise floor, a directly counted false-discovery rate, statistically enforced walk-forward discipline. Realism was never the goal; a verifiable answer key was.

And the finding that will stay with me is the separation of the two axes. The strongest model in the study is the best forecaster and the worst scientist — it extracts the most real signal and also invents the most fake structure where none exists. Capability and epistemic discipline are not the same thing, they do not improve together for free, and a benchmark that only measures one of them is telling you half the story. The skill of not claiming — of returning an empty submission when the universe is empty — remains, across every model tested, the scarce one.