Built on Wade · 1,000 agents · 61 forecast dates · 2 arms

The same agents, scored on what they knew and on what we know now

Each agent forecast US GDP growth 61 times, reading economic data through one endpoint. A session token fixes the date it reads as of. In the control arm the same agent reads the same months at today's revised values, which is what a data store without publication dates hands every backtest.

638 / 1,000agents score better with hindsight than they really did
0 / 61,000as-known reads that included a value published after their date
60,979 / 61,000control reads that did
20,140 /srequests served with 2,000 agents connected, on two cores

Every agent, both ways. Query them.

Forecast error against the first official estimate, over 56 quarters. On the dashed line an agent scores the same either way. Below it, hindsight made the agent look better than it was. The terminal runs DuckDB in your browser over the same results. Any query that returns agent_id lights those agents up on the chart.

Looks better with hindsightLooks the same or worseFrom your query
Hover or tap a point for the agent behind it.
loading DuckDB….help · .tables · .clear

Who you would have picked

Rank the agents on revised data and five of the top ten were not in the real top 100. Only 52 of the hindsight top 100 belong there.

The crowd against GDPNow

Root mean squared error in percentage points, same 56 quarters. The agents are four-variable regressions with no model calls, so the crowd is there to load the endpoint. It isn't trying to beat the Atlanta Fed.

ForecasterAs knownHindsight
GDPNow on the same date1.051.05
Mean of the 100 agents with the best record so far1.561.51
Mean of all 1,000 agents1.631.60
Median single agent2.142.06

How fast the endpoint answers

2,000 connections held open, each request reading five series (about 1,700 values, 30 KB). Two server processes on a 4-core VM that also ran the load generator.

LoadReq /sp50p99
Swarm dates, one read a second each2,0730.45 ms1.5 ms
Swarm dates, back to back20,14085 ms473 ms
Any day 2011 to 2026, every 2 s1,0481.6 ms13.6 ms
Any day 2011 to 2026, back to back2,564807 ms948 ms

Back-to-back rows measure throughput, so their latency is time queued behind 2,000 other agents. The full replay, 122,000 sessions and 3.9 GB of responses, took 71 seconds, and every forecast matched the same agents computed inside the store.

The intent checker

Wade turns an ambiguous question into explicit readings, a person picks one, and a separate checker reads each SQL query against what the question meant, across ten dimensions such as time window, grain and filters. The model scores the dimensions. The verdict is computed in code. It was tested on 60 candidate queries across 14 analytics questions, 36 of them built to be wrong in exactly one way.

36 / 36wrong queries flagged
33 / 36blamed on the exact dimension that was wrong
22 / 24correct queries passed

How it works

  1. The store2.95 million dated facts for 469 US series: every revision FRED and the Philadelphia Fed hold, each stamped with the day it was published.
  2. The sessionsAn agent opens a session with an as-of date. Every read in that session returns only what had been published by then. The control arm gets the same months at today's values.
  3. The agentsEach agent is dealt four series and refits a regression every quarter on the data as its session shows it. Scores leave out 2020 and 2025 Q3.
  4. WadeWade pins each investigation: the question, the chosen reading, the data snapshot and every step, in an append-only log. Results are content-hashed, so a rerun passes only when it produces the same bytes.