Each agent forecast US GDP growth 61 times, reading economic data through one endpoint. A session token fixes the date it reads as of. In the control arm the same agent reads the same months at today's revised values, which is what a data store without publication dates hands every backtest.
Forecast error against the first official estimate, over 56 quarters. On the dashed line an agent scores the same either way. Below it, hindsight made the agent look better than it was. The terminal runs DuckDB in your browser over the same results. Any query that returns agent_id lights those agents up on the chart.
Rank the agents on revised data and five of the top ten were not in the real top 100. Only 52 of the hindsight top 100 belong there.
Root mean squared error in percentage points, same 56 quarters. The agents are four-variable regressions with no model calls, so the crowd is there to load the endpoint. It isn't trying to beat the Atlanta Fed.
| Forecaster | As known | Hindsight |
|---|---|---|
| GDPNow on the same date | 1.05 | 1.05 |
| Mean of the 100 agents with the best record so far | 1.56 | 1.51 |
| Mean of all 1,000 agents | 1.63 | 1.60 |
| Median single agent | 2.14 | 2.06 |
2,000 connections held open, each request reading five series (about 1,700 values, 30 KB). Two server processes on a 4-core VM that also ran the load generator.
| Load | Req /s | p50 | p99 |
|---|---|---|---|
| Swarm dates, one read a second each | 2,073 | 0.45 ms | 1.5 ms |
| Swarm dates, back to back | 20,140 | 85 ms | 473 ms |
| Any day 2011 to 2026, every 2 s | 1,048 | 1.6 ms | 13.6 ms |
| Any day 2011 to 2026, back to back | 2,564 | 807 ms | 948 ms |
Back-to-back rows measure throughput, so their latency is time queued behind 2,000 other agents. The full replay, 122,000 sessions and 3.9 GB of responses, took 71 seconds, and every forecast matched the same agents computed inside the store.
Wade turns an ambiguous question into explicit readings, a person picks one, and a separate checker reads each SQL query against what the question meant, across ten dimensions such as time window, grain and filters. The model scores the dimensions. The verdict is computed in code. It was tested on 60 candidate queries across 14 analytics questions, 36 of them built to be wrong in exactly one way.