feat(replay): R2 outcomes + calibration metrics
Scores every archived estimate against reality — the sample multiplier the phase plan calls for: Brier/log-loss of estimated_prob benchmarked against the market price (prior_prob) on the same rows, over ALL evaluations with a resolved outcome, not just executed trades. - schema.sql: market_outcomes (one row per resolved market; outcome = final YES price 1.0/0.0, UMA-final only) - bot/outcomes.py: CLI (python -m bot.outcomes) with two phases — fetch resolutions for archived markets via the existing get_market_resolution() (open/disputed/ambiguous markets simply retry next invocation; no data-loss urgency, Gamma reports past resolutions at any time), then compute calibration: Brier micro (per evaluation) / macro (per market — the honest sample size given ~1 eval/min autocorrelation), log-loss with 1e-9 clipping, per-category breakdown. --run-id scores a replay run's re-estimates instead of the archive (counterfactual calibration). - db.py: 4 accessors (pending markets, outcome upsert, coverage, calibration rows for archive or run) - tests: 12 new (116 total green); compute_calibration is a pure function driven by literals No prod behavior change: the bot never imports bot.outcomes; the only shared surface is the idempotent schema migration (dry-run BEGIN/ROLLBACK clean against prod). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
6c544e46e2
commit
124b6d8558
@@ -431,3 +431,24 @@ CREATE TABLE IF NOT EXISTS replay_decisions (
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_replay_decisions_run ON replay_decisions(run_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_replay_decisions_mkt ON replay_decisions(market_id);
|
||||
|
||||
-- ─────────────────────────────────────────────────────────────────────────────
|
||||
-- Replay R2: outcomes + calibration metrics
|
||||
--
|
||||
-- One row per resolved market, fetched from the Gamma API via
|
||||
-- get_market_resolution() (UMA-final only: a market closed but still in
|
||||
-- proposal/dispute is not stored). outcome is the final YES price:
|
||||
-- 1.0 = YES won, 0.0 = NO won.
|
||||
--
|
||||
-- Joining signals (or replay_decisions) to market_outcomes scores every
|
||||
-- archived estimate against reality — Brier / log-loss of estimated_prob
|
||||
-- benchmarked against the market price (prior_prob) on the same rows,
|
||||
-- answering "does the model add value over the market?" across ALL
|
||||
-- evaluations, not just executed trades.
|
||||
-- ─────────────────────────────────────────────────────────────────────────────
|
||||
CREATE TABLE IF NOT EXISTS market_outcomes (
|
||||
market_id TEXT PRIMARY KEY,
|
||||
outcome DOUBLE PRECISION NOT NULL,
|
||||
resolved_at TIMESTAMPTZ,
|
||||
fetched_at TIMESTAMPTZ DEFAULT NOW()
|
||||
);
|
||||
|
||||
Reference in New Issue
Block a user