Archived
chemavxandClaude Fable 5
124b6d8558
Scores every archived estimate against reality — the sample multiplier the phase plan calls for: Brier/log-loss of estimated_prob benchmarked against the market price (prior_prob) on the same rows, over ALL evaluations with a resolved outcome, not just executed trades. - schema.sql: market_outcomes (one row per resolved market; outcome = final YES price 1.0/0.0, UMA-final only) - bot/outcomes.py: CLI (python -m bot.outcomes) with two phases — fetch resolutions for archived markets via the existing get_market_resolution() (open/disputed/ambiguous markets simply retry next invocation; no data-loss urgency, Gamma reports past resolutions at any time), then compute calibration: Brier micro (per evaluation) / macro (per market — the honest sample size given ~1 eval/min autocorrelation), log-loss with 1e-9 clipping, per-category breakdown. --run-id scores a replay run's re-estimates instead of the archive (counterfactual calibration). - db.py: 4 accessors (pending markets, outcome upsert, coverage, calibration rows for archive or run) - tests: 12 new (116 total green); compute_calibration is a pure function driven by literals No prod behavior change: the bot never imports bot.outcomes; the only shared surface is the idempotent schema migration (dry-run BEGIN/ROLLBACK clean against prod). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>