chemavx/polymarket-bot
1
0
Fork 0
Code Issues Pull Requests 10 Actions Packages Projects Releases 1 Wiki Activity

feat(strategy): GNews guardrail — clamp news-only shifts to prior±0.25

Browse Source 
Post-mortem NVIDIA 631181: one uncorroborated high-weight signal (legacy
Manifold 0.13 at weight 0.6) flipped a 0.845 market to 0.431 and lost.
With Manifold observational-only and macro signals gated behind
is_non_price, GNews (weight 1.5) is the only live signal able to move
politics markets 20-30 pp against the order-book consensus.  This adds a
catastrophic fuse, not a fine calibration:

- apply_news_guardrail(): when |news_lo| >= NEWS_MATERIAL_LOGODDS_THRESHOLD
  (0.10) and every other signal (fg, mom, btc_dom, mfld) is below it,
  clamp the posterior to prior ± MAX_NEWS_ONLY_PROB_SHIFT (0.25).  Any
  corroborating material signal disables the clamp.  Config via env
  (NEWS_GUARDRAIL_ENABLED=true by default).
- edge_gross/edge_net computed from the clamped posterior; raw_final_prob
  preserved in reasoning (persisted via trades.reasoning — no schema
  migration) and in the NEWS_MATERIAL log line.
- guardrail_changed_trade_decision: raw edge crossed the regime gate but
  the clamped edge no longer does (fuse prevented a trade).  Note: with
  the default 0.25 band the clamped edge_net is 0.21, above every regime
  minimum, so the flag only fires with a tighter configured band.
- Observability gated on materiality: NEWS_MATERIAL per-market line and a
  compact NEWS SUMMARY cycle line, only when with_news > 0 — no flood
  from the ~145 news-less markets per cycle.
- 9 deterministic tests (extreme clamp, in-band passthrough, corroboration,
  inclusive threshold, disabled, changed_decision).

No changes to NEWS_LOGODDS_WEIGHT, Manifold flags, edge thresholds,
sizing, payout, resolution, or historical trades.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
chemavx
2026-07-01 20:26:02 +00:00
co-authored by Claude Fable 5
parent 9e21ecac21
commit 7f84bc3ec7
3 changed files with 416 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
bot/main.py
+14
View File
@@ -277,6 +277,20 @@ async def run_trading_loop(
manifold_summary,
)
# NEWS SUMMARY — one compact line, only on cycles where at least
# one market had a material GNews contribution (never an empty
# section on news-less cycles).
if stats["news_with_material"] > 0:
log.info(
"NEWS SUMMARY | with_news=%d | avg_shift=%+.2f | "
"max_shift=%+.2f | guardrail_applied=%d | changed_decisions=%d",
stats["news_with_material"],
stats["news_avg_shift"],
stats["news_max_shift"],
stats["news_guardrail_applied"],
stats["news_changed_decisions"],
)
# 9. Update daily metrics
await metrics.update_daily_summary()
bot/strategy/bayesian.py
+155 -12
View File
@@ -84,6 +84,27 @@ def _env_bool(name: str, default: bool) -> bool:
MANIFOLD_SIGNAL_ENABLED = _env_bool("MANIFOLD_SIGNAL_ENABLED", False)
MANIFOLD_AUDIT_ENABLED = _env_bool("MANIFOLD_AUDIT_ENABLED", True)
# ── GNews guardrail (catastrophic fuse) ────────────────────────────────────────
# Post-mortem NVIDIA 631181: a single strong signal (legacy Manifold 0.13 at
# weight 0.6) flipped a 0.845 market to 0.431 and lost. With Manifold now
# observational-only and macro signals gated behind is_non_price, GNews
# (weight 1.5) is the only live signal that can move politics markets 20-30 pp
# against the order-book consensus. This is NOT a fine calibration — it is a
# fuse against the extreme case: one uncorroborated signal violently inverting
# the market.
#
# NEWS_GUARDRAIL_ENABLED: master switch for the fuse.
# MAX_NEWS_ONLY_PROB_SHIFT: when GNews is the ONLY material signal, the final
# probability is clamped to prior ± this value. 0.25 still allows a 25 pp
# move (edge_net 0.21 after costs) — trades still happen, sizing is bounded.
# NEWS_MATERIAL_LOGODDS_THRESHOLD: a signal counts as *material* iff its
# |log-odds contribution| >= this value. Below it, a signal is noise and
# does NOT count as corroboration. If ANY other signal (fg, momentum,
# btc_dom, manifold) is material, the fuse does not apply.
NEWS_GUARDRAIL_ENABLED = _env_bool("NEWS_GUARDRAIL_ENABLED", True)
MAX_NEWS_ONLY_PROB_SHIFT = float(os.getenv("MAX_NEWS_ONLY_PROB_SHIFT", "0.25"))
NEWS_MATERIAL_LOGODDS_THRESHOLD = float(os.getenv("NEWS_MATERIAL_LOGODDS_THRESHOLD", "0.10"))
# GNews free tier: 100 req/day. We limit to 5 queries per trading cycle
# (politics markets only) and rely on 6 h cache to stay within budget.
MAX_NEWS_QUERIES_PER_CYCLE = 5
@@ -179,6 +200,42 @@ def has_token(text: str, token: str) -> bool:
# Phase 3 — GNews priority scoring
# ─────────────────────────────────────────────────────────────────────────────
def apply_news_guardrail(
prior: float,
raw_final_prob: float,
feat_news_lo: float,
other_feats_lo: tuple[float, ...],
) -> tuple[float, bool]:
"""
GNews guardrail (catastrophic fuse).
Clamp raw_final_prob to prior ± MAX_NEWS_ONLY_PROB_SHIFT when ALL hold:
1. NEWS_GUARDRAIL_ENABLED
2. |feat_news_lo| >= NEWS_MATERIAL_LOGODDS_THRESHOLD (news is material)
3. every other signal's |log-odds contribution| is below the threshold
(GNews is the ONLY material signal — no corroboration)
Returns (final_prob, guardrail_applied). guardrail_applied is True only
when the clamp actually changed the value; a raw_final_prob already inside
the band passes through untouched with applied=False.
Module globals are read at call time so tests can monkeypatch them.
"""
if not NEWS_GUARDRAIL_ENABLED:
return raw_final_prob, False
if abs(feat_news_lo) < NEWS_MATERIAL_LOGODDS_THRESHOLD:
return raw_final_prob, False
if any(abs(v) >= NEWS_MATERIAL_LOGODDS_THRESHOLD for v in other_feats_lo):
return raw_final_prob, False # corroborated — fuse does not apply
clamped = min(
max(raw_final_prob, prior - MAX_NEWS_ONLY_PROB_SHIFT),
prior + MAX_NEWS_ONLY_PROB_SHIFT,
)
if clamped == raw_final_prob:
return raw_final_prob, False
return clamped, True
def gnews_priority(market: Market, news: "NewsClient") -> float:
"""
Score a market for GNews query priority (higher = more valuable to query).
@@ -300,6 +357,10 @@ class BayesianStrategy:
# (edge_gross, edge_net, regime_min) for every market that reached the
# edge computation stage (passed prior-extreme, family, unsupported filters)
self._evaluated_edges: list[tuple[float, float, float]] = []
# GNews guardrail observability — only markets with material news
self._news_shifts: list[float] = [] # final_prob - prior, signed
self._news_guardrail_applied: int = 0
self._news_changed_decisions: int = 0
def reset_cycle(self) -> None:
"""Call once at the start of each trading cycle to reset per-cycle counters."""
@@ -311,6 +372,9 @@ class BayesianStrategy:
self._manifold_fetched = 0
self._manifold_on_trade = 0
self._evaluated_edges = []
self._news_shifts = []
self._news_guardrail_applied = 0
self._news_changed_decisions = 0
def get_cycle_stats(self) -> dict:
"""Return per-cycle counters for the [CYCLE SUMMARY] log block."""
@@ -330,6 +394,14 @@ class BayesianStrategy:
"gross_gt_004": sum(1 for g in all_gross if g > 0.04),
"manifold_matches_accepted": self._manifold_on_trade,
"manifold_matches_rejected": self._manifold_fetched - self._manifold_on_trade,
# GNews guardrail — markets with |news_lo| >= NEWS_MATERIAL_LOGODDS_THRESHOLD
"news_with_material": len(self._news_shifts),
"news_avg_shift": (sum(self._news_shifts) / len(self._news_shifts))
if self._news_shifts else 0.0,
"news_max_shift": max(self._news_shifts, key=abs)
if self._news_shifts else 0.0,
"news_guardrail_applied": self._news_guardrail_applied,
"news_changed_decisions": self._news_changed_decisions,
}
async def evaluate(
@@ -503,6 +575,7 @@ class BayesianStrategy:
# Phase 3: caller has pre-sorted markets by gnews_priority() so the
# highest-value markets reach this block first.
news_log_adj = 0.0
news_sentiment = 0.0
# self._news.enabled gates the whole block: with no GNews API key the
# client is a no-op, so we must not consume (or report) query budget for
# it — see NewsClient.enabled.
@@ -511,6 +584,7 @@ class BayesianStrategy:
self._news_queries_this_cycle += 1
sentiment = await self._news.get_sentiment(market.question)
if abs(sentiment) > 0.05:
news_sentiment = sentiment
news_log_adj = sentiment * NEWS_LOGODDS_WEIGHT
sources.append(f"GNews: {sentiment:+.2f}")
else:
@@ -652,8 +726,31 @@ class BayesianStrategy:
# Posterior via log-odds updating
log_odds_prior = math.log(prior / (1 - prior))
total_adj = sum(adjustments)
estimated_prob = _sigmoid(log_odds_prior + total_adj * 2 + news_log_adj + manifold_log_adj)
estimated_prob = max(0.05, min(0.95, estimated_prob))
# raw_final_prob: posterior BEFORE the news guardrail.
raw_final_prob = _sigmoid(log_odds_prior + total_adj * 2 + news_log_adj + manifold_log_adj)
raw_final_prob = max(0.05, min(0.95, raw_final_prob))
# Per-feature log-odds contributions (Phase 6) — computed here (not
# after the edge gate) because the guardrail below needs them to decide
# signal materiality.
# fg / mom / btc_dom: probability-delta × 2 → log-odds.
# news / mfld: already log-odds (LOGODDS_WEIGHT already applied).
feat_fg_lo = _fg_contribution * 2
feat_mom_lo = _momentum_contribution * 2
feat_news_lo = news_log_adj
feat_mfld_lo = manifold_log_adj
feat_btc_dom_lo = _btc_dom_contribution * 2
# ── GNews guardrail (catastrophic fuse) ──────────────────────────────
# When GNews is the ONLY material signal, clamp the posterior to
# prior ± MAX_NEWS_ONLY_PROB_SHIFT. estimated_prob (post-guardrail) is
# what edge/trading uses; raw_final_prob is kept for observability.
estimated_prob, news_guardrail_applied = apply_news_guardrail(
prior,
raw_final_prob,
feat_news_lo,
(feat_fg_lo, feat_mom_lo, feat_btc_dom_lo, feat_mfld_lo),
)
# ── Phase 1: edge_gross and edge_net ─────────────────────────────────
raw_edge = estimated_prob - market.yes_price
@@ -675,15 +772,6 @@ class BayesianStrategy:
if manifold_log_adj != 0.0:
confidence = min(confidence_cap, confidence + 0.08)
# Per-feature log-odds contributions (Phase 6).
# fg / mom / btc_dom: probability-delta × 2 → log-odds.
# news / mfld: already log-odds (LOGODDS_WEIGHT already applied).
feat_fg_lo = _fg_contribution * 2
feat_mom_lo = _momentum_contribution * 2
feat_news_lo = news_log_adj
feat_mfld_lo = manifold_log_adj
feat_btc_dom_lo = _btc_dom_contribution * 2
feat_str = (
f"fg_lo={feat_fg_lo:+.4f} mom_lo={feat_mom_lo:+.4f} "
f"news_lo={feat_news_lo:+.4f} mfld_lo={feat_mfld_lo:+.4f} "
@@ -695,6 +783,48 @@ class BayesianStrategy:
passed_net = edge_net >= regime_min
can_trade = passed_net and confidence >= MIN_CONFIDENCE
# ── Guardrail decision impact ────────────────────────────────────────
# True when the un-clamped posterior's edge crossed the regime gate but
# the clamped one no longer does — i.e. the fuse PREVENTED a trade.
# Confidence is invariant under the clamp (it depends only on signal
# agreement), so the edge gate is the only component that can flip.
guardrail_changed_trade_decision = False
if news_guardrail_applied:
raw_edge_net = abs(raw_final_prob - market.yes_price) - TOTAL_COST_RATE
guardrail_changed_trade_decision = (
raw_edge_net >= regime_min and edge_net < regime_min
)
# ── Guardrail observability — ONLY markets with material news ───────
# Gated on materiality so the ~145 markets/cycle without news don't
# flood the logs. posterior_before_news = everything except GNews.
news_is_material = abs(feat_news_lo) >= NEWS_MATERIAL_LOGODDS_THRESHOLD
if news_is_material:
posterior_before_news = max(0.05, min(0.95, _sigmoid(
log_odds_prior + total_adj * 2 + manifold_log_adj
)))
self._news_shifts.append(estimated_prob - prior)
if news_guardrail_applied:
self._news_guardrail_applied += 1
if guardrail_changed_trade_decision:
self._news_changed_decisions += 1
log.info(
"NEWS_MATERIAL %-50s | cat=%-12s | family=%-28s | "
"prior=%.3f | before_news=%.3f | raw=%.3f | final=%.3f | "
"sent=%+.2f | news_lo=%+.4f | "
"edge_before_news=%.3f | edge_after_raw=%.3f | edge_after_guardrail=%.3f | "
"guardrail=%s | changed_decision=%s | max_shift=%.2f",
market.question[:50], category, family,
prior, posterior_before_news, raw_final_prob, estimated_prob,
news_sentiment, feat_news_lo,
abs(posterior_before_news - market.yes_price),
abs(raw_final_prob - market.yes_price),
edge_gross,
"applied" if news_guardrail_applied else "none",
str(guardrail_changed_trade_decision).lower(),
MAX_NEWS_ONLY_PROB_SHIFT,
)
if not can_trade:
# Increment the appropriate edge-net counter
if edge_net <= 0:
@@ -723,8 +853,21 @@ class BayesianStrategy:
)
return None
# When GNews participated, expose raw vs final and the guardrail verdict
# (Task 4 of the guardrail spec); otherwise keep the legacy format.
if news_log_adj != 0.0:
prob_part = (
f"Prior=poly({prior:.3f}) → raw={raw_final_prob:.3f} "
f"→ final={estimated_prob:.3f} | "
f"GNews sent={news_sentiment:+.2f} | "
f"guardrail={'applied' if news_guardrail_applied else 'none'} | "
f"changed_decision={str(guardrail_changed_trade_decision).lower()} | "
f"max_shift={MAX_NEWS_ONLY_PROB_SHIFT:.2f} | "
)
else:
prob_part = f"Prior=poly({prior:.3f}) → estimate={estimated_prob:.3f} | "
reasoning = (
f"Prior=poly({prior:.3f}) → estimate={estimated_prob:.3f} | "
prob_part +
f"Poly price={market.yes_price:.3f} | "
f"edge_gross={edge_gross:+.3f} | edge_net={edge_net:+.3f} | "
f"regime_min={regime_min:.2f} | days={days} | "
tests/test_news_guardrail.py
+247
View File
@@ -0,0 +1,247 @@
"""
Tests for the GNews guardrail (catastrophic fuse).
Post-mortem NVIDIA 631181: one uncorroborated signal at high weight flipped a
0.845 market to 0.431. With Manifold observational-only and macro signals
gated behind is_non_price, GNews is the only live signal able to move politics
markets 20-30 pp against the order-book consensus. The fuse clamps the
posterior to prior ± MAX_NEWS_ONLY_PROB_SHIFT when GNews is the ONLY material
signal (|log-odds| >= NEWS_MATERIAL_LOGODDS_THRESHOLD); any other material
signal counts as corroboration and disables the clamp.
Politics markets have no macro adjustments, so full-path tests exercise the
"GNews only" branch naturally; the corroboration branch is tested through the
pure helper apply_news_guardrail().
evaluate() emits a NEWS_MATERIAL log line for every market whose news
contribution is material (trade or skip); tests parse it via caplog.
"""
import asyncio
import logging
import math
import re
import pytest
import bot.strategy.bayesian as bayesian
from bot.data.external import ExternalSignals
from bot.data.polymarket import Market
from bot.strategy.bayesian import (
NEWS_LOGODDS_WEIGHT,
BayesianStrategy,
apply_news_guardrail,
)
NEWS_MATERIAL_RE = re.compile(
r"NEWS_MATERIAL.*raw=(\d+\.\d+) \| final=(\d+\.\d+).*"
r"guardrail=(applied|none) \| changed_decision=(true|false)"
)
def _logodds(p: float) -> float:
return math.log(p / (1 - p))
def _sentiment_for(prior: float, target_raw: float) -> float:
"""Sentiment that moves `prior` to exactly `target_raw` via GNews alone."""
return (_logodds(target_raw) - _logodds(prior)) / NEWS_LOGODDS_WEIGHT
class FakeNews:
"""Deterministic NewsClient stub returning a fixed sentiment."""
enabled = True
def __init__(self, sentiment: float) -> None:
self._sentiment = sentiment
async def get_sentiment(self, question: str) -> float:
return self._sentiment
def get_freshness(self, question: str) -> float:
return 1.0
def _make_market(yes_price: float) -> Market:
return Market(
id="mkt-guardrail-1",
condition_id="cond-guardrail-1",
question="Will John Smith win the election?",
yes_token_id="yes-tok",
no_token_id="no-tok",
yes_price=yes_price,
no_price=1.0 - yes_price,
volume_24h=50_000.0,
end_date="2026-07-15T00:00:00Z", # politics <30 d → regime_min 0.08
active=True,
category="politics",
)
def _make_signals() -> ExternalSignals:
# Neutral macro environment; irrelevant for politics (gated) but explicit.
return ExternalSignals(
btc_price=100_000.0,
btc_change_24h=0.0,
eth_price=4_000.0,
eth_change_24h=0.0,
btc_dominance=50.0,
fear_greed_index=50,
fear_greed_label="neutral",
total_market_cap_change=0.0,
valid=True,
)
def _evaluate(yes_price: float, sentiment: float, caplog) -> tuple[
BayesianStrategy, tuple[float, float, str, str]
]:
"""Run evaluate() on a politics market and parse the NEWS_MATERIAL line."""
strategy = BayesianStrategy(news=FakeNews(sentiment), manifold=None, db=None)
market = _make_market(yes_price)
with caplog.at_level(logging.INFO, logger="bot.strategy.bayesian"):
asyncio.run(strategy.evaluate(market, _make_signals(), occupied_families=set()))
for record in caplog.records:
m = NEWS_MATERIAL_RE.search(record.getMessage())
if m:
return strategy, (
float(m.group(1)), float(m.group(2)), m.group(3), m.group(4)
)
pytest.fail(
"No NEWS_MATERIAL log line found; got: "
f"{[r.getMessage() for r in caplog.records]}"
)
# ─────────────────────────────────────────────────────────────────────────────
# Test 1 — extreme uncorroborated shift: clamp to prior - MAX_NEWS_ONLY_PROB_SHIFT
# ─────────────────────────────────────────────────────────────────────────────
def test_extreme_news_only_shift_is_clamped(caplog):
"""prior=0.845, raw 0.431 (NVIDIA signature) → final clamped to 0.595."""
strategy, (raw, final, guardrail, _) = _evaluate(
yes_price=0.845, sentiment=_sentiment_for(0.845, 0.431), caplog=caplog
)
assert raw == pytest.approx(0.431, abs=1e-3)
assert guardrail == "applied"
assert final >= 0.595
assert final == pytest.approx(0.845 - bayesian.MAX_NEWS_ONLY_PROB_SHIFT, abs=1e-3)
assert strategy.get_cycle_stats()["news_guardrail_applied"] == 1
assert strategy.get_cycle_stats()["news_with_material"] == 1
# ─────────────────────────────────────────────────────────────────────────────
# Test 2 — moderate shift inside the band: passes through untouched
# ─────────────────────────────────────────────────────────────────────────────
def test_moderate_news_shift_inside_band_not_clamped(caplog):
"""prior=0.50, raw 0.62 → within ±0.25 band → final=0.62, no clamp."""
strategy, (raw, final, guardrail, _) = _evaluate(
yes_price=0.50, sentiment=_sentiment_for(0.50, 0.62), caplog=caplog
)
assert raw == pytest.approx(0.62, abs=1e-3)
assert final == pytest.approx(0.62, abs=1e-3)
assert guardrail == "none"
assert strategy.get_cycle_stats()["news_guardrail_applied"] == 0
# Still counted as a material-news market for the NEWS SUMMARY.
assert strategy.get_cycle_stats()["news_with_material"] == 1
# ─────────────────────────────────────────────────────────────────────────────
# Test 3 — corroboration: any other material signal disables the fuse
# ─────────────────────────────────────────────────────────────────────────────
def test_corroborated_news_not_clamped():
"""GNews material + another signal >= threshold → raw passes without clamp."""
news_lo = _logodds(0.20) - _logodds(0.50) # ≈ -1.386, clearly material
final, applied = apply_news_guardrail(
prior=0.50,
raw_final_prob=0.20,
feat_news_lo=news_lo,
other_feats_lo=(0.0, 0.15, 0.0, 0.0), # one corroborating signal
)
assert final == 0.20
assert applied is False
def test_corroboration_threshold_is_inclusive():
"""|other| == threshold exactly counts as corroboration (>=, not >)."""
final, applied = apply_news_guardrail(
prior=0.50,
raw_final_prob=0.20,
feat_news_lo=-1.386,
other_feats_lo=(bayesian.NEWS_MATERIAL_LOGODDS_THRESHOLD, 0.0, 0.0, 0.0),
)
assert final == 0.20
assert applied is False
def test_uncorroborated_helper_clamps():
"""Same shift with only noise elsewhere → clamped to prior - 0.25."""
final, applied = apply_news_guardrail(
prior=0.50,
raw_final_prob=0.20,
feat_news_lo=-1.386,
other_feats_lo=(0.05, -0.09, 0.0, 0.0), # all below threshold → noise
)
assert final == pytest.approx(0.25)
assert applied is True
def test_sub_material_news_never_clamped():
"""|news_lo| below threshold → fuse not armed, whatever the shift."""
final, applied = apply_news_guardrail(
prior=0.50,
raw_final_prob=0.10,
feat_news_lo=0.09,
other_feats_lo=(0.0, 0.0, 0.0, 0.0),
)
assert final == 0.10
assert applied is False
def test_guardrail_disabled_passthrough(monkeypatch):
monkeypatch.setattr(bayesian, "NEWS_GUARDRAIL_ENABLED", False)
final, applied = apply_news_guardrail(
prior=0.845,
raw_final_prob=0.431,
feat_news_lo=-1.974,
other_feats_lo=(0.0, 0.0, 0.0, 0.0),
)
assert final == 0.431
assert applied is False
# ─────────────────────────────────────────────────────────────────────────────
# Test 4 — changed_decision: the clamp moves the edge from tradeable to not
# ─────────────────────────────────────────────────────────────────────────────
def test_guardrail_changed_trade_decision(monkeypatch, caplog):
"""
With max_shift=0.10 the clamped edge (0.10 gross, 0.06 net) falls below the
politics <30 d regime gate (0.08) while the raw edge (0.414 gross, 0.374
net) crossed it → the fuse prevented the trade → changed_decision=true.
(With the default 0.25 the clamped edge_net is 0.21, above every regime
minimum, so the flag can only fire with a tighter configured band.)
"""
monkeypatch.setattr(bayesian, "MAX_NEWS_ONLY_PROB_SHIFT", 0.10)
strategy, (raw, final, guardrail, changed) = _evaluate(
yes_price=0.845, sentiment=_sentiment_for(0.845, 0.431), caplog=caplog
)
assert raw == pytest.approx(0.431, abs=1e-3)
assert final == pytest.approx(0.745, abs=1e-3)
assert guardrail == "applied"
assert changed == "true"
stats = strategy.get_cycle_stats()
assert stats["news_changed_decisions"] == 1
assert stats["news_guardrail_applied"] == 1
def test_default_band_does_not_change_decision(caplog):
"""Default 0.25 band: clamp binds but edge_net 0.21 still crosses the gate."""
_, (_, _, guardrail, changed) = _evaluate(
yes_price=0.845, sentiment=_sentiment_for(0.845, 0.431), caplog=caplog
)
assert guardrail == "applied"
assert changed == "false"