The signals and markets tables existed since Phase 2/5 but never had a
writer; the replay engine (phase plan line 2.1) needs a per-(market, cycle)
archive of what the strategy saw and decided. This wires them up:
- signals: one row per evaluated market per cycle, now carrying INPUTS
(news_sentiment, feat_*_lo, volume_24h, days_to_resolution) plus the
existing outputs (probs, edges, gates, skip_reason). skip_reason is
granular: unsupported/no_signals/prior_extreme/family/edge_net/
confidence/reentry_guard. news_budget_skipped distinguishes "GNews not
asked" (5-query budget) from "no news".
- ext_snapshots: one row per cycle with the ExternalSignals snapshot;
signals rows join on cycle_ts.
- markets: metadata upserted each cycle (replay rebuilds Market from it).
- Retention: prune > SIGNALS_RETENTION_DAYS (default 90) once a day.
- SIGNAL_RECORDER_ENABLED (default true) gates all DB writes; every write
is try/except — the recorder can never break trading.
Strategy changes are purely additive (record accumulation at each exit
path of evaluate()); no weights, thresholds, gates or sizing touched,
per the freeze in the current phase plan.
Tests: 10 new deterministic tests (85 total passing). Schema migration
dry-run validated against prod postgres inside a rolled-back transaction.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Post-mortem NVIDIA 631181: one uncorroborated high-weight signal (legacy
Manifold 0.13 at weight 0.6) flipped a 0.845 market to 0.431 and lost.
With Manifold observational-only and macro signals gated behind
is_non_price, GNews (weight 1.5) is the only live signal able to move
politics markets 20-30 pp against the order-book consensus. This adds a
catastrophic fuse, not a fine calibration:
- apply_news_guardrail(): when |news_lo| >= NEWS_MATERIAL_LOGODDS_THRESHOLD
(0.10) and every other signal (fg, mom, btc_dom, mfld) is below it,
clamp the posterior to prior ± MAX_NEWS_ONLY_PROB_SHIFT (0.25). Any
corroborating material signal disables the clamp. Config via env
(NEWS_GUARDRAIL_ENABLED=true by default).
- edge_gross/edge_net computed from the clamped posterior; raw_final_prob
preserved in reasoning (persisted via trades.reasoning — no schema
migration) and in the NEWS_MATERIAL log line.
- guardrail_changed_trade_decision: raw edge crossed the regime gate but
the clamped edge no longer does (fuse prevented a trade). Note: with
the default 0.25 band the clamped edge_net is 0.21, above every regime
minimum, so the flag only fires with a tighter configured band.
- Observability gated on materiality: NEWS_MATERIAL per-market line and a
compact NEWS SUMMARY cycle line, only when with_news > 0 — no flood
from the ~145 news-less markets per cycle.
- 9 deterministic tests (extreme clamp, in-band passthrough, corroboration,
inclusive threshold, disabled, changed_decision).
No changes to NEWS_LOGODDS_WEIGHT, Manifold flags, edge thresholds,
sizing, payout, resolution, or historical trades.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two minor faults found during the GNews capture/prioritisation diagnostic:
1. Hyphens/dashes reached the GNews query verbatim. '-' is GNews's exclusion
operator, so a token like "El-Sayed" returned HTTP 400 and wasted a query.
_PUNCT_RE now strips '-', en dash and em dash to spaces.
2. The per-cycle GNews budget counter incremented in evaluate() before
get_sentiment() checked the API key, so with no key configured the
[CYCLE SUMMARY] reported a phantom "gnews_queries_used: 5/5" with zero real
requests. Added NewsClient.enabled and gated the GNews block on it; with no
key the counter stays 0/5 and no spurious SKIP_GNEWS_PRIORITY is logged.
No behaviour change when a key is present.
Prioritisation itself was confirmed correct and is left untouched: politics
markets are sorted by gnews_priority DESC and prior-extreme markets return
before the budget is consumed, so no query is ever spent on a market that
cannot trade.
Tests: tests/test_news_query.py (4 new); full suite 66 passed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase 3 excluded momentum and Fear & Greed from politics/tech/events
markets; Phase 4 fixed ticker detection. But the BTC-dominance signal was
still applied to non-price markets that legitimately mention a ticker
('Will the ETH ETF be approved?'), despite having no demonstrated causality
for non-price outcomes. Reuse the existing is_non_price gate so the
contribution stays 0.0 -> feat_btc_dom_lo = 0.0 for those markets.
Price-market behavior unchanged: ETH/altcoin/general-crypto markets keep
the +/-0.03 dominance adjustment.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Substring matching over question_lower flagged non-crypto markets as crypto:
'dissolved' matched 'sol', 'Canada' matched 'ada', 'Seth' matched 'eth'.
Those false flags armed the BTC-dominance signal (btc_dom_lo=+0.06 observed
on politics markets in production).
Short tickers (btc, eth, sol, xrp, doge, ltc, bnb, ada, avax) now go through
has_token(), which requires non-alphanumeric boundaries so 'ETH', '$ETH' and
'ETH/USD' still match. Long unambiguous names (bitcoin, ethereum, solana,
cardano, ...) remain substring checks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
For politics/tech/events markets there is no above/below price notion, so
is_price_above defaulted to False (or flipped on accidental wording like
"reach") and sign-inverted the macro adjustments: BTC +5% or high Fear&Greed
subtracted probability from YES on "Will X win the election?" markets.
Skip both signals entirely for non-price markets: contributions stay 0.0,
feat_mom_lo / feat_fg_lo persist as 0.0. Price markets (BTC/ETH/crypto)
keep the exact current behavior, including the below-market sign flip.
Removes the now-dead BTC(sentiment) momentum branch and its 0.5 attenuator.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Per-category coverage audit showed coverage_rate=0.0 across every category in the
bot's current universe, so any edge Manifold produced was false edge. Retire it as
an ACTIVE trading signal while keeping the full audit/coverage/cooldown trail for a
future reactivation decision.
Two module-level flags in bayesian.py (read from env):
- MANIFOLD_SIGNAL_ENABLED (default False): when False, Manifold never touches the
edge model — manifold_log_adj stays 0.0 (no posterior shift), no confidence bump,
feat_mfld_lo=0.0 (so it can never be the dominant feature), no trade contribution,
and mfld_audit_id is not propagated so the audit's used_in_trade stays FALSE.
- MANIFOLD_AUDIT_ENABLED (default True): matcher still runs; audit/coverage rows and
cooldowns are still written. The matcher is only called when a flag is on.
When signal is disabled, logs and reasoning carry "Manifold: observational_only".
Endpoints /api/metrics/manifold-matches and /api/metrics/manifold-coverage,
cooldowns, audit tables and existing trades are unchanged. No code or tables removed.
Other signals, thresholds, exposure, risk manager and the executor are untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The trading loop re-evaluated the same ~22 politics/tech markets every ~60s,
flooding manifold_match_audit with ~76k rows (~3,500 attempts/market) of which
none carried new information, making the metrics uninterpretable.
Add per-market persistent cooldowns:
- New table manifold_eval_cooldown (poly_market_id PK, last_evaluated_at,
last_status, retry_after, cooldown_reason) created via run_migrations.
- bayesian.evaluate() consults the cooldown BEFORE calling the matcher and skips
the call entirely while now() < retry_after — no matcher call, no audit row,
no signal (equivalent to no_results). After a real evaluation it upserts the
cooldown with a verdict-dependent backoff: no_results/low_score/outcome_mismatch/
ambiguous_inversion -> 24h, conditional_market -> 7d, accepted -> 1h.
- manifold.py stays a pure client/matcher with no DB access.
- New db methods get_manifold_cooldown / upsert_manifold_cooldown.
- /api/metrics/manifold-matches summary gains unique_markets
{evaluated, accepted, coverage_rate} for per-market (not per-attempt) coverage.
Matching logic, MANIFOLD_MATCHER_VERSION, MANIFOLD_LOGODDS_WEIGHT, edge/exposure
thresholds and existing audit rows are unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add MANIFOLD_MATCHER_VERSION="v3_outcome_guard" tag persisted to
manifold_match_audit.matcher_version so metrics can isolate current-matcher
stats from pre-versioning records, whose accepted matches the outcome
guard would now reject.
- schema: add matcher_version column + index; idempotent startup backfill
tagging NULL rows as legacy_pre_outcome_guard (no outcome types) or
v2_outcome_guard_no_version (has outcome type, version not persisted)
- save_manifold_audit: write matcher_version on every new record
- get_manifold_matches: split summary into current_version / all_time /
legacy; recent_matches now carry matcher_version
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reject false-positive matches where Jaccard overlap is high but the outcome is
not equivalent (e.g. Poly nomination vs Manifold "If X is nominee, will he win").
- _is_conditional(): detect conditional Manifold markets (If/Conditional on/
Assuming/Given that prefixes + mid-sentence " if ...," clauses) -> reject with
reason "conditional_market".
- _classify_outcome(): classify into nomination|primary_win|general_win|
conditional|other; reject when poly/mfld types differ or either is conditional
-> reason "outcome_mismatch: poly=... manifold=...".
- Persist poly_outcome_type/mfld_outcome_type on ManifoldMatchResult, in
manifold_match_audit (CREATE + idempotent ALTER), save_manifold_audit() and
the bayesian call site.
- Tests covering classification, conditional detection and the Graham Platner
regression (now rejected); valid nomination<->nomination still accepted.
Untouched: _MATCH_THRESHOLD (0.40), MANIFOLD_LOGODDS_WEIGHT, edge thresholds,
exposure, trading logic.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds feat_fg_lo / feat_mom_lo / feat_news_lo / feat_mfld_lo / feat_btc_dom_lo
to every trade, all normalized to log-odds contribution for direct comparability.
- fg / mom / btc_dom: raw probability-delta × 2 → log-odds
- news / mfld: already log-odds (LOGODDS_WEIGHT already applied), no scaling
- btc_dom tracked separately in bayesian.py instead of bundled in total_adj
- reasoning string updated to fg_lo= / mom_lo= notation for self-documentation
Schema: 5 new DOUBLE PRECISION columns + 2 partial indexes
Stack: TradingSignal → Order → Trade → save_trade all carry feat fields
Startup: backfill_feature_columns() recovers fg/mom/news/mfld from old
reasoning strings (×2 applied to fg/mom); btc_dom_lo stays NULL for legacy
API: /api/metrics/features — triggered/material split per feature with
two-level thresholds (0.05 for fg/mom/btc_dom, 0.10 for news/mfld)
API: /api/trades/legacy — exposes pre-Phase-1 trades (edge_net IS NULL)
API: _enrich_trade backward-compat: reads DB columns first, falls back to
reasoning regex with unit conversion for pre-Phase-6 trades
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signal 5: ManifoldClient queries Manifold Markets API for a matching binary
market by keyword overlap (threshold 0.25) and applies a log-odds adjustment
proportional to the divergence from the Polymarket prior.
manifold_log_adj = (log_odds(manifold_prob) - log_odds(prior)) × 0.6
A 30pp divergence (Manifold 0.75 vs Poly 0.45) produces edge_gross ≈ 0.19,
clearing the politics far-horizon regime_min=0.12 after costs. Confidence
boosted +0.08 when Manifold match found.
Per-feature observability: every SKIP_EDGE_NET and TRADE log line now includes
fg=±X.XXX mom=±X.XXX mfld=±X.XXXX news=±X.XXXX
so the contribution of each signal to edge is auditable per market.
Files: bot/data/manifold.py (new), bot/strategy/bayesian.py, bot/main.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Phase 1 — Edge neto real (paper.py, bayesian.py, risk/manager.py, db.py):
- Trade records now store edge_gross, edge_net, prior_prob, final_prob,
mid_price, spread_estimate, commission, family_key
- edge_net = edge_gross - SPREAD_ESTIMATE(0.02) - COMMISSION_RATE(0.02)
NOTE: both constants are heuristics, not exact Polymarket exchange costs
- Execution gate changed from edge_gross > MIN_EDGE to edge_net > regime_min_edge
Phase 2 — Market families (polymarket.py):
- market_family_key(market) groups related markets:
texas-republican-2026, fed-april-2026, openai-2026, etc.
- At most 1 trade per family per cycle; occupied_families propagated via main.py
- Family key logged on every TRADE and SKIP line
Phase 3 — GNews priority (news.py, bayesian.py, main.py):
- NewsClient.get_freshness() returns 1.0/0.75/0.40/0.10 by cache age
- gnews_priority(market, news) = uncertainty × volume_score × freshness
- Politics markets sorted by priority DESC before eval so best markets get
the 5-query/cycle GNews budget first
Phase 4 — Regime min-edge by category/horizon (bayesian.py):
- politics >60d → 0.12, 30-60d → 0.10, <30d → 0.08
- tech / crypto/finance → 0.10
- All thresholds applied to edge_net (not edge_gross)
Phase 5 — Observability (bayesian.py, main.py):
- Structured skip labels: SKIP_UNSUPPORTED, SKIP_NO_SIGNALS,
SKIP_PRIOR_EXTREME, SKIP_FAMILY, SKIP_GNEWS_PRIORITY, SKIP_EDGE_NET
- TRADE lines now include family_key, edge_gross, edge_net, regime_min, days
- schema.sql: 8 new cols on trades, 7 new cols on signals (via ALTER TABLE IF NOT EXISTS)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Markets where Polymarket consensus is near-certain leave no room for our
signals to generate MIN_EDGE=0.10 — evaluating them wastes GNews quota and
produces noise. Filter them out early with a clear log reason.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- CACHE_TTL: 4h → 6h (≤36 req/day with ≤9 politics markets)
- GNews only called for is_politics markets (BTC/F&G cover crypto/macro)
- MAX_NEWS_QUERIES_PER_CYCLE=5: BayesianStrategy.reset_cycle() called each
iteration; counter increments only on actual API call (cache hits free)
- 2s asyncio.sleep in news.py finally block after each real HTTP request
- main.py sorts markets: politics first by end_date ascending, so soonest-
resolving markets consume the 5-query budget before others
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
bot/data/news.py (new):
- NewsClient with in-memory cache (TTL=4h) to stay within 100 req/day limit
- _build_query(): strips dates, punctuation and stopwords from market question
- _score_headlines(): keyword-based pos/neg vote per article, averaged ∈ [-1, +1]
- Degrades to 0.0 on missing key, 403 quota, or network error
bot/strategy/bayesian.py:
- BayesianStrategy(news=NewsClient) — optional, backwards compatible
- Signal 4: GNews sentiment applied as direct log-odds shift (weight=1.5)
so a ±1.0 sentiment score moves a 50% prior to 82%/18%
- +0.10 confidence boost when news signal is present
- NEWS_LOGODDS_WEIGHT constant documented at module level
bot/main.py:
- Instantiate NewsClient, pass to BayesianStrategy, close in finally block
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Every market now emits an INFO line:
TRADE/SKIP <question> | cat=... | prior=... | est=... | edge=... | conf=... | dir=... | signals=... [| reason=...]
Unsupported-category and no-external-signals early exits also log at INFO
so the full evaluation funnel is visible without changing log level.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- polymarket.py: add keyword lists for politics (election, trump, ukraine…),
tech (AI, OpenAI, Apple, nvidia…), and events (super bowl, oscar, spacex…);
introduce _detect_category() so all four categories flow through a single
code path; filter already-expired markets (end_dt < now) in addition to
the existing future-cutoff filter; log per-category counts at startup
- bayesian.py: extend is_any_supported to include is_politics / is_tech /
is_events; use BTC as a risk-sentiment proxy for non-crypto categories
(halved weight to reflect weaker correlation); cap confidence_cap at 0.65
for macro/politics/tech/events; MIN_EDGE stays at 0.10
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
chemavx