processor.py: simplify _score_quality prompt to single axis —
"how relevant is this text to topic X?" — instead of averaging
relevance + density + credibility, which let off-topic but
well-written content pass through
exhaustive.py: pre-compute topic keywords (stopword-filtered) at
scraper init; filter child URLs (discovered during crawl, depth>0)
to only add ones whose URL path or title contains a topic keyword;
seed URLs (depth=0, from DDG/Wikipedia/Reddit) are always included
since those searches are already topic-scoped
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- bot.py: add cmd_process handler to manually trigger chunk processing
on the last session; register CommandHandler("process")
- processor.py: log exceptions from asyncio.gather instead of silently
dropping them; add per-chunk quality score debug logging; warn when
all chunks filtered by quality threshold with actionable hint;
raise fallback score to 0.6 so Ollama failures don't filter chunks
- exhaustive.py: replace bot User-Agent with full browser UA + headers
for REDDIT_HEADERS; downgrade Reddit 403 from warning to info since
server IPs are routinely blocked; use content_type=None on json()
to avoid aiohttp content-type mismatch errors
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ChemaVX