researchowl/KNOWN-ISSUES.md

ResearchOwl — Known Issues & Operational Gotchas

Brotli / aiohttp incompatibility

Do NOT install any brotli backend (Brotli, brotlicffi) while on aiohttp 3.14.x. aiohttp's incremental brotli decompressor is broken and fails intermittently depending on stream chunking ("Can not decode content-encoding: br" on VALID streams — the same bytes decompress fine offline; httpx is unaffected).

Extra trap: merely installing a backend makes aiohttp advertise br in the default Accept-Encoding of every session that doesn't set one explicitly. On 2026-07-04 a scraper fix installed brotlicffi and inadvertently enabled br project-wide: Ghost/Cloudflare responded in brotli, the decode blew up AFTER Ghost had already accepted the POST, and the publish fallback re-published → duplicate drafts + silently lost SEO notices.

Fix applied (commits 397546a + 76c927f + 7d07375):

• no brotli backend in requirements.txt (comment there explains why),
• explicit Accept-Encoding: gzip, deflate on all Ghost/autofill aiohttp sessions and on the scraper HEADERS (57f341f),
• duplicate guard in _publish_blog_to_ghost: on fallback-after-exception it checks find_draft_by_title() before re-publishing.

Re-test with disclosure.org before ever re-enabling br (e.g. after an aiohttp upgrade).

SQLite WAL mode + read-only mounts

The database runs in WAL mode, so even read-only access needs the -shm file writable, or must open with sqlite3 "file:...?immutable=1". Backups (daily CronJob researchowl-db-backup, 03:00 Europe/Madrid, PVC researchowl-backups, 7-day retention) inherit WAL mode: to inspect one from a read-only mount use ?immutable=1; to restore, copy it to a writable location first.

DDG (duckduckgo_search) blocks the event loop

DDGS() is synchronous (blocking requests inside). Never call it directly from async code — always wrap in loop.run_in_executor() (see _ddg_text_sync / _ddg_videos_sync in src/scraper/exhaustive.py). Direct calls froze the entire Telegram bot during searches until fixed on 2026-07-04 (8dfd011).

Google News RSS is a dead end from this infrastructure

news.google.com/rss entry links point to /rss/articles/CBMi… redirects that hit a consent wall from EU IPs, and the inner AU_yqL id is only resolvable via Google's private batchexecute API. Do not retry. The news seed uses Bing News RSS instead (ENABLE_NEWS_SEED, real publisher URL in the ?url= param of apiclick.aspx — unwrapped by _unwrap_news_link).