researchowl/KNOWN-ISSUES.md

ResearchOwl — Known Issues & Operational Gotchas

Brotli / aiohttp incompatibility

Do NOT install any brotli backend (Brotli, brotlicffi) while on aiohttp 3.14.x. aiohttp's incremental brotli decompressor is broken and fails intermittently depending on stream chunking ("Can not decode content-encoding: br" on VALID streams — the same bytes decompress fine offline; httpx is unaffected).

Extra trap: merely installing a backend makes aiohttp advertise br in the default Accept-Encoding of every session that doesn't set one explicitly. On 2026-07-04 a scraper fix installed brotlicffi and inadvertently enabled br project-wide: Ghost/Cloudflare responded in brotli, the decode blew up AFTER Ghost had already accepted the POST, and the publish fallback re-published → duplicate drafts + silently lost SEO notices.

Fix applied (commits 397546a + 76c927f + 7d07375, guard relocated 2026-07-05):

• no brotli backend in requirements.txt (comment there explains why),
• explicit Accept-Encoding: gzip, deflate on all Ghost/autofill aiohttp sessions and on the scraper HEADERS (57f341f),
• duplicate guard INSIDE GhostPublisher.publish_draft (covers every caller, including /publish): if the POST is accepted (2xx) but reading the response fails, it recovers the just-created draft via find_draft_by_title(title, since=attempt_start) instead of raising. The guard only fires after an accepted POST — a pre-POST failure (Ollama down, menu fetch, links) never triggers it, so a stale same-title draft from a previous run can no longer swallow freshly generated content.

Note: with no backend installed there is NO brotli fallback at all — a server that responds Content-Encoding: br without it being advertised (misbehaving CDN) fails undecodable and that source is lost. Accepted trade-off.

The header value lives in one place: SAFE_ACCEPT_ENCODING in src/config.py. Every aiohttp session/request must use it explicitly — never rely on aiohttp's default Accept-Encoding, which silently grows br if a backend appears.

Re-test with disclosure.org before ever re-enabling br (e.g. after an aiohttp upgrade).

SQLite WAL mode + read-only mounts

The database runs in WAL mode, so even read-only access needs the -shm file writable, or must open with sqlite3 "file:...?immutable=1". Backups (daily CronJob researchowl-db-backup, 03:00 Europe/Madrid, PVC researchowl-backups, 7-day retention) inherit WAL mode: to inspect one from a read-only mount use ?immutable=1; to restore, copy it to a writable location first.

DDG (duckduckgo_search) blocks the event loop

DDGS() is synchronous (blocking requests inside). Never call it directly from async code — always wrap in loop.run_in_executor() (see _ddg_text_sync / _ddg_videos_sync in src/scraper/exhaustive.py). Direct calls froze the entire Telegram bot during searches until fixed on 2026-07-04 (8dfd011).

Google News RSS is a dead end from this infrastructure

news.google.com/rss entry links point to /rss/articles/CBMi… redirects that hit a consent wall from EU IPs, and the inner AU_yqL id is only resolvable via Google's private batchexecute API. Do not retry. The news seed uses Bing News RSS instead (ENABLE_NEWS_SEED, real publisher URL in the ?url= param of apiclick.aspx — unwrapped by _unwrap_news_link).

Large sources can OOM-kill the pod

On 2026-07-10 the pod was OOMKilled (memory limit was 1Gi) mid-research: a batch of 20 concurrent sources hit a 98k-word document plus several large PDFs at once, and pdfplumber's parse spiked RAM past the limit. The in-memory research task died with the pod and its session sat in running forever.

Mitigations now in place:

• Memory limit raised to 2Gi (k8s-manifests/researchowl/deployment.yaml).
• PDFs capped at 15MB (was 50MB), checked both via Content-Length and actual body size; pdfplumber runs in run_in_executor (it is sync + CPU-heavy and also froze the event loop, same class of bug as DDGS) and flushes its page cache per page.
• Extracted content is truncated to max_content_length (300k chars) before hitting source_contents.
• On startup the bot marks orphaned running sessions as interrupted.

If a research still dies, the scraped sources survive in the DB: /process re-chunks and scores them without re-scraping.