docs: KNOWN-ISSUES.md — gotchas operacionales (brotli/aiohttp, WAL, DDGS, Google News) [skip ci]

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-04 20:53:34 +00:00
parent 7d07375dd1
commit 68ae6b436c
1 changed files with 48 additions and 0 deletions
@@ -0,0 +1,48 @@
+# ResearchOwl — Known Issues & Operational Gotchas
+
+## Brotli / aiohttp incompatibility
+
+Do NOT install any brotli backend (`Brotli`, `brotlicffi`) while on aiohttp 3.14.x.
+aiohttp's incremental brotli decompressor is broken and fails intermittently
+depending on stream chunking ("Can not decode content-encoding: br" on VALID
+streams — the same bytes decompress fine offline; httpx is unaffected).
+
+Extra trap: merely *installing* a backend makes aiohttp advertise `br` in the
+default `Accept-Encoding` of every session that doesn't set one explicitly.
+On 2026-07-04 a scraper fix installed brotlicffi and inadvertently enabled br
+project-wide: Ghost/Cloudflare responded in brotli, the decode blew up AFTER
+Ghost had already accepted the POST, and the publish fallback re-published →
+duplicate drafts + silently lost SEO notices.
+
+Fix applied (commits `397546a` + `76c927f` + `7d07375`):
+- no brotli backend in requirements.txt (comment there explains why),
+- explicit `Accept-Encoding: gzip, deflate` on all Ghost/autofill aiohttp
+  sessions and on the scraper HEADERS (`57f341f`),
+- duplicate guard in `_publish_blog_to_ghost`: on fallback-after-exception it
+  checks `find_draft_by_title()` before re-publishing.
+
+Re-test with disclosure.org before ever re-enabling br (e.g. after an aiohttp
+upgrade).
+
+## SQLite WAL mode + read-only mounts
+
+The database runs in WAL mode, so even read-only access needs the `-shm` file
+writable, or must open with `sqlite3 "file:...?immutable=1"`. Backups (daily
+CronJob `researchowl-db-backup`, 03:00 Europe/Madrid, PVC `researchowl-backups`,
+7-day retention) inherit WAL mode: to inspect one from a read-only mount use
+`?immutable=1`; to restore, copy it to a writable location first.
+
+## DDG (duckduckgo_search) blocks the event loop
+
+`DDGS()` is synchronous (blocking requests inside). Never call it directly from
+async code — always wrap in `loop.run_in_executor()` (see `_ddg_text_sync` /
+`_ddg_videos_sync` in `src/scraper/exhaustive.py`). Direct calls froze the
+entire Telegram bot during searches until fixed on 2026-07-04 (`8dfd011`).
+
+## Google News RSS is a dead end from this infrastructure
+
+`news.google.com/rss` entry links point to `/rss/articles/CBMi…` redirects that
+hit a consent wall from EU IPs, and the inner `AU_yqL` id is only resolvable via
+Google's private batchexecute API. Do not retry. The news seed uses Bing News
+RSS instead (`ENABLE_NEWS_SEED`, real publisher URL in the `?url=` param of
+apiclick.aspx — unwrapped by `_unwrap_news_link`).