From 68ae6b436c0f1ebcace614892560c7852993ac84 Mon Sep 17 00:00:00 2001 From: ChemaVX Date: Sat, 4 Jul 2026 20:53:34 +0000 Subject: [PATCH] =?UTF-8?q?docs:=20KNOWN-ISSUES.md=20=E2=80=94=20gotchas?= =?UTF-8?q?=20operacionales=20(brotli/aiohttp,=20WAL,=20DDGS,=20Google=20N?= =?UTF-8?q?ews)=20[skip=20ci]?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- KNOWN-ISSUES.md | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100644 KNOWN-ISSUES.md diff --git a/KNOWN-ISSUES.md b/KNOWN-ISSUES.md new file mode 100644 index 0000000..85544a1 --- /dev/null +++ b/KNOWN-ISSUES.md @@ -0,0 +1,48 @@ +# ResearchOwl — Known Issues & Operational Gotchas + +## Brotli / aiohttp incompatibility + +Do NOT install any brotli backend (`Brotli`, `brotlicffi`) while on aiohttp 3.14.x. +aiohttp's incremental brotli decompressor is broken and fails intermittently +depending on stream chunking ("Can not decode content-encoding: br" on VALID +streams — the same bytes decompress fine offline; httpx is unaffected). + +Extra trap: merely *installing* a backend makes aiohttp advertise `br` in the +default `Accept-Encoding` of every session that doesn't set one explicitly. +On 2026-07-04 a scraper fix installed brotlicffi and inadvertently enabled br +project-wide: Ghost/Cloudflare responded in brotli, the decode blew up AFTER +Ghost had already accepted the POST, and the publish fallback re-published → +duplicate drafts + silently lost SEO notices. + +Fix applied (commits `397546a` + `76c927f` + `7d07375`): +- no brotli backend in requirements.txt (comment there explains why), +- explicit `Accept-Encoding: gzip, deflate` on all Ghost/autofill aiohttp + sessions and on the scraper HEADERS (`57f341f`), +- duplicate guard in `_publish_blog_to_ghost`: on fallback-after-exception it + checks `find_draft_by_title()` before re-publishing. + +Re-test with disclosure.org before ever re-enabling br (e.g. after an aiohttp +upgrade). + +## SQLite WAL mode + read-only mounts + +The database runs in WAL mode, so even read-only access needs the `-shm` file +writable, or must open with `sqlite3 "file:...?immutable=1"`. Backups (daily +CronJob `researchowl-db-backup`, 03:00 Europe/Madrid, PVC `researchowl-backups`, +7-day retention) inherit WAL mode: to inspect one from a read-only mount use +`?immutable=1`; to restore, copy it to a writable location first. + +## DDG (duckduckgo_search) blocks the event loop + +`DDGS()` is synchronous (blocking requests inside). Never call it directly from +async code — always wrap in `loop.run_in_executor()` (see `_ddg_text_sync` / +`_ddg_videos_sync` in `src/scraper/exhaustive.py`). Direct calls froze the +entire Telegram bot during searches until fixed on 2026-07-04 (`8dfd011`). + +## Google News RSS is a dead end from this infrastructure + +`news.google.com/rss` entry links point to `/rss/articles/CBMi…` redirects that +hit a consent wall from EU IPs, and the inner `AU_yqL` id is only resolvable via +Google's private batchexecute API. Do not retry. The news seed uses Bing News +RSS instead (`ENABLE_NEWS_SEED`, real publisher URL in the `?url=` param of +apiclick.aspx — unwrapped by `_unwrap_news_link`).