Files

T

ChemaVXandClaude Fable 5 634f38e016 fix(scraper): SAFE_ACCEPT_ENCODING compartida — cubre la sesión SearXNG olvidada

La sesión aiohttp de _search_searxng (camino primario de búsqueda) construía
sus headers sin Accept-Encoding: heredaba el default de aiohttp, que volvería
a anunciar br si algún día se reinstala un backend brotli. El valor vive ahora
en src/config.py (SAFE_ACCEPT_ENCODING) en vez de copiado literal por sitio.

También corrige el comentario de HEADERS que afirmaba que brotlicffi seguía
instalado como fallback — 397546a lo quitó: un br no anunciado hoy falla sin
recuperación y la fuente se pierde.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

2026-07-05 15:56:07 +00:00

3.3 KiB

Raw Blame History

ResearchOwl — Known Issues & Operational Gotchas

Brotli / aiohttp incompatibility

Do NOT install any brotli backend (Brotli, brotlicffi) while on aiohttp 3.14.x. aiohttp's incremental brotli decompressor is broken and fails intermittently depending on stream chunking ("Can not decode content-encoding: br" on VALID streams — the same bytes decompress fine offline; httpx is unaffected).

Extra trap: merely installing a backend makes aiohttp advertise br in the default Accept-Encoding of every session that doesn't set one explicitly. On 2026-07-04 a scraper fix installed brotlicffi and inadvertently enabled br project-wide: Ghost/Cloudflare responded in brotli, the decode blew up AFTER Ghost had already accepted the POST, and the publish fallback re-published → duplicate drafts + silently lost SEO notices.

Fix applied (commits 397546a + 76c927f + 7d07375, guard relocated 2026-07-05):

no brotli backend in requirements.txt (comment there explains why),
explicit Accept-Encoding: gzip, deflate on all Ghost/autofill aiohttp sessions and on the scraper HEADERS (57f341f),
duplicate guard INSIDE GhostPublisher.publish_draft (covers every caller, including /publish): if the POST is accepted (2xx) but reading the response fails, it recovers the just-created draft via find_draft_by_title(title, since=attempt_start) instead of raising. The guard only fires after an accepted POST — a pre-POST failure (Ollama down, menu fetch, links) never triggers it, so a stale same-title draft from a previous run can no longer swallow freshly generated content.

Note: with no backend installed there is NO brotli fallback at all — a server that responds Content-Encoding: br without it being advertised (misbehaving CDN) fails undecodable and that source is lost. Accepted trade-off.

The header value lives in one place: SAFE_ACCEPT_ENCODING in src/config.py. Every aiohttp session/request must use it explicitly — never rely on aiohttp's default Accept-Encoding, which silently grows br if a backend appears.

Re-test with disclosure.org before ever re-enabling br (e.g. after an aiohttp upgrade).

SQLite WAL mode + read-only mounts

The database runs in WAL mode, so even read-only access needs the -shm file writable, or must open with sqlite3 "file:...?immutable=1". Backups (daily CronJob researchowl-db-backup, 03:00 Europe/Madrid, PVC researchowl-backups, 7-day retention) inherit WAL mode: to inspect one from a read-only mount use ?immutable=1; to restore, copy it to a writable location first.

DDG (duckduckgo_search) blocks the event loop

DDGS() is synchronous (blocking requests inside). Never call it directly from async code — always wrap in loop.run_in_executor() (see _ddg_text_sync / _ddg_videos_sync in src/scraper/exhaustive.py). Direct calls froze the entire Telegram bot during searches until fixed on 2026-07-04 (8dfd011).

Google News RSS is a dead end from this infrastructure

news.google.com/rss entry links point to /rss/articles/CBMi… redirects that hit a consent wall from EU IPs, and the inner AU_yqL id is only resolvable via Google's private batchexecute API. Do not retry. The news seed uses Bing News RSS instead (ENABLE_NEWS_SEED, real publisher URL in the ?url= param of apiclick.aspx — unwrapped by _unwrap_news_link).