# ResearchOwl — Known Issues & Operational Gotchas

## Brotli / aiohttp incompatibility

Do NOT install any brotli backend (`Brotli`, `brotlicffi`) while on aiohttp 3.14.x.
aiohttp's incremental brotli decompressor is broken and fails intermittently
depending on stream chunking ("Can not decode content-encoding: br" on VALID
streams — the same bytes decompress fine offline; httpx is unaffected).

Extra trap: merely *installing* a backend makes aiohttp advertise `br` in the
default `Accept-Encoding` of every session that doesn't set one explicitly.
On 2026-07-04 a scraper fix installed brotlicffi and inadvertently enabled br
project-wide: Ghost/Cloudflare responded in brotli, the decode blew up AFTER
Ghost had already accepted the POST, and the publish fallback re-published →
duplicate drafts + silently lost SEO notices.

Fix applied (commits `397546a` + `76c927f` + `7d07375`, guard relocated
2026-07-05):
- no brotli backend in requirements.txt (comment there explains why),
- explicit `Accept-Encoding: gzip, deflate` on all Ghost/autofill aiohttp
  sessions and on the scraper HEADERS (`57f341f`),
- duplicate guard INSIDE `GhostPublisher.publish_draft` (covers every caller,
  including `/publish`): if the POST is accepted (2xx) but reading the response
  fails, it recovers the just-created draft via `find_draft_by_title(title,
  since=attempt_start)` instead of raising. The guard only fires after an
  accepted POST — a pre-POST failure (Ollama down, menu fetch, links) never
  triggers it, so a stale same-title draft from a previous run can no longer
  swallow freshly generated content.

Note: with no backend installed there is NO brotli fallback at all — a server
that responds `Content-Encoding: br` without it being advertised (misbehaving
CDN) fails undecodable and that source is lost. Accepted trade-off.

The header value lives in one place: `SAFE_ACCEPT_ENCODING` in `src/config.py`.
Every aiohttp session/request must use it explicitly — never rely on aiohttp's
default Accept-Encoding, which silently grows `br` if a backend appears.

Re-test with disclosure.org before ever re-enabling br (e.g. after an aiohttp
upgrade).

## SQLite WAL mode + read-only mounts

The database runs in WAL mode, so even read-only access needs the `-shm` file
writable, or must open with `sqlite3 "file:...?immutable=1"`. Backups (daily
CronJob `researchowl-db-backup`, 03:00 Europe/Madrid, PVC `researchowl-backups`,
7-day retention) inherit WAL mode: to inspect one from a read-only mount use
`?immutable=1`; to restore, copy it to a writable location first.

## DDG (duckduckgo_search) blocks the event loop

`DDGS()` is synchronous (blocking requests inside). Never call it directly from
async code — always wrap in `loop.run_in_executor()` (see `_ddg_text_sync` /
`_ddg_videos_sync` in `src/scraper/exhaustive.py`). Direct calls froze the
entire Telegram bot during searches until fixed on 2026-07-04 (`8dfd011`).

## Google News RSS is a dead end from this infrastructure

`news.google.com/rss` entry links point to `/rss/articles/CBMi…` redirects that
hit a consent wall from EU IPs, and the inner `AU_yqL` id is only resolvable via
Google's private batchexecute API. Do not retry. The news seed uses Bing News
RSS instead (`ENABLE_NEWS_SEED`, real publisher URL in the `?url=` param of
apiclick.aspx — unwrapped by `_unwrap_news_link`).