49 lines
2.5 KiB
Markdown
49 lines
2.5 KiB
Markdown
# ResearchOwl — Known Issues & Operational Gotchas
|
|
|
|
## Brotli / aiohttp incompatibility
|
|
|
|
Do NOT install any brotli backend (`Brotli`, `brotlicffi`) while on aiohttp 3.14.x.
|
|
aiohttp's incremental brotli decompressor is broken and fails intermittently
|
|
depending on stream chunking ("Can not decode content-encoding: br" on VALID
|
|
streams — the same bytes decompress fine offline; httpx is unaffected).
|
|
|
|
Extra trap: merely *installing* a backend makes aiohttp advertise `br` in the
|
|
default `Accept-Encoding` of every session that doesn't set one explicitly.
|
|
On 2026-07-04 a scraper fix installed brotlicffi and inadvertently enabled br
|
|
project-wide: Ghost/Cloudflare responded in brotli, the decode blew up AFTER
|
|
Ghost had already accepted the POST, and the publish fallback re-published →
|
|
duplicate drafts + silently lost SEO notices.
|
|
|
|
Fix applied (commits `397546a` + `76c927f` + `7d07375`):
|
|
- no brotli backend in requirements.txt (comment there explains why),
|
|
- explicit `Accept-Encoding: gzip, deflate` on all Ghost/autofill aiohttp
|
|
sessions and on the scraper HEADERS (`57f341f`),
|
|
- duplicate guard in `_publish_blog_to_ghost`: on fallback-after-exception it
|
|
checks `find_draft_by_title()` before re-publishing.
|
|
|
|
Re-test with disclosure.org before ever re-enabling br (e.g. after an aiohttp
|
|
upgrade).
|
|
|
|
## SQLite WAL mode + read-only mounts
|
|
|
|
The database runs in WAL mode, so even read-only access needs the `-shm` file
|
|
writable, or must open with `sqlite3 "file:...?immutable=1"`. Backups (daily
|
|
CronJob `researchowl-db-backup`, 03:00 Europe/Madrid, PVC `researchowl-backups`,
|
|
7-day retention) inherit WAL mode: to inspect one from a read-only mount use
|
|
`?immutable=1`; to restore, copy it to a writable location first.
|
|
|
|
## DDG (duckduckgo_search) blocks the event loop
|
|
|
|
`DDGS()` is synchronous (blocking requests inside). Never call it directly from
|
|
async code — always wrap in `loop.run_in_executor()` (see `_ddg_text_sync` /
|
|
`_ddg_videos_sync` in `src/scraper/exhaustive.py`). Direct calls froze the
|
|
entire Telegram bot during searches until fixed on 2026-07-04 (`8dfd011`).
|
|
|
|
## Google News RSS is a dead end from this infrastructure
|
|
|
|
`news.google.com/rss` entry links point to `/rss/articles/CBMi…` redirects that
|
|
hit a consent wall from EU IPs, and the inner `AU_yqL` id is only resolvable via
|
|
Google's private batchexecute API. Do not retry. The news seed uses Bing News
|
|
RSS instead (`ENABLE_NEWS_SEED`, real publisher URL in the `?url=` param of
|
|
apiclick.aspx — unwrapped by `_unwrap_news_link`).
|