ChemaVXandClaude Fable 5
4ee9ad064e
Build & Deploy ResearchOwl / build-and-push (push) Successful in 7s
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
84 lines
4.3 KiB
Markdown
84 lines
4.3 KiB
Markdown
# ResearchOwl — Known Issues & Operational Gotchas
|
|
|
|
## Brotli / aiohttp incompatibility
|
|
|
|
Do NOT install any brotli backend (`Brotli`, `brotlicffi`) while on aiohttp 3.14.x.
|
|
aiohttp's incremental brotli decompressor is broken and fails intermittently
|
|
depending on stream chunking ("Can not decode content-encoding: br" on VALID
|
|
streams — the same bytes decompress fine offline; httpx is unaffected).
|
|
|
|
Extra trap: merely *installing* a backend makes aiohttp advertise `br` in the
|
|
default `Accept-Encoding` of every session that doesn't set one explicitly.
|
|
On 2026-07-04 a scraper fix installed brotlicffi and inadvertently enabled br
|
|
project-wide: Ghost/Cloudflare responded in brotli, the decode blew up AFTER
|
|
Ghost had already accepted the POST, and the publish fallback re-published →
|
|
duplicate drafts + silently lost SEO notices.
|
|
|
|
Fix applied (commits `397546a` + `76c927f` + `7d07375`, guard relocated
|
|
2026-07-05):
|
|
- no brotli backend in requirements.txt (comment there explains why),
|
|
- explicit `Accept-Encoding: gzip, deflate` on all Ghost/autofill aiohttp
|
|
sessions and on the scraper HEADERS (`57f341f`),
|
|
- duplicate guard INSIDE `GhostPublisher.publish_draft` (covers every caller,
|
|
including `/publish`): if the POST is accepted (2xx) but reading the response
|
|
fails, it recovers the just-created draft via `find_draft_by_title(title,
|
|
since=attempt_start)` instead of raising. The guard only fires after an
|
|
accepted POST — a pre-POST failure (Ollama down, menu fetch, links) never
|
|
triggers it, so a stale same-title draft from a previous run can no longer
|
|
swallow freshly generated content.
|
|
|
|
Note: with no backend installed there is NO brotli fallback at all — a server
|
|
that responds `Content-Encoding: br` without it being advertised (misbehaving
|
|
CDN) fails undecodable and that source is lost. Accepted trade-off.
|
|
|
|
The header value lives in one place: `SAFE_ACCEPT_ENCODING` in `src/config.py`.
|
|
Every aiohttp session/request must use it explicitly — never rely on aiohttp's
|
|
default Accept-Encoding, which silently grows `br` if a backend appears.
|
|
|
|
Re-test with disclosure.org before ever re-enabling br (e.g. after an aiohttp
|
|
upgrade).
|
|
|
|
## SQLite WAL mode + read-only mounts
|
|
|
|
The database runs in WAL mode, so even read-only access needs the `-shm` file
|
|
writable, or must open with `sqlite3 "file:...?immutable=1"`. Backups (daily
|
|
CronJob `researchowl-db-backup`, 03:00 Europe/Madrid, PVC `researchowl-backups`,
|
|
7-day retention) inherit WAL mode: to inspect one from a read-only mount use
|
|
`?immutable=1`; to restore, copy it to a writable location first.
|
|
|
|
## DDG (duckduckgo_search) blocks the event loop
|
|
|
|
`DDGS()` is synchronous (blocking requests inside). Never call it directly from
|
|
async code — always wrap in `loop.run_in_executor()` (see `_ddg_text_sync` /
|
|
`_ddg_videos_sync` in `src/scraper/exhaustive.py`). Direct calls froze the
|
|
entire Telegram bot during searches until fixed on 2026-07-04 (`8dfd011`).
|
|
|
|
## Google News RSS is a dead end from this infrastructure
|
|
|
|
`news.google.com/rss` entry links point to `/rss/articles/CBMi…` redirects that
|
|
hit a consent wall from EU IPs, and the inner `AU_yqL` id is only resolvable via
|
|
Google's private batchexecute API. Do not retry. The news seed uses Bing News
|
|
RSS instead (`ENABLE_NEWS_SEED`, real publisher URL in the `?url=` param of
|
|
apiclick.aspx — unwrapped by `_unwrap_news_link`).
|
|
|
|
## Large sources can OOM-kill the pod
|
|
|
|
On 2026-07-10 the pod was OOMKilled (memory limit was 1Gi) mid-research: a
|
|
batch of 20 concurrent sources hit a 98k-word document plus several large PDFs
|
|
at once, and pdfplumber's parse spiked RAM past the limit. The in-memory
|
|
research task died with the pod and its session sat in `running` forever.
|
|
|
|
Mitigations now in place:
|
|
|
|
- Memory limit raised to 2Gi (`k8s-manifests/researchowl/deployment.yaml`).
|
|
- PDFs capped at 15MB (was 50MB), checked both via Content-Length and actual
|
|
body size; pdfplumber runs in `run_in_executor` (it is sync + CPU-heavy and
|
|
also froze the event loop, same class of bug as DDGS) and flushes its page
|
|
cache per page.
|
|
- Extracted content is truncated to `max_content_length` (300k chars) before
|
|
hitting `source_contents`.
|
|
- On startup the bot marks orphaned `running` sessions as `interrupted`.
|
|
|
|
If a research still dies, the scraped sources survive in the DB: `/process`
|
|
re-chunks and scores them without re-scraping.
|