ChemaVXandClaude Fable 5
3d74857d85
El guard anterior vivía en el fallback de autofill y casaba con CUALQUIER draft del mismo título: un fallo pre-POST (Ollama caído) con un draft viejo del mismo topic descartaba en silencio el contenido recién generado. Ahora la recuperación vive dentro de publish_draft y solo se activa cuando el POST fue aceptado (2xx) y falla la lectura de la respuesta — el escenario exacto del incidente br del 2026-07-04 — con filtro since=attempt_start en find_draft_by_title. Cubre a todos los callers: autofill, bare publish y /publish (antes sin proteger: reintento del usuario = draft duplicado). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
55 lines
2.8 KiB
Markdown
55 lines
2.8 KiB
Markdown
# ResearchOwl — Known Issues & Operational Gotchas
|
|
|
|
## Brotli / aiohttp incompatibility
|
|
|
|
Do NOT install any brotli backend (`Brotli`, `brotlicffi`) while on aiohttp 3.14.x.
|
|
aiohttp's incremental brotli decompressor is broken and fails intermittently
|
|
depending on stream chunking ("Can not decode content-encoding: br" on VALID
|
|
streams — the same bytes decompress fine offline; httpx is unaffected).
|
|
|
|
Extra trap: merely *installing* a backend makes aiohttp advertise `br` in the
|
|
default `Accept-Encoding` of every session that doesn't set one explicitly.
|
|
On 2026-07-04 a scraper fix installed brotlicffi and inadvertently enabled br
|
|
project-wide: Ghost/Cloudflare responded in brotli, the decode blew up AFTER
|
|
Ghost had already accepted the POST, and the publish fallback re-published →
|
|
duplicate drafts + silently lost SEO notices.
|
|
|
|
Fix applied (commits `397546a` + `76c927f` + `7d07375`, guard relocated
|
|
2026-07-05):
|
|
- no brotli backend in requirements.txt (comment there explains why),
|
|
- explicit `Accept-Encoding: gzip, deflate` on all Ghost/autofill aiohttp
|
|
sessions and on the scraper HEADERS (`57f341f`),
|
|
- duplicate guard INSIDE `GhostPublisher.publish_draft` (covers every caller,
|
|
including `/publish`): if the POST is accepted (2xx) but reading the response
|
|
fails, it recovers the just-created draft via `find_draft_by_title(title,
|
|
since=attempt_start)` instead of raising. The guard only fires after an
|
|
accepted POST — a pre-POST failure (Ollama down, menu fetch, links) never
|
|
triggers it, so a stale same-title draft from a previous run can no longer
|
|
swallow freshly generated content.
|
|
|
|
Re-test with disclosure.org before ever re-enabling br (e.g. after an aiohttp
|
|
upgrade).
|
|
|
|
## SQLite WAL mode + read-only mounts
|
|
|
|
The database runs in WAL mode, so even read-only access needs the `-shm` file
|
|
writable, or must open with `sqlite3 "file:...?immutable=1"`. Backups (daily
|
|
CronJob `researchowl-db-backup`, 03:00 Europe/Madrid, PVC `researchowl-backups`,
|
|
7-day retention) inherit WAL mode: to inspect one from a read-only mount use
|
|
`?immutable=1`; to restore, copy it to a writable location first.
|
|
|
|
## DDG (duckduckgo_search) blocks the event loop
|
|
|
|
`DDGS()` is synchronous (blocking requests inside). Never call it directly from
|
|
async code — always wrap in `loop.run_in_executor()` (see `_ddg_text_sync` /
|
|
`_ddg_videos_sync` in `src/scraper/exhaustive.py`). Direct calls froze the
|
|
entire Telegram bot during searches until fixed on 2026-07-04 (`8dfd011`).
|
|
|
|
## Google News RSS is a dead end from this infrastructure
|
|
|
|
`news.google.com/rss` entry links point to `/rss/articles/CBMi…` redirects that
|
|
hit a consent wall from EU IPs, and the inner `AU_yqL` id is only resolvable via
|
|
Google's private batchexecute API. Do not retry. The news seed uses Bing News
|
|
RSS instead (`ENABLE_NEWS_SEED`, real publisher URL in the `?url=` param of
|
|
apiclick.aspx — unwrapped by `_unwrap_news_link`).
|