feat(seo): aviso de colisión de tema en /generate blog en

Al publicar el draft EN en Ghost, el título propuesto se compara contra
los posts published+scheduled del sitio (corpus vía Admin API — incluye
la cola programada, justo el caso del doble Kecksburg del 2026-07-10) con
el topic_collision vendorizado. Si colisiona, el notice de Telegram lleva
un bloque "🚨 Posible colisión de tema" en las tres rutas (live, dryrun,
bare). Nunca bloquea: el draft se crea igual, el humano decide (fusionar,
retitular o enlazar a propósito). Solo lang=en (stopwords inglesas).
Títulos ajenos saneados de entidades Markdown (regla de _safe_send).
Aislamiento: cualquier fallo del check → notice sin bloque y warning en
logs, jamás rompe la publicación.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

KNOWN-ISSUES.md

@@ -0,0 +1,83 @@
+# ResearchOwl — Known Issues & Operational Gotchas
+
+## Brotli / aiohttp incompatibility
+
+Do NOT install any brotli backend (`Brotli`, `brotlicffi`) while on aiohttp 3.14.x.
+aiohttp's incremental brotli decompressor is broken and fails intermittently
+depending on stream chunking ("Can not decode content-encoding: br" on VALID
+streams — the same bytes decompress fine offline; httpx is unaffected).
+
+Extra trap: merely *installing* a backend makes aiohttp advertise `br` in the
+default `Accept-Encoding` of every session that doesn't set one explicitly.
+On 2026-07-04 a scraper fix installed brotlicffi and inadvertently enabled br
+project-wide: Ghost/Cloudflare responded in brotli, the decode blew up AFTER
+Ghost had already accepted the POST, and the publish fallback re-published →
+duplicate drafts + silently lost SEO notices.
+
+Fix applied (commits `397546a` + `76c927f` + `7d07375`, guard relocated
+2026-07-05):
+- no brotli backend in requirements.txt (comment there explains why),
+- explicit `Accept-Encoding: gzip, deflate` on all Ghost/autofill aiohttp
+  sessions and on the scraper HEADERS (`57f341f`),
+- duplicate guard INSIDE `GhostPublisher.publish_draft` (covers every caller,
+  including `/publish`): if the POST is accepted (2xx) but reading the response
+  fails, it recovers the just-created draft via `find_draft_by_title(title,
+  since=attempt_start)` instead of raising. The guard only fires after an
+  accepted POST — a pre-POST failure (Ollama down, menu fetch, links) never
+  triggers it, so a stale same-title draft from a previous run can no longer
+  swallow freshly generated content.
+
+Note: with no backend installed there is NO brotli fallback at all — a server
+that responds `Content-Encoding: br` without it being advertised (misbehaving
+CDN) fails undecodable and that source is lost. Accepted trade-off.
+
+The header value lives in one place: `SAFE_ACCEPT_ENCODING` in `src/config.py`.
+Every aiohttp session/request must use it explicitly — never rely on aiohttp's
+default Accept-Encoding, which silently grows `br` if a backend appears.
+
+Re-test with disclosure.org before ever re-enabling br (e.g. after an aiohttp
+upgrade).
+
+## SQLite WAL mode + read-only mounts
+
+The database runs in WAL mode, so even read-only access needs the `-shm` file
+writable, or must open with `sqlite3 "file:...?immutable=1"`. Backups (daily
+CronJob `researchowl-db-backup`, 03:00 Europe/Madrid, PVC `researchowl-backups`,
+7-day retention) inherit WAL mode: to inspect one from a read-only mount use
+`?immutable=1`; to restore, copy it to a writable location first.
+
+## DDG (duckduckgo_search) blocks the event loop
+
+`DDGS()` is synchronous (blocking requests inside). Never call it directly from
+async code — always wrap in `loop.run_in_executor()` (see `_ddg_text_sync` /
+`_ddg_videos_sync` in `src/scraper/exhaustive.py`). Direct calls froze the
+entire Telegram bot during searches until fixed on 2026-07-04 (`8dfd011`).
+
+## Google News RSS is a dead end from this infrastructure
+
+`news.google.com/rss` entry links point to `/rss/articles/CBMi…` redirects that
+hit a consent wall from EU IPs, and the inner `AU_yqL` id is only resolvable via
+Google's private batchexecute API. Do not retry. The news seed uses Bing News
+RSS instead (`ENABLE_NEWS_SEED`, real publisher URL in the `?url=` param of
+apiclick.aspx — unwrapped by `_unwrap_news_link`).
+
+## Large sources can OOM-kill the pod
+
+On 2026-07-10 the pod was OOMKilled (memory limit was 1Gi) mid-research: a
+batch of 20 concurrent sources hit a 98k-word document plus several large PDFs
+at once, and pdfplumber's parse spiked RAM past the limit. The in-memory
+research task died with the pod and its session sat in `running` forever.
+
+Mitigations now in place:
+
+- Memory limit raised to 2Gi (`k8s-manifests/researchowl/deployment.yaml`).
+- PDFs capped at 15MB (was 50MB), checked both via Content-Length and actual
+  body size; pdfplumber runs in `run_in_executor` (it is sync + CPU-heavy and
+  also froze the event loop, same class of bug as DDGS) and flushes its page
+  cache per page.
+- Extracted content is truncated to `max_content_length` (300k chars) before
+  hitting `source_contents`.
+- On startup the bot marks orphaned `running` sessions as `interrupted`.
+
+If a research still dies, the scraped sources survive in the DB: `/process`
+re-chunks and scores them without re-scraping.

requirements.txt

@@ -1,22 +1,25 @@
 # Core
-fastapi==0.115.0
+python-telegram-bot==22.8
-uvicorn==0.30.0
+httpx==0.28.1
-python-telegram-bot==21.5
+aiohttp==3.14.1
-httpx==0.27.0
+# NO instalar backend brotli (Brotli/brotlicffi): el decode incremental br de
-aiohttp==3.10.0
+# aiohttp 3.14 está roto (falla con streams válidos según el troceo), y con un
+# backend presente aiohttp anuncia br POR DEFECTO en toda sesión sin
+# Accept-Encoding explícito — el 2026-07-04 eso rompió publish_draft/autofill
+# contra Ghost/Cloudflare (drafts duplicados). Sin backend, nada anuncia br.
 
 # Scraping
-beautifulsoup4==4.12.3
+beautifulsoup4==4.15.0
-lxml==5.2.2
+lxml==5.4.0
 trafilatura==1.12.2
 youtube-transcript-api==0.6.3
-pdfplumber==0.11.9
+pdfplumber==0.11.10
 feedparser==6.0.12
-duckduckgo-search==6.2.6
+duckduckgo-search==6.4.2
 
 # Storage & Embeddings
 sqlite-vec==0.1.9
-aiosqlite==0.20.0
+aiosqlite==0.22.1
 
 # Processing
 tiktoken==0.7.0
@@ -27,12 +30,12 @@ scikit-learn==1.5.1
 anthropic>=0.40.0
 
 # PDF export
-markdown==3.7
+markdown==3.10.2
 reportlab==4.2.5
 
 # Utilities
-pydantic==2.8.0
+pydantic==2.13.4
-pydantic-settings==2.4.0
+pydantic-settings==2.14.2
 tenacity==9.0.0
 structlog==24.4.0
-python-dotenv==1.0.1
+python-dotenv==1.2.2

src/bot/bot.py

@@ -10,7 +10,7 @@ from typing import Optional
 from zoneinfo import ZoneInfo
 
 import structlog
-from telegram import Update, Message
+from telegram import Update, Message, LinkPreviewOptions
 from telegram.ext import (
     Application, CommandHandler, MessageHandler,
     filters, ContextTypes
@@ -397,7 +397,7 @@ async def cmd_generate(update: Update, ctx: ContextTypes.DEFAULT_TYPE):
                 await update.message.reply_text(
                     generator.last_publish_notice,
                     parse_mode=ParseMode.MARKDOWN,
-                    disable_web_page_preview=True,
+                    link_preview_options=LinkPreviewOptions(is_disabled=True),
                 )
             except Exception as e:
                 logger.warning("Failed to send SEO summary message", error=str(e))
@@ -526,7 +526,10 @@ async def cmd_news(update: Update, ctx: ContextTypes.DEFAULT_TYPE):
 
         items = [item_from_row(r) for r in recent]
         for chunk in format_digest(items):
-            await update.message.reply_text(chunk, disable_web_page_preview=False)
+            # Previews activos a propósito (el digest gana con la tarjeta del link).
+            await update.message.reply_text(
+                chunk, link_preview_options=LinkPreviewOptions(is_disabled=False)
+            )
     finally:
         await db_conn.close()
 
@@ -848,6 +851,27 @@ async def cmd_help(update: Update, ctx: ContextTypes.DEFAULT_TYPE):
 
 # ─── Bot setup ────────────────────────────────────────────────────────────────
 
+async def _mark_interrupted_on_startup(app: Application) -> None:
+    """Las tareas de research viven solo en memoria (_active_tasks): un
+    reinicio del pod las mata sin tocar la DB, y sus sesiones quedan en
+    'running' para siempre — parecen activas en /status y get_active_session.
+    """
+    db_conn = await get_db()
+    try:
+        cursor = await db_conn.execute(
+            "UPDATE research_sessions SET status = ?, updated_at = ? WHERE status = ?",
+            (ResearchStatus.INTERRUPTED, time.time(), ResearchStatus.RUNNING),
+        )
+        await db_conn.commit()
+        if cursor.rowcount:
+            logger.info("Orphaned running sessions marked interrupted",
+                        count=cursor.rowcount)
+    except Exception as e:
+        logger.warning("Interrupted-mark failed — bot continues", error=str(e))
+    finally:
+        await db_conn.close()
+
+
 async def _purge_on_startup(app: Application) -> None:
     db_conn = await get_db()
     try:
@@ -969,7 +993,7 @@ async def _scheduler_loop(app: Application):
                                 for c in chunks:
                                     await app.bot.send_message(
                                         chat_id=news_chat_id, text=c,
-                                        disable_web_page_preview=False,
+                                        link_preview_options=LinkPreviewOptions(is_disabled=False),
                                     )
                                 # Marca todos los pendientes (incluidos los "…y N más")
                                 await db.mark_news_notified([r["id"] for r in pending])
@@ -991,6 +1015,7 @@ async def _start_scheduler(app: Application) -> None:
 
 
 async def _on_startup(app: Application) -> None:
+    await _mark_interrupted_on_startup(app)
     await _purge_on_startup(app)
     await _start_scheduler(app)
 

src/config.py

@@ -1,52 +1,72 @@
-from pydantic_settings import BaseSettings
+from pydantic_settings import BaseSettings, SettingsConfigDict
 from pydantic import Field
 from typing import Optional
 
+# Accept-Encoding para TODA petición aiohttp del proyecto. Nunca anunciar "br":
+# el descompresor incremental de aiohttp 3.14 está roto, y con un backend brotli
+# instalado aiohttp lo anunciaría POR DEFECTO en cualquier sesión sin
+# Accept-Encoding explícito (incidente 2026-07-04, ver KNOWN-ISSUES.md).
+# Toda sesión/petición nueva debe usar esta constante, no un literal.
+SAFE_ACCEPT_ENCODING = "gzip, deflate"
+
 
 class Settings(BaseSettings):
     # Telegram
-    telegram_bot_token: str = Field(..., env="TELEGRAM_BOT_TOKEN")
+    telegram_bot_token: str = Field(...)
-    telegram_allowed_users: str = Field("", env="TELEGRAM_ALLOWED_USERS")  # comma-separated user IDs
+    telegram_allowed_users: str = Field("")  # comma-separated user IDs
 
     # Ollama
-    ollama_url: str = Field("http://ollama.chemavx.xyz", env="OLLAMA_URL")
+    ollama_url: str = Field("http://ollama.chemavx.xyz")
-    ollama_model: str = Field("qwen2.5:3b", env="OLLAMA_MODEL")
+    ollama_model: str = Field("qwen2.5:3b")
-    ollama_embed_model: str = Field("qwen2.5:3b", env="OLLAMA_EMBED_MODEL")
+    ollama_embed_model: str = Field("qwen2.5:3b")
 
     # Claude fallback (optional)
-    anthropic_api_key: Optional[str] = Field(None, env="ANTHROPIC_API_KEY")
+    anthropic_api_key: Optional[str] = Field(None)
-    claude_model: str = Field("claude-haiku-4-5", env="CLAUDE_MODEL")
+    claude_model: str = Field("claude-haiku-4-5")
 
     # Database
-    db_path: str = Field("/data/researchowl.db", env="DB_PATH")
+    db_path: str = Field("/data/researchowl.db")
 
     # Scraping
-    max_depth: int = Field(3, env="MAX_DEPTH")                      # recursion depth
+    max_depth: int = Field(3)                      # recursion depth
-    max_sources: int = Field(150, env="MAX_SOURCES")                # hard cap
+    max_sources: int = Field(150)                # hard cap
-    max_pages_per_search: int = Field(5, env="MAX_PAGES_PER_SEARCH")
+    max_pages_per_search: int = Field(5)
     searxng_url: str = Field(
         "http://searxng-svc.researchowl.svc.cluster.local:8080/search",
-        env="SEARXNG_URL"
     )
-    request_timeout: int = Field(30, env="REQUEST_TIMEOUT")
+    # Motores que el scraper pide a SearXNG (todos vienen configurados por los
-    request_delay: float = Field(1.0, env="REQUEST_DELAY")          # seconds between requests
+    # defaults de la instancia, use_default_settings: true). Verificados en vivo
-    min_content_length: int = Field(200, env="MIN_CONTENT_LENGTH")  # chars
+    # 2026-07-03: la lista ampliada duplica resultados y dominios únicos (~2x)
+    # por ~+2s/query, y da redundancia cuando brave/DDG caen en rate limit o
+    # CAPTCHA. Excluidos: qwant (access denied), yahoo (roto), wikidata (0
+    # resultados en queries de texto), wayback machine (duplica contenido vivo).
+    searxng_engines: str = Field(
+        "duckduckgo,google,bing,brave,startpage,mojeek,presearch,"
+        "google news,bing news,internet archive scholar"
+    )
+    request_timeout: int = Field(30)
+    request_delay: float = Field(1.0)          # seconds between requests
+    min_content_length: int = Field(200)  # chars
+    # Libros/dumps enteros (100k+ palabras) inflan RAM y DB sin aportar al RAG
+    max_content_length: int = Field(300_000)  # chars
 
     # Fuentes opcionales — desactivadas por defecto: la IP del homelab está
     # bloqueada por Reddit (403) y YouTube (transcripts vacíos), eran peso muerto.
-    enable_youtube: bool = Field(False, env="ENABLE_YOUTUBE")
+    enable_youtube: bool = Field(False)
-    enable_reddit: bool = Field(False, env="ENABLE_REDDIT")
+    enable_reddit: bool = Field(False)
+    # Bing News RSS como semilla de actualidad — off hasta activarlo a propósito
+    enable_news_seed: bool = Field(False)
 
     # Processing
-    chunk_size: int = Field(800, env="CHUNK_SIZE")                  # tokens per chunk
+    chunk_size: int = Field(800)                  # tokens per chunk
-    chunk_overlap: int = Field(100, env="CHUNK_OVERLAP")
+    chunk_overlap: int = Field(100)
-    quality_threshold: float = Field(0.3, env="QUALITY_THRESHOLD")  # 0-1, chunks below discarded
+    quality_threshold: float = Field(0.3)  # 0-1, chunks below discarded
 
     # Ghost CMS
-    ghost_url: Optional[str] = Field(None, env="GHOST_URL")
+    ghost_url: Optional[str] = Field(None)
-    ghost_api_key: Optional[str] = Field(None, env="GHOST_API_KEY")
+    ghost_api_key: Optional[str] = Field(None)
-    ghost_url_en: str = Field("", env="GHOST_URL_EN")
+    ghost_url_en: str = Field("")
-    ghost_api_key_en: str = Field("", env="GHOST_API_KEY_EN")
+    ghost_api_key_en: str = Field("")
 
     # SEO autofill — "off" | "on" | "dryrun" (default off).
     #   off    = today's exact behavior (bare draft, no second LLM call).
@@ -55,28 +75,27 @@ class Settings(BaseSettings):
     #   dryrun = runs the full pipeline + reports the proposed SEO to Telegram, but
     #            writes a BARE draft (no seo fields), for inspection before trusting
     #            the write path. A `/generate blog en dry` arg forces dryrun per-call.
-    seo_autofill: str = Field("off", env="SEO_AUTOFILL")
+    seo_autofill: str = Field("off")
 
     # News monitor (RSS de noticias UAP/OVNI) — inerte hasta NEWS_ENABLED=true.
-    NEWS_ENABLED: bool = Field(False, env="NEWS_ENABLED")
+    NEWS_ENABLED: bool = Field(False)
-    NEWS_POLL_INTERVAL_HOURS: int = Field(6, env="NEWS_POLL_INTERVAL_HOURS")
+    NEWS_POLL_INTERVAL_HOURS: int = Field(6)
     # Chat al que el monitor empuja novedades; si None -> 1er TELEGRAM_ALLOWED_USERS
     # (ver propiedad news_chat_id).
-    NEWS_CHAT_ID: Optional[int] = Field(None, env="NEWS_CHAT_ID")
+    NEWS_CHAT_ID: Optional[int] = Field(None)
     NEWS_KEYWORDS: str = Field(
         "ovni,ovnis,uap,ufo,desclasificado,desclasificacion,declassified,"
         "avistamiento,sighting,extraterrestre,non-human,nhi,grusch,"
         "aaro,pursue,fenomeno aereo,whistleblower",
-        env="NEWS_KEYWORDS",
     )
-    NEWS_MAX_ITEMS: int = Field(15, env="NEWS_MAX_ITEMS")
+    NEWS_MAX_ITEMS: int = Field(15)
 
     # Alerts
-    cost_alert_threshold: float = Field(0.15, env="COST_ALERT_THRESHOLD")
+    cost_alert_threshold: float = Field(0.15)
 
     # App
-    log_level: str = Field("INFO", env="LOG_LEVEL")
+    log_level: str = Field("INFO")
-    timezone: str = Field("Europe/Madrid", env="TIMEZONE")
+    timezone: str = Field("Europe/Madrid")
 
     @property
     def allowed_user_ids(self) -> list[int]:
@@ -102,8 +121,7 @@ class Settings(BaseSettings):
     def seo_autofill_dryrun(self) -> bool:
         return (self.seo_autofill or "").strip().lower() == "dryrun"
 
-    class Config:
+    model_config = SettingsConfigDict(env_file=".env")
-        env_file = ".env"
 
 
 settings = Settings()

src/db/database.py

@@ -18,6 +18,7 @@ class ResearchStatus(str, Enum):
     SATURATED = "saturated"
     FINISHED = "finished"
     ERROR = "error"
+    INTERRUPTED = "interrupted"  # el pod se reinició con el research en marcha
 
 
 class OutputType(str, Enum):

src/generator/generator.py

@@ -9,9 +9,10 @@ import json
 import re
 import time
 
+import aiohttp
 import structlog
 
-from src.config import settings
+from src.config import settings, SAFE_ACCEPT_ENCODING
 from src.llm import get_anthropic_client
 from src.processor.processor import OllamaClient, ContentProcessor
 from src.db.database import ResearchDB, OutputType
@@ -385,6 +386,24 @@ def _seo_dryrun_message(ghost: "GhostPublisher", post: dict, seo: dict, inserted
     )
 
 
+def _norm_title(t: str) -> str:
+    """Normaliza un título para compararlo con lo que Ghost almacenó: colapsa
+    whitespace y trunca a 255 chars (límite de posts.title en Ghost — un H1
+    largo del LLM puede volver truncado/normalizado, y la igualdad exacta
+    fallaría justo en los casos que el guard debe cubrir)."""
+    return " ".join((t or "").split())[:255]
+
+
+def _parse_ghost_ts(ts: str) -> float | None:
+    """created_at de la Admin API (ISO 8601, p.ej. 2026-07-04T20:26:00.000+00:00
+    o sufijo Z) → epoch. None si no parsea."""
+    try:
+        from datetime import datetime
+        return datetime.fromisoformat(ts.replace("Z", "+00:00")).timestamp()
+    except Exception:
+        return None
+
+
 class GhostPublisher:
     def __init__(self, lang: str = "es"):
         self.lang = lang
@@ -419,6 +438,64 @@ class GhostPublisher:
         )
         return f"{signing}.{sig}"
 
+    def _admin_headers(self) -> dict:
+        """Headers canónicos de la Admin API (token JWT fresco por llamada).
+        Único sitio donde se construyen: cualquier cambio (Accept-Version,
+        encoding) se hace aquí y cubre todas las llamadas a Ghost."""
+        return {
+            "Authorization": f"Ghost {self._make_token()}",
+            "Accept-Version": "v5.0",
+            # Nunca heredar el default de aiohttp: con un backend brotli
+            # instalado anunciaría br y el decode roto de aiohttp 3.14 rompe
+            # la lectura de respuestas (incidente 2026-07-04, KNOWN-ISSUES.md).
+            "Accept-Encoding": SAFE_ACCEPT_ENCODING,
+        }
+
+    async def _admin_get(self, query: str, timeout: float = 15) -> dict | None:
+        """GET a {url}/ghost/api/admin/{query} con auth + headers canónicos.
+        None si non-200 (logueado); las excepciones de red/parseo suben tal
+        cual para que cada caller aplique su propia política best-effort."""
+        async with aiohttp.ClientSession(
+                timeout=aiohttp.ClientTimeout(total=timeout)) as sess:
+            async with sess.get(f"{self.url}/ghost/api/admin/{query}",
+                                headers=self._admin_headers()) as resp:
+                if resp.status != 200:
+                    body = await resp.text()
+                    logger.warning("Ghost admin GET non-200", query=query[:80],
+                                   status=resp.status, body=body[:200])
+                    return None
+                return await resp.json()
+
+    async def find_draft_by_title(self, title: str,
+                                  since: float | None = None) -> dict | None:
+        """Busca entre los drafts recientes uno con título exacto.
+
+        Mitad de recuperación del guard anti-duplicados de publish_draft: si el
+        POST fue aceptado por Ghost pero la lectura de la respuesta falló
+        (p.ej. decode roto), el draft existe aunque el caller viera una
+        excepción. Con `since` (epoch) solo casan drafts creados en este
+        intento — un draft viejo del mismo título de una run anterior NO cuenta
+        como "ya publicado". La comparación usa _norm_title (whitespace
+        colapsado + truncado a 255) para sobrevivir la normalización de Ghost.
+        Best-effort: None si no hay match o si falla."""
+        data = await self._admin_get(
+            "posts/?filter=status:draft&order=created_at%20desc&limit=15"
+            "&fields=id,title,slug,created_at"
+        )
+        if data is None:
+            return None
+        wanted = _norm_title(title)
+        for p in data.get("posts", []):
+            if _norm_title(p.get("title") or "") != wanted:
+                continue
+            if since is not None:
+                created = _parse_ghost_ts(p.get("created_at") or "")
+                # 60s de margen por posible desfase de reloj Ghost/bot.
+                if created is None or created < since - 60:
+                    continue
+            return p
+        return None
+
     async def publish_draft(self, title: str, markdown_content: str,
                             tags: list[str] | None = None,
                             seo: dict | None = None,
@@ -432,8 +509,6 @@ class GhostPublisher:
           * seo — a generate_seo_fields dict; when given, its meta/OG/Twitter fields
             are added to the post and its (allow-list) tags are used.
         """
-        import aiohttp as _aio
-
         # Caller-supplied linked HTML wins; otherwise convert markdown as before.
         html = body_html if body_html is not None else self._build_html(markdown_content)
 
@@ -454,7 +529,6 @@ class GhostPublisher:
             "sections": [[10, 0]],
         })
 
-        token = self._make_token()
         # Language-aware default tag (fixes the hardcoded "investigacion" for EN).
         tag_names = tags or [_DEFAULT_TAG.get(self.lang, "investigation")]
         post_obj = {
@@ -474,19 +548,40 @@ class GhostPublisher:
                 "twitter_description": seo["twitter_description"],
             })
         body = {"posts": [post_obj]}
-        async with _aio.ClientSession() as sess:
+        attempt_start = time.time()
+        async with aiohttp.ClientSession() as sess:
             async with sess.post(
                 f"{self.url}/ghost/api/admin/posts/",
                 json=body,
-                headers={
+                headers=self._admin_headers(),
-                    "Authorization": f"Ghost {token}",
-                    "Accept-Version": "v5.0",
-                },
             ) as resp:
                 if resp.status not in (200, 201):
                     text = await resp.text()
                     raise ValueError(f"Ghost API {resp.status}: {text[:300]}")
+                # Guard anti-duplicados (incidente 2026-07-04): el 2xx confirma
+                # que Ghost YA creó el draft; si la lectura del body falla
+                # (p.ej. decode br roto), recuperamos el draft recién creado en
+                # vez de propagar — propagar hacía que el caller (o el usuario
+                # con /publish) reintentara y duplicara. El guard vive AQUÍ para
+                # cubrir a todos los callers, y solo se activa tras un POST
+                # aceptado: un fallo pre-POST jamás lo dispara.
+                try:
                     return await resp.json()
+                except Exception as read_err:
+                    existing = None
+                    try:
+                        existing = await self.find_draft_by_title(
+                            title, since=attempt_start)
+                    except Exception as find_err:
+                        logger.warning("publish_draft: draft-exists check failed",
+                                       error=str(find_err))
+                    if existing:
+                        logger.warning(
+                            "publish_draft: POST aceptado pero lectura de la "
+                            "respuesta falló; draft recuperado sin re-publicar",
+                            post_id=existing["id"], error=str(read_err))
+                        return {"posts": [existing]}
+                    raise
 
 
 # ─── Output generation ────────────────────────────────────────────────────────
@@ -517,6 +612,7 @@ class OutputGenerator:
             return ""
         title = _extract_title(full_output) or topic
         mode = _resolve_seo_mode(seo_override)
+        collision_note = await self._collision_note(lang, title)
 
         if mode in ("on", "dryrun"):
             try:
@@ -541,19 +637,28 @@ class OutputGenerator:
                     # surface the proposal so Jose can inspect before trusting writes.
                     result = await ghost.publish_draft(title, full_output)
                     post = result["posts"][0]
-                    self.last_publish_notice = _seo_dryrun_message(ghost, post, seo, inserted_pairs)
+                    self.last_publish_notice = (
+                        _seo_dryrun_message(ghost, post, seo, inserted_pairs)
+                        + collision_note)
                 else:
                     result = await ghost.publish_draft(
                         title, full_output, tags=seo["tags"], seo=seo,
                         body_html=linked_html)
                     post = result["posts"][0]
-                    self.last_publish_notice = _seo_live_message(ghost, post, seo, inserted_pairs)
+                    self.last_publish_notice = (
+                        _seo_live_message(ghost, post, seo, inserted_pairs)
+                        + collision_note)
                 logger.info("Auto-published blog to Ghost",
                             mode=mode, post_id=post["id"], links=len(inserted_pairs))
                 return ""
             except Exception as e:
                 logger.warning("SEO autofill failed; falling back to bare draft",
                                error=str(e))
+                # Sin guard anti-duplicados aquí: vive DENTRO de publish_draft
+                # (solo se activa tras un POST aceptado). Un fallo pre-POST
+                # (Ollama caído, menú, links) cae aquí y DEBE publicar el
+                # contenido nuevo — un guard por título casaba con drafts
+                # viejos del mismo topic y lo descartaba en silencio.
                 # fall through to the bare-draft publish below
 
         # OFF mode, or autofill failed → today's exact bare-draft publish.
@@ -561,11 +666,27 @@ class OutputGenerator:
             result = await ghost.publish_draft(title, full_output)
             post = result["posts"][0]
             logger.info("Auto-published blog to Ghost (bare)", post_id=post["id"])
-            return _bare_ghost_notice(ghost, post)
+            return _bare_ghost_notice(ghost, post) + collision_note
         except Exception as e:
             logger.warning("Auto-publish to Ghost failed", error=str(e))
             return ""
 
+    async def _collision_note(self, lang: str, title: str) -> str:
+        """Aviso de canibalización del título propuesto contra los posts
+        published+scheduled del sitio. Solo EN (las stopwords del motor son
+        inglesas). Nunca lanza y nunca bloquea: el draft se publica igual y
+        el aviso viaja en el notice de Telegram.
+        """
+        if lang != "en":
+            return ""
+        try:
+            from src.seo.autofill import collision_notice, fetch_collision_corpus
+            corpus = await fetch_collision_corpus(lang)
+            return collision_notice(title, corpus) or ""
+        except Exception as e:
+            logger.warning("Topic collision check failed — skipped", error=str(e))
+            return ""
+
     async def generate(self, session_id: int, output_type: OutputType,
                        progress_callback=None, lang: str = "es",
                        seo_override: str | None = None) -> str:

src/news/monitor.py

@@ -21,7 +21,12 @@ UA = "ExclusionZone-NewsBot/1.0 (+https://theexclusionzone.com)"
 
 FEEDS: list[dict] = [
     {"name": "Gran Misterio",      "url": "https://granmisterio.org/feed/",                                                     "lang": "es"},
-    {"name": "Espacio Misterio",   "url": "https://www.espaciomisterio.com/feed/",                                              "lang": "es"},
+    # Reemplazos ES tras retirar Espacio Misterio (XML roto). Verificados 2026-07-03:
+    # MysteryPlanet y Marcianitos publican a diario/casi a diario; El Ojo Crítico es
+    # mensual (poco volumen a propósito). Ufopolis/MundoEsoterico descartados (muerto/roto).
+    {"name": "MysteryPlanet",      "url": "https://mysteryplanet.com.ar/site/feed/",                                            "lang": "es"},
+    {"name": "Marcianitos Verdes", "url": "https://marcianitosverdes.haaan.com/feed/",                                          "lang": "es"},
+    {"name": "El Ojo Crítico",     "url": "https://elojocritico.info/feed/",                                                    "lang": "es"},
     {"name": "Josep Guijarro (YT)", "url": "https://www.youtube.com/feeds/videos.xml?channel_id=UCIlGy26zi-BbX6zDYAMQ9Qg",      "lang": "es"},
     {"name": "r/UFOs",             "url": "https://www.reddit.com/r/UFOs/new/.rss",                                             "lang": "en"},
     {"name": "r/UFOB",             "url": "https://www.reddit.com/r/UFOB/new/.rss",                                             "lang": "en"},
@@ -37,6 +42,16 @@ _SHORT_TOKEN_MAXLEN = 4
 # Límite de Telegram 4096; troceamos por debajo con margen.
 _TG_LIMIT = 4000
 
+# Reddit sin OAuth ratelimita por IP con ventana larga (>30s verificado): la 2ª
+# petición seguida a reddit.com devuelve 429 SIEMPRE, así que solo se pollea un
+# feed de reddit por run, rotando. Contador por proceso: tras un reinicio se
+# empieza por el primero, lo cual es inocuo.
+_reddit_turn = 0
+
+
+def _is_reddit(feed: dict) -> bool:
+    return "reddit.com" in feed["url"]
+
 
 @dataclass
 class NewsItem:
@@ -90,13 +105,32 @@ async def poll_feeds(db, settings) -> list[NewsItem]:
     Best-effort: cada feed va en su propio try/except — uno caído nunca tumba el
     run. feedparser.parse es bloqueante (I/O de red), así que se ejecuta en un
     hilo para no congelar el event loop del bot durante el poll.
+
+    Reddit: solo se pollea UN feed de reddit.com por run, rotando entre ellos
+    (ver _reddit_turn) — dos peticiones seguidas garantizan un 429 en la 2ª.
     """
+    global _reddit_turn
     keywords = [k.strip() for k in settings.NEWS_KEYWORDS.split(",") if k.strip()]
     new_items: list[NewsItem] = []
 
+    reddit_feeds = [f for f in FEEDS if _is_reddit(f)]
+    skip: set[str] = set()
+    if len(reddit_feeds) > 1:
+        chosen = reddit_feeds[_reddit_turn % len(reddit_feeds)]
+        _reddit_turn += 1
+        skip = {f["name"] for f in reddit_feeds if f["name"] != chosen["name"]}
+
     for feed in FEEDS:
+        if feed["name"] in skip:
+            continue
         try:
             parsed = await asyncio.to_thread(feedparser.parse, feed["url"], agent=UA)
+            status = getattr(parsed, "status", None)
+            # Un 429/403 llega con bozo=False y 0 entries: sin este guard se
+            # tragaba en silencio (así estuvo r/UFOB sin sembrar desde F1).
+            if status and status >= 400:
+                log.warning("feed %s HTTP %s; salto", feed["name"], status)
+                continue
             if parsed.bozo and not parsed.entries:
                 log.warning("feed bozo sin entries: %s (%s)",
                             feed["name"], getattr(parsed, "bozo_exception", ""))

src/scraper/exhaustive.py

@@ -7,7 +7,7 @@ import random
 import re
 import time
 from typing import Optional
-from urllib.parse import urljoin, urlparse, quote_plus
+from urllib.parse import urljoin, urlparse, parse_qs, quote, quote_plus
 
 import aiohttp
 import feedparser
@@ -18,7 +18,7 @@ from duckduckgo_search import DDGS
 from youtube_transcript_api import YouTubeTranscriptApi, NoTranscriptFound
 from tenacity import retry, stop_after_attempt, wait_exponential
 
-from src.config import settings
+from src.config import settings, SAFE_ACCEPT_ENCODING
 from src.db.database import ResearchDB
 
 logger = structlog.get_logger()
@@ -27,7 +27,13 @@ HEADERS = {
     "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
     "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
     "Accept-Language": "en-US,en;q=0.9,es;q=0.8",
-    "Accept-Encoding": "gzip, deflate, br",
+    # Sin "br": el descompresor incremental de aiohttp 3.14 falla con streams
+    # brotli válidos según el troceo de chunks (disclosure.org, todaywhy.com;
+    # los mismos bytes descomprimen bien offline). Con gzip el camino zlib es
+    # sólido. OJO: ya NO hay ningún backend brotli instalado (397546a quitó
+    # brotlicffi) — si un servidor responde br sin que se haya anunciado,
+    # aiohttp falla sin recuperación posible y esa fuente se pierde.
+    "Accept-Encoding": SAFE_ACCEPT_ENCODING,
     "DNT": "1",
 }
 
@@ -36,7 +42,7 @@ REDDIT_HEADERS = {
     "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
     "Accept": "application/json, text/javascript, */*; q=0.01",
     "Accept-Language": "en-US,en;q=0.9",
-    "Accept-Encoding": "gzip, deflate, br",
+    "Accept-Encoding": SAFE_ACCEPT_ENCODING,  # sin "br", ver HEADERS
     "Referer": "https://www.reddit.com/",
     "X-Requested-With": "XMLHttpRequest",
 }
@@ -53,6 +59,10 @@ YOUTUBE_RE = re.compile(r"(?:youtube\.com/watch\?v=|youtu\.be/)([a-zA-Z0-9_-]{11
 PDF_RE = re.compile(r"\.pdf(\?|$)", re.IGNORECASE)
 REDDIT_RE = re.compile(r"reddit\.com/(r/\w+/comments/\w+)")
 WIKIPEDIA_RE = re.compile(r"wikipedia\.org/wiki/(.+)")
+# Segmentos de path típicos de feeds, con límite de palabra: el substring puro
+# clasificaba /feedback o /atomic como rss.
+RSS_RE = re.compile(r"/(rss|feeds?|atom)(/|\.xml|\.rss|$)|\.rss$|format=rss",
+                    re.IGNORECASE)
 
 
 async def fetch_with_retry(fetch_fn, source_name: str, max_retries: int = 3):
@@ -82,7 +92,7 @@ def detect_source_type(url: str) -> str:
         return "wikipedia"
     if "arxiv.org" in url:
         return "arxiv"
-    if any(d in url for d in ["rss", "feed", "atom"]):
+    if RSS_RE.search(url):
         return "rss"
     return "web"
 
@@ -100,6 +110,25 @@ def is_blacklisted(url: str) -> bool:
         return True
 
 
+def _unwrap_news_link(link: str) -> str:
+    """Bing News envuelve los links de entry en apiclick.aspx; la URL real del
+    publisher viene en texto plano en el parámetro ?url=. Cualquier otro link
+    pasa sin tocar. Si es un apiclick pero no trae URL usable, devuelve "" —
+    el caller lo descarta por startswith("http") — en vez de dejar pasar el
+    redirect de bing.com, que se rasparía como página de consent/basura."""
+    try:
+        parsed = urlparse(link)
+        if (parsed.netloc.lower().endswith("bing.com")
+                and parsed.path == "/news/apiclick.aspx"):
+            real = parse_qs(parsed.query).get("url", [""])[0].strip()
+            if real.startswith("//"):  # protocol-relative
+                real = "https:" + real
+            return real if real.startswith("http") else ""
+    except Exception:
+        return ""
+    return link
+
+
 def normalize_url(url: str) -> str:
     # Strip only the fragment. NO borrar el query string: rompía URLs de
     # YouTube (watch?v=...) y artículos que enrutan por query (?id=, ?p=).
@@ -167,11 +196,14 @@ class ExhaustiveScraper:
         tasks = [
             self._seed_search(),
             self._seed_wikipedia(),
+            self._seed_archive(),
         ]
         if settings.enable_reddit:
             tasks.append(self._seed_reddit())
         if settings.enable_youtube:
             tasks.append(self._seed_youtube())
+        if settings.enable_news_seed:
+            tasks.append(self._seed_news())
         await asyncio.gather(*tasks, return_exceptions=True)
 
     async def _generate_ddg_queries(self) -> list[str]:
@@ -229,13 +261,17 @@ class ExhaustiveScraper:
         params = {
             "q": query,
             "format": "json",
-            "engines": "duckduckgo,google,bing,brave",
+            "engines": settings.searxng_engines,
             "language": "all",
         }
         headers = {
             "Accept": "application/json",
             "X-Forwarded-For": "127.0.0.1",
             "User-Agent": "ResearchOwl/1.0",
+            # Sin esto la sesión hereda el default de aiohttp, que anunciaría
+            # br si algún día se reinstala un backend brotli — y el decode roto
+            # de 3.14 tumbaría TODAS las búsquedas SearXNG (el camino primario).
+            "Accept-Encoding": SAFE_ACCEPT_ENCODING,
         }
         try:
             async with aiohttp.ClientSession() as session:
@@ -261,6 +297,21 @@ class ExhaustiveScraper:
             logger.warning("SearXNG failed", query=query, error=str(e))
             return []
 
+    # DDGS es síncrono (requests bloqueante por dentro): llamarlo directamente
+    # desde código async congela el event loop entero — bot de Telegram incluido —
+    # mientras dura la búsqueda. Estos helpers deben ejecutarse SIEMPRE vía
+    # run_in_executor.
+
+    @staticmethod
+    def _ddg_text_sync(query: str) -> list[dict]:
+        with DDGS() as ddgs:
+            return list(ddgs.text(query, max_results=settings.max_pages_per_search))
+
+    @staticmethod
+    def _ddg_videos_sync(query: str) -> list[dict]:
+        with DDGS() as ddgs:
+            return list(ddgs.videos(query, max_results=10))
+
     async def _seed_search(self):
         """SearXNG primary + DDG fallback per query"""
         queries = await self._generate_ddg_queries()
@@ -271,12 +322,10 @@ class ExhaustiveScraper:
             if not results:
                 logger.info("SearXNG vacío, usando DDG", query=query)
                 try:
-                    with DDGS() as ddgs:
+                    loop = asyncio.get_running_loop()
-                        ddg_results = list(ddgs.text(
+                    results = await loop.run_in_executor(
-                            query,
+                        None, self._ddg_text_sync, query
-                            max_results=settings.max_pages_per_search
+                    )
-                        ))
-                    results = ddg_results
                     logger.info("DDG fallback ok", query=query, results=len(results))
                 except Exception as e:
                     logger.warning("DDG fallback failed", query=query, error=str(e))
@@ -294,30 +343,92 @@ class ExhaustiveScraper:
             await asyncio.sleep(random.uniform(1, 3))
 
     async def _seed_wikipedia(self):
-        """Search Wikipedia API for correct article URLs.
+        """Siembra Wikipedia en AMBOS idiomas siempre, con búsqueda full-text
-        Tries English first, falls back to Spanish if no results found."""
+        (list=search). El opensearch anterior era prefix-de-título y devolvía 0
+        para topics verbosos ("incidente ovni de Manises 1979") aunque el
+        artículo exista; y el break tras en dejaba es solo como fallback.
+        Cada idioma va aislado: un fallo no afecta al otro."""
         http = await self._get_http()
-        added = 0
 
         for lang in ("en", "es"):
             try:
                 api_url = (
-                    f"https://{lang}.wikipedia.org/w/api.php?action=opensearch"
+                    f"https://{lang}.wikipedia.org/w/api.php?action=query"
-                    f"&search={quote_plus(self.topic)}&limit=10&format=json"
+                    f"&list=search&srsearch={quote_plus(self.topic)}"
+                    f"&srlimit=8&format=json"
                 )
                 async with http.get(api_url) as resp:
                     data = await resp.json()
-                    urls = data[3] if len(data) > 3 else []
+                hits = data.get("query", {}).get("search", [])
-                    for url in urls:
+                added = 0
-                        if url:
+                for h in hits:
-                            await self.db.add_source(self.session_id, url, "wikipedia", depth=0)
+                    title = h.get("title")
+                    if not title:
+                        continue
+                    url = (f"https://{lang}.wikipedia.org/wiki/"
+                           f"{quote(title.replace(' ', '_'))}")
+                    await self.db.add_source(self.session_id, url, "wikipedia",
+                                             depth=0, title=title)
                     added += 1
-                logger.info("Wikipedia seed", lang=lang, found=len(urls))
+                logger.info("Wikipedia seed", lang=lang, found=added)
-                if added > 0:
-                    break  # English results found — no need to try Spanish
             except Exception as e:
                 logger.warning("Wikipedia API seed failed", lang=lang, error=str(e))
 
+    async def _seed_archive(self):
+        """Siembra Internet Archive (advancedsearch, mediatype:texts): documentos
+        FOIA, informes y libros históricos. Se siembra la página /details/<id>;
+        sus PDFs los descubre la recursión (la propia página los enlaza) y los
+        extrae _extract_pdf. Sin fallback de query: el AND estricto de archive.org
+        con el topic completo prima precisión — relajarlo mete ruido (verificado:
+        'Manises' a secas devuelve loza valenciana). Best-effort: nunca rompe el seed."""
+        try:
+            http = await self._get_http()
+            q = quote_plus(f"({self.topic}) AND mediatype:(texts)")
+            url = (
+                f"https://archive.org/advancedsearch.php?q={q}"
+                "&fl[]=identifier&fl[]=title&rows=12&page=1&output=json"
+            )
+            async with http.get(url) as resp:
+                if resp.status != 200:
+                    logger.warning("Archive.org seed non-200", status=resp.status)
+                    return
+                data = await resp.json(content_type=None)
+            docs = data.get("response", {}).get("docs", [])
+            added = 0
+            for d in docs:
+                ident = d.get("identifier")
+                if not ident:
+                    continue
+                item_url = f"https://archive.org/details/{ident}"
+                title = d.get("title")
+                await self.db.add_source(
+                    self.session_id, item_url, "web", depth=0,
+                    title=title if isinstance(title, str) else None
+                )
+                added += 1
+            logger.info("Archive.org seed", found=len(docs), added=added)
+        except Exception as e:
+            logger.warning("Archive.org seed failed", error=str(e))
+
+    async def _seed_news(self):
+        """Siembra Bing News RSS para cobertura de actualidad. Solo registra
+        el feed como fuente tipo rss: _extract_rss (dispatch del pipeline) lo
+        parsea y siembra sus entries — sin duplicar lógica de parseo aquí.
+        Bing y no Google News: los links de Google (news.google.com/rss/articles/
+        CBMi…) van a un muro de consent + redirect cifrado solo resoluble vía su
+        API interna batchexecute (verificado 2026-07-04); los de Bing llevan la
+        URL real del publisher en ?url= (ver _unwrap_news_link).
+        Best-effort: nunca rompe el seed."""
+        try:
+            url = f"https://www.bing.com/news/search?q={quote_plus(self.topic)}&format=rss"
+            await self.db.add_source(
+                self.session_id, url, "rss", depth=0,
+                title=f"Bing News: {self.topic}"
+            )
+            logger.info("News seed added", topic=self.topic)
+        except Exception as e:
+            logger.warning("News seed failed", error=str(e))
+
     async def _seed_reddit(self):
         """Search Reddit — sequential to avoid rate limiting"""
         try:
@@ -347,11 +458,11 @@ class ExhaustiveScraper:
     async def _seed_youtube(self):
         """Search YouTube via DDG for video transcripts"""
         try:
-            with DDGS() as ddgs:
+            loop = asyncio.get_running_loop()
-                results = list(ddgs.videos(
+            results = await loop.run_in_executor(
-                    f"{self.topic} documentary explanation",
+                None, self._ddg_videos_sync,
-                    max_results=10
+                f"{self.topic} documentary explanation"
-                ))
+            )
             for r in results:
                 url = r.get("content", "")
                 if "youtube.com" in url or "youtu.be" in url:
@@ -472,6 +583,9 @@ class ExhaustiveScraper:
                         lambda: self._extract_reddit(url), url
                     )
                     await asyncio.sleep(settings.request_delay)
+                elif source_type == "rss":
+                    # El feed no tiene contenido propio: es puro descubrimiento.
+                    return await self._extract_rss(source_id, url, source["depth"])
                 elif source_type == "pdf":
                     content, title = await fetch_with_retry(
                         lambda: self._extract_pdf(url), url
@@ -514,6 +628,11 @@ class ExhaustiveScraper:
                                         error="Content too short or empty")
             return
 
+        if len(content) > settings.max_content_length:
+            logger.info("Content truncated", source_id=source_id,
+                        original_length=len(content), url=url[:60])
+            content = content[:settings.max_content_length]
+
         word_count = len(content.split())
 
         await self.db.save_source_content(source_id, content)
@@ -669,30 +788,85 @@ class ExhaustiveScraper:
             logger.warning("Reddit extraction failed", url=url, error=str(e))
             return None, None
 
+    async def _extract_rss(self, source_id: int, url: str, depth: int) -> int:
+        """RSS/Atom: siembra las entries del feed como fuentes nuevas (filtradas
+        por relevancia de título/URL) y marca el feed como skipped — no tiene
+        contenido propio que trocear. feedparser.parse es bloqueante: va en hilo,
+        igual que en el monitor de noticias. Devuelve nº de fuentes añadidas."""
+        parsed = await asyncio.to_thread(feedparser.parse, url,
+                                         agent=HEADERS["User-Agent"])
+        entries = parsed.entries or []
+        added = 0
+        if depth < settings.max_depth:
+            # Se escanean TODAS las entries y el cap aplica a las añadidas:
+            # capar antes de filtrar perdía las relevantes que no caen entre
+            # las 20 más recientes (verificado con thedebrief: 0/20 vs 3/100).
+            for e in entries:
+                if added >= 20:  # cap, como los 30 links/página de web
+                    break
+                link = normalize_url(_unwrap_news_link(getattr(e, "link", "") or ""))
+                title = getattr(e, "title", "") or ""
+                if (not link.startswith("http") or is_blacklisted(link)
+                        or not self._url_is_relevant(link, title)):
+                    continue
+                if await self.db.source_exists(self.session_id, link):
+                    continue
+                await self.db.add_source(
+                    self.session_id, link, detect_source_type(link),
+                    depth=depth + 1, title=title or None
+                )
+                added += 1
+        await self.db.update_source(
+            source_id, status="skipped",
+            error=f"rss feed: {added} entries sembradas"
+        )
+        logger.info("RSS feed seeded", url=url[:60],
+                    entries=len(entries), added=added)
+        return added
+
+    # pdfplumber es síncrono y CPU-intensivo: parsear inline congela el event
+    # loop, y con PDFs grandes el pico de RAM puede matar el pod (OOM con
+    # límite de 1-2Gi). Ejecutar SIEMPRE vía run_in_executor.
+
+    @staticmethod
+    def _parse_pdf_sync(path: str) -> str:
+        import pdfplumber
+
+        with pdfplumber.open(path) as pdf:
+            pages = []
+            for page in pdf.pages[:50]:  # max 50 pages
+                pages.append(page.extract_text() or "")
+                page.flush_cache()  # pdfplumber cachea objetos de página: liberar
+            return "\n\n".join(pages)
+
     async def _extract_pdf(self, url: str) -> tuple[Optional[str], Optional[str]]:
         """Download and extract PDF text"""
-        import pdfplumber
         import tempfile
         import os
 
+        max_pdf_bytes = 15 * 1024 * 1024  # varios PDFs concurrentes en RAM: cap agresivo
+
         http = await self._get_http()
         try:
             async with http.get(url) as resp:
                 if resp.status != 200:
                     return None, None
                 content_length = int(resp.headers.get("content-length", 0))
-                if content_length > 50 * 1024 * 1024:  # skip PDFs > 50MB
+                if content_length > max_pdf_bytes:
                     return None, None
                 pdf_bytes = await resp.read()
+                # Sin Content-Length el check anterior no protege
+                if len(pdf_bytes) > max_pdf_bytes:
+                    return None, None
 
             with tempfile.NamedTemporaryFile(suffix=".pdf", delete=False) as f:
                 f.write(pdf_bytes)
                 tmp_path = f.name
+            del pdf_bytes
 
             try:
-                with pdfplumber.open(tmp_path) as pdf:
+                loop = asyncio.get_running_loop()
-                    pages = [p.extract_text() or "" for p in pdf.pages[:50]]  # max 50 pages
+                text = await loop.run_in_executor(None, self._parse_pdf_sync, tmp_path)
-                    text = "\n\n".join(pages)
                 return text, url.split("/")[-1]
             finally:
                 os.unlink(tmp_path)

src/seo/.rules.sha256

@@ -1 +1 @@
-b8faa93b1f7727d3870e18f69b68283d9a97ed9da819ef0cfa79a60cc2c4ab70
+73872aaf4079e5a0b690fbbe7de7e7967edc2297c8bad79142308714452c60ef

src/seo/autofill.py

@@ -37,14 +37,19 @@ SITE_BY_LANG = {
 
 # Tag allow-list — the model may ONLY pick from these; invented tags are dropped
 # deterministically after the call (never trust the LLM to self-limit). EN is the
-# live vocabulary Jose confirmed; ES is not yet constrained, so we don't enforce
+# live vocabulary Jose confirmed; ES is the live core taxonomy on zonadeexclusion
-# it (model picks freely there until an ES vocab is pinned down).
+# (fetched 2026-07-03), mirroring EN. Deliberately excluded from ES: the one-post
+# long-tail tags the model invented before constraining, and "investigacion-2"
+# (a Ghost case-collision duplicate of "investigacion").
 ALLOWED_TAGS = {
     "en": [
         "uap", "declassified", "military-cases", "classic-cases",
         "investigation", "spain-cases", "latin-america",
     ],
-    # "es": [...]  # TODO: pin the live ES tag vocabulary before constraining.
+    "es": [
+        "uap", "desclasificados", "casos-militares", "casos-clasicos",
+        "investigacion", "casos-espana",
+    ],
 }
 
 # Language-aware default tag when nothing from the allow-list fits / the model
@@ -70,33 +75,20 @@ async def fetch_published_menu(lang: str) -> list[dict]:
     try:
         # Lazy import to avoid a heavy/circular import at module load.
         from src.generator.generator import GhostPublisher
-        import aiohttp as _aio
 
         pub = GhostPublisher(lang=lang)
         if not pub.is_configured():
             logger.warning("seo.menu: Ghost not configured for lang", lang=lang)
             return []
 
-        token = pub._make_token()
+        # _admin_get lleva los headers canónicos (auth, Accept-Version y el
-        url = (
+        # Accept-Encoding sin br — ver KNOWN-ISSUES.md) y loguea los non-200.
-            f"{pub.url}/ghost/api/admin/posts/"
+        data = await pub._admin_get(
-            "?filter=status:published&fields=id,slug,title&limit=all"
+            "posts/?filter=status:published&fields=id,slug,title&limit=all",
+            timeout=30,
         )
-        timeout = _aio.ClientTimeout(total=30)
+        if data is None:
-        async with _aio.ClientSession(timeout=timeout) as sess:
-            async with sess.get(
-                url,
-                headers={
-                    "Authorization": f"Ghost {token}",
-                    "Accept-Version": "v5.0",
-                },
-            ) as resp:
-                if resp.status != 200:
-                    body = await resp.text()
-                    logger.warning("seo.menu: non-200 from Ghost",
-                                   status=resp.status, body=body[:200])
             return []
-                data = await resp.json()
 
         menu = [
             {"slug": p["slug"], "title": p.get("title", "")}
@@ -116,9 +108,13 @@ async def fetch_published_menu(lang: str) -> list[dict]:
 def _system_prompt(lang: str) -> str:
     allow = ", ".join(ALLOWED_TAGS.get(lang, []))
     out_lang = "SPANISH" if lang == "es" else "ENGLISH"
+    # The anti-legacy "never use investigacion" rule is EN-only: on the ES site
+    # "investigacion" IS the canonical tag (and the allow-list already excludes
+    # the "investigacion-2" duplicate).
+    legacy_clause = ", never use \"investigacion\"" if lang == "en" else ""
     tag_clause = (
         f"TAGS: choose 2-4 tags ONLY from this exact list (never invent a tag, "
-        f"never translate it, never use \"investigacion\"): {allow}.\n"
+        f"never translate it{legacy_clause}): {allow}.\n"
         if allow else
         "TAGS: 2-4 lowercase-hyphenated topical tags appropriate to the article.\n"
     )
@@ -654,3 +650,62 @@ def insert_internal_links(
                          phrase=phrase, slug=slug)
 
     return "".join(tokens), inserted_pairs
+
+# ─── 4. Topic collision (aviso pre-publish) ──────────────────────────────────
+
+_MD_UNSAFE = re.compile(r"[*_`\[\]]")
+
+
+def _slugify_title(title: str) -> str:
+    """Aproximación del slug que Ghost generará del título — el draft aún no
+    tiene slug real, y topic_collision usa el slug como una de sus señales."""
+    return re.sub(r"[^a-z0-9]+", "-", title.lower()).strip("-")
+
+
+async def fetch_collision_corpus(lang: str) -> list[dict]:
+    """Posts published+scheduled del sitio (id, slug, title, status) para el
+    check de colisión de tema. A diferencia de fetch_published_menu incluye
+    los PROGRAMADOS: chocar con la cola de vacaciones es justo el caso a
+    cazar (2026-07-10: segundo Kecksburg publicado con otro ya en cola).
+    Aislamiento total: cualquier fallo → [] y log, nunca raise.
+    """
+    try:
+        # Lazy import to avoid a heavy/circular import at module load.
+        from src.generator.generator import GhostPublisher
+
+        pub = GhostPublisher(lang=lang)
+        if not pub.is_configured():
+            return []
+        data = await pub._admin_get(
+            "posts/?filter=status:[published,scheduled]"
+            "&fields=id,slug,title,status&limit=all",
+            timeout=30,
+        )
+        if data is None:
+            return []
+        corpus = [p for p in data.get("posts", []) if p.get("slug")]
+        logger.info("seo.collision: corpus fetched", lang=lang, count=len(corpus))
+        return corpus
+    except Exception as e:  # noqa: BLE001 — isolation guarantee
+        logger.warning("seo.collision: corpus fetch failed", lang=lang, error=str(e))
+        return []
+
+
+def collision_notice(title: str, corpus: list[dict]) -> str | None:
+    """Aviso (Markdown seguro para Telegram) si el título propuesto colisiona
+    con un post existente, vía el motor vendorizado R.topic_collision.
+    None si no hay colisión. Los títulos ajenos se sanean de entidades
+    Markdown para no romper el parseo del mensaje (ver regla de _safe_send).
+    """
+    if not corpus:
+        return None
+    candidate = {"id": None, "title": title, "slug": _slugify_title(title)}
+    violations = R.topic_collision(candidate, corpus)
+    if not violations:
+        return None
+    lines = [_MD_UNSAFE.sub("", v.message) for v in violations[:3]]
+    return (
+        "\n\n🚨 *Posible colisión de tema* — el draft se ha creado igualmente:\n"
+        + "\n".join(f"• {line}" for line in lines)
+        + "\nAntes de publicar: fusionar, retitular a otro ángulo o enlazar a propósito."
+    )

src/seo/rules.py

@@ -195,6 +195,112 @@ def r_jsonld(p):
     return [Violation("jsonld.missing", MED, "no BlogPosting JSON-LD", "add JSON-LD")]
 
 
+# ---- topic collision (corpus-aware; NOT in RULES) ---------------------------
+# Two posts about the same case cannibalize each other in the SERP (2026-07-10:
+# a second Kecksburg post was published while another sat scheduled; a "When
+# Nuclear ... Went/Go Silent" near-twin title was already queued). RULES functions
+# are (post) -> violations; this one also needs the rest of the site, so callers
+# (seo_validate.py) pass the corpus explicitly: published + scheduled posts as
+# dicts with at least {id, title, slug}.
+
+TOPIC_STOPWORDS = {
+    # english glue
+    "the", "a", "an", "of", "and", "in", "at", "on", "to", "that", "what",
+    "when", "who", "why", "how", "its", "his", "her", "their", "our", "one",
+    "still", "cant", "couldnt", "went", "go", "goes", "most", "from", "with",
+    "they", "them", "these", "this", "are", "were", "was", "is", "be", "been",
+    "has", "have", "had", "but", "for", "all", "than", "then", "ever", "never",
+    # domain-generic (present in half the catalog — carry no case identity)
+    "ufo", "ufos", "uap", "uaps", "incident", "incidents", "case", "cases",
+    "file", "files", "mystery", "declassified", "declassification", "pentagon",
+    "government", "military", "congress", "secret", "program", "investigation",
+    "evidence", "witness", "witnesses", "document", "documents", "documented",
+    "unexplained", "encounter", "sighting", "sightings", "alien", "aliens",
+    "phenomena", "aerial", "unidentified", "extraordinary", "americas",
+    "american", "video", "footage",
+}
+# 0.70 calibrated 2026-07-10: the "When Nuclear Weapons Go / Arsenal Went
+# Silent" near-twin pair scores 0.742 (char-level penalizes weapons/arsenal);
+# the closest legit-distinct pair in the catalog scores 0.65.
+TITLE_HOOK_SIM_MIN = 0.70   # SequenceMatcher on the pre-colon hook
+SLUG_JACCARD_MIN   = 0.5    # shared slug-token ratio
+# Years >= this are "news era", not case identity: every contemporary post
+# carries the current year (PURSUE 2026, Grusch 2026...) without being the same
+# story. Case years in the catalog run 1947-2019.
+NEWS_YEAR_MIN = 2020
+
+_YEAR_RE = re.compile(r"\b(19|20)\d{2}\b")
+
+
+def _tokens(text):
+    return set(re.findall(r"[a-z0-9]+", _s(text).lower()))
+
+
+def _case_years(title, slug):
+    """Historical case years (pre news-era) found in title+slug."""
+    return {m.group(0) for m in _YEAR_RE.finditer(title + " " + slug)
+            if int(m.group(0)) < NEWS_YEAR_MIN}
+
+
+def _sig_tokens(title, slug):
+    """Case-identity tokens: title+slug minus glue/domain words, years and
+    fragments shorter than 3 chars (possessive 's', initials...)."""
+    toks = _tokens(title) | _tokens(slug.replace("-", " "))
+    return {t for t in toks
+            if len(t) >= 3 and t not in TOPIC_STOPWORDS and not _YEAR_RE.fullmatch(t)}
+
+
+def _hook(title):
+    return _s(title).split(":")[0].strip().lower()
+
+
+def topic_collision(post, corpus):
+    """Compare one candidate post against the site corpus → list[Violation].
+
+    Fires when the candidate and an existing post look like the same story:
+      - share a case year AND a case-identity token (Kecksburg+1965), or
+      - their pre-colon title hooks read nearly the same, or
+      - their slugs share most of their tokens.
+    """
+    from difflib import SequenceMatcher
+
+    out = []
+    c_years = _case_years(_s(post.get("title")), _s(post.get("slug")))
+    c_sig = _sig_tokens(post.get("title"), _s(post.get("slug")))
+    c_hook = _hook(post.get("title"))
+    c_slug_toks = _tokens(_s(post.get("slug")).replace("-", " "))
+
+    for other in corpus:
+        if other.get("id") == post.get("id"):
+            continue
+        o_title, o_slug = _s(other.get("title")), _s(other.get("slug"))
+        o_years = _case_years(o_title, o_slug)
+        o_sig = _sig_tokens(o_title, o_slug)
+
+        reasons = []
+        if (c_years & o_years) and (c_sig & o_sig):
+            shared = ", ".join(sorted(c_sig & o_sig)[:3] + sorted(c_years & o_years))
+            reasons.append((HIGH, f"same case + year ({shared})"))
+        hook_sim = SequenceMatcher(None, c_hook, _hook(o_title)).ratio()
+        if c_hook and hook_sim >= TITLE_HOOK_SIM_MIN:
+            reasons.append((MED, f"title hooks {hook_sim:.0%} similar"))
+        o_slug_toks = _tokens(o_slug.replace("-", " "))
+        union = c_slug_toks | o_slug_toks
+        if union:
+            jac = len(c_slug_toks & o_slug_toks) / len(union)
+            if jac >= SLUG_JACCARD_MIN:
+                reasons.append((MED, f"slugs {jac:.0%} overlapping"))
+
+        if reasons:
+            sev = max(s for s, _ in reasons)
+            why = "; ".join(r for _, r in reasons)
+            out.append(Violation(
+                "topic.collision", sev,
+                f"collides with [{other.get('status', '?')}] \"{o_title[:60]}\" — {why}",
+                "merge, retitle to a distinct angle, or interlink deliberately"))
+    return out
+
+
 RULES = [
     r_meta_title,
     r_meta_description,

tests/test_scraper.py

@@ -11,6 +11,16 @@ def test_detect_source_type():
     assert detect_source_type("https://example.com/article") == "web"
 
 
+def test_detect_source_type_rss():
+    assert detect_source_type("https://thedebrief.org/feed/") == "rss"
+    assert detect_source_type("https://example.com/rss") == "rss"
+    assert detect_source_type("https://example.com/feeds/all.atom.xml") == "rss"
+    assert detect_source_type("https://www.liberationtimes.com/?format=rss") == "rss"
+    # Falsos positivos del substring viejo: no son feeds.
+    assert detect_source_type("https://example.com/feedback") == "web"
+    assert detect_source_type("https://example.com/atomic-theory") == "web"
+
+
 def test_is_blacklisted():
     assert is_blacklisted("https://facebook.com/something") == True
     assert is_blacklisted("https://en.wikipedia.org/wiki/Test") == False
@@ -26,3 +36,22 @@ def test_simple_chunk():
     chunks = simple_chunk(text, chunk_size=100, overlap=20)
     assert len(chunks) > 1
     assert all(isinstance(c, str) for c in chunks)
+
+
+def test_unwrap_news_link():
+    from src.scraper.exhaustive import _unwrap_news_link
+    wrapped = ("http://www.bing.com/news/apiclick.aspx?ref=FexRss&aid=&tid=x"
+               "&url=https://example.com/article&c=1&mkt=es-es")
+    assert _unwrap_news_link(wrapped) == "https://example.com/article"
+    # Links normales pasan intactos
+    assert _unwrap_news_link("https://thedebrief.org/foo") == "https://thedebrief.org/foo"
+    # url= protocol-relative se normaliza a https
+    assert _unwrap_news_link(
+        "http://www.bing.com/news/apiclick.aspx?url=//example.com/article"
+    ) == "https://example.com/article"
+    # apiclick sin url= usable se descarta ("" → el caller lo salta), nunca se
+    # deja pasar el redirect de bing.com como fuente
+    assert _unwrap_news_link("http://www.bing.com/news/apiclick.aspx?ref=x") == ""
+    assert _unwrap_news_link(
+        "http://www.bing.com/news/apiclick.aspx?url=javascript:alert(1)"
+    ) == ""

tests/test_seo_autofill.py

@@ -0,0 +1,83 @@
+from src.seo.autofill import ALLOWED_TAGS, DEFAULT_TAG, _coerce, _system_prompt
+
+BASE = {
+    "meta_title": "t",
+    "meta_description": "d",
+    "custom_excerpt": "e",
+    "image_query": "q",
+    "image_context": "c",
+}
+
+
+def test_coerce_es_drops_invented_tags():
+    obj = dict(BASE, tags=["uap", "humanoides", "Desclasificados", "investigacion-2"])
+    out = _coerce(obj, "es")
+    assert out["tags"] == ["uap", "desclasificados"]
+
+
+def test_coerce_es_falls_back_to_default():
+    obj = dict(BASE, tags=["pentagono", "encuentros-cercanos"])
+    out = _coerce(obj, "es")
+    assert out["tags"] == [DEFAULT_TAG["es"]]
+
+
+def test_coerce_en_still_constrained():
+    obj = dict(BASE, tags=["uap", "investigacion"])
+    out = _coerce(obj, "en")
+    assert out["tags"] == ["uap"]
+
+
+def test_system_prompt_lists_allowed_tags_per_lang():
+    es = _system_prompt("es")
+    en = _system_prompt("en")
+    for tag in ALLOWED_TAGS["es"]:
+        assert tag in es
+    # La regla anti-legacy 'never use "investigacion"' es solo para EN: en ES
+    # "investigacion" es el tag canónico del allow-list.
+    assert 'never use "investigacion"' in en
+    assert 'never use "investigacion"' not in es
+    assert "ONLY from this exact list" in es
+
+
+# ─── topic collision ─────────────────────────────────────────────────────────
+
+from src.seo.autofill import collision_notice, _slugify_title
+
+CORPUS = [
+    {"id": "1", "status": "published", "slug": "kecksburg-1965-acorn-ufo-missing-nasa-files",
+     "title": 'Kecksburg 1965: The Acorn-Shaped Object, the Missing NASA Files, and "Pennsylvania\'s Roswell"'},
+    {"id": "2", "status": "scheduled", "slug": "uss-russell-2019-pyramid-uap-channel-islands",
+     "title": "USS Russell 2019: The Pyramid UAP Video and the Channel Islands Drone Swarm"},
+]
+
+
+def test_collision_fires_on_same_case_and_year():
+    note = collision_notice("Kecksburg 1965: New Acorn Evidence", CORPUS)
+    assert note is not None
+    assert "kecksburg" in note.lower()
+    assert "1965" in note
+
+
+def test_collision_none_on_distinct_case():
+    assert collision_notice("Tehran 1976: The Jet-Disabling Encounter", CORPUS) is None
+
+
+def test_collision_none_on_empty_corpus():
+    assert collision_notice("Kecksburg 1965: Anything", []) is None
+
+
+def test_collision_note_is_markdown_safe():
+    corpus = [{"id": "9", "status": "published", "slug": "weird-1990-case",
+               "title": "Weird *1990* [Case] with_underscores and `ticks`"}]
+    note = collision_notice("Weird 1990: Case Revisited", corpus)
+    assert note is not None
+    # las entidades Markdown de títulos ajenos se sanean (solo quedan las nuestras)
+    bullets = [line for line in note.split("\n") if line.startswith("• ")]
+    assert bullets
+    for line in bullets:
+        for ch in "*_`[]":
+            assert ch not in line
+
+
+def test_slugify_title():
+    assert _slugify_title("USS Russell 2019: The Pyramid UAP!") == "uss-russell-2019-the-pyramid-uap"