feat(scraper): F2 — Wikipedia bilingüe siempre y búsqueda full-text
Build & Deploy ResearchOwl / build-and-push (push) Successful in 6s

Dos arreglos en _seed_wikipedia, motivados por la validación de F1
(found=0 en ambos idiomas para 'incidente ovni de Manises 1979' pese
a existir el artículo en es.wikipedia):

- opensearch es prefix-de-título: topics verbosos devuelven 0. Se pasa
  a action=query&list=search (full-text, srlimit=8), que para el mismo
  topic encuentra 'Manises UFO incident' (en) e 'Incidente OVNI de
  Manises' (es) + relacionados. Verificado contra la API en ambos.
- Se elimina el break tras en: ahora siembra en + es SIEMPRE, cada
  idioma aislado en su try/except.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
ChemaVX
2026-07-03 15:40:08 +00:00
co-authored by Claude Fable 5
parent 200de018e4
commit 9dae61fde8
+21 -14
View File
@@ -7,7 +7,7 @@ import random
import re import re
import time import time
from typing import Optional from typing import Optional
from urllib.parse import urljoin, urlparse, quote_plus from urllib.parse import urljoin, urlparse, quote, quote_plus
import aiohttp import aiohttp
import feedparser import feedparser
@@ -294,27 +294,34 @@ class ExhaustiveScraper:
await asyncio.sleep(random.uniform(1, 3)) await asyncio.sleep(random.uniform(1, 3))
async def _seed_wikipedia(self): async def _seed_wikipedia(self):
"""Search Wikipedia API for correct article URLs. """Siembra Wikipedia en AMBOS idiomas siempre, con búsqueda full-text
Tries English first, falls back to Spanish if no results found.""" (list=search). El opensearch anterior era prefix-de-título y devolvía 0
para topics verbosos ("incidente ovni de Manises 1979") aunque el
artículo exista; y el break tras en dejaba es solo como fallback.
Cada idioma va aislado: un fallo no afecta al otro."""
http = await self._get_http() http = await self._get_http()
added = 0
for lang in ("en", "es"): for lang in ("en", "es"):
try: try:
api_url = ( api_url = (
f"https://{lang}.wikipedia.org/w/api.php?action=opensearch" f"https://{lang}.wikipedia.org/w/api.php?action=query"
f"&search={quote_plus(self.topic)}&limit=10&format=json" f"&list=search&srsearch={quote_plus(self.topic)}"
f"&srlimit=8&format=json"
) )
async with http.get(api_url) as resp: async with http.get(api_url) as resp:
data = await resp.json() data = await resp.json()
urls = data[3] if len(data) > 3 else [] hits = data.get("query", {}).get("search", [])
for url in urls: added = 0
if url: for h in hits:
await self.db.add_source(self.session_id, url, "wikipedia", depth=0) title = h.get("title")
added += 1 if not title:
logger.info("Wikipedia seed", lang=lang, found=len(urls)) continue
if added > 0: url = (f"https://{lang}.wikipedia.org/wiki/"
break # English results found — no need to try Spanish f"{quote(title.replace(' ', '_'))}")
await self.db.add_source(self.session_id, url, "wikipedia",
depth=0, title=title)
added += 1
logger.info("Wikipedia seed", lang=lang, found=added)
except Exception as e: except Exception as e:
logger.warning("Wikipedia API seed failed", lang=lang, error=str(e)) logger.warning("Wikipedia API seed failed", lang=lang, error=str(e))