feat(scraper): F2 — Wikipedia bilingüe siempre y búsqueda full-text
Build & Deploy ResearchOwl / build-and-push (push) Successful in 6s
Build & Deploy ResearchOwl / build-and-push (push) Successful in 6s
Dos arreglos en _seed_wikipedia, motivados por la validación de F1 (found=0 en ambos idiomas para 'incidente ovni de Manises 1979' pese a existir el artículo en es.wikipedia): - opensearch es prefix-de-título: topics verbosos devuelven 0. Se pasa a action=query&list=search (full-text, srlimit=8), que para el mismo topic encuentra 'Manises UFO incident' (en) e 'Incidente OVNI de Manises' (es) + relacionados. Verificado contra la API en ambos. - Se elimina el break tras en: ahora siembra en + es SIEMPRE, cada idioma aislado en su try/except. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
200de018e4
commit
9dae61fde8
+20
-13
@@ -7,7 +7,7 @@ import random
|
||||
import re
|
||||
import time
|
||||
from typing import Optional
|
||||
from urllib.parse import urljoin, urlparse, quote_plus
|
||||
from urllib.parse import urljoin, urlparse, quote, quote_plus
|
||||
|
||||
import aiohttp
|
||||
import feedparser
|
||||
@@ -294,27 +294,34 @@ class ExhaustiveScraper:
|
||||
await asyncio.sleep(random.uniform(1, 3))
|
||||
|
||||
async def _seed_wikipedia(self):
|
||||
"""Search Wikipedia API for correct article URLs.
|
||||
Tries English first, falls back to Spanish if no results found."""
|
||||
"""Siembra Wikipedia en AMBOS idiomas siempre, con búsqueda full-text
|
||||
(list=search). El opensearch anterior era prefix-de-título y devolvía 0
|
||||
para topics verbosos ("incidente ovni de Manises 1979") aunque el
|
||||
artículo exista; y el break tras en dejaba es solo como fallback.
|
||||
Cada idioma va aislado: un fallo no afecta al otro."""
|
||||
http = await self._get_http()
|
||||
added = 0
|
||||
|
||||
for lang in ("en", "es"):
|
||||
try:
|
||||
api_url = (
|
||||
f"https://{lang}.wikipedia.org/w/api.php?action=opensearch"
|
||||
f"&search={quote_plus(self.topic)}&limit=10&format=json"
|
||||
f"https://{lang}.wikipedia.org/w/api.php?action=query"
|
||||
f"&list=search&srsearch={quote_plus(self.topic)}"
|
||||
f"&srlimit=8&format=json"
|
||||
)
|
||||
async with http.get(api_url) as resp:
|
||||
data = await resp.json()
|
||||
urls = data[3] if len(data) > 3 else []
|
||||
for url in urls:
|
||||
if url:
|
||||
await self.db.add_source(self.session_id, url, "wikipedia", depth=0)
|
||||
hits = data.get("query", {}).get("search", [])
|
||||
added = 0
|
||||
for h in hits:
|
||||
title = h.get("title")
|
||||
if not title:
|
||||
continue
|
||||
url = (f"https://{lang}.wikipedia.org/wiki/"
|
||||
f"{quote(title.replace(' ', '_'))}")
|
||||
await self.db.add_source(self.session_id, url, "wikipedia",
|
||||
depth=0, title=title)
|
||||
added += 1
|
||||
logger.info("Wikipedia seed", lang=lang, found=len(urls))
|
||||
if added > 0:
|
||||
break # English results found — no need to try Spanish
|
||||
logger.info("Wikipedia seed", lang=lang, found=added)
|
||||
except Exception as e:
|
||||
logger.warning("Wikipedia API seed failed", lang=lang, error=str(e))
|
||||
|
||||
|
||||
Reference in New Issue
Block a user