feat(scraper): F4 — siembra de Internet Archive (mediatype:texts)
Build & Deploy ResearchOwl / build-and-push (push) Successful in 6s
Build & Deploy ResearchOwl / build-and-push (push) Successful in 6s
_seed_archive() consulta advancedsearch con el topic completo y siembra las páginas /details/<id> (rows=12); los PDFs de cada item los descubre la recursión y los extrae _extract_pdf. Verificado contra la API: 'Roswell incident declassified' devuelve informes del Proyecto Mogul y documentos del CIA Reading Room. Sin fallback de query a propósito: el AND estricto prima precisión — relajar términos mete ruido (verificado: 'Manises' a secas devuelve libros de loza valenciana). Topics sin material en Archive siembran 0 y el resto de seeds cubren. Aislado en try/except propio, además del return_exceptions del gather. Cierra el plan F1-F4 de mejora del descubrimiento de fuentes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
0ce9933aaa
commit
c701628144
@@ -171,6 +171,7 @@ class ExhaustiveScraper:
|
|||||||
tasks = [
|
tasks = [
|
||||||
self._seed_search(),
|
self._seed_search(),
|
||||||
self._seed_wikipedia(),
|
self._seed_wikipedia(),
|
||||||
|
self._seed_archive(),
|
||||||
]
|
]
|
||||||
if settings.enable_reddit:
|
if settings.enable_reddit:
|
||||||
tasks.append(self._seed_reddit())
|
tasks.append(self._seed_reddit())
|
||||||
@@ -329,6 +330,42 @@ class ExhaustiveScraper:
|
|||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.warning("Wikipedia API seed failed", lang=lang, error=str(e))
|
logger.warning("Wikipedia API seed failed", lang=lang, error=str(e))
|
||||||
|
|
||||||
|
async def _seed_archive(self):
|
||||||
|
"""Siembra Internet Archive (advancedsearch, mediatype:texts): documentos
|
||||||
|
FOIA, informes y libros históricos. Se siembra la página /details/<id>;
|
||||||
|
sus PDFs los descubre la recursión (la propia página los enlaza) y los
|
||||||
|
extrae _extract_pdf. Sin fallback de query: el AND estricto de archive.org
|
||||||
|
con el topic completo prima precisión — relajarlo mete ruido (verificado:
|
||||||
|
'Manises' a secas devuelve loza valenciana). Best-effort: nunca rompe el seed."""
|
||||||
|
try:
|
||||||
|
http = await self._get_http()
|
||||||
|
q = quote_plus(f"({self.topic}) AND mediatype:(texts)")
|
||||||
|
url = (
|
||||||
|
f"https://archive.org/advancedsearch.php?q={q}"
|
||||||
|
"&fl[]=identifier&fl[]=title&rows=12&page=1&output=json"
|
||||||
|
)
|
||||||
|
async with http.get(url) as resp:
|
||||||
|
if resp.status != 200:
|
||||||
|
logger.warning("Archive.org seed non-200", status=resp.status)
|
||||||
|
return
|
||||||
|
data = await resp.json(content_type=None)
|
||||||
|
docs = data.get("response", {}).get("docs", [])
|
||||||
|
added = 0
|
||||||
|
for d in docs:
|
||||||
|
ident = d.get("identifier")
|
||||||
|
if not ident:
|
||||||
|
continue
|
||||||
|
item_url = f"https://archive.org/details/{ident}"
|
||||||
|
title = d.get("title")
|
||||||
|
await self.db.add_source(
|
||||||
|
self.session_id, item_url, "web", depth=0,
|
||||||
|
title=title if isinstance(title, str) else None
|
||||||
|
)
|
||||||
|
added += 1
|
||||||
|
logger.info("Archive.org seed", found=len(docs), added=added)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning("Archive.org seed failed", error=str(e))
|
||||||
|
|
||||||
async def _seed_reddit(self):
|
async def _seed_reddit(self):
|
||||||
"""Search Reddit — sequential to avoid rate limiting"""
|
"""Search Reddit — sequential to avoid rate limiting"""
|
||||||
try:
|
try:
|
||||||
|
|||||||
Reference in New Issue
Block a user