feat(scraper): seed de noticias Bing News RSS tras ENABLE_NEWS_SEED (off)
Build & Deploy ResearchOwl / build-and-push (push) Successful in 7s
Build & Deploy ResearchOwl / build-and-push (push) Successful in 7s
Nuevo _seed_news: registra el feed de Bing News como fuente rss y _extract_rss (F3) siembra sus entries — sin lógica de parseo duplicada. _unwrap_news_link extrae la URL real del publisher del ?url= de apiclick.aspx. Bing y no Google News como pedía el plan: verificado en vivo que los links de news.google.com/rss/articles/ acaban en muro de consent (UE) y el id AU_yqL solo se resuelve vía batchexecute interno — habría sembrado URLs muertas. Los de Bing llevan la URL real en texto plano. Flag off por defecto: se activará deliberadamente vía deployment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
8dfd0116b2
commit
a23810b9cc
@@ -36,3 +36,15 @@ def test_simple_chunk():
|
||||
chunks = simple_chunk(text, chunk_size=100, overlap=20)
|
||||
assert len(chunks) > 1
|
||||
assert all(isinstance(c, str) for c in chunks)
|
||||
|
||||
|
||||
def test_unwrap_news_link():
|
||||
from src.scraper.exhaustive import _unwrap_news_link
|
||||
wrapped = ("http://www.bing.com/news/apiclick.aspx?ref=FexRss&aid=&tid=x"
|
||||
"&url=https://example.com/article&c=1&mkt=es-es")
|
||||
assert _unwrap_news_link(wrapped) == "https://example.com/article"
|
||||
# Links normales pasan intactos
|
||||
assert _unwrap_news_link("https://thedebrief.org/foo") == "https://thedebrief.org/foo"
|
||||
# apiclick sin url= no revienta
|
||||
assert _unwrap_news_link("http://www.bing.com/news/apiclick.aspx?ref=x") \
|
||||
== "http://www.bing.com/news/apiclick.aspx?ref=x"
|
||||
|
||||
Reference in New Issue
Block a user