fix(scraper): _unwrap_news_link normaliza //url y descarta apiclick sin URL usable

Un ?url= protocol-relative (//publisher.com/...) caía por el startswith('http')
y devolvía el link apiclick.aspx crudo, que pasaba blacklist y relevancia y
acababa raspado como página de redirect/consent de Bing. Ahora se antepone
https: a los protocol-relative y, si no se puede extraer URL válida de un
apiclick, se devuelve '' para que el caller lo salte.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
ChemaVX
2026-07-05 15:58:40 +00:00
co-authored by Claude Fable 5
parent b58a3ce118
commit fbe8ae1885
2 changed files with 18 additions and 8 deletions
+10 -3
View File
@@ -45,6 +45,13 @@ def test_unwrap_news_link():
assert _unwrap_news_link(wrapped) == "https://example.com/article"
# Links normales pasan intactos
assert _unwrap_news_link("https://thedebrief.org/foo") == "https://thedebrief.org/foo"
# apiclick sin url= no revienta
assert _unwrap_news_link("http://www.bing.com/news/apiclick.aspx?ref=x") \
== "http://www.bing.com/news/apiclick.aspx?ref=x"
# url= protocol-relative se normaliza a https
assert _unwrap_news_link(
"http://www.bing.com/news/apiclick.aspx?url=//example.com/article"
) == "https://example.com/article"
# apiclick sin url= usable se descarta ("" → el caller lo salta), nunca se
# deja pasar el redirect de bing.com como fuente
assert _unwrap_news_link("http://www.bing.com/news/apiclick.aspx?ref=x") == ""
assert _unwrap_news_link(
"http://www.bing.com/news/apiclick.aspx?url=javascript:alert(1)"
) == ""