fix(scraper): _unwrap_news_link normaliza //url y descarta apiclick sin URL usable
Un ?url= protocol-relative (//publisher.com/...) caía por el startswith('http')
y devolvía el link apiclick.aspx crudo, que pasaba blacklist y relevancia y
acababa raspado como página de redirect/consent de Bing. Ahora se antepone
https: a los protocol-relative y, si no se puede extraer URL válida de un
apiclick, se devuelve '' para que el caller lo salte.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
b58a3ce118
commit
fbe8ae1885
+10
-3
@@ -45,6 +45,13 @@ def test_unwrap_news_link():
|
||||
assert _unwrap_news_link(wrapped) == "https://example.com/article"
|
||||
# Links normales pasan intactos
|
||||
assert _unwrap_news_link("https://thedebrief.org/foo") == "https://thedebrief.org/foo"
|
||||
# apiclick sin url= no revienta
|
||||
assert _unwrap_news_link("http://www.bing.com/news/apiclick.aspx?ref=x") \
|
||||
== "http://www.bing.com/news/apiclick.aspx?ref=x"
|
||||
# url= protocol-relative se normaliza a https
|
||||
assert _unwrap_news_link(
|
||||
"http://www.bing.com/news/apiclick.aspx?url=//example.com/article"
|
||||
) == "https://example.com/article"
|
||||
# apiclick sin url= usable se descarta ("" → el caller lo salta), nunca se
|
||||
# deja pasar el redirect de bing.com como fuente
|
||||
assert _unwrap_news_link("http://www.bing.com/news/apiclick.aspx?ref=x") == ""
|
||||
assert _unwrap_news_link(
|
||||
"http://www.bing.com/news/apiclick.aspx?url=javascript:alert(1)"
|
||||
) == ""
|
||||
|
||||
Reference in New Issue
Block a user