fix(scraper): no anunciar br en Accept-Encoding — decode incremental roto en aiohttp 3.14
Build & Deploy ResearchOwl / build-and-push (push) Successful in 7s
Build & Deploy ResearchOwl / build-and-push (push) Successful in 7s
Diagnóstico definitivo: los streams br de disclosure.org/todaywhy.com son VÁLIDOS (los bytes crudos descomprimen offline con brotlicffi), pero el descompresor incremental de aiohttp 3.14.1 falla según el troceo de chunks — intermitente entre sitios e intentos, con Brotli y con brotlicffi. Sin anunciar br los servidores mandan gzip (verificado: los 4 sitios afectados, 200 + contenido íntegro) y el camino zlib es sólido. brotlicffi se queda como red por si llega br no anunciado. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
6c807212a0
commit
57f341fc22
@@ -27,7 +27,12 @@ HEADERS = {
|
|||||||
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
|
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
|
||||||
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
|
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
|
||||||
"Accept-Language": "en-US,en;q=0.9,es;q=0.8",
|
"Accept-Language": "en-US,en;q=0.9,es;q=0.8",
|
||||||
"Accept-Encoding": "gzip, deflate, br",
|
# Sin "br": el descompresor incremental de aiohttp 3.14 falla con streams
|
||||||
|
# brotli válidos según el troceo de chunks (disclosure.org, todaywhy.com;
|
||||||
|
# los mismos bytes descomprimen bien offline con brotlicffi). Con gzip el
|
||||||
|
# camino zlib es sólido. brotlicffi queda instalado por si algún servidor
|
||||||
|
# manda br sin que se anuncie.
|
||||||
|
"Accept-Encoding": "gzip, deflate",
|
||||||
"DNT": "1",
|
"DNT": "1",
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -36,7 +41,7 @@ REDDIT_HEADERS = {
|
|||||||
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
|
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
|
||||||
"Accept": "application/json, text/javascript, */*; q=0.01",
|
"Accept": "application/json, text/javascript, */*; q=0.01",
|
||||||
"Accept-Language": "en-US,en;q=0.9",
|
"Accept-Language": "en-US,en;q=0.9",
|
||||||
"Accept-Encoding": "gzip, deflate, br",
|
"Accept-Encoding": "gzip, deflate", # sin "br", ver HEADERS
|
||||||
"Referer": "https://www.reddit.com/",
|
"Referer": "https://www.reddit.com/",
|
||||||
"X-Requested-With": "XMLHttpRequest",
|
"X-Requested-With": "XMLHttpRequest",
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user