5 ms·

I worked on a large web scraper for several years and JavaScript almost never needs to be executed. The only times I've had were to extract obfuscated links tha

I worked on a large web scraper for several years and JavaScript almost never needs to be executed. The only times I've had were to extract obfuscated links that are revealed by some bit twiddling code, specific to each request, and this was achievable by forking out to deno.

shubb 5y ago
I think javascript comes up because cloudflare use some kind of javascript challenge as part of the DDOS protection. There are python libraries that know how to deal with it, or you can use some level of headless browser. https://github.com/VeNoMouS/cloudscraper https://github.com/VeNoMouS/cloudscraper
missblit 5y ago
This is highly domain (and sometimes User-Agent) dependent and in my experience JS is required more and more. e.g. good luck trying to get much out of youtube.com (or any other video site) without executing JS.
- maltz 5y ago
  YouTube has "var ytInitialData" & "var ytInitialPlayerResponse" params hardcoded in HTML. No need to run JS!