4 ms·

Anyone who suggests a tool that can't understand JavaScript doesn't know what they are talking about You should be using Headless Chrome or Headless Firefox wi

• bdcravens 9y ago
  I haven't dug deep recently, but if you need to automate browser download dialog this wasn't possible with Headless Chrome. (I'd love to find out that this has changed, and you can control it as well as you can with Selenium)
• sp0rk 9y ago
  There are a great many sites that degrade gracefully when JS support is not available. It makes absolutely no sense to waste the resources required to run a full headless browser when simple HTTP requests will retrieve the same information faster, more efficiently, and in a way that's easier to parallelize.
  • xur17 9y ago
    A lot of times you can also watch the api calls JS pages (or apps) make and retrieve nice structured json data. I personally avoid executing js unless it's necessary, as it adds more complexity, and is noticeably more brittle.
    • jordanpg 9y ago
      Using an undocumented API, however, carries significant risk for production operations.
      • flukus 9y ago
        If you're web scraping then you've already decided that this risk is worthwhile, it's already an undocumented API.
  • jordanpg 9y ago
    Yes, but a great many sites don't, and for those, you need Selenium + browser, full stop.