3 ms·
Of course curl can. And curl was the best option 15, even 10 years ago. But to scrape a complicated site with curl (or any other non-browser scrapping utility)
by MatthewPhillips 15y ago
Of course curl can. And curl was the best option 15, even 10 years ago. But to scrape a complicated site with curl (or any other non-browser scrapping utility) you wind up basically rewriting the application you're attempting to scrape.
Browsers sole purpose is to run web apps. It doesn't make sense to use anything else.
- baudehlo 15y agoAt the low end of the scale I agree. But once you start scaling it's very difficult to run more than around 40 concurrent sessions with a browser-based scraper. With a pure Nodejs scraper we can run over 1000 parallel sessions per CPU.