4 ms·
"curl" can scrape sites like that. Just watch the Chrome "Network" window and see what requests get made. Then use that. Scraping doesn't need a browser.
by baudehlo 15y ago
"curl" can scrape sites like that. Just watch the Chrome "Network" window and see what requests get made. Then use that. Scraping doesn't need a browser.
- MatthewPhillips 15y agoOf course curl can. And curl was the best option 15, even 10 years ago. But to scrape a complicated site with curl (or any other non-browser scrapping utility) you wind up basically rewriting the application you're attempting to scrape. Browsers sole purpose is to run web apps. It doesn't make sense to use anything else.
- baudehlo 15y agoAt the low end of the scale I agree. But once you start scaling it's very difficult to run more than around 40 concurrent sessions with a browser-based scraper. With a pure Nodejs scraper we can run over 1000 parallel sessions per CPU.