3 ms·
From my experience headless browser scraping is in the order of 100x slower due to increased bandwidth, cpu and memory. I would seriously suggest the other, sta
by samwillis 4y ago
From my experience headless browser scraping is in the order of 100x slower due to increased bandwidth, cpu and memory. I would seriously suggest the other, start with traditional scraping and if you can’t make it work then go headless. The difference in speed is to the point that going a longer route (maybe having to scrape 10x more pages for example) to the content that enables you to not use headless will probably still end up quicker.
On the memory side, headless you end up with far more memory leaks, having to manage stopping and starting new browser instances while maintaining scraping state. The devops overhead is probably 10x more with headless.
- taosx 4y agoAgreed (especially on the devops overhead), some options to lower the bandwidth usage using request blocking: - block ads - blocks videos, images - css stylesheets In the past I've built scraper infrastructure (headless pools, credential stores, proxy managers, agent profiles) and managed to get a pretty efficient service by tracking cpu,memory,network usage per each job and writing specialized versions. I got pretty far trying to automatically generate specialized scrapers from previous requests but I moved to other projects. Scraping becomes boring really fast if you don't use the data in meaningful ways.
- MonaroVXR 4y agoWhat is traditional?
- elorant 4y agoIf you don't load images, and you disable 3rd party JavaScript you can save a lot of loading time with a headless. I'm using a preconfigured profile in Firefox with uMatrix installed and sites load at least 50% faster. Add another 20-30% from not loading images, and suddenly using a headless becomes affordable.