4 ms·
I've just discovered Headless Chrome crawler and it works pretty well. Not sure how well it will scale, but I'll index a few hundred sites only. https://github
by tiborsaas 6y ago
I've just discovered Headless Chrome crawler and it works pretty well. Not sure how well it will scale, but I'll index a few hundred sites only.
https://github.com/yujiosaka/headless-chrome-crawler https://github.com/yujiosaka/headless-chrome-crawler
- mnmkng 6y agoI have not tried the Headless Chrome Crawler personally, but try the Apify SDK out https://github.com/apify/apify-js https://github.com/apify/apify-js if the Headless Chrome crawler does not scale well enough. We use it to scrape billions of pages every month.