5 ms·
Nicely done! I have been writing crawlers for a while now and executing Javascript is very expensive and slow, even for Google. When I crawl the web I usually r
by latenightcoding 10y ago
Nicely done!
I have been writing crawlers for a while now and executing Javascript is very expensive and slow, even for Google. When I crawl the web I usually run javascript from a headless browser only on top priority sites.
- elorant 10y agoWhat I find very useful is to run Firefox through Selenium with a plugin installed to disable pictures. Then it's blazing fast.