4 ms·
Doesn't seem to noticably slow down my test bot. Headful crawling already takes ~10sec/page so an extra 0.5sec is hardly that big a deal.
by nikisweeting 2y ago
Doesn't seem to noticably slow down my test bot. Headful crawling already takes ~10sec/page so an extra 0.5sec is hardly that big a deal.
- akoboldfrying 2y ago"Headful"?
- danielheath 2y agoAs opposed to “headless” - meaning it runs a whole browser.
- akoboldfrying 2y agoThanks, TIL.
- jeroenhd 2y agoHow effective a PoW firewall is, depends on how conservatively the worst offenders configure their browser engine resource limits. A small scraper may be able to afford spending the extra CPU cycles, but if you're like AI training bots, sometimes sending hundreds of browser instances at a time, the math becomes different. From what I've read about the results, it seems like the approach is effective against the very worst scrapers and bots out there.