3 ms·
I wonder how well this will actually work. The core problem is that alot of crawlers aren't spending their money. They are part of a botnet so they are just sp
by shanemhansen 1y ago
I wonder how well this will actually work.
The core problem is that alot of crawlers aren't spending their money. They are part of a botnet so they are just spending the victim's money.
But hopefully most of the crawlers aren't botnets or funded by free VC money so they have an economic incentive to avoid crawling systems requiring proof-of-work.
- Havoc 1y agoPretty sure the AI crawlers aren't botnets
- thayne 1y agoI'd bet some of them, run by less scrupulous entities, are.
- theshrike79 1y agoIt's not about stopping, more about slowing down. (numbers for presentation purposes only) If every request goes from 10ms to 100ms - a human won't care as they'll click like 5-6 times total, while reading the output between page load, but a bot will crawl the site 10 times slower.