4 ms·
> You limit data scraping by blocking things a human user couldn't do, like access a thousand posts a minute. This is aimed directly at reducing normal activity
by marginalia_nu 3y ago
> You limit data scraping by blocking things a human user couldn't do, like access a thousand posts a minute. This is aimed directly at reducing normal activity across the whole system.
This isn't accurate at all.
Standard operating procedure for large-scale scraping is to use a botnet, it's a sybil attack where you can have tens or even hundreds of thousands of IPs ticking away at 10 requests per minute or some such. Done correctly, it's nearly impossible to detect which nodes are complicit from their access patterns or user agents.
That said, rate limits work relatively poorly against this. A proof of work approach like HashCash might work though.
- CamperBob2 3y agoWouldn't limiting the number of IP addresses that can log in as a single user avoid the botnet problem? That said, someone mentioned that the login page alone involves 11 MB of data transfer, so maybe that is the problem they are confronted with.
- tlogan 3y agoThe solution is not simple but there is one (but it is complex). The question is are you going to spend time and money to implement the right solution or just pick up a cheaper one? Could it be that they just stop paying bills for Akrose and they shut down their service?