3 ms·
Can I say all of them? Actually, something that may be even more important is free and open access to APIs. I'm okay with registration procedures for larger vo
by kqr 9y ago
Can I say all of them?
Actually, something that may be even more important is free and open access to APIs. I'm okay with registration procedures for larger volumes, but not for hobby use. It seems to me an unnecessary obstacle. If the hit rates are similar to what a user with a web browser would produce why is my script forced to register when the web user is not?
- setr 9y agoThe problem is you can usually just get around the limiter pretty easily, isn't it? Like one site I was scraping did rate limiting by ip; I just spawned 5 aws boxes to do the scraping. And with aws pricing model, there was no real reason to stop at only 5 boxes, since each request was independent.. If I were planning to make a profit on that data, I might well have spawned 1000 boxes to speed things up (took 2 weeks of 24/7 scraping on 5)
- MoBattah 9y agoCan you elaborate?
- setr 9y agoIf you do non-registration rate limiting on an api, then you still need some identifier to calculate api usage; its not hard to find an identifier for a single machine (ie IP address, which isn't actually but close enough), But without registration, theres no way I can think of to associate multiple machines with a single person. And presumably if you're rate-limiting on individual users of the api (and not the total usage across all users), then you're trying to maximize the number of users accessing the api simultaneously (ie you don't want 1 user maxing out resources, denying all other users). But with things like cloud computing, its trivial for a single user to have an absurd number of machines (legally). AWS charges on compute-hour, not number of machines. Running 1 instance for 50 hours and 50 instances for 1 hour has the same cost. But for the server, that difference is significant; you apply rate limiting because you don't want to be hammered for a short duration followed by nothing, at least not from one user. Registration can also be bypassed somewhat trivially (temporary emails and aliases), but its a good deal more effort than bypassing non-registration rate limiting. And presumably most people who want to scrape a dataset large enough to bother bypassing the limiter are nautrally, by their job, aware of what I've described. Bypassing registration takes a good deal more work, and more specific tooling So tl;dr Rate limiting by ip only affects 1 machine, but not necessarily 1 user Rate limiting by registration affects n machines, with 1 user Before cloud vps, 1 machine basically correlated to 1 user. Now its trivial for n machines to correlate to 1 user.