2 ms·
Another implicit social contract is that you can tell whether a request is coming from a commercial or non-commercial source based on the originating ISP. This
by Retr0id 1y ago
Another implicit social contract is that you can tell whether a request is coming from a commercial or non-commercial source based on the originating ISP. This was always a heuristic but it was more reliable in the past.
If 1000 AWS boxes start hammering your API you might raise an eyebrow, but 1000 requests coming from residential ISPs around the world could be an organic surge in demand for your service.
Residential proxy services break this - which has been happening on some level for a long time, but the AI-training-set arms race has driven up demand and thus also supply.
It's quite easy to block all of AWS, for example, but it's less easy to figure out which residential IPs are part of a commercially-operated botnet.
- SoftTalker 1y ago> it's less easy to figure out which residential IPs are part of a commercially-operated botnet Is the client navigating the site faster than humanly possible? It's a bot. This seems like a simple test. > 1000 requests coming from residential ISPs around the world could be an organic surge But probably isn't.
- Retr0id 1y ago> This seems like a simple test. Not when the singular bot has a pool of millions of IPs to originate each request from. If you think there's an easy solution here, productize it and make billions.
- SoftTalker 1y agoIPs that you've never seen before are hitting a single random page deep within your site are bots, or first-time followers of a search engine link. Grey list them and respond slowly. If they are seen again at normal human rates, unthrottle them.
- danielscrubs 1y agoAka, make the web worse for everyone. Thanks for playing.