3 ms·

I did just this. Using a $2k a year database from a smaller provider that isn't maxmind, claude and I built a pretty slick ASN based categorization system. I ca

I did just this. Using a $2k a year database from a smaller provider that isn't maxmind, claude and I built a pretty slick ASN based categorization system. I can categorize an ASN as a residential IP, a service provider, a legit crawler/scraper, etc. For anything that is suspicious, I dynamically use turnstile to gate access to our service. Turns out there's no ISP for any VPN, they just contract with a shitload of mom and pop shady colocation services across the world. We collect signals that help determine good vs bad networks. For example, large amounts of requests to .php endpoints, large amounts of empty accounts from the same /24 subnet, etc etc. All these signals let us automatically determine risk, and then put up a challenge. Authenticated users never see the challenge even if they are on a risky network (VPN 99.9% of the time), unless the network has been identified as 100% malicious, then it gets a full block. Here's a small snapshot of the dashboard: https://cos.ridewithgps.com/screenshots/6a7c54d0-12Aug26-358916819.png https://cos.ridewithgps.com/screenshots/6a7c54d0-12Aug26-358... This was probably a total of 3-4 days of work, spread out over a couple months of iterative claude led hacking. I didn't know exactly what to build, but had some of the key architectural ideas in my head. Opus+Faable made easy work of it all, and ended up guiding some really slick improvements for performance. I would say this has dropped about 20% of all traffic to our service, though it turns out turnstile is a massive target for bots, so replacing that with something custom is next on the list.

inigyou 2mo ago
Contracting with their colocation facilities is exactly how that's supposed to work. If you don't actually operate a wide area network then you aren't supposed to be registered in these databases and have IP blocks. The exception is people who do anycast, but VPN companies don't. You know all these guys just switch to residential proxies if they detect a site is blocking data centers, right? Because that's a very common thing to do.
- cullenking 2mo ago
  Not sure what you mean by your first comment - there is no technical reason that I know of that prevents a VPN provider from having their own ASN and address space. As for the latter comment....not sure what your implication is. Yes, bot/spam mitigation is whackamole, but there are consequences for not playing the game of whackamole. Luckily residential proxies are few and far between so far, but they will grow in popularity. When they do, and I can't get by with the occasional individual residential IP ban, we'll come up with other methods to handle. Luckily the signal is strong with vulnerability scanning, which makes it pretty easy to automate. The only reason to put up whole ASN mitigation (captcha/turnstile, outright bans) is just efficiency. Nothing stopping individual IP banning. The scrapers are the tricky ones, since they more easily hide in legit traffic. However legit traffic has patterns that scrapers do not emulate (at least for a service like ours with millions of pieces of user generated content that's easily walkable), so you can still pull out the signal. It's just a little trickier. Definitely a continual arms race though.
  - inigyou 2mo ago
    Majority of scraper traffic right now is from residential proxies
    - cullenking 2mo ago
      That’s not the pattern I am seeing, but I might be outside the norm. The majority of bot action (scraper, spam) comes from vpn providers by a long shot, ignoring (Chinese ASNs, Indian mobile ISPs etc). I see very little consumer isp action except cheap international providers, which are still swamped by vpn traffic.
alam2000 2mo ago
[dead]

inigyou 2mo ago

Contracting with their colocation facilities is exactly how that's supposed to work. If you don't actually operate a wide area network then you aren't supposed to be registered in these databases and have IP blocks. The exception is people who do anycast, but VPN companies don't. You know all these guys just switch to residential proxies if they detect a site is blocking data centers, right? Because that's a very common thing to do.

cullenking 2mo ago
Not sure what you mean by your first comment - there is no technical reason that I know of that prevents a VPN provider from having their own ASN and address space. As for the latter comment....not sure what your implication is. Yes, bot/spam mitigation is whackamole, but there are consequences for not playing the game of whackamole. Luckily residential proxies are few and far between so far, but they will grow in popularity. When they do, and I can't get by with the occasional individual residential IP ban, we'll come up with other methods to handle. Luckily the signal is strong with vulnerability scanning, which makes it pretty easy to automate. The only reason to put up whole ASN mitigation (captcha/turnstile, outright bans) is just efficiency. Nothing stopping individual IP banning. The scrapers are the tricky ones, since they more easily hide in legit traffic. However legit traffic has patterns that scrapers do not emulate (at least for a service like ours with millions of pieces of user generated content that's easily walkable), so you can still pull out the signal. It's just a little trickier. Definitely a continual arms race though.
- inigyou 2mo ago
  Majority of scraper traffic right now is from residential proxies
  - cullenking 2mo ago
    That’s not the pattern I am seeing, but I might be outside the norm. The majority of bot action (scraper, spam) comes from vpn providers by a long shot, ignoring (Chinese ASNs, Indian mobile ISPs etc). I see very little consumer isp action except cheap international providers, which are still swamped by vpn traffic.

alam2000 2mo ago

[dead]