4 ms·

Many of those user-agents listed are often faked. Look up which ASN owns their IP. If I block most VPS providers most of the faked bots vanish. There are sti

by Bender 2mo ago

Many of those user-agents listed are often faked. Look up which ASN owns their IP. If I block most VPS providers most of the faked bots vanish. There are still some running from residential and phones using hijacked code (readers that are not really just readers but really multipurpose proxies). On that note, do not trust the linked source code but rather decompile the live code your phone is running and have AI analyze it.

gavinhking 2mo ago
Yeah, that's exactly what these visits are: faked user agents that fail IP verification or Web Bot Auth. What's interesting is the surge across so many websites in the last week.
- Bender 2mo ago
  There are many possibilities but one of them could be some new vuln was released and they are looking for it. That would require looking at the URL's they are requesting. Botters run their own purpose built campaigns. Do you also have a summary of URL's requested by unique counts?
  - gavinhking 2mo ago
    Looks like many of the paths relate to AI coding tools. There are some examples below the chart
    - hluska 2mo ago
      You keep repeating this about a small minority of the tools that were posted.
- nik282000 2mo ago
  I've had a similar bump in scanners in the past week, more than half of it is coming from MS and Google owned IPs and all of them are spoofing AI agents.
bflesch 2mo ago
Same for the origin IP address. The fiber leaving your country is tapped, and those people can inject packets with any origin IP that they want. Your ISP has no way to check if their peer actually received a certain packet from a certain country or not. From a technical perspective, all this "china/russia" attribution is built on a quite shaky foundation. As a sysadmin you'd never know if it would be the British crown attacking your European company instead. Not minimizing nation state cyber crime here, but the packet goes through many hands with different incentives.
- pixl97 2mo ago
  Problem here is there are not single fibers attaching (most) countries, but a bunch of them. If you control both the ingress and egress for some particular users it's possible, but if you don't then your probing packing may end up back in China with a lot of evidence of backscatter.
  - bflesch 2mo ago
    I'd be surprised if there is a single route from EU to non-EU countries which does not pass through British control.
    - inigyou 2mo ago
      Does Britain own all fiber links between Switzerland and France?
      - bflesch 2mo ago
        Unfortunately I can't check how traffic flows from France to Switzerland because I'm not in France. My traffic from Germany passes through a British-owned hop on its way to Switzerland. My German ISP is British as well so either way it wouldn't make a difference, they basically have all traffic twice.
codegeek 2mo ago
Is there an easy way to block any requests originating from VPS etc instead of residential/commercial IP from legitimate users ? I know cloudflare does a few things but I really want to figure out a way to block any request say at nginx or caddy (reverse proxy) from reaching origin servers if they are not from an IP that is not a VPS etc.
- basilikum 2mo ago
  > /commercial IP from legitimate users No, because legitimate users do not just use residential and "commercial" IPs. Like me, right now
  - VladVladikoff 2mo ago
    You are the 0.001%
    - basilikum 2mo ago
      Much more than 0.001% of people care about their privacy or (the larger portion) do not have unfiltered access to the internet.
      - Bender 2mo ago
        I second this. When I have tested blocking VPS/data-centers to my silly blog there were about a dozen people on HN [1] that could not view my site out of the roughly ~17,000 (not counting bots) that could. It's not a big number but those are real people and they count. I am going to move full blocking to a test node that people can play with but I have to finish working with Claude to revise someones repo is is no longer maintained because one does not simply put an anonymous chan board on the great wide open internets without some critical thinking. [1] - https://news.ycombinator.com/item?id=49060945 https://news.ycombinator.com/item?id=49060945
        VladVladikoff 2mo ago
cullenking 2mo ago
I did just this. Using a $2k a year database from a smaller provider that isn't maxmind, claude and I built a pretty slick ASN based categorization system. I can categorize an ASN as a residential IP, a service provider, a legit crawler/scraper, etc. For anything that is suspicious, I dynamically use turnstile to gate access to our service. Turns out there's no ISP for any VPN, they just contract with a shitload of mom and pop shady colocation services across the world. We collect signals that help determine good vs bad networks. For example, large amounts of requests to .php endpoints, large amounts of empty accounts from the same /24 subnet, etc etc. All these signals let us automatically determine risk, and then put up a challenge. Authenticated users never see the challenge even if they are on a risky network (VPN 99.9% of the time), unless the network has been identified as 100% malicious, then it gets a full block. Here's a small snapshot of the dashboard: https://cos.ridewithgps.com/screenshots/6a7c54d0-12Aug26-358916819.png https://cos.ridewithgps.com/screenshots/6a7c54d0-12Aug26-358... This was probably a total of 3-4 days of work, spread out over a couple months of iterative claude led hacking. I didn't know exactly what to build, but had some of the key architectural ideas in my head. Opus+Faable made easy work of it all, and ended up guiding some really slick improvements for performance. I would say this has dropped about 20% of all traffic to our service, though it turns out turnstile is a massive target for bots, so replacing that with something custom is next on the list.
- inigyou 2mo ago
  Contracting with their colocation facilities is exactly how that's supposed to work. If you don't actually operate a wide area network then you aren't supposed to be registered in these databases and have IP blocks. The exception is people who do anycast, but VPN companies don't. You know all these guys just switch to residential proxies if they detect a site is blocking data centers, right? Because that's a very common thing to do.
  - cullenking 2mo ago
    Not sure what you mean by your first comment - there is no technical reason that I know of that prevents a VPN provider from having their own ASN and address space. As for the latter comment....not sure what your implication is. Yes, bot/spam mitigation is whackamole, but there are consequences for not playing the game of whackamole. Luckily residential proxies are few and far between so far, but they will grow in popularity. When they do, and I can't get by with the occasional individual residential IP ban, we'll come up with other methods to handle. Luckily the signal is strong with vulnerability scanning, which makes it pretty easy to automate. The only reason to put up whole ASN mitigation (captcha/turnstile, outright bans) is just efficiency. Nothing stopping individual IP banning. The scrapers are the tricky ones, since they more easily hide in legit traffic. However legit traffic has patterns that scrapers do not emulate (at least for a service like ours with millions of pieces of user generated content that's easily walkable), so you can still pull out the signal. It's just a little trickier. Definitely a continual arms race though.

by Bender 2mo ago

gavinhking 2mo ago

Yeah, that's exactly what these visits are: faked user agents that fail IP verification or Web Bot Auth. What's interesting is the surge across so many websites in the last week.

Bender 2mo ago
There are many possibilities but one of them could be some new vuln was released and they are looking for it. That would require looking at the URL's they are requesting. Botters run their own purpose built campaigns. Do you also have a summary of URL's requested by unique counts?
- gavinhking 2mo ago
  Looks like many of the paths relate to AI coding tools. There are some examples below the chart
  - hluska 2mo ago
    You keep repeating this about a small minority of the tools that were posted.
nik282000 2mo ago
I've had a similar bump in scanners in the past week, more than half of it is coming from MS and Google owned IPs and all of them are spoofing AI agents.

bflesch 2mo ago

Same for the origin IP address. The fiber leaving your country is tapped, and those people can inject packets with any origin IP that they want. Your ISP has no way to check if their peer actually received a certain packet from a certain country or not. From a technical perspective, all this "china/russia" attribution is built on a quite shaky foundation. As a sysadmin you'd never know if it would be the British crown attacking your European company instead. Not minimizing nation state cyber crime here, but the packet goes through many hands with different incentives.

pixl97 2mo ago
Problem here is there are not single fibers attaching (most) countries, but a bunch of them. If you control both the ingress and egress for some particular users it's possible, but if you don't then your probing packing may end up back in China with a lot of evidence of backscatter.
- bflesch 2mo ago
  I'd be surprised if there is a single route from EU to non-EU countries which does not pass through British control.
  - inigyou 2mo ago
    Does Britain own all fiber links between Switzerland and France?
    - bflesch 2mo ago
      Unfortunately I can't check how traffic flows from France to Switzerland because I'm not in France. My traffic from Germany passes through a British-owned hop on its way to Switzerland. My German ISP is British as well so either way it wouldn't make a difference, they basically have all traffic twice.

codegeek 2mo ago

Is there an easy way to block any requests originating from VPS etc instead of residential/commercial IP from legitimate users ? I know cloudflare does a few things but I really want to figure out a way to block any request say at nginx or caddy (reverse proxy) from reaching origin servers if they are not from an IP that is not a VPS etc.

basilikum 2mo ago
> /commercial IP from legitimate users No, because legitimate users do not just use residential and "commercial" IPs. Like me, right now
- VladVladikoff 2mo ago
  You are the 0.001%
  - basilikum 2mo ago
    Much more than 0.001% of people care about their privacy or (the larger portion) do not have unfiltered access to the internet.
    - Bender 2mo ago
      I second this. When I have tested blocking VPS/data-centers to my silly blog there were about a dozen people on HN [1] that could not view my site out of the roughly ~17,000 (not counting bots) that could. It's not a big number but those are real people and they count. I am going to move full blocking to a test node that people can play with but I have to finish working with Claude to revise someones repo is is no longer maintained because one does not simply put an anonymous chan board on the great wide open internets without some critical thinking. [1] - https://news.ycombinator.com/item?id=49060945 https://news.ycombinator.com/item?id=49060945
      - VladVladikoff 2mo ago

cullenking 2mo ago

I did just this. Using a $2k a year database from a smaller provider that isn't maxmind, claude and I built a pretty slick ASN based categorization system. I can categorize an ASN as a residential IP, a service provider, a legit crawler/scraper, etc. For anything that is suspicious, I dynamically use turnstile to gate access to our service. Turns out there's no ISP for any VPN, they just contract with a shitload of mom and pop shady colocation services across the world. We collect signals that help determine good vs bad networks. For example, large amounts of requests to .php endpoints, large amounts of empty accounts from the same /24 subnet, etc etc. All these signals let us automatically determine risk, and then put up a challenge. Authenticated users never see the challenge even if they are on a risky network (VPN 99.9% of the time), unless the network has been identified as 100% malicious, then it gets a full block. Here's a small snapshot of the dashboard: https://cos.ridewithgps.com/screenshots/6a7c54d0-12Aug26-358916819.png https://cos.ridewithgps.com/screenshots/6a7c54d0-12Aug26-358... This was probably a total of 3-4 days of work, spread out over a couple months of iterative claude led hacking. I didn't know exactly what to build, but had some of the key architectural ideas in my head. Opus+Faable made easy work of it all, and ended up guiding some really slick improvements for performance. I would say this has dropped about 20% of all traffic to our service, though it turns out turnstile is a massive target for bots, so replacing that with something custom is next on the list.

inigyou 2mo ago
Contracting with their colocation facilities is exactly how that's supposed to work. If you don't actually operate a wide area network then you aren't supposed to be registered in these databases and have IP blocks. The exception is people who do anycast, but VPN companies don't. You know all these guys just switch to residential proxies if they detect a site is blocking data centers, right? Because that's a very common thing to do.
- cullenking 2mo ago
  Not sure what you mean by your first comment - there is no technical reason that I know of that prevents a VPN provider from having their own ASN and address space. As for the latter comment....not sure what your implication is. Yes, bot/spam mitigation is whackamole, but there are consequences for not playing the game of whackamole. Luckily residential proxies are few and far between so far, but they will grow in popularity. When they do, and I can't get by with the occasional individual residential IP ban, we'll come up with other methods to handle. Luckily the signal is strong with vulnerability scanning, which makes it pretty easy to automate. The only reason to put up whole ASN mitigation (captcha/turnstile, outright bans) is just efficiency. Nothing stopping individual IP banning. The scrapers are the tricky ones, since they more easily hide in legit traffic. However legit traffic has patterns that scrapers do not emulate (at least for a service like ours with millions of pieces of user generated content that's easily walkable), so you can still pull out the signal. It's just a little trickier. Definitely a continual arms race though.