3 ms·

How does that explain blocks that happen to less common browsers and/or less common platforms?

derefr 3y ago
Heuristics are about optimizing between false positives and false negatives. Many headless-browser stealth techniques involve rotating between the signatures and reflected metrics of real — but niche and/or ancient — User-Agents. (For some reason, the developers of these stealth systems think that variety beats commonality. Maybe it makes sense if they're specifically trying to overcome Apache mod_security's signature-based UA blocking or something.) It turns out that when you actually see one of these UAs in your server logs, it's far more (99.99%) likely to be a stealthed bot that picked that UA out of a bag, than it is to be an actual niche/ancient UA. In the case of the niche UAs, this is a tragedy of the commons. In the case of the ancient UAs, though, there's no downside to blocking them entirely — because if the traffic is going through Cloudflare at all, then you're already requiring of the client a minimum version of TLS that the real old UAs can't even speak. So the only things actually saying they're that old device — but managing to get through an HTTP request at all — are stealthed bots.
- runnerup 3y ago
  > Many headless-browser stealth techniques involve rotating between the signatures and reflected metrics of real — but niche and/or ancient — User-Agents I work in this space and we just use fingerprints we collect from actual users over the previous month. The hardest work is: 1) reverse-engineering the javascript of the CAPTCHA/fingerprinting solutions so that we can collect and encapsulate the fingerprints correctly in a way that looks native to cloudflare/recaptcha/etc, and 2) Training AI models to solve the captchas well enough. Sounds like most of the people you catch are using ancient user-agents. But I doubt most of the people you want to catch are.
  - derefr 3y ago
    I mean, when I'm talking about stealth techniques, I'm not talking about what I'm seeing in my own server logs, but rather about what I find built into various bits of, ahem, "anti-detect software for affiliate marketing" tech that I dredge up from fraud Telegram groups, carding marketplaces, etc. that the attackers I do catch seem to frequent. (Gotta stay five steps ahead!) I suppose there are verticals where the data is so valuable, and the garden around it so walled-in, that you could build a whole IT business with a custom scraping stack just around extracting that data to then resell it. (I presume that's the business you're in.) But for most verticals, the "attackers" you'll see in your logs aren't people building a data-broker business, and so aren't building their own secret-sauce anonymity from scratch; rather, they're end-users who want to do an end-run around your rate-limits, commit promotion fraud, etc., and so want to buy anonymity as a product, script-kiddie style. And "anonymity as a product", sold publicly (rather than through high-value contracts) tends to suck. It's script-kiddies buying from script-kiddies, with no real engineering in sight. > I work in this space and we just use fingerprints we collect from actual users over the previous month. Are you sure you're not in a citogenesis cycle? How sure are you that some of those "real users" aren't your peers' stealthed bots, who in turn picked up those fingerprints from unknowningly observing other stealthed bots in their logs, who...
    - runnerup 3y ago
      > Are you sure you're not in a citogenesis cycle? How sure are you that some of those "real users" aren't your peers' stealthed bots, who in turn picked up those fingerprints from unknowningly observing other stealthed bots in their logs, who... Doesn't matter, they work (at least when used in combination with high-quality proxy IP's). If they stopped working I'd do something else. We only apply hard science when absolutely needed, otherwise it's mostly wire and duct tape holding things together -- ruthless focus on creating business value. We definitely only sell this via high-value contracts, so you're probably mostly correct there. Though puppeteer-stealth deserves at least a quiet shout-out for not completely sucking. That said, we do pay attention to a lot of the research in the field, even if we only apply the absolute bare minimum needed to create business value. Eric Wustrow[0] at UC Boulder does really, really good work in an adjacent space, and we've found some his papers/software to be helpful, as well as those of some of the colleagues he works most closely with. I don't think he'd love our applications of his research, but our technological needs dovetail well with the needs of the anti-censorship research that he works on. If you were interested in the degree of "citogenesis", I think that's something that academic researchers like Wustrow et. al would be very well-positioned to investigate. Highly recommend any of their papers, they make front page of HN surprisingly often. 0: https://ericw.us/trow/ https://ericw.us/trow/

Heuristics are about optimizing between false positives and false negatives. Many headless-browser stealth techniques involve rotating between the signatures and reflected metrics of real — but niche and/or ancient — User-Agents. (For some reason, the developers of these stealth systems think that variety beats commonality. Maybe it makes sense if they're specifically trying to overcome Apache mod_security's signature-based UA blocking or something.) It turns out that when you actually see one of these UAs in your server logs, it's far more (99.99%) likely to be a stealthed bot that picked that UA out of a bag, than it is to be an actual niche/ancient UA. In the case of the niche UAs, this is a tragedy of the commons. In the case of the ancient UAs, though, there's no downside to blocking them entirely — because if the traffic is going through Cloudflare at all, then you're already requiring of the client a minimum version of TLS that the real old UAs can't even speak. So the only things actually saying they're that old device — but managing to get through an HTTP request at all — are stealthed bots.

runnerup 3y ago
> Many headless-browser stealth techniques involve rotating between the signatures and reflected metrics of real — but niche and/or ancient — User-Agents I work in this space and we just use fingerprints we collect from actual users over the previous month. The hardest work is: 1) reverse-engineering the javascript of the CAPTCHA/fingerprinting solutions so that we can collect and encapsulate the fingerprints correctly in a way that looks native to cloudflare/recaptcha/etc, and 2) Training AI models to solve the captchas well enough. Sounds like most of the people you catch are using ancient user-agents. But I doubt most of the people you want to catch are.
- derefr 3y ago
  I mean, when I'm talking about stealth techniques, I'm not talking about what I'm seeing in my own server logs, but rather about what I find built into various bits of, ahem, "anti-detect software for affiliate marketing" tech that I dredge up from fraud Telegram groups, carding marketplaces, etc. that the attackers I do catch seem to frequent. (Gotta stay five steps ahead!) I suppose there are verticals where the data is so valuable, and the garden around it so walled-in, that you could build a whole IT business with a custom scraping stack just around extracting that data to then resell it. (I presume that's the business you're in.) But for most verticals, the "attackers" you'll see in your logs aren't people building a data-broker business, and so aren't building their own secret-sauce anonymity from scratch; rather, they're end-users who want to do an end-run around your rate-limits, commit promotion fraud, etc., and so want to buy anonymity as a product, script-kiddie style. And "anonymity as a product", sold publicly (rather than through high-value contracts) tends to suck. It's script-kiddies buying from script-kiddies, with no real engineering in sight. > I work in this space and we just use fingerprints we collect from actual users over the previous month. Are you sure you're not in a citogenesis cycle? How sure are you that some of those "real users" aren't your peers' stealthed bots, who in turn picked up those fingerprints from unknowningly observing other stealthed bots in their logs, who...
  - runnerup 3y ago
    > Are you sure you're not in a citogenesis cycle? How sure are you that some of those "real users" aren't your peers' stealthed bots, who in turn picked up those fingerprints from unknowningly observing other stealthed bots in their logs, who... Doesn't matter, they work (at least when used in combination with high-quality proxy IP's). If they stopped working I'd do something else. We only apply hard science when absolutely needed, otherwise it's mostly wire and duct tape holding things together -- ruthless focus on creating business value. We definitely only sell this via high-value contracts, so you're probably mostly correct there. Though puppeteer-stealth deserves at least a quiet shout-out for not completely sucking. That said, we do pay attention to a lot of the research in the field, even if we only apply the absolute bare minimum needed to create business value. Eric Wustrow[0] at UC Boulder does really, really good work in an adjacent space, and we've found some his papers/software to be helpful, as well as those of some of the colleagues he works most closely with. I don't think he'd love our applications of his research, but our technological needs dovetail well with the needs of the anti-censorship research that he works on. If you were interested in the degree of "citogenesis", I think that's something that academic researchers like Wustrow et. al would be very well-positioned to investigate. Highly recommend any of their papers, they make front page of HN surprisingly often. 0: https://ericw.us/trow/ https://ericw.us/trow/