5 ms·

As someone with lots of anti-anti-botting knowledge - both are ineffective. Even if it's a "global rate limit" I'll find out the value (never ran into someone

As someone with lots of anti-anti-botting knowledge - both are ineffective. Even if it's a "global rate limit" I'll find out the value (never ran into someone randomizing it) and jump on the web request faster than anyone else RIGHT as it comes up. With CAPTCHAs I'll bypass with a solving service and/or computer vision if it's easy, or even just get past the noCAPTCHA solutions with primed browser instances from credible networks. But don't kid yourself - that would not solve botting at all.

carty76ers 7y ago
> But don't kid yourself - that would not solve botting at all. What would solve it? Or rather, what is the best defensive measure these days?
- folkhack 7y ago
  At this time - I honestly don't know. Even the reputation-based stuff is laughable and one can hide a Puppeteer instance with good originating networks, and spoofing a ton of details in the browser. Even if that's a no-go you can also automate plain-old-Chromium/Chrome with extensions and run it in a headless session through something like Xpra. I'm experimenting with Firefox solutions as well. All-in-all, I've never been stopped - and that's not me stroking my ego... there's TONS of resources out there for this stuff that are just a DuckDuckGo search away. The biggest thing is if they start aggressively fingerprinting bots, they're going to start blocking user real people. It's all based on a score - and getting a good score is just a matter of a credible proxy, CAPTCHA bypassing services, and making a browser look highly credible. --- For a "real" answer of some value - as a web developer myself, I'd try to make it as expensive as possible for them. Which specifically would be to implement a non-standard CAPTCHA solution and do rate/conversion-limiting per-network. The reason I didn't say this up-front is because it's not a solid solution - it's just increasing the barrier of difficulty and cost for those that are trying to automate around your solution.
svdr 7y ago
So what does help against bots?
- jsjw7sbw 7y ago
  Lottery like yosemite camp 4 has. I really enjoyed my experience with that. You could make some lotteries months or weeks ahead to match different needs.
- folkhack 7y ago
  Answered further up in the thread: https://news.ycombinator.com/item?id=21631112 https://news.ycombinator.com/item?id=21631112 It's a cat and mouse game for sure, but the answer is usually nothing if the person is sophisticated enough. You can only increase the difficulty/cost (covered in that comment).
gitgud 7y ago
> But don't kid yourself - that would not solve botting at all. Surely capatcha's and rate limiting raises the bar for people botting. It couldn't make it any worse right?
- CaptainMarvel 7y ago
  If it doesn’t make it any better, then captcha makes it works for real humans
  - chrisweekly 7y ago
    works -> worse
  - folkhack 7y ago
    Yep - and as someone who's ran a lot of conversion-based online solutions this is 100% true. Even when you account for automated sign-ups etc. the inclusion of a CAPTCHA will ding your rates.
- folkhack 7y ago
  It increases the barrier and cost for sure! But the thing is if you get someone who's even remotely sophisticated we can get past this sorta stuff in short order. For something that's highly desirable like tickets, Nike drops, or apparently campsites there are many people with sufficient ability to bypass this stuff.
jachee 7y ago
I like how the sibling comments all expect an abuser to reveal how to prevent their abuse. To paraphrase Sinclair: It is difficult to get a man to divulge how to prevent something, when his salary depends on his not preventing it.
- kortilla 7y ago
  That’s not a paraphrase of Sinclair. Sinclair’s saying is about people not understanding something that conflicts with their income. Refusing to give away a secret has absolutely no overlap with Sinclair’s saying.
  - jachee 7y ago
    I guess I should have said "adapt" or "inspired by" or "riffing on" or something to indicate to the literal-minded that I was merely copying Sinclair's phraseology, rather than his meaning. I thought "paraphrase" was sufficient, but I appear to have misjudged.
- folkhack 7y ago
  I'm an open book! Ask any question that you would like! Here's the thing - all of this info is out there (largely in other HN threads on this topic) and I'm nothing special in my field-of-knowledge ;) I'm confident if people fully prevented my "abuse" they'd start to block actual users... simple as that.
  - yetanotherjosh 7y ago
    What's your take on ML for bot classification? How successful has that family of strategies been in your opinion? One could speculate on what particular features of client behavior a model would hone in on to detect a bot, but it would actually likely be unexpected behavioral oddities not shared by legit clients that a human developer's intuition wouldn't think of.
    - folkhack 7y ago
nython 7y ago
> As someone with lots of anti-anti-botting knowledge Any recommendations for books/other information sources? Currently doing some backend work for a company that mainly does scraping and a lot of this seems to be based on the tribal knowledge of the resident old wise one.
- folkhack 7y ago
  I sorta disagree on this being tribal knowledge - a lot of this stuff is out there if you're willing to dig a bit. Tons of it comes down to network reputation and having a legitimate-looking bot. If you're scraping at scale it's an infrastructure problem just as much as it is a fingerprinting one. Stuff like https://news.ycombinator.com/item?id=20479015 https://news.ycombinator.com/item?id=20479015 is a goldmine for me and it usually is a fun weekend working around said methods in a lab-like environment. > resident old wise one He's just invested the time in picking this skillset up - it's definitely not just something you're an expert on after a few small projects. It takes years of having things break over, and over, and over [...] There's a small Discord server that I setup for people who do a lot of RPA/web scraping if you're interested in joining a "tribe". My contact info is accessible through my profile if you're interested =)
  - nython 7y ago
    re: tribal knowledge - didn't mean the field as a whole, just that bus factor is pretty low at the place I'm currently at.

carty76ers 7y ago

> But don't kid yourself - that would not solve botting at all. What would solve it? Or rather, what is the best defensive measure these days?

folkhack 7y ago
At this time - I honestly don't know. Even the reputation-based stuff is laughable and one can hide a Puppeteer instance with good originating networks, and spoofing a ton of details in the browser. Even if that's a no-go you can also automate plain-old-Chromium/Chrome with extensions and run it in a headless session through something like Xpra. I'm experimenting with Firefox solutions as well. All-in-all, I've never been stopped - and that's not me stroking my ego... there's TONS of resources out there for this stuff that are just a DuckDuckGo search away. The biggest thing is if they start aggressively fingerprinting bots, they're going to start blocking user real people. It's all based on a score - and getting a good score is just a matter of a credible proxy, CAPTCHA bypassing services, and making a browser look highly credible. --- For a "real" answer of some value - as a web developer myself, I'd try to make it as expensive as possible for them. Which specifically would be to implement a non-standard CAPTCHA solution and do rate/conversion-limiting per-network. The reason I didn't say this up-front is because it's not a solid solution - it's just increasing the barrier of difficulty and cost for those that are trying to automate around your solution.

svdr 7y ago

So what does help against bots?

jsjw7sbw 7y ago
Lottery like yosemite camp 4 has. I really enjoyed my experience with that. You could make some lotteries months or weeks ahead to match different needs.
folkhack 7y ago
Answered further up in the thread: https://news.ycombinator.com/item?id=21631112 https://news.ycombinator.com/item?id=21631112 It's a cat and mouse game for sure, but the answer is usually nothing if the person is sophisticated enough. You can only increase the difficulty/cost (covered in that comment).

gitgud 7y ago

> But don't kid yourself - that would not solve botting at all. Surely capatcha's and rate limiting raises the bar for people botting. It couldn't make it any worse right?

CaptainMarvel 7y ago
If it doesn’t make it any better, then captcha makes it works for real humans
- chrisweekly 7y ago
  works -> worse
- folkhack 7y ago
  Yep - and as someone who's ran a lot of conversion-based online solutions this is 100% true. Even when you account for automated sign-ups etc. the inclusion of a CAPTCHA will ding your rates.
folkhack 7y ago
It increases the barrier and cost for sure! But the thing is if you get someone who's even remotely sophisticated we can get past this sorta stuff in short order. For something that's highly desirable like tickets, Nike drops, or apparently campsites there are many people with sufficient ability to bypass this stuff.

jachee 7y ago

I like how the sibling comments all expect an abuser to reveal how to prevent their abuse. To paraphrase Sinclair: It is difficult to get a man to divulge how to prevent something, when his salary depends on his not preventing it.

kortilla 7y ago
That’s not a paraphrase of Sinclair. Sinclair’s saying is about people not understanding something that conflicts with their income. Refusing to give away a secret has absolutely no overlap with Sinclair’s saying.
- jachee 7y ago
  I guess I should have said "adapt" or "inspired by" or "riffing on" or something to indicate to the literal-minded that I was merely copying Sinclair's phraseology, rather than his meaning. I thought "paraphrase" was sufficient, but I appear to have misjudged.
folkhack 7y ago
I'm an open book! Ask any question that you would like! Here's the thing - all of this info is out there (largely in other HN threads on this topic) and I'm nothing special in my field-of-knowledge ;) I'm confident if people fully prevented my "abuse" they'd start to block actual users... simple as that.
- yetanotherjosh 7y ago
  What's your take on ML for bot classification? How successful has that family of strategies been in your opinion? One could speculate on what particular features of client behavior a model would hone in on to detect a bot, but it would actually likely be unexpected behavioral oddities not shared by legit clients that a human developer's intuition wouldn't think of.
  - folkhack 7y ago

nython 7y ago

> As someone with lots of anti-anti-botting knowledge Any recommendations for books/other information sources? Currently doing some backend work for a company that mainly does scraping and a lot of this seems to be based on the tribal knowledge of the resident old wise one.

folkhack 7y ago
I sorta disagree on this being tribal knowledge - a lot of this stuff is out there if you're willing to dig a bit. Tons of it comes down to network reputation and having a legitimate-looking bot. If you're scraping at scale it's an infrastructure problem just as much as it is a fingerprinting one. Stuff like https://news.ycombinator.com/item?id=20479015 https://news.ycombinator.com/item?id=20479015 is a goldmine for me and it usually is a fun weekend working around said methods in a lab-like environment. > resident old wise one He's just invested the time in picking this skillset up - it's definitely not just something you're an expert on after a few small projects. It takes years of having things break over, and over, and over [...] There's a small Discord server that I setup for people who do a lot of RPA/web scraping if you're interested in joining a "tribe". My contact info is accessible through my profile if you're interested =)
- nython 7y ago
  re: tribal knowledge - didn't mean the field as a whole, just that bus factor is pretty low at the place I'm currently at.