6 ms·

Not reCAPTCHA, please. It’s horrendous in any environment that blocks any real amount of tracking. Rate limiting would solve the bot issue without inconvenienci

Not reCAPTCHA, please. It’s horrendous in any environment that blocks any real amount of tracking. Rate limiting would solve the bot issue without inconveniencing regular users.

tjohns 7y ago
Rate limiting would not prevent individuals running their own bots, or a service that uses a different IP for each request.
- nedwin 7y ago
  Which is exactly what's happening.
folkhack 7y ago
As someone with lots of anti-anti-botting knowledge - both are ineffective. Even if it's a "global rate limit" I'll find out the value (never ran into someone randomizing it) and jump on the web request faster than anyone else RIGHT as it comes up. With CAPTCHAs I'll bypass with a solving service and/or computer vision if it's easy, or even just get past the noCAPTCHA solutions with primed browser instances from credible networks. But don't kid yourself - that would not solve botting at all.
- carty76ers 7y ago
  > But don't kid yourself - that would not solve botting at all. What would solve it? Or rather, what is the best defensive measure these days?
  - folkhack 7y ago
    At this time - I honestly don't know. Even the reputation-based stuff is laughable and one can hide a Puppeteer instance with good originating networks, and spoofing a ton of details in the browser. Even if that's a no-go you can also automate plain-old-Chromium/Chrome with extensions and run it in a headless session through something like Xpra. I'm experimenting with Firefox solutions as well. All-in-all, I've never been stopped - and that's not me stroking my ego... there's TONS of resources out there for this stuff that are just a DuckDuckGo search away. The biggest thing is if they start aggressively fingerprinting bots, they're going to start blocking user real people. It's all based on a score - and getting a good score is just a matter of a credible proxy, CAPTCHA bypassing services, and making a browser look highly credible. --- For a "real" answer of some value - as a web developer myself, I'd try to make it as expensive as possible for them. Which specifically would be to implement a non-standard CAPTCHA solution and do rate/conversion-limiting per-network. The reason I didn't say this up-front is because it's not a solid solution - it's just increasing the barrier of difficulty and cost for those that are trying to automate around your solution.
- svdr 7y ago
  So what does help against bots?
  - jsjw7sbw 7y ago
    Lottery like yosemite camp 4 has. I really enjoyed my experience with that. You could make some lotteries months or weeks ahead to match different needs.
  - folkhack 7y ago
    Answered further up in the thread: https://news.ycombinator.com/item?id=21631112 https://news.ycombinator.com/item?id=21631112 It's a cat and mouse game for sure, but the answer is usually nothing if the person is sophisticated enough. You can only increase the difficulty/cost (covered in that comment).
- gitgud 7y ago
  > But don't kid yourself - that would not solve botting at all. Surely capatcha's and rate limiting raises the bar for people botting. It couldn't make it any worse right?
  - CaptainMarvel 7y ago
    If it doesn’t make it any better, then captcha makes it works for real humans
    - chrisweekly 7y ago
      works -> worse
    - folkhack 7y ago
      Yep - and as someone who's ran a lot of conversion-based online solutions this is 100% true. Even when you account for automated sign-ups etc. the inclusion of a CAPTCHA will ding your rates.
  - folkhack 7y ago
    It increases the barrier and cost for sure! But the thing is if you get someone who's even remotely sophisticated we can get past this sorta stuff in short order. For something that's highly desirable like tickets, Nike drops, or apparently campsites there are many people with sufficient ability to bypass this stuff.
- jachee 7y ago
  I like how the sibling comments all expect an abuser to reveal how to prevent their abuse. To paraphrase Sinclair: It is difficult to get a man to divulge how to prevent something, when his salary depends on his not preventing it.
  - kortilla 7y ago
    That’s not a paraphrase of Sinclair. Sinclair’s saying is about people not understanding something that conflicts with their income. Refusing to give away a secret has absolutely no overlap with Sinclair’s saying.
    - jachee 7y ago
      I guess I should have said "adapt" or "inspired by" or "riffing on" or something to indicate to the literal-minded that I was merely copying Sinclair's phraseology, rather than his meaning. I thought "paraphrase" was sufficient, but I appear to have misjudged.
  - folkhack 7y ago
    I'm an open book! Ask any question that you would like! Here's the thing - all of this info is out there (largely in other HN threads on this topic) and I'm nothing special in my field-of-knowledge ;) I'm confident if people fully prevented my "abuse" they'd start to block actual users... simple as that.
    - yetanotherjosh 7y ago
      What's your take on ML for bot classification? How successful has that family of strategies been in your opinion? One could speculate on what particular features of client behavior a model would hone in on to detect a bot, but it would actually likely be unexpected behavioral oddities not shared by legit clients that a human developer's intuition wouldn't think of.
      - folkhack 7y ago
- nython 7y ago
  > As someone with lots of anti-anti-botting knowledge Any recommendations for books/other information sources? Currently doing some backend work for a company that mainly does scraping and a lot of this seems to be based on the tribal knowledge of the resident old wise one.
  - folkhack 7y ago
    I sorta disagree on this being tribal knowledge - a lot of this stuff is out there if you're willing to dig a bit. Tons of it comes down to network reputation and having a legitimate-looking bot. If you're scraping at scale it's an infrastructure problem just as much as it is a fingerprinting one. Stuff like https://news.ycombinator.com/item?id=20479015 https://news.ycombinator.com/item?id=20479015 is a goldmine for me and it usually is a fun weekend working around said methods in a lab-like environment. > resident old wise one He's just invested the time in picking this skillset up - it's definitely not just something you're an expert on after a few small projects. It takes years of having things break over, and over, and over [...] There's a small Discord server that I setup for people who do a lot of RPA/web scraping if you're interested in joining a "tribe". My contact info is accessible through my profile if you're interested =)
    - nython 7y ago
      re: tribal knowledge - didn't mean the field as a whole, just that bus factor is pretty low at the place I'm currently at.
big_chungus 7y ago
The reason I hate recaptcha: it's not just a rate-limit for one site, but for the entire web. Run a google dork? You'll have to find three more fire hydrants the next time you're signing up for a service. Install an ad blocker? Six more fire hydrants. Log out of google? Twelve more. Heaven forbid if I _actually_ scrape something. Hitting one site harder than an average user shouldn't force me to fill out captchas on every site until my "reputation score" or whatever is back up. Feels like social credit for web sites.
- xur17 7y ago
  Yeah, I'm getting incredibly frustrated with it as well. I block tracking on the web, so I'm constantly doing multiple pages of free identification for Google. If I were building a bot for a site that uses captchas, I'd just use one of the many paid services that solve captchas. Last I checked, it was ~$0.003 per solve, hardly a blocker. To be honest, I wish I could easily use one of these solvers in my normal web browser...
  - Semaphor 7y ago
    As someone who actually needs/wants to visit about 50% of the recaptcha-using sites: Do the audio challenge. Type something close, the audio challenge barely cares what you type, it has fewer repetitions than the image challenge and is just generally easier.
    - rbritton 7y ago
      I've increasingly had reCAPTCHA refuse to give me an audio challenge.
- mortdeus 7y ago
  I was having fun with recaptha the other day on Dan Harmon's (of rick and morty fame) site called Channel 101. His sign up page was all sarcastic and the recaptcha test asked { Image of distorted random letters and numbers } "Are you human?" [text box] Prove it... And I kept trying answers like. "Okay blood's drawn, where do i send the sample?" "uhhh so I'm blind..." (no voice alt test) "a robot testing a human for humanity? (this is going to go nowhere good quick...) "Morty, its your grandpa Rick! Let me in Morty. belch" "I identify as a pro exclusionary Cylon. My prefered pronouns are su, pid 1, root, or if you prefer just refer to me as Zuck" (okay i didn't get to enter that last one in. too long) "Isn't putting random numbers and letters the kind of answer a ROBOT would make. It's only an answer written in THEIR language! o.O Unfortunately Mr. Harmon's test had a fatal bug or something because I never could get past the sign up page. :'( Or maybe I'm not human or i'm just missing something? idk.
dang 7y ago
We detached this subthread from https://news.ycombinator.com/item?id=21625449 https://news.ycombinator.com/item?id=21625449.

Not reCAPTCHA, please. It’s horrendous in any environment that blocks any real amount of tracking. Rate limiting would solve the bot issue without inconveniencing regular users.

tjohns 7y ago

Rate limiting would not prevent individuals running their own bots, or a service that uses a different IP for each request.

nedwin 7y ago
Which is exactly what's happening.

folkhack 7y ago

As someone with lots of anti-anti-botting knowledge - both are ineffective. Even if it's a "global rate limit" I'll find out the value (never ran into someone randomizing it) and jump on the web request faster than anyone else RIGHT as it comes up. With CAPTCHAs I'll bypass with a solving service and/or computer vision if it's easy, or even just get past the noCAPTCHA solutions with primed browser instances from credible networks. But don't kid yourself - that would not solve botting at all.

carty76ers 7y ago
> But don't kid yourself - that would not solve botting at all. What would solve it? Or rather, what is the best defensive measure these days?
- folkhack 7y ago
  At this time - I honestly don't know. Even the reputation-based stuff is laughable and one can hide a Puppeteer instance with good originating networks, and spoofing a ton of details in the browser. Even if that's a no-go you can also automate plain-old-Chromium/Chrome with extensions and run it in a headless session through something like Xpra. I'm experimenting with Firefox solutions as well. All-in-all, I've never been stopped - and that's not me stroking my ego... there's TONS of resources out there for this stuff that are just a DuckDuckGo search away. The biggest thing is if they start aggressively fingerprinting bots, they're going to start blocking user real people. It's all based on a score - and getting a good score is just a matter of a credible proxy, CAPTCHA bypassing services, and making a browser look highly credible. --- For a "real" answer of some value - as a web developer myself, I'd try to make it as expensive as possible for them. Which specifically would be to implement a non-standard CAPTCHA solution and do rate/conversion-limiting per-network. The reason I didn't say this up-front is because it's not a solid solution - it's just increasing the barrier of difficulty and cost for those that are trying to automate around your solution.
svdr 7y ago
So what does help against bots?
- jsjw7sbw 7y ago
  Lottery like yosemite camp 4 has. I really enjoyed my experience with that. You could make some lotteries months or weeks ahead to match different needs.
- folkhack 7y ago
  Answered further up in the thread: https://news.ycombinator.com/item?id=21631112 https://news.ycombinator.com/item?id=21631112 It's a cat and mouse game for sure, but the answer is usually nothing if the person is sophisticated enough. You can only increase the difficulty/cost (covered in that comment).
gitgud 7y ago
> But don't kid yourself - that would not solve botting at all. Surely capatcha's and rate limiting raises the bar for people botting. It couldn't make it any worse right?
- CaptainMarvel 7y ago
  If it doesn’t make it any better, then captcha makes it works for real humans
  - chrisweekly 7y ago
    works -> worse
  - folkhack 7y ago
    Yep - and as someone who's ran a lot of conversion-based online solutions this is 100% true. Even when you account for automated sign-ups etc. the inclusion of a CAPTCHA will ding your rates.
- folkhack 7y ago
  It increases the barrier and cost for sure! But the thing is if you get someone who's even remotely sophisticated we can get past this sorta stuff in short order. For something that's highly desirable like tickets, Nike drops, or apparently campsites there are many people with sufficient ability to bypass this stuff.
jachee 7y ago
I like how the sibling comments all expect an abuser to reveal how to prevent their abuse. To paraphrase Sinclair: It is difficult to get a man to divulge how to prevent something, when his salary depends on his not preventing it.
- kortilla 7y ago
  That’s not a paraphrase of Sinclair. Sinclair’s saying is about people not understanding something that conflicts with their income. Refusing to give away a secret has absolutely no overlap with Sinclair’s saying.
  - jachee 7y ago
    I guess I should have said "adapt" or "inspired by" or "riffing on" or something to indicate to the literal-minded that I was merely copying Sinclair's phraseology, rather than his meaning. I thought "paraphrase" was sufficient, but I appear to have misjudged.
- folkhack 7y ago
  I'm an open book! Ask any question that you would like! Here's the thing - all of this info is out there (largely in other HN threads on this topic) and I'm nothing special in my field-of-knowledge ;) I'm confident if people fully prevented my "abuse" they'd start to block actual users... simple as that.
  - yetanotherjosh 7y ago
    What's your take on ML for bot classification? How successful has that family of strategies been in your opinion? One could speculate on what particular features of client behavior a model would hone in on to detect a bot, but it would actually likely be unexpected behavioral oddities not shared by legit clients that a human developer's intuition wouldn't think of.
    - folkhack 7y ago
nython 7y ago
> As someone with lots of anti-anti-botting knowledge Any recommendations for books/other information sources? Currently doing some backend work for a company that mainly does scraping and a lot of this seems to be based on the tribal knowledge of the resident old wise one.
- folkhack 7y ago
  I sorta disagree on this being tribal knowledge - a lot of this stuff is out there if you're willing to dig a bit. Tons of it comes down to network reputation and having a legitimate-looking bot. If you're scraping at scale it's an infrastructure problem just as much as it is a fingerprinting one. Stuff like https://news.ycombinator.com/item?id=20479015 https://news.ycombinator.com/item?id=20479015 is a goldmine for me and it usually is a fun weekend working around said methods in a lab-like environment. > resident old wise one He's just invested the time in picking this skillset up - it's definitely not just something you're an expert on after a few small projects. It takes years of having things break over, and over, and over [...] There's a small Discord server that I setup for people who do a lot of RPA/web scraping if you're interested in joining a "tribe". My contact info is accessible through my profile if you're interested =)
  - nython 7y ago
    re: tribal knowledge - didn't mean the field as a whole, just that bus factor is pretty low at the place I'm currently at.

big_chungus 7y ago

The reason I hate recaptcha: it's not just a rate-limit for one site, but for the entire web. Run a google dork? You'll have to find three more fire hydrants the next time you're signing up for a service. Install an ad blocker? Six more fire hydrants. Log out of google? Twelve more. Heaven forbid if I _actually_ scrape something. Hitting one site harder than an average user shouldn't force me to fill out captchas on every site until my "reputation score" or whatever is back up. Feels like social credit for web sites.

xur17 7y ago
Yeah, I'm getting incredibly frustrated with it as well. I block tracking on the web, so I'm constantly doing multiple pages of free identification for Google. If I were building a bot for a site that uses captchas, I'd just use one of the many paid services that solve captchas. Last I checked, it was ~$0.003 per solve, hardly a blocker. To be honest, I wish I could easily use one of these solvers in my normal web browser...
- Semaphor 7y ago
  As someone who actually needs/wants to visit about 50% of the recaptcha-using sites: Do the audio challenge. Type something close, the audio challenge barely cares what you type, it has fewer repetitions than the image challenge and is just generally easier.
  - rbritton 7y ago
    I've increasingly had reCAPTCHA refuse to give me an audio challenge.
mortdeus 7y ago
I was having fun with recaptha the other day on Dan Harmon's (of rick and morty fame) site called Channel 101. His sign up page was all sarcastic and the recaptcha test asked { Image of distorted random letters and numbers } "Are you human?" [text box] Prove it... And I kept trying answers like. "Okay blood's drawn, where do i send the sample?" "uhhh so I'm blind..." (no voice alt test) "a robot testing a human for humanity? (this is going to go nowhere good quick...) "Morty, its your grandpa Rick! Let me in Morty. belch" "I identify as a pro exclusionary Cylon. My prefered pronouns are su, pid 1, root, or if you prefer just refer to me as Zuck" (okay i didn't get to enter that last one in. too long) "Isn't putting random numbers and letters the kind of answer a ROBOT would make. It's only an answer written in THEIR language! o.O Unfortunately Mr. Harmon's test had a fatal bug or something because I never could get past the sign up page. :'( Or maybe I'm not human or i'm just missing something? idk.

dang 7y ago

We detached this subthread from https://news.ycombinator.com/item?id=21625449 https://news.ycombinator.com/item?id=21625449.