27 ms·
An Empirical Study and Evaluation of Modern CAPTCHAs
- mattmaroon 3y agoHow many years until sites actually remove CAPTCHAs though?
- intellectronica 3y agoYou see where this is heading: after superintelligence is achieved, CAPTCHAs will be designed to be questions that humans get wrong but AI has no problem with.
- mike-cardwell 3y agoA superintelligent AI would be able to imitate a human, getting the answers incorrect in exactly the way needed. However, I'm not entirely sure what kind of system a superintelligent AI would need to access which would be protected by a captcha.
- steve1977 3y agoAt least that would still be a proper CAPTCHA, in that it tells computers and humans apart…
- llamaInSouth 3y agoI already had issues with captchas (specially on tor).... so now its going to get worst?
- muzani 3y agoTor is practically unusable for me because it triggers so many captchas. I end up using Mullvad Browser, which is similar but byo VPN.
- shadowgovt 3y agoThe price we pay for obfuscating the trust signals on our connection is that our connection is untrusted. As an American, I have a similar experience when I travel across the Atlantic. It's always funny to me when I land in the UK, start using websites I use normally at home, and get cookie verification modals from hell to breakfast.
- maksimur 3y agoCan't vouch for other Europeans but I got used to them to the point my arm moves automatically where needed before clicking, even accounting for extra modals. I almost don't register them anymore.
- olliej 3y agoIt's really amazing when we still get those text ones and nowadays you can literally select the text in many of the images and copy/paste into the input field.
- yaomtc 3y agoI've never seen a text version that lets me select the text, that's bizarre
- _rutinerad 3y agoI’m assuming that he means that on (for example) Mac you can select text from any image and copy paste it. https://uk.pcmag.com/macos/138058/not-just-iphone-how-to-use-apples-live-text-feature-from-your-mac https://uk.pcmag.com/macos/138058/not-just-iphone-how-to-use...
- olliej 3y ago100% correct, I assumed windows also let people do that given text recognition is apparently trivial now (to the extent it's annoying - trying to drag images and get text selection instead is annoying :-/)
- serf 3y agoWindows has gone sort of the opposite way, copy/paste is now often hindered if the engine recognizes the string to be sensitive or otherwise un-wise to copy to your clipboard. I've had a few instances on Windows 11 and surrounding software where ctrl-C as well as the context menu entry for 'Copy' were greyed out for this reason when skimming through logfiles, presumably because there was something about the line that triggered the MS "that's a password!" regex; stuuuupid stuff.
- reddalo 3y agoI think Windows now does it as well, but of course (as all things Windows) it works only in very few apps (forget Win32 ones, for example).
- Proven 3y ago[dead]
- mdale 3y agoI think captchas disappear next year or so. Already was soft human determination.
- topspin 3y agoWhat replaces captchas? Are there any not excessively burdensome tests that a standard issue human can pass that a machine somehow cannot? I'm assuming the "find all the bicycles" tests are also obsolete.
- ReactiveJelly 3y agoSadly, probably something like TPMs or email logins (from a reputable email provider of course, one who requires SMS to sign up, from a reputable phone provider of course, one who doesn't offer free VoIP numbers and requires a credit card to sign up, from a reputable card brand of course, not a burner card)
- radium3d 3y agofrom a reputable card brand who doesn't allow usage of stolen cards? lol maybe the internet just implodes.
- nomel 3y agoSomething realtime, like video, is beyond most models at the moment. After that, realtime input, like little mini game you have to show proficiency at by scoring 5. I think the mini game approach could be fun. It could probably work for a year or two. :-\
- serf 3y agothe minigame thing has been defeated for a long time. it's trivial to solve when there are only so many subsets of a game, however randomized the starting states are. I guess there is a silver-lining in the premise of AI generated one-time-use games for that sake, but then there is a significant "can a human even do this?" problem to conquer at that point.. and worse the same AI tech is going to be established on the opposite side of the wall trying to defeat the thing. I think it'll all boil down to some sort of state-license fallback method like "please enter a CC or ID number to continue" -- which is ultimately a defeat of the user, unfortunately.
- anonzzzies 3y agoI guess validating a payment card is going to be the next step to sign up for whatever. Don’t allow pre paid BINs and let’s go. Gonna be pretty miserable, however someone needs to find something as I currently would rather pay 0.01$ instead of solving a captcha. Especially the select all the bicycles; it’s a waste of life.
- 2Gkashmiri 3y agolook up indian UPI. "validating payment card" and all that snazzy bits are error prone, old, archaic and cost a fortune to businesses. in upi system, you are presented with a QR code or you input your UPI ID, you click pay and it gets through. if you are worried about "fraud protection", why rely on an intermediary like ebay or credit card company and instead should take up with your bank or the seller or courts.
- EGreg 3y agoThere is literally nothing you can do to prevent bot accounts online now, other than requiring people to show up to events periodically. And even then, they can just use bots AFTER they’ve validated their accounts. The Internet will become a dark forest, and since that is where all of our communication and transactions happen of any significance, that’s pretty much game over for the significance of human activity. Think I am overstating the fact? It already happened with wall street trading. First, institutions prefer bots to human. Then, you will come to prefer bots to humans. Then every human will be surrounded with 999 bots and unable to change anything or appeal to any significant number of humans to change anything.
- shwouchk 3y agoPlease. Last time I had to solve a captcha it was wasted 15 minutes (not exaggerating!) of my life, clicking on an endless stream of bikes, motorcycles, buses and stoplights. As punishment for using a vpn.
- pasc1878 3y agoI've managed that without a VPN - although I do have poor sight. It also does not help that the shown busses, water hydrants, pavements look totally unfamiliar to me. (Why aren't captures taken from all over the world Indian busses would be fun - London ones would be too boring)
- visarga 3y agosimple - if the user solves is too well, reject
- muzani 3y agoval delay = 500 + Math.random() * 3000
- JumpCrisscross 3y agoSomeone will get rich turning this into a browser plug-in.
- LoganDark 3y agoYou can already buy captcha solves through browser plugins. The only difference is they currently use clickfarms full of underpaid workers from third-world countries
- RockRobotRock 3y agoThese sites like 2captcha and deathbycaptcha let anyone sign up to be a worker and start solving captchas for $$. If you can run AI that solves captcha just as well, you can literally print money.
- LoganDark 3y ago2captcha gives you $0.50 per "1-2 hours". Is that really worth all the work? deathbycaptcha does not let anyone simply sign up to work.
- RockRobotRock 3y ago> 2captcha gives you $0.50 per "1-2 hours". Is that really worth all the work? That's rough, but you could scale it up I guess. Didn't know that about dbc, thank you.
- beAbU 3y agoNope, the moment this becomes a viable solution then spammers will pick it up, making captcha useless amost overnight. Websites will very quickly pivot to alternative solutions like payment card verfification, etc.
- shadowgovt 3y agoOr to the next unsolved problem in machine learning. The whole point of ReCAPTCHA, at least, is to convert all this human labor into training data.
- k4rli 3y agoAre there any open local models for basic alphanumeric picture captchas to save on 2captcha?
- bawolff 3y agoA surprising number can be solved with teseract and simple preprocessing (e.g. thresholding, expand and contract lines). For more complex cases, not AI but consider the attack in https://www.usenix.org/system/files/conference/woot14/woot14-bursztein.pdf https://www.usenix.org/system/files/conference/woot14/woot14...
- hiAndrewQuinn 3y agoGood. Hardware authentication is where it's at.
- rgrieselhuber 3y agoLike the Clipper Chip?
- hooverd 3y agoOops! Looks like you're not using a government approved OS and browser.
- caymanjim 3y agoGoogle CAPTCHAs were designed and deployed as a mechanism to train AIs. That's why they are the way they are. Any security theater surrounding them is entirely incidental. So it's no surprise that the AIs are now good at solving them. We've trained them for years.
- leobg 3y agoI still find it funny that Google, with the advantage of having millions of Internet users train their AI like galley slaves for free, hasn’t yet been able to crack vision driven self driving. Tesla had no such advantage when training their FSD to recognize traffic lights, bicycles, motorcycles, etc.
- mike_d 3y ago> hasn’t yet been able to crack vision driven self driving But they have? For years Google Street view has read signs, house numbers, phone numbers of businesses, etc. from the environment. It is safe to assume they have this built into Waymo as well. I assume you might be trying to reference "vision only" self-driving, which is a fantasy made up by Elon Musk because nobody would sell him LiDAR sensors cheaply. https://www.thedrive.com/tech/43779/this-tesla-model-y-dummy-crash-shows-exactly-why-lidar-matters https://www.thedrive.com/tech/43779/this-tesla-model-y-dummy...
- leobg 3y agoThis is a meme. “Sour grape Elon, touting vision because no one will sell him LiDAR sensors. Which are the gold standard sensors that solve self driving.” How exactly does LiDAR tell you whether the thing in question can move (dog) or not (trash can)? How does it allow a neural net to infer intent? You’ll actually have to solve vision. Even if you had LiDAR. There’s no way around it. And once you’ve solved it, LiDAR becomes superfluous. Chesterton’s fence.
- mike_d 3y ago> How exactly does LiDAR tell you whether the thing in question can move LIDAR is continuously scanning, usually multiple times a second. It is irrelevant if the object is a trash can or a dog if it has a trajectory into the street. > And once you’ve solved it, LiDAR becomes superfluous. Only if you have the low Musk level standards of simply being equal to a human. There are plenty of jobs robots can do better than humans and driving is one of them. But it does require LIDAR and/or radar. https://abc7news.com/tesla-s-autopilot-self-driving-car-officials-investigating-teslas-autopilot-feature-after-fatal-crash/1410042/ https://abc7news.com/tesla-s-autopilot-self-driving-car-offi...
- hknmtt 3y agono need for captchas, just implement throttling per ip. like bcrypt dues for passwords. if a bot fills up a form(or whatever), so be it, but it won't be able to do it for another N seconds or minutes..so the problem then is lowered from per try, which can be thousands of submissions, all the way down to per period and per ip.
- ArtTimeInvestor 3y agoThe solution could be a cryptocurrency which can be mined in the browser. Hashcash, which was one of the inspirations for Bitcoin, was initially invented to prevent email spam. Consumer devices have a lot of spare CPU and RAM. So a proof-of-work algorithm which consumes those resources for a minute might work? If it generates $0.01 for the website owner in that minute, maybe that would work?
- insanitybit 3y agoWouldn't any proof of work be just as easy for a computer to achieve as a human?
- trompetenaccoun 3y ago'Proof of Work' as it's generally understood is done by computers only. But I guess I understand what you're asking and the answer is yes, that is a problem. For Sybil resistance it's better to know if someone is a unique human, not if they're a machine that has paid the toll: https://en.wikipedia.org/wiki/Proof_of_personhood https://en.wikipedia.org/wiki/Proof_of_personhood There are exotic solutions like the 'Idena Network'. But sadly I have to admit the best solution I've seen so far is Sam Altman's Worldcoin. Not that I'm a fan, I still hope we can find something better than scanning everyone's eyeball.
- CaptainFever 3y agoYes, but it's more of an anti spam measure.
- SturgeonsLaw 3y agoYeah but it would cost spammers who want to impersonate a large number of humans at once
- swinglock 3y agoProof of work can already be implemented without a token. Tor has such a feature for denial of service protection. https://blog.torproject.org/introducing-proof-of-work-defense-for-onion-services/ https://blog.torproject.org/introducing-proof-of-work-defens... A benefit of a token is you can recycle previous proof of work by using a small amount of Bitcoin, which could be transferred using Lightning. The value could also be transferred back some amount of time after registration given no bad behavior, allowing for larger sums than a cent, which could provide better protection.
- peter_retief 3y agoIs this really so surprising? Probably a better captcha would be sign of life not puzzles.
- bamboozled 3y agoGoogle created this problem, let’s see them solve it.
- gary_0 3y agoDoes HN ever require CAPTCHAs? It seems to do pretty well with its basic but battle-tested moderation/antispam tools, and rate-limiting that seems to repel all but the most concerted DDoS attacks. I don't think HN has any unreasonable restrictions on scraping or third-party clients, either. And it manages to serve 5M unique visitors a month and 10M views a day[0]. [0] https://news.ycombinator.com/item?id=33454140 https://news.ycombinator.com/item?id=33454140
- jamiek88 3y agoOn one machine! :)
- nextaccountic 3y agoOn one thread even Pretty sure it's an AST interpreter too (metacircular eval - apply, as in SICP)
- bongobingo1 3y agoI cant tell if the audience of HN are more likely to script something untoward against HN, be that DDOS or just "check out my product" spam, because its a bunch of hackers - or less likely to do it because (maybe) we like having nice things, or figure the audience is too in the know to fall for boring crypto spam.
- SXX 3y agoHN audience is rich enough to just pay $10 for 1000 solved CAPTCHAs of any complexity since those services are human powered.
- ShamelessC 3y agoThey go down somewhat frequently. I think it’s like four 9’s? I’m not sure why they insist on running just a few machines though. They have more than enough money and probably make up the difference by the advertising for YC that they get.
- urig 3y agoHow did the OP get from the article linked to the title of this post?
- rezonant 3y agoI'm also wondering this. I don't think it has anything to do with AI solves.
- tgv 3y agoSection 5.5: "Table 3 contrasts our measured human solving times and accuracy against those of automated bots reported in the literature." Although it's not clear to me that the humans all really were humans.
- tunnuz 3y agoWhat a surprise since CAPTCHAs were created to gather data to train AIs
- ixmerof 3y agoHave they tried with puzzles used by Rockstar Games or HBO Max to reset a password? They are impossible to solve, asking to solve 17 questions and more and still failing you to retry with higher count. Even the audio version is quite innovative
- rezonant 3y agoAs best as I can tell this study explores many facets of how humans solve captchas. I couldn't find anything about AIs outperforming humans in the study. Can someone give me a section reference? Solving reCAPTCHA v2/v3 requires more than just clicking the box and an image puzzle. If that was all it was we would be overrun by now. Lots of folks commenting that the title's statement makes sense because CAPTCHAs are meant to train AIs. While this is broadly true, that's a nice side effect. The way modern CAPTCHAs like reCaptcha V2+ work, is they monitor behavioral analytics-- from things like your browsing history to how your mouse moves on the page. This is why most of the time, most people only need to click a box. I'm not sure there's a LMM out there that includes mouse movement as a modality. The kinds of AIs that are designed to beat CAPTCHAs also don't have the data from Google et al to use to train, unless we're concerned Google is training it's own bots to bypass CAPTCHAs, I suppose it's not inconceivable?
- mherrmann 3y agoIt's in Table 3.
- rezonant 3y agoThank you. The data in that table (for reCAPTCHA, citation 63) is from another paper from 2016 which is focused on solving the actual user-presented problems. It doesn't (directly at least) say they achieved a reliable automation of captcha acceptance, though. https://ieeexplore.ieee.org/document/7467367 https://ieeexplore.ieee.org/document/7467367 From the abstract: > Through extensive experimentation, we identify flaws that allow adversaries to effortlessly influence the risk analysis, bypass restrictions, and deploy large-scale attacks. Subsequently, we design a novel low-cost attack that leverages deep learning technologies for the semantic annotation of images. I'd suspect reCaptcha has been updated in the 7 years since to address shortcomings. Another entry in the table (citation 45) is from 2020 and talks about using an object detection AI to solve the image tests. This again looks like it's focused on the task, not the primary mechanism (behavioral analytics).
- croemer 3y agoYeah, the study is really not about AI solving captchas but how humans solve them. Quite a clickbait title - but those do well on HN unfortunately.
- codedrivendev 3y agoI think I prefer the recent CAPTCHAs (where you solve a puzzle by rotating an item, or finding the matching item). The older ones from years ago (deciphering mangled text and trying to work out if it is an `i`, `1` or `l` were more annoying)
- dasrecht 3y agoSo we now proof that we're human by failing those tests?
- jon_richards 3y agoThe problem with designing a bear proof trash can is that there’s significant overlap between the smartest bears and the dumbest tourists.
- croes 3y agoSo if you fail, you are human.
- steve1977 3y agoJust as Turing intended?
- sebtron 3y agoGreat news, can we please get rid of CAPTCHAs now?
- kromem 3y agoNo, we'll still have them, but now sites will only allow you in if you kind of suck at them.
- rightbyte 3y agoThat is already the case. On some questions you can't answer the correct answer, but have to guess what most other would answer.
- kromem 3y agoIt's been particularly frustrating with the picture ones broken up into squares. I tend to be careful to select any square that contains any of what's being asked, but I clearly must be the minority as it always fails unless I select the minimum number of valid squares and ignore the slight overlap on surrounding ones.
- baol 3y agoThe main road to tell computers and humans apart will soon be that computers are a bit too fast and accurate to be humans.
- tomschwiha 3y agoWe could simply reverse captchas now: if the captcha is solved its a roboter, otherwise its a human.
- bamboozled 3y agoWe can’t program a bot to fail ?
- amelius 3y agoYes, we use Copilot for that.
- jonplackett 3y agoI find Amazon’s captchas so hard now! I have to do the audio one
- mherrmann 3y agoThe relevant data for the claim of the headline is in Table 3. On all the tasks with enough data, bots were both faster and more accurate than humans.
- renonce 3y agoYeah, the claim of the headline comes from the first sentence in Section 5.5. I think either the title should match the paper's title or that should be pointed out as part of the submission - not sure how HN's title guidelines work.
- alexnewman 3y agoAll of these papers miss that captchas have multiple levels of difficultly. People who get an enterprise account or work closely with the captcha providers will find very different results. Many captcha providers now decide what captchas to send out, in hard mode based on what LLMs cannot solve Captchas are purposely not made too hard as people like pex.com need to be able to bypass them for copyright enforcement. Note I’m biased as I was a founder of hcaptcha
- croemer 3y agoBot operators can already pay human captcha solvers as the paper mentions. So all this does is potentially replace those humans with AI, driving down prices for bot operators. As prices for bot operators decrease, website operators will increase the challenge and drive up effort for the intended website audience (humans) who are solving captchas instead of paying bots. In the end, the website operators will have to stop using captchas as the intended website audience will no longer be willing to solve harder captchas. Website operators can use alternatives, like asking for micro-payments, high enough to discourage most bot operators.
- tarruda 3y ago> Website operators can use alternatives, like asking for micro-payments Similarly to how dApps work in ethereum-like blockchains?
- drexlspivey 3y agoMicropayments is not possible when stripe/visa/paypal charge a 30 cents minimum fee
- kuon 3y agoI hate captcha, it takes ages and I often fail the google one. I would happily pay to have them removed from my browsing. I don't use AliExpress often, but now I can't as the captcha just plain doesn't work.
- NotSammyHagar 3y agoI find captchas extremely painful, because of ambiguity and not loading all the pictures. I wait for a minute and some never show. When they do load, so manyare pics of bicycles and motorcycles and cross walks. Are you supposed to click on the tiny piece that goes tojust past another tile or not? You can't refresh one that doesn't load, I think most of them start over if you refresh. Like other people reported, if you ever use tor, it's very common for the captchas to just not load. They just kind of hang without showing the pictures. Regular websites generally just work fine on tor, it seems to be a captcha problem.
- xlbuttplug2 3y ago> Are you supposed to click on the tiny piece that goes tojust past another tile or not? I ask myself this every time.
- dgellow 3y agoPretty sure the hesitation is what makes us humans :)
- DeathArrow 3y agoGreat, so maybe we would find a less annoying bot detection technique than captchas.
- PicassoCTs 3y agowell captchas are not there to keep bots out, they are free click work for google?
- PointThink 3y agoMandatory xkcd https://xkcd.com/810/ https://xkcd.com/810/
- anonym0us88 3y ago[dead]
- lapcat 3y agoI predicted this 7 years ago: "How will the machines take over? When CAPTCHAs become so hard that only AI can solve them, humans will be completely locked out of the net." https://twitter.com/lapcatsoftware/status/771857826130034688 https://twitter.com/lapcatsoftware/status/771857826130034688
- armchairhacker 3y agoI thought this was already happening ~7 years ago. The "what text is in this image captchas" got a lot less common a while ago, and I think this was partly the reason why.
- Zopieux 3y agoCan the mods please fix the completely wrong and click bait title? Zero AI mentioned there.
- wslh 3y agoIt does not suprise me since lately I have a lot of mistakes with CAPTCHAs. Mainly the ones with characters with different colors, superposed, and rotated. I think there are some that we as humans just guess because the final image is not clear enough. I think in the same way AI can beat us recognizing unfocused photos.
- jbd0 3y agoI have been locked out of websites for solving a captcha so quickly that it thought I was a bot. So we went from requiring humans to solve a puzzle that bots can't to now requiring that humans solve the puzzle slower than bots do.
- Aissen 3y agoThe most funny thing about this limit is that it's self-reinforcing. Bots will learn to sleep() and wiggle the mouse. Humans will learn to wait. Everyone will be worse off.
- RockofStrength 3y agoCAPTCHAs are used as a literal Turing test; that's their whole purpose. From the get-go their usefulness window had a looming expiration date.
- YeGoblynQueenne 3y agoMisleadingly editorialised title. Actual title and abstract (which doesn't say anything about AIs "now" outperforming humans): An Empirical Study & Evaluation of Modern CAPTCHAs * For nearly two decades, CAPTCHAs have been widely used as a means of protection against bots. Throughout the years, as their use grew, techniques to defeat or bypass CAPTCHAs have continued to improve. Meanwhile, CAPTCHAs have also evolved in terms of sophistication and diversity, becoming increasingly difficult to solve for both bots (machines) and humans. Given this long-standing and still-ongoing arms race, it is critical to investigate how long it takes legitimate users to solve modern CAPTCHAs, and how they are perceived by those users.* * In this work, we explore CAPTCHAs in the wild by evaluating users' solving performance and perceptions of unmodified currently-deployed CAPTCHAs. We obtain this data through manual inspection of popular websites and user studies in which 1,400 participants collectively solved 14,000 CAPTCHAs. Results show significant differences between the most popular types of CAPTCHAs: surprisingly, solving time and user perception are not always correlated. We performed a comparative study to investigate the effect of experimental context -- specifically the difference between solving CAPTCHAs directly versus solving them as part of a more natural task, such as account creation. Whilst there were several potential confounding factors, our results show that experimental context could have an impact on this task, and must be taken into account in future CAPTCHA studies. Finally, we investigate CAPTCHA-induced user task abandonment by analyzing participants who start and do not complete the task.* @dang, could you please correct the title? Thanks.
- tim333 3y agoCoincidentally Worldcoin is up 30% today. Maybe cryptographic/biometric proof of being a human will be useful after all?
- notnullorvoid 3y agoNo thanks, I'd rather not live in a dystopian nightmare where Sam Altman is in control of assigning proof of humanity. Worldcoin will undoubtedly end up assigning identities to AIs for profit anyway, and/or there will be swaths of identities being sold on the black market.
- tim333 3y agoWell - there doesn't have to be a Sam + Worldcoin monopoly on these things. Anyone else could launch a similar proof / ID system and websites and the like could accept any they feel like accepting.
- barbazoo 3y agoI wish captcha providers universally had to provide a way to shut down their use by bad actors. Here in Canada I get tons of scam texts pointing me to a fake banking or postal service website asking me to pay a fake bill. I want to ddos them with fake payment data but they’re all protected by hcaptcha.
- deleted 3y ago[deleted]
- kderbyma 3y agoI don't know....lately I just don't even try and purposefully make mistakes on it by leaving out one or two just to fuck with the captcha
- Duanemclemore 3y ago"and that's the story of the invention of the Voight-Kampff Test, kids!"
- fnordpiglet 3y agoMy wife is entirely unable to solve a captcha. Her solution to any captcha is to get me to do it for her while she loudly swears at the creators. I welcome being able to outsource this task to AI.
- dang 3y agoSubmitted title was "AI bots are now outperforming humans in solving CAPTCHAs", which broke HN's title rule: "Please use the original title, unless it is misleading or linkbait; don't editorialize." Submitters: If you want to say what you think is important about an article, that's fine, but do it by adding a comment to the thread. Then your view will be on a level playing field with everyone else's: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&sort=byDate&type=comment&query=%22level%20playing%20field%22%20by:dang https://hn.algolia.com/?dateRange=all&page=0&prefix=false&so...
- Geee 3y agoMy pet theory is that our whole simulated world is actually a huge captcha. Captchas keep evolving until you have to live an entire lifetime as a human to prove that you're a human. When you die you wake up and get access to a website.
- Izkata 3y agoSo that's what 42 was for!