6 ms·
Benchmarking leading AI agents against Google reCAPTCHA v2
- cedws 11mo agoWould performance improve if the tiles were stitched together and fed to a vision model, and then tiles are selected based on a bounding box?
- mdahardy 11mo agoThat's a cool idea. I bet it would work better.
- PaulHoule 11mo agoI know people were solving CAPTCHAS with neural nets (with PHP no less!) back in 2009.
- golfer 11mo agoIndeed, captcha vs captcha bot solvers has been an ongoing war for a long time. Considering all the cybercrime and ubiquitous online fraud today, it's pretty impressive that captchas have held the line as long as they have.
- mdahardy 11mo agoYou could definitely do better than we do here - this was just a test of how well these general-purpose systems are out-of-the-box
- xnx 11mo agoSeems like Google Gemini is tied for the best and is the cheapest way to solve Google's reCAPTCHA. Will be interesting to see how Gemini 3 does later this year.
- bena 11mo agoMakes sense, what do you think it was trained on?
- mdahardy 11mo agoAfter watching hundreds of these runs, Gemini was by far the least frustrating model to observe.
- dgacmu 11mo agoIn my admittedly limited-domain tests, Gemini did _far_ better at image recognition tasks than any of the other models. (This was about 9 months ago, though, so who knows what the current state of things). Google has one of the best internal labeled image datasets, if not the best, and I suspect this is all related.
- Xenoamorphous 11mo agoI’m sure they do better than me. Sometimes I get stuck on an endless loop of buses and fire hydrants. Also, when they ask you to identify traffic lights, do you select the post? And when it’s motor/bycicles, do you select the guy riding it?
- datadrivenangel 11mo agoThat's not due to accuracy, you're getting tarpitted for not looking human enough.
- sixhobbits 11mo agoDidn't look a lot into this but I think the fact that humans are willing to do this in the "cents per thousand" or something range means that it's really hard to get much interest in automating it
- Semaphor 11mo agoThere's a browser extension to solve them. Buster.
- mdahardy 11mo agoWhile running this I looked at hundreds and hundreds of captchas. And I still get rejected on like 20% of them when I do them. I truly don't understand their algorithm lol
- Sayrus 11mo agoTesting those same captcha on Google Chrome improved my accuracy by at least an order of magnitude. Either that or it was never about the buses and fire hydrants.
- ACCount37 11mo agoIt's a known "issue" of reCaptcha, and many other systems like it. If it thinks you're a bot, it will "fail" the first few correct solves before it lets you through. The worst offenders will just loop you forever, no matter how many solves you get right.
- guluarte 11mo agoin other words reasoning call fill the context window with crap
- flakiness 11mo agoTo be honest I'm surprised how well it holds. I expected close-to-total collapse. It'll be a matter of time I guess, but still.
- mdahardy 11mo agoSame! As we talk about in the article, the failures were less from raw model intelligence/ability than from challenges with timing and dynamic interfaces
- swyx 11mo agoi mean did you see the cross-tile numbers
- criddell 11mo agoI wonder if any of the agents hit the audio button and listened to the instructions? In my experience, that can be pretty helpful.
- WhereIsTheTruth 11mo ago3 models only, can we really call that a benchmark?
- mdahardy 11mo agoyes
- deleted 11mo ago[deleted]
- WhereIsTheTruth 11mo agoIgnorance Is Bliss
- cindyllm 11mo ago[dead]
- maknee 11mo agointeresting results. why does reload/cross-tile have worse results? would be nice to see some examples of failed results (how close did it to solving?)
- mdahardy 11mo agoWe have an example of a failed cross-tile result in the article - the models seem like they're much better at detecting whether something is in an image vs. identifying the boundaries of those items. This probably has to do with how they're trained - if you train on descriptions/image pairs, I'm not sure how well that does at learning boundaries. Reload are challenging because of how the agent-action loop works. But the models were pretty good at identifying when a tile contained an item.
- Youden 11mo agoI'm also curious what the success rates are for humans. Personally I find those two the most bothersome as well. Cross-tile because it's not always clear which parts of the object count and reload because it's so damn slow.
- ajsnigrutin 11mo agoSo, when do we reach a level where AI is better than humans and we remove captcha from pages alltogether? If you don't want bots to read content, don't put it online, you're just inconveniencing real people now.
- cubefox 11mo agoThey can also sign up and post spam/scams. There are a lot of those spam bots on YouTube, and there probably would be a lot more without any bot protection. Another issue is aggressive scrapers effectively DOSing a website. Some defense against bots is necessary.
- deleted 11mo ago[deleted]
- 1gn15 11mo agoI use manual verification on first post (note: can probably use an LLM to automate this), or just not have a comment section in the first place. That way, you moderate based on content, not identity or mental ability (which can be discriminatory and is also a losing game, as seen in TFA). Either that, or just be honest and allow anonymous posting lol
- deleted 11mo ago[deleted]
- kjok 11mo agoIf not today, models will get better at solving captchas in the near future. IMHO, the real concern, however, is cheap captcha solving services.
- arbol 11mo agoThe solvers are a problem but they give themselves away when they incorrectly fake devices or run out of context. I run a bot detection SaaS and we've had some success blocking them. Their advertised solve times are also wildly inaccurate. They take ages to return a successful token, if at all. The number of companies providing bot mitigation is also growing rapidly, making it difficult for the solvers to stay on top of reverse engineering etc.
- kjok 11mo ago> when they incorrectly fake devices And how often does this happen? Do you have any proof? Most YC companies building browser agents have built-in captcha solvers.
- arbol 11mo agoThat's a good question. I haven't checked the stats to see how often it happens but I will make a note to return with some info. We're dealing with the entire internet, not just YC companies, and many scrapers / solvers will pass up a user agent that doesn't quite match the JS capabilities you would expect of the browser version. Some solving companies allow you to pass up user agent , which causes inconsistencies as they're not changing their stack to match the user agent you supply. Under the hood they're running whatever version of headless Chrome they're currently pinned to.
- mehdibl 11mo agoOk and then? Those models were not trained for this purpose. It's like the last hype over using generative AI for trading. You might use it for sentiment analysis, summarization and data pre-processing. But classic forecast models will outperform them if you feed them the right metrics.
- Legend2440 11mo agoIt is relevant because they are trained for the purpose of browser use and completing tasks on websites. Being able to bypass captchas is important for using many websites. It would be nice to see comparisons to some special-purpose CAPTCHA solvers though.
- bagacrap 11mo agoAnd more broadly, if an agent is supposed to do everything a human can on the web, its ability to solve a captcha is likely a decent litmus test.
- daveguy 11mo agoThese are all multi-modal models, right? And the vision capabilities are particularly touted in Gemini. https://ai.google.dev/gemini-api/docs/image-understanding https://ai.google.dev/gemini-api/docs/image-understanding
- jngiam1 11mo agoI hypothesize that these AI agents are all likely higher than human performance now.
- daveguy 11mo agoWow. Cross-tile performance was 0-2%. That's the challenge where you select all of the tiles containing an item where the single item is in a subset of tiles. As opposed to all the tiles that contain the item type (static - 60% max) and the reload version (21% max). Seems to really highlight how far these things are from reasoning or human level intelligence. Although to be fair, the cross-tile is the one I perform worst on too (but more like 90+% rather than 2%).
- RobertDeNiro 11mo agoI think the prompt is probably at fault here. You can use LLMs for object segmentation and they do fairly well, less than 1% seems too low.
- mdahardy 11mo agoThe cross-tile challenges were quite robust - every model struggled with them, and we tried with several iterations of the prompt. I'm sure you could improve with specialized systems, but the models out-of-the-box definitely struggle with segmentation
- alexnewman 11mo agoHcaptcha cofounder here. Enterprise users have a lot of fancy configuration behind the scenes. I wonder if they coordinated with recaptcha or just assume there sitekey in the same as others
- amirhirsch 11mo agoThis was done on the re-captcha demo page no invisible fingerprinting, behavioral test, or user classification.
- alexnewman 11mo agoAh yea probably not a good test then. Good point
- rkagerer 11mo agoForget whether humans can't distinguish your AI from another human. The real Turing test is whether your AI passes all the various flavors of captcha checks.
- timshell 11mo agoOne of the writers here. We believe the real Turing Test is whether your AI performs a CAPTCHA like a human would/does.
- theoldgreybeard 11mo agoI’ve used LLMs to solve captchas for shits and giggles, just taking a screenshot and pasting it into ChatGPT and having it tell me what squares to click and I think it solves them better than I do. Can we just get rid of them now, they are so annoying and basically useless.
- throwawayu5pg 11mo agostatic, cross-tile and reload. recaptcha call window pings LPRs.
- jameslk 11mo agoIs it assumed that humans perform 100% against this captcha? Because being one of those humans it’s been closer to 50% for me I’m guessing Google is evaluating more than whether the answer was correct enough (ie does my browser and behavior look like a bot?), so that may be a factor
- sjapps 11mo ago[dead]
- padolsey 11mo agoTo this day I hate captchas. Back when it was genuinely helping to improve OCR for old books, I loved that in the same way I loved folding@home, but now I just see these widgets as a fundamentally exclusionary and ableist blocker. People with cognitive, sight, motor, (and many other) impairments are at a severe disadvantage (and no, audio isn't a remedy, it is just shifting to other ableisms). You can add as many aria labels as you like but if you're relying on captchas, you are not accessible. It really upsets me that these are now increasing in popularity. They are not the solution. I don't know what is, but this aint it.
- TulliusCicero 11mo ago> In general, all models performed best on Static challenges and worst on Cross-tile challenges. I also perform poorly on cross-tile, I never know whether to count a tiny bit of a bicycle in a square as "a bike in that square".
- akimbostrawman 11mo agoAt this point i am convinced all captchas almost entirely rely on ip reputation. Even on linux with hardened firefox you can get stuck in a infinite loop with one IP but then switch to another one that let's you in after 0-2 tries.
- tim333 11mo agoI'd have a job with the first cross-tile one shown saying select squares with motorcycles. Does the square above the handle bars appearing to maybe contain part of a rear view mirror count? I'm not surprised the LLMs were failing on those.
- VectorLock 11mo agoIs calling Browser Use and "open source framework" a bit misleading it looks like a commercial product that requires an API key to use even if you run the source?