4 ms·

I'd be interested in knowing how good it is at solving visual captchas, do we foresee a huge rise in automated bypasses?

by Usu 3y ago

GaggiX 3y ago
Solving CAPTCHAs at the moment is more inexpensive using humans than using GPT-4 API.
- yeldarb 3y ago
  If true, this is wild. I suppose a human could spend 10 seconds per Captcha, so they could do 360 per hour. Add some overhead for not being operating at peak performance every minute of every hour & call it 250. Let's say you can hire someone for $2, that works out to a bit over a penny per Captcha. I don't think OpenAI has published pricing for GPT-4 Vision yet, but if we assume it's on par with GPT-4, and uses only 1000 of the 8000 possible tokens to process an image that's 3 cents per Captcha. Doesn't seem completely unreasonable that at-scale humans may actually be cheaper than LLMs at this point. My mind is a little blown.
  - Andoryuuta 3y ago
    You'd be surprised, or perhaps horrified, by how cheap (self-proclaimed) human-based captcha solving services are. If you just search for "captcha solving service" the first few results that come up offer 1000 solves of text-based captchas for <= $1 USD, (puzzle / JS browser challenge captchas are charged much higher). Whether these are actually human based, or just impressive OCR services, it seems like they are still much more cost effective than GPT-4 is for now.
    - altcognito 3y ago
      I imagine they are a mix.
  - eiiot 3y ago
    The way these work is usually presenting an existing captcha to another human who doesn’t even know they’re solving the captcha. For example, sites hosting pirated content serve fake captchas as a way to make money.
zerojames 3y ago
We have just added a section on this! TL;DR: GPT-4V isn't great at this task at the moment :)
- qingcharles 3y ago
  Is it possible they hobbled it a bit? I know CAPTCHA solving was one of the reasons they delayed the roll-out of this feature.
  - gs17 3y ago
    Given that it fails by hallucinating the structure of the challenge instead of refusing to solve a CAPTCHA, I doubt they've intentionally reduced the capability. Although the example in your sibling comment implies it should have enough information to do it.
- circuit10 3y ago
  Back when they leaked it via a Discord bot I found it worked better when you ask it to first describe each box Without doing that: https://cdn.discordapp.com/attachments/964175221089259591/1156683009836196050/image.png https://cdn.discordapp.com/attachments/964175221089259591/11... With it: https://cdn.discordapp.com/attachments/964175221089259591/1156683180053647360/image.png https://cdn.discordapp.com/attachments/964175221089259591/11... (though it's only one example so it could be coincidence)

I'd be interested in knowing how good it is at solving visual captchas, do we foresee a huge rise in automated bypasses?

by Usu 3y ago

I'd be interested in knowing how good it is at solving visual captchas, do we foresee a huge rise in automated bypasses?

GaggiX 3y ago

Solving CAPTCHAs at the moment is more inexpensive using humans than using GPT-4 API.

yeldarb 3y ago
If true, this is wild. I suppose a human could spend 10 seconds per Captcha, so they could do 360 per hour. Add some overhead for not being operating at peak performance every minute of every hour & call it 250. Let's say you can hire someone for $2, that works out to a bit over a penny per Captcha. I don't think OpenAI has published pricing for GPT-4 Vision yet, but if we assume it's on par with GPT-4, and uses only 1000 of the 8000 possible tokens to process an image that's 3 cents per Captcha. Doesn't seem completely unreasonable that at-scale humans may actually be cheaper than LLMs at this point. My mind is a little blown.
- Andoryuuta 3y ago
  You'd be surprised, or perhaps horrified, by how cheap (self-proclaimed) human-based captcha solving services are. If you just search for "captcha solving service" the first few results that come up offer 1000 solves of text-based captchas for <= $1 USD, (puzzle / JS browser challenge captchas are charged much higher). Whether these are actually human based, or just impressive OCR services, it seems like they are still much more cost effective than GPT-4 is for now.
  - altcognito 3y ago
    I imagine they are a mix.
- eiiot 3y ago
  The way these work is usually presenting an existing captcha to another human who doesn’t even know they’re solving the captcha. For example, sites hosting pirated content serve fake captchas as a way to make money.

zerojames 3y ago

We have just added a section on this! TL;DR: GPT-4V isn't great at this task at the moment :)

qingcharles 3y ago
Is it possible they hobbled it a bit? I know CAPTCHA solving was one of the reasons they delayed the roll-out of this feature.
- gs17 3y ago
  Given that it fails by hallucinating the structure of the challenge instead of refusing to solve a CAPTCHA, I doubt they've intentionally reduced the capability. Although the example in your sibling comment implies it should have enough information to do it.
circuit10 3y ago
Back when they leaked it via a Discord bot I found it worked better when you ask it to first describe each box Without doing that: https://cdn.discordapp.com/attachments/964175221089259591/1156683009836196050/image.png https://cdn.discordapp.com/attachments/964175221089259591/11... With it: https://cdn.discordapp.com/attachments/964175221089259591/1156683180053647360/image.png https://cdn.discordapp.com/attachments/964175221089259591/11... (though it's only one example so it could be coincidence)