9 ms·

How not to design a CAPTCHA

310 points by DrewHintz 15y ago

adamtulinius 15y ago
A few years ago, or so i think, people went all crazy talking about a replacement for captcha's: Show a range of images, and make the user pick the image described by a block of text. How come nobody adopted that approach?
- a3_nm 15y ago
  If you only have a limited collection of images to pick from, then bots could get decent scores by picking at random. A better approach might be to ask users to pick matching images (ie. 2^N possible choices).
- DrewHintz 15y ago
  What would the system use for its corpus of images and text descriptions? The corpus would have to be significantly larger than what any given attacker could manually identify. Once an attacker has manually identified an image+text combination, they could store the combination and use it to solve any future CAPTCHAs with the same image+text.
- lbrandy 15y ago
  Because the math doesn't work. Most "next-gen" captcha fundamentally fail (by orders of magnitude) one of the many pillars that make captchas scale.... 1. Is it trivial for a human to answer correctly? This affects growth. 2. Can humans do it quickly? This affects growth. 3. How is the random guess-rate? This better be abysmal. 4. How good is the “opposing” technology? 5. How is the guess rate of a sophisticated attacker, using said technology? 6. How much human input is required to create your captcha? You better be asymptotically better than human-solving the captcha. 7. What are the cultural and accessibility issues?
  - Joakal 15y ago
    8. The user may have a slow computer. I remember suggestions of using computing power to slow down guess-rates. Probably related to bitcoins. However, it doesn't work since some users don't seek better computer performance.
- jerf 15y ago
  "Anyone can invent a security system that he himself cannot break." - http://www.schneier.com/blog/archives/2011/04/schneiers_law.html http://www.schneier.com/blog/archives/2011/04/schneiers_law.... Any CAPTCHA scheme that can be solved by enumeration of all possible answers is a failure, because there are cost effective ways to hit a CAPTCHA over and over again, with cheap humans, and build the enumeration table. This is where the "pick the image with a cute thing" in it scheme falls down. In this case, once the enumeration of description -> image(s) is determined, you lose. Any scheme that involves humans some how creating tags or labeling images or writing text will generally be enumerable as well, because they can trivially out-manpower you. Also, many CAPTCHA schemes use a model of spammer in which the spammer isn't permitted to be clever. If there is a pattern, in the real world the spammer is "allowed" to exploit it. There are 2^64 different ways to add two 32-bit numbers to each other, but that doesn't mean that you can beat a spammer just by asking the user to do a simple addition, because when I say "enumerate" I mean it more in the computer science sense, not the literal sense. They can and will create something that parses the problem and does it, so for instance for my stupid "add two random 32-bit numbers" example the CAPTCHA is actually easier for a computer than a human. CAPTCHAs are hard and getting steadily harder... at least, if you require them to work. Security theater is easy.
- ams6110 15y ago
  Mainly because, to quote Spolsky, Users don't read instructions. If the captcha is ANYTHING other than immediately obvious, a signficant number users will not be able to pass it.
Turing_Machine 15y ago
A slightly less clueless (but still clueless) approach to CAPTCHA design is to 1) make the CAPTCHA case-sensitive, 2) use letters for which the lower-case representation is very similar to upper-case, and/or use both zero and the letter O, 1 and the letter l, and so on, 3) use an image munging algorithm that makes it next to impossible to disambiguate the cases in 2).
- Xk 15y ago
  The problem with captchas is they have to be readable to humans. Sure, a captcha of "lI0Ol1o" would would probably be unreadable to a computer ... but it would be to a human too. We're quickly approaching the point that image recognition is getting as good at solving image captchas as humans are, and when we do, we'll need to find some other way to do it.
  - tst 15y ago
    Reminds me of the infamous Rapidshare Captcha with cats. [1] There were hard to solve. However, also for humans. [1]: http://www.labnol.org/internet/favorites/cats-inside-rapidshare-captcha-images/3064/ http://www.labnol.org/internet/favorites/cats-inside-rapidsh...
    - soult 15y ago
      > However, also for humans. Which is exactly why they added those. Premium accounts do not require the user to enter a captcha for every download. So every user who was annoyed by the "cats captcha" was a potential customer.
- hammock 15y ago
  What I think is cool are the captchas that make fake words that actually look like they could be real words (as opposed to a random string of text). Makes it easier for a human to read and figure out, but no easier for a bot. I dont know how they do that.
  - 15y ago
ghurlman 15y ago
Sony... some part of me had really hoped that they would overreact to the hacking movement against them, and lock themselves down like Ft. Knox. Instead, it would seem they're taking the "we'll get hacked anyway, so let's not waste our time" approach.
- dennisgorelik 15y ago
  The Sony's CAPTCHA we are discussing here was likely written years ago (before Sony security vulnerability scandal). It just indicates pathetic state of Sony Security development team - something that cannot be changed overnight.
Slackwise 15y ago
I work in medical IT. You'd be surprised how many government sites do similar. An example would be https://sso.state.mi.us/som/dch/enroll/reg_page1.jsp https://sso.state.mi.us/som/dch/enroll/reg_page1.jsp (You can enter any fake name/email, this is only step one of the registration script. The next page has the captch in question.) The captcha is plaintext, right on the page. The data from the captcha isn't even sent to the server, it is processed locally via JavaScript. So, the bots don't even have to do anything, but humans have to input a meaningless number... <input type="text" name="inputNumber" class="entry-field" size="5" tabindex="3">  document.write('<div id="layerNum" class="verifyNumber" align="center">'); document.write('<b>'+str+'</b>'); document.write('<img src="generateGIF.jsp?number='+str+'">'); document.write('</div>'); document.write('<input size="5" type="hidden" name="rdNumber" value="'+str+'">');  <input type="submit" value="Continue" name="submit" onclick="return Valid();">  function Valid(){ // ... if(chkRandomNumber()){ return true; }else{ return false; } // ... } function chkRandomNumber(){ str1=document.all.rdNumber.value; str2=document.all.inputNumber.value; if(str1!=str2){ alert("Please check and type the number as shown in the box"); return false; }else{ return true; } }
- codabrink 15y ago
  Wow, that is very surprising. Is it that the web development industry hurting that much for good programmers, or are just the wrong people being hired?
  - ignifero 15y ago
    The second. They probably have scammed the client in other ways as well.
    - sanderjd 15y ago
      "Scammed" is probably not the right word here - at least to me, it conveys a malicious intent, while mistakes like this are merely ignorance. I'm sure most of us have made mistakes just as stupid as this, despite working hard to earn our pay.
sthatipamala 15y ago
Completely OT: I find it interesting that this post and several other HN posts this week are hosted on Google Plus. I definitely would not have predicted that G+ would encroach on the LiveJournal/Tumblr space.
- carbocation 15y ago
  On a similar tangent to your OT post: we're getting to the point where seeing (plus.google.com) would be useful, since it conveys quite a different meaning to me from (google.com).
  - nantes 15y ago
    If you use Chrome, you might be interested in http://news.ycombinator.com/item?id=2240646 http://news.ycombinator.com/item?id=2240646
    - carbocation 15y ago
      Actually, yes, that did the trick for me; thanks! (Though I still think it's a feature that everyone would appreciate, so if it could go into news.yc, that would be even better.)
      - nantes 15y ago
        You will hear no disagreement from me. Glad it worked for you.
  - deleted 15y ago
    [deleted]
  - dmazin 15y ago
    I've noticed this issue too; I've clicked a link a number of times now thinking it came from Google but it ended up merely being hosted at Plus.
  - there 15y ago
    http://arclanguage.org/item?id=14829 http://arclanguage.org/item?id=14829 http://news.ycombinator.com/item?id=2727178 http://news.ycombinator.com/item?id=2727178
RyanMcGreal 15y ago
On a site I administer that used to be deluged in spam, I managed to eliminate it with a three-pass filter: 1. Simple mathematical question, e.g. "What do you get if you add five and three?" Answer is processed on the server. 2. Hidden form field that is supposed to remain blank. 3. Blacklist of common spam words.
- patrickyeon 15y ago
  2 and 3 are decent, as long as you don't have commenters trying to discuss something spammy (depends on the site community). #1 only works because your site isn't big enough for anybody to specifically target, though. I'm not saying it's bad (so long as it works, it's by definition at least "good enough"), just don't expect it to scale.
  - ppod 15y ago
    A system to solve problems like #1 was actually one of the very first tasks solved by early AI research. It was a PhD in MIT in 1964. http://en.wikipedia.org/wiki/STUDENT_%28computer_program%29 http://en.wikipedia.org/wiki/STUDENT_%28computer_program%29
  - RyanMcGreal 15y ago
    I should note that registered users get to skip the captcha. Right now the site gets around 1,500 visitors and 3,500 pages a day, and growth has been steady and incremental for some years.
- __david__ 15y ago
  On a forum I run (phpbb3) I eliminated 99% of the spam by adding 1 field that says "enter 42 here to prove you are human". No image, no hidden field, nothing. We still get the occasional spammer but the real problem was our phpbb3 board showing up in the automated spam programs. As soon as we were slightly different than the default install, nearly all the spam stopped. The interesting thing was that even the built-in captcha didn't stop the spam--it was worth cracking since everyone uses it.
yid 15y ago
If anyone ever wondered what the phrase "cargo cult science" referred to, this is a prime example. They're going through all the motions, but sadly their understanding of the universe is gratuitously flawed.
- drenei 15y ago
  +1 for cargo cults: http://en.wikipedia.org/wiki/Cargo_cult http://en.wikipedia.org/wiki/Cargo_cult. Its a great idea to keep in mind for a creator/designer/programmer. People/users/everyone all too often intuit through imitation.
rlf 15y ago
I can't believe Google is criticizing how Sony does CAPTCHAs when I've been complaining for years about how difficult Google's are to read. But as to their point, based on Sony's recent security issues, it doesn't sound like Sony has a very good IT department.
- kijinbear 15y ago
  It's not Google criticizing Sony, it's Andrew Hintz posting on his Google+ page.
mixmastamyk 15y ago
Jesus, rootkits, psn, and now plaintext captchas ... the dev/it clowns at sony need to be fired en masse.
alexitosrv 15y ago
If you are in this, maybe you could find interesting this review of a paper from googlers to approach a CAPTCHA design, in which humans are asked to select the right image rotation: http://glinden.blogspot.com/2009/05/exploiting-spammers-to-make-computers.html http://glinden.blogspot.com/2009/05/exploiting-spammers-to-m... As always, one of the most interesting part of truly great CAPTCHA systems is that they are advancing the state of the art in image recognition. But on the other hand we still have scams like this, and no real solutions.
desaiguddu 15y ago
Need help for Open Sourcing the CAPTCHA research project. I have covered few points of CAPTCHA design in my presentation. Here is my CAPTCHA research paper: http://news.ycombinator.org/item?id=2754436 http://news.ycombinator.org/item?id=2754436 http://www.slideshare.net/desaiguddu/drag-and-drop-captcha-a-better-approach-to-captcha http://www.slideshare.net/desaiguddu/drag-and-drop-captcha-a...
Kwpolska 15y ago
DON'T use a bloody CAPTCHA.
snorkel 15y ago
What about just asking the user "Why would a benevolent God allow evil to exist?" and then the server checks if the answer mentions "freewill"
dfc 15y ago
On the subject of terrible captcha systems. I found the following gem while looking for OSS games for linux: "You are born into WHAT? (answer is one english word)* [1] It is not entirely clear to me what the expected answer is. A google search for "you are born into" does not return any answer that is clearly correct. If I had to guess I would go with "sin" but I am hoping that nobody would be so ignorant as to design a captcha system that assumes a certain cultural/religious background. [1] http://garden.sourceforge.net/drupal/?q=image/tid/3 http://garden.sourceforge.net/drupal/?q=image/tid/3

310 points by DrewHintz 15y ago

A few years ago, or so i think, people went all crazy talking about a replacement for captcha's: Show a range of images, and make the user pick the image described by a block of text. How come nobody adopted that approach?

a3_nm 15y ago
If you only have a limited collection of images to pick from, then bots could get decent scores by picking at random. A better approach might be to ask users to pick matching images (ie. 2^N possible choices).
DrewHintz 15y ago
What would the system use for its corpus of images and text descriptions? The corpus would have to be significantly larger than what any given attacker could manually identify. Once an attacker has manually identified an image+text combination, they could store the combination and use it to solve any future CAPTCHAs with the same image+text.
lbrandy 15y ago
Because the math doesn't work. Most "next-gen" captcha fundamentally fail (by orders of magnitude) one of the many pillars that make captchas scale.... 1. Is it trivial for a human to answer correctly? This affects growth. 2. Can humans do it quickly? This affects growth. 3. How is the random guess-rate? This better be abysmal. 4. How good is the “opposing” technology? 5. How is the guess rate of a sophisticated attacker, using said technology? 6. How much human input is required to create your captcha? You better be asymptotically better than human-solving the captcha. 7. What are the cultural and accessibility issues?
- Joakal 15y ago
  8. The user may have a slow computer. I remember suggestions of using computing power to slow down guess-rates. Probably related to bitcoins. However, it doesn't work since some users don't seek better computer performance.
jerf 15y ago
"Anyone can invent a security system that he himself cannot break." - http://www.schneier.com/blog/archives/2011/04/schneiers_law.html http://www.schneier.com/blog/archives/2011/04/schneiers_law.... Any CAPTCHA scheme that can be solved by enumeration of all possible answers is a failure, because there are cost effective ways to hit a CAPTCHA over and over again, with cheap humans, and build the enumeration table. This is where the "pick the image with a cute thing" in it scheme falls down. In this case, once the enumeration of description -> image(s) is determined, you lose. Any scheme that involves humans some how creating tags or labeling images or writing text will generally be enumerable as well, because they can trivially out-manpower you. Also, many CAPTCHA schemes use a model of spammer in which the spammer isn't permitted to be clever. If there is a pattern, in the real world the spammer is "allowed" to exploit it. There are 2^64 different ways to add two 32-bit numbers to each other, but that doesn't mean that you can beat a spammer just by asking the user to do a simple addition, because when I say "enumerate" I mean it more in the computer science sense, not the literal sense. They can and will create something that parses the problem and does it, so for instance for my stupid "add two random 32-bit numbers" example the CAPTCHA is actually easier for a computer than a human. CAPTCHAs are hard and getting steadily harder... at least, if you require them to work. Security theater is easy.
ams6110 15y ago
Mainly because, to quote Spolsky, Users don't read instructions. If the captcha is ANYTHING other than immediately obvious, a signficant number users will not be able to pass it.

Turing_Machine 15y ago

A slightly less clueless (but still clueless) approach to CAPTCHA design is to 1) make the CAPTCHA case-sensitive, 2) use letters for which the lower-case representation is very similar to upper-case, and/or use both zero and the letter O, 1 and the letter l, and so on, 3) use an image munging algorithm that makes it next to impossible to disambiguate the cases in 2).

Xk 15y ago
The problem with captchas is they have to be readable to humans. Sure, a captcha of "lI0Ol1o" would would probably be unreadable to a computer ... but it would be to a human too. We're quickly approaching the point that image recognition is getting as good at solving image captchas as humans are, and when we do, we'll need to find some other way to do it.
- tst 15y ago
  Reminds me of the infamous Rapidshare Captcha with cats. [1] There were hard to solve. However, also for humans. [1]: http://www.labnol.org/internet/favorites/cats-inside-rapidshare-captcha-images/3064/ http://www.labnol.org/internet/favorites/cats-inside-rapidsh...
  - soult 15y ago
    > However, also for humans. Which is exactly why they added those. Premium accounts do not require the user to enter a captcha for every download. So every user who was annoyed by the "cats captcha" was a potential customer.
hammock 15y ago
What I think is cool are the captchas that make fake words that actually look like they could be real words (as opposed to a random string of text). Makes it easier for a human to read and figure out, but no easier for a bot. I dont know how they do that.
- 15y ago

ghurlman 15y ago

Sony... some part of me had really hoped that they would overreact to the hacking movement against them, and lock themselves down like Ft. Knox. Instead, it would seem they're taking the "we'll get hacked anyway, so let's not waste our time" approach.

dennisgorelik 15y ago
The Sony's CAPTCHA we are discussing here was likely written years ago (before Sony security vulnerability scandal). It just indicates pathetic state of Sony Security development team - something that cannot be changed overnight.

Slackwise 15y ago

I work in medical IT. You'd be surprised how many government sites do similar. An example would be https://sso.state.mi.us/som/dch/enroll/reg_page1.jsp https://sso.state.mi.us/som/dch/enroll/reg_page1.jsp (You can enter any fake name/email, this is only step one of the registration script. The next page has the captch in question.) The captcha is plaintext, right on the page. The data from the captcha isn't even sent to the server, it is processed locally via JavaScript. So, the bots don't even have to do anything, but humans have to input a meaningless number... <input type="text" name="inputNumber" class="entry-field" size="5" tabindex="3">  document.write('<div id="layerNum" class="verifyNumber" align="center">'); document.write('<b>'+str+'</b>'); document.write('<img src="generateGIF.jsp?number='+str+'">'); document.write('</div>'); document.write('<input size="5" type="hidden" name="rdNumber" value="'+str+'">');  <input type="submit" value="Continue" name="submit" onclick="return Valid();">  function Valid(){ // ... if(chkRandomNumber()){ return true; }else{ return false; } // ... } function chkRandomNumber(){ str1=document.all.rdNumber.value; str2=document.all.inputNumber.value; if(str1!=str2){ alert("Please check and type the number as shown in the box"); return false; }else{ return true; } }

codabrink 15y ago
Wow, that is very surprising. Is it that the web development industry hurting that much for good programmers, or are just the wrong people being hired?
- ignifero 15y ago
  The second. They probably have scammed the client in other ways as well.
  - sanderjd 15y ago
    "Scammed" is probably not the right word here - at least to me, it conveys a malicious intent, while mistakes like this are merely ignorance. I'm sure most of us have made mistakes just as stupid as this, despite working hard to earn our pay.

sthatipamala 15y ago

Completely OT: I find it interesting that this post and several other HN posts this week are hosted on Google Plus. I definitely would not have predicted that G+ would encroach on the LiveJournal/Tumblr space.

carbocation 15y ago
On a similar tangent to your OT post: we're getting to the point where seeing (plus.google.com) would be useful, since it conveys quite a different meaning to me from (google.com).
- nantes 15y ago
  If you use Chrome, you might be interested in http://news.ycombinator.com/item?id=2240646 http://news.ycombinator.com/item?id=2240646
  - carbocation 15y ago
    Actually, yes, that did the trick for me; thanks! (Though I still think it's a feature that everyone would appreciate, so if it could go into news.yc, that would be even better.)
    - nantes 15y ago
      You will hear no disagreement from me. Glad it worked for you.
- deleted 15y ago
  [deleted]
- dmazin 15y ago
  I've noticed this issue too; I've clicked a link a number of times now thinking it came from Google but it ended up merely being hosted at Plus.
- there 15y ago
  http://arclanguage.org/item?id=14829 http://arclanguage.org/item?id=14829 http://news.ycombinator.com/item?id=2727178 http://news.ycombinator.com/item?id=2727178

RyanMcGreal 15y ago

On a site I administer that used to be deluged in spam, I managed to eliminate it with a three-pass filter: 1. Simple mathematical question, e.g. "What do you get if you add five and three?" Answer is processed on the server. 2. Hidden form field that is supposed to remain blank. 3. Blacklist of common spam words.

patrickyeon 15y ago
2 and 3 are decent, as long as you don't have commenters trying to discuss something spammy (depends on the site community). #1 only works because your site isn't big enough for anybody to specifically target, though. I'm not saying it's bad (so long as it works, it's by definition at least "good enough"), just don't expect it to scale.
- ppod 15y ago
  A system to solve problems like #1 was actually one of the very first tasks solved by early AI research. It was a PhD in MIT in 1964. http://en.wikipedia.org/wiki/STUDENT_%28computer_program%29 http://en.wikipedia.org/wiki/STUDENT_%28computer_program%29
- RyanMcGreal 15y ago
  I should note that registered users get to skip the captcha. Right now the site gets around 1,500 visitors and 3,500 pages a day, and growth has been steady and incremental for some years.
__david__ 15y ago
On a forum I run (phpbb3) I eliminated 99% of the spam by adding 1 field that says "enter 42 here to prove you are human". No image, no hidden field, nothing. We still get the occasional spammer but the real problem was our phpbb3 board showing up in the automated spam programs. As soon as we were slightly different than the default install, nearly all the spam stopped. The interesting thing was that even the built-in captcha didn't stop the spam--it was worth cracking since everyone uses it.

yid 15y ago

If anyone ever wondered what the phrase "cargo cult science" referred to, this is a prime example. They're going through all the motions, but sadly their understanding of the universe is gratuitously flawed.

drenei 15y ago
+1 for cargo cults: http://en.wikipedia.org/wiki/Cargo_cult http://en.wikipedia.org/wiki/Cargo_cult. Its a great idea to keep in mind for a creator/designer/programmer. People/users/everyone all too often intuit through imitation.

rlf 15y ago

I can't believe Google is criticizing how Sony does CAPTCHAs when I've been complaining for years about how difficult Google's are to read. But as to their point, based on Sony's recent security issues, it doesn't sound like Sony has a very good IT department.

kijinbear 15y ago
It's not Google criticizing Sony, it's Andrew Hintz posting on his Google+ page.

mixmastamyk 15y ago

Jesus, rootkits, psn, and now plaintext captchas ... the dev/it clowns at sony need to be fired en masse.

alexitosrv 15y ago

If you are in this, maybe you could find interesting this review of a paper from googlers to approach a CAPTCHA design, in which humans are asked to select the right image rotation: http://glinden.blogspot.com/2009/05/exploiting-spammers-to-make-computers.html http://glinden.blogspot.com/2009/05/exploiting-spammers-to-m... As always, one of the most interesting part of truly great CAPTCHA systems is that they are advancing the state of the art in image recognition. But on the other hand we still have scams like this, and no real solutions.

desaiguddu 15y ago

Need help for Open Sourcing the CAPTCHA research project. I have covered few points of CAPTCHA design in my presentation. Here is my CAPTCHA research paper: http://news.ycombinator.org/item?id=2754436 http://news.ycombinator.org/item?id=2754436 http://www.slideshare.net/desaiguddu/drag-and-drop-captcha-a-better-approach-to-captcha http://www.slideshare.net/desaiguddu/drag-and-drop-captcha-a...

Kwpolska 15y ago

DON'T use a bloody CAPTCHA.

snorkel 15y ago

What about just asking the user "Why would a benevolent God allow evil to exist?" and then the server checks if the answer mentions "freewill"

dfc 15y ago

On the subject of terrible captcha systems. I found the following gem while looking for OSS games for linux: "You are born into WHAT? (answer is one english word)* [1] It is not entirely clear to me what the expected answer is. A google search for "you are born into" does not return any answer that is clearly correct. If I had to guess I would go with "sin" but I am hoping that nobody would be so ignorant as to design a captcha system that assumes a certain cultural/religious background. [1] http://garden.sourceforge.net/drupal/?q=image/tid/3 http://garden.sourceforge.net/drupal/?q=image/tid/3