5 ms·

I think recaptcha and captcha in general are very overused today, as they cause way too much inconvenience for the user. Why discriminate against robots so much

I think recaptcha and captcha in general are very overused today, as they cause way too much inconvenience for the user. Why discriminate against robots so much? What's so wrong about crawling or using automated tools? With today's networks and hardware performance most websites shouldn't concern themselves with denial-of-service type of attacks, unless they're past a certain threshold of popularity.

taftster 8y ago
One thing is account creation. It's usually a bad idea that user accounts are created by non-humans. In general, anything that writes to a database may have concerns about robot/automation/abuse. I agree with you, for "read only" content, let the robots crawl.
benrbray 8y ago
Well, it's useful on places where genuine human interaction is important, like Craiglist or Reddit. (in reality though reddit has a huge bot / astroturfing problem anyway, so perhaps the value of captchas really is overstated)
remus 8y ago
There's nothing inherently wrong with crawlers and automated tools, but there's a lot of poorly written scrapers out there that can easily generate a huge volume of traffic along with the associated maintenance. Why waste time catering to people who are too lazy to write a decent scraper? Most people would prefer to spend their time working on things that will benefit the other 99% of their users.
learc83 8y ago
At work we have a scraper that likes to use a particularly expensive search query using hundreds of different ips dozens of times per second, all so that they can scrape data that is freely available from us as an XML feed at a different url. Every time I found a way to fingerprint and block them, they'd change their bot to avoid detection. Captcha for rate limiting seemed like the least bad option. However, eventually I decided that instead of blocking them, I'd return random results from the db. They weren't checking the data too thoroughly because to this day they haven't changed the bot to avoid my most recent pattern detection. They're still merrily scraping useless data.
- seotut2 8y ago
  Rate limiting can be achieved without actually inconveniencing regular users. Put a exponentially growing delay on the server's response for requests coming in too quick succession.
  - learc83 8y ago
    The problem was that with the sheer number of ip addresses they were using, and with the rate that some normal users used that particular endpoint, regular users would have been inconvenienced--either by being forced to wait or by being forced to do a captcha. I would have set it to only show the captcha if the delay was active, so users would effectively have had a choice of wait or captcha.
- herpderp3dtwerp 8y ago
  Awesome idea giving them crappy data. I do the same thing with loyalty programs. Someone has already registered XXX-555-1234 as a phone number in your area code. I like to think I'm adding a little bit of chaos to their data tracking.
- robk 8y ago
  Why not add a "PLEASE USE THE XML DUMMY!" line to the returned results when you figure out it's them?
  - learc83 8y ago
    Tried that. They didn't seem to care.
SquareWheel 8y ago
Because 99% of bots on the web are spam bots. Naked email addresses are skimmed and cataloged. Any unprotected contact form will receive dozens of spam messages per day. Forums constantly get new members that only post linkspam. There's very few friendly bots like archive.org, and those respect meta robots tags anyway.

taftster 8y ago

One thing is account creation. It's usually a bad idea that user accounts are created by non-humans. In general, anything that writes to a database may have concerns about robot/automation/abuse. I agree with you, for "read only" content, let the robots crawl.

benrbray 8y ago

Well, it's useful on places where genuine human interaction is important, like Craiglist or Reddit. (in reality though reddit has a huge bot / astroturfing problem anyway, so perhaps the value of captchas really is overstated)

remus 8y ago

There's nothing inherently wrong with crawlers and automated tools, but there's a lot of poorly written scrapers out there that can easily generate a huge volume of traffic along with the associated maintenance. Why waste time catering to people who are too lazy to write a decent scraper? Most people would prefer to spend their time working on things that will benefit the other 99% of their users.

learc83 8y ago

At work we have a scraper that likes to use a particularly expensive search query using hundreds of different ips dozens of times per second, all so that they can scrape data that is freely available from us as an XML feed at a different url. Every time I found a way to fingerprint and block them, they'd change their bot to avoid detection. Captcha for rate limiting seemed like the least bad option. However, eventually I decided that instead of blocking them, I'd return random results from the db. They weren't checking the data too thoroughly because to this day they haven't changed the bot to avoid my most recent pattern detection. They're still merrily scraping useless data.

seotut2 8y ago
Rate limiting can be achieved without actually inconveniencing regular users. Put a exponentially growing delay on the server's response for requests coming in too quick succession.
- learc83 8y ago
  The problem was that with the sheer number of ip addresses they were using, and with the rate that some normal users used that particular endpoint, regular users would have been inconvenienced--either by being forced to wait or by being forced to do a captcha. I would have set it to only show the captcha if the delay was active, so users would effectively have had a choice of wait or captcha.
herpderp3dtwerp 8y ago
Awesome idea giving them crappy data. I do the same thing with loyalty programs. Someone has already registered XXX-555-1234 as a phone number in your area code. I like to think I'm adding a little bit of chaos to their data tracking.
robk 8y ago
Why not add a "PLEASE USE THE XML DUMMY!" line to the returned results when you figure out it's them?
- learc83 8y ago
  Tried that. They didn't seem to care.

SquareWheel 8y ago

Because 99% of bots on the web are spam bots. Naked email addresses are skimmed and cataloged. Any unprotected contact form will receive dozens of spam messages per day. Forums constantly get new members that only post linkspam. There's very few friendly bots like archive.org, and those respect meta robots tags anyway.