7 ms·

"To make sure our team understood what a pain CAPTCHAs could be, I blacklisted all the IP addresses used in CloudFlare's office so our employees would need to p

"To make sure our team understood what a pain CAPTCHAs could be, I blacklisted all the IP addresses used in CloudFlare's office so our employees would need to pass a CAPTCHA every time they wanted to visit any of our customers' sites." Say what you will about cloudflare, that's an impressive move.

jgrahamc 11y ago
What he didn't say was that he did it without warning and it lasted 30 days. Imagine what that was like for the technical support and customer success teams who were helping customers with their sites.
- deleted 11y ago
  [deleted]
- brianwawok 11y ago
  I am a strong believer in dogfooding. If it wasn't a pain you felt, it wouldn't be prioritized to be fixed.
  - sievebrain 11y ago
    Dogfooding is great, if there actually is a fix to the issue at hand. If the issue is "anonymous internet access gets abused a lot" then I'm not sure what a tech support guy is meant to usefully contribute to ending the dogfooding pain.
    - throwaway7767 11y ago
      > Dogfooding is great, if there actually is a fix to the issue at hand. If the issue is "anonymous internet access gets abused a lot" then I'm not sure what a tech support guy is meant to usefully contribute to ending the dogfooding pain. The problem being solved was not "anonymous internet access gets abused a lot", it was "the mechanism we use to combat abuse is too aggressive for some segments of our users and thereby denying service to actual humans, and these challenges are invisibile to our engineers and employees because they don't browse on connections that trigger the system". As the blog post shows (and this is backed up by my experience as a user of Tor) CloudFlare significantly improved its handling of this in response. So I would say this is a success. I am sure that pressure from employees, including support guys, caused this long-running issue to finally get the attention it deserved.
      - eastdakota 11y ago
        Exactly.
- mdip 11y ago
  Imagine what that was like for the technical support and customer success teams who were helping customers with their sites. It sucks. But that's exactly the point. Using the internet is a mission critical thing for many people and for some that means using Tor Browser or similar to get around oppressive governments. This sounds like a really effective way to make sure the things you're doing are impacting your customers minimally. I bet you guys screamed loudly and were heard more clearly than a random customer trying to use Tor day-to-day.
mdip 11y ago
Impressive, yes, but I'm going to hazard a guess that they didn't route all of that traffic through Tor to experience the CAPTCHA with the bandwidth constraints imposed by using an exit node. The CAPTCHAs are (mostly) easy to solve, but all of the ones I was presented with were "pick the right one out of 9 different images" and loading the entire CAPTCHA in Tor Browser took several seconds (and many revealed a new image after clicking one of the 9). This is then repeated at least once (I received three on one site, I'm guessing because I didn't know if the picture was a store front or just the front of some building). After completing the challenge I was given a connection error and had to repeat the entire thing again in one case. There are much lower bandwidth CAPTCHAs out there and those should be favored over these large image-based ones for connections originating from the block of addresses represented by Tor exit nodes.
- pfg 11y ago
  The entire CloudFlare recaptcha page is < 100 KB, and that's including the images. If you're annoyed by the time it takes for that page to load, you won't be happy browsing any site via Tor - even Google is > 200 KB.
  - lallysingh 11y ago
    How much of each is cached?
    - pfg 11y ago
      Both requested with a clean cache using Tor Browser Bundle. Not sure if TBB does any caching at all anyway, seems like that would make fingerprinting easier, but I haven't checked. FWIW, the actual captcha image is ~20 KB.
  - mdip 11y ago
    You're right, they're not really that big. They load extraordinarily slowly. Each load one at a time from upper left to lower right and you can watch them download -- they don't just "appear", they show up like they would on a slow dial-up connection. Once you get onto the site, it loads more slowly than in a non-Tor connection, but a news site I hit loaded everything at about the same speed as the little reCAPTCHA form, so I'm left wondering if it's something related to reCAPTCHA.
diskcat 11y ago
Sounds like the "old wise sage master teaching grasshopper employees a lesson" pattern. But has he such low opinions of his employee that he couldn't have simply told them "captchas are bad UX, try it out if you don't believe me"? it just seems condescending to me
- Klathmon 11y ago
  I didn't see it like that at all... It was more of a "let's really feel the pain from this and try to come up with a better solution". And at the end of it they didn't. They felt the pain of having to do it for just about everything, and still couldn't come up with a better solution other than "improve the captcha system". And from there they gave some suggestions on how they could work with tor to give a better experience. Simply saying "captchas suck, find a better way" would have gotten nowhere because the people involved wouldn't personally know the pain points, they wouldn't know the exact issues, or what exactly about it was the most annoying. (was it the captcha itself, or the frequency you had to fill it out? Did it completely break some of their workflows, and what were the ways they ended up getting around those breakages, and can they prevent them? Perhaps they found the redirect page broke some browsers/software/extensions!) I would kill to have a one-on-one with a typical "user" of my products. A long sit down meeting to understand exactly what they want/need above "i want it to work better". Making your developers/staff users of your product is a great way to do that and to really internalize it.
DanielDent 11y ago
Others have already mentioned how Tor bandwidth/latency issues could make things substantially worse for 'real' Tor users outside of their simulation. But the bigger issue I would wonder about is Google's reputation systems. Google does not treat all CAPTCHA requests equally. An office full of CloudFlare employees peaceably going about their daily browsing is going to get a much easier CAPTCHA situation than a Tor IP containing a mix of automated nefarious activity and individuals peaceably browsing from infomatically repressed countries.

jgrahamc 11y ago

What he didn't say was that he did it without warning and it lasted 30 days. Imagine what that was like for the technical support and customer success teams who were helping customers with their sites.

deleted 11y ago
[deleted]
brianwawok 11y ago
I am a strong believer in dogfooding. If it wasn't a pain you felt, it wouldn't be prioritized to be fixed.
- sievebrain 11y ago
  Dogfooding is great, if there actually is a fix to the issue at hand. If the issue is "anonymous internet access gets abused a lot" then I'm not sure what a tech support guy is meant to usefully contribute to ending the dogfooding pain.
  - throwaway7767 11y ago
    > Dogfooding is great, if there actually is a fix to the issue at hand. If the issue is "anonymous internet access gets abused a lot" then I'm not sure what a tech support guy is meant to usefully contribute to ending the dogfooding pain. The problem being solved was not "anonymous internet access gets abused a lot", it was "the mechanism we use to combat abuse is too aggressive for some segments of our users and thereby denying service to actual humans, and these challenges are invisibile to our engineers and employees because they don't browse on connections that trigger the system". As the blog post shows (and this is backed up by my experience as a user of Tor) CloudFlare significantly improved its handling of this in response. So I would say this is a success. I am sure that pressure from employees, including support guys, caused this long-running issue to finally get the attention it deserved.
    - eastdakota 11y ago
      Exactly.
mdip 11y ago
Imagine what that was like for the technical support and customer success teams who were helping customers with their sites. It sucks. But that's exactly the point. Using the internet is a mission critical thing for many people and for some that means using Tor Browser or similar to get around oppressive governments. This sounds like a really effective way to make sure the things you're doing are impacting your customers minimally. I bet you guys screamed loudly and were heard more clearly than a random customer trying to use Tor day-to-day.

mdip 11y ago

Impressive, yes, but I'm going to hazard a guess that they didn't route all of that traffic through Tor to experience the CAPTCHA with the bandwidth constraints imposed by using an exit node. The CAPTCHAs are (mostly) easy to solve, but all of the ones I was presented with were "pick the right one out of 9 different images" and loading the entire CAPTCHA in Tor Browser took several seconds (and many revealed a new image after clicking one of the 9). This is then repeated at least once (I received three on one site, I'm guessing because I didn't know if the picture was a store front or just the front of some building). After completing the challenge I was given a connection error and had to repeat the entire thing again in one case. There are much lower bandwidth CAPTCHAs out there and those should be favored over these large image-based ones for connections originating from the block of addresses represented by Tor exit nodes.

pfg 11y ago
The entire CloudFlare recaptcha page is < 100 KB, and that's including the images. If you're annoyed by the time it takes for that page to load, you won't be happy browsing any site via Tor - even Google is > 200 KB.
- lallysingh 11y ago
  How much of each is cached?
  - pfg 11y ago
    Both requested with a clean cache using Tor Browser Bundle. Not sure if TBB does any caching at all anyway, seems like that would make fingerprinting easier, but I haven't checked. FWIW, the actual captcha image is ~20 KB.
- mdip 11y ago
  You're right, they're not really that big. They load extraordinarily slowly. Each load one at a time from upper left to lower right and you can watch them download -- they don't just "appear", they show up like they would on a slow dial-up connection. Once you get onto the site, it loads more slowly than in a non-Tor connection, but a news site I hit loaded everything at about the same speed as the little reCAPTCHA form, so I'm left wondering if it's something related to reCAPTCHA.

diskcat 11y ago

Sounds like the "old wise sage master teaching grasshopper employees a lesson" pattern. But has he such low opinions of his employee that he couldn't have simply told them "captchas are bad UX, try it out if you don't believe me"? it just seems condescending to me

Klathmon 11y ago
I didn't see it like that at all... It was more of a "let's really feel the pain from this and try to come up with a better solution". And at the end of it they didn't. They felt the pain of having to do it for just about everything, and still couldn't come up with a better solution other than "improve the captcha system". And from there they gave some suggestions on how they could work with tor to give a better experience. Simply saying "captchas suck, find a better way" would have gotten nowhere because the people involved wouldn't personally know the pain points, they wouldn't know the exact issues, or what exactly about it was the most annoying. (was it the captcha itself, or the frequency you had to fill it out? Did it completely break some of their workflows, and what were the ways they ended up getting around those breakages, and can they prevent them? Perhaps they found the redirect page broke some browsers/software/extensions!) I would kill to have a one-on-one with a typical "user" of my products. A long sit down meeting to understand exactly what they want/need above "i want it to work better". Making your developers/staff users of your product is a great way to do that and to really internalize it.

DanielDent 11y ago

Others have already mentioned how Tor bandwidth/latency issues could make things substantially worse for 'real' Tor users outside of their simulation. But the bigger issue I would wonder about is Google's reputation systems. Google does not treat all CAPTCHA requests equally. An office full of CloudFlare employees peaceably going about their daily browsing is going to get a much easier CAPTCHA situation than a Tor IP containing a mix of automated nefarious activity and individuals peaceably browsing from infomatically repressed countries.