4 ms·

The main problem with CloudFlare is how dumb their "protection" is. It doesn't make sense at all to block Tor users from just accessing read-only content, like

The main problem with CloudFlare is how dumb their "protection" is. It doesn't make sense at all to block Tor users from just accessing read-only content, like CloudFlare does today. Forms/login pages/comment boxes etc should be protected of course, and most people wouldn't have anything against solving a captcha for logging in, but preventing people from just reading stuff anonymously/securely is borderline evil from a user experience point of view. However it's obviously much easier from an engineering standpoint though to just block people outright.

angry-hacker 11y ago
Have you ever had problems with malicious botnets/spam targeting your site and they are all behind TOR? It's not really that simple. If an IP, or IP range, behaves badly - goodbye. It's a TOR problem to solve it, it's not the problem of web servers and also CloudFlare.
- kristofferR 11y ago
  That's my whole point. Feel free to require a captcha for the login form etc and other pages where botnets/spam can be a problem, but don't ban people from pages where they only have read-access anyway. It's like preventing people from reading books just because a few authors write a lot of crappy/illegal stuff. Sure - make it a little bit harder to "become an author" (by using a captcha), but don't make it harder to read stuff.
pfg 11y ago
This was addressed in CloudFlare's blog post: > One suggestion has been that we treat GET requests for static content differently than we do more risky requests like POSTs. We actually already do treat more dangerous requests differently than less risky requests. The problem is Tor exit nodes often have very bad reputations due to all the malicious requests they send, and you can do a lot of harm just with GETs. Content scraping, ad click fraud, and vulnerability scanning are all threats our customers ask us to protect them from and all only take GET requests.
- kristofferR 11y ago
  Hm, I didn't notice that part and I hadn't thought about ad click fraud and vulnerability scanning. Preventing content scraping for publicly accessible content is kind of dumb and impossible in my view anyway though. For the few that care about making scraping more cumbersome the option should be there to enable the captchas, but I don't think it should be a default. Ad click fraud is a harder problem to figure out, but should be a issue for the ad-networks to deal with and not the sites with the ads. I'm torn about preventing vulnerability scanning. While I do it on my server using mod_security, it's also a form of "security by obscurity", which I'm not that much a fan of. Anyway, disabling captchas completely should be a very visible option. The protections on "dangerous GET requests" are probably only needed for a small minority of CloudFlare sites, so what should be the default setting becomes a question of what's most valued the most - safety for the many (making Tor usage easy and not cumbersome) or security for the few (who rely on the default settings protecting them from "niche" security issues). I think making Tor usage easy by default would be best for the security of the web at large, but I can see why CloudFlare has different priorities.
- kuschku 11y ago
  That argument only holds true if the person operating the site has no idea at all about the HTTP standards. * GET requests have to be idempotent. * Security by Obscurity is not Security. * Content Scraping is nothing you have to protect against, or should protect against – DRM just does not work.
  - pfg 11y ago
    Right, but this is about abuse and solutions for that. Obviously, blocking Tor is not going to prevent a determined attacker from scraping your site or trying some SQLi vectors. It might, however, prevent a large number of bots from scraping your site for emails, scanning for vulnerabilities, or doing click fraud. It's not a perfect solution, but those rarely exist. CloudFlare sees a lot of malicious traffic, so they probably have a better view of what works and what doesn't compared to everyone else.
    - 11y ago
adrr 11y ago
How do you stop people from scraping your site? Databrokers frequently will mine social sites to build profiles on people. There's legitimate reasons to block TOR for read-only content.
- zzzcpan 11y ago
  Scrapers visit many unique URLs. Such request patterns should be fairly easily distinguishable from the rest of the visitors. To do it on a large scale in realtime something like loglog[1] counter could be used. [1] https://en.wikipedia.org/wiki/Flajolet%E2%80%93Martin_algorithm https://en.wikipedia.org/wiki/Flajolet%E2%80%93Martin_algori...
  - eastdakota 11y ago
    How do you id the scraper from request to request? Cookie? Bots can just discard. IP? Now we're back to the same problem. Some sort of super-cookie browser fingerprinting? Sure, maybe possible, but it's super sketchy and sacrifices anonymity which is important.
    - zzzcpan 11y ago
      I don't think it is necessary to identify the scraper. Something simple should work, like counting how many unique URLs for a particular website got accessed in some period of time and comparing that to similar thing, but counted for known non-bot users (normalized). If one is much bigger than the other one - start requiring CAPTCHAs, but only for those requests, that are not part of known non-bot users set of our probabilistic counter of requests or even a separate bloom filter, depending on what is going to be more efficient and more accurate.
- fabulist 11y ago
  It is 2016. The web is public. People scrape sites. Don't show them things they shouldn't see. Don't pretend you're going to stop 80legs from scraping you by blocking Tor.

angry-hacker 11y ago

Have you ever had problems with malicious botnets/spam targeting your site and they are all behind TOR? It's not really that simple. If an IP, or IP range, behaves badly - goodbye. It's a TOR problem to solve it, it's not the problem of web servers and also CloudFlare.

kristofferR 11y ago
That's my whole point. Feel free to require a captcha for the login form etc and other pages where botnets/spam can be a problem, but don't ban people from pages where they only have read-access anyway. It's like preventing people from reading books just because a few authors write a lot of crappy/illegal stuff. Sure - make it a little bit harder to "become an author" (by using a captcha), but don't make it harder to read stuff.

pfg 11y ago

This was addressed in CloudFlare's blog post: > One suggestion has been that we treat GET requests for static content differently than we do more risky requests like POSTs. We actually already do treat more dangerous requests differently than less risky requests. The problem is Tor exit nodes often have very bad reputations due to all the malicious requests they send, and you can do a lot of harm just with GETs. Content scraping, ad click fraud, and vulnerability scanning are all threats our customers ask us to protect them from and all only take GET requests.

kristofferR 11y ago
Hm, I didn't notice that part and I hadn't thought about ad click fraud and vulnerability scanning. Preventing content scraping for publicly accessible content is kind of dumb and impossible in my view anyway though. For the few that care about making scraping more cumbersome the option should be there to enable the captchas, but I don't think it should be a default. Ad click fraud is a harder problem to figure out, but should be a issue for the ad-networks to deal with and not the sites with the ads. I'm torn about preventing vulnerability scanning. While I do it on my server using mod_security, it's also a form of "security by obscurity", which I'm not that much a fan of. Anyway, disabling captchas completely should be a very visible option. The protections on "dangerous GET requests" are probably only needed for a small minority of CloudFlare sites, so what should be the default setting becomes a question of what's most valued the most - safety for the many (making Tor usage easy and not cumbersome) or security for the few (who rely on the default settings protecting them from "niche" security issues). I think making Tor usage easy by default would be best for the security of the web at large, but I can see why CloudFlare has different priorities.
kuschku 11y ago
That argument only holds true if the person operating the site has no idea at all about the HTTP standards. * GET requests have to be idempotent. * Security by Obscurity is not Security. * Content Scraping is nothing you have to protect against, or should protect against – DRM just does not work.
- pfg 11y ago
  Right, but this is about abuse and solutions for that. Obviously, blocking Tor is not going to prevent a determined attacker from scraping your site or trying some SQLi vectors. It might, however, prevent a large number of bots from scraping your site for emails, scanning for vulnerabilities, or doing click fraud. It's not a perfect solution, but those rarely exist. CloudFlare sees a lot of malicious traffic, so they probably have a better view of what works and what doesn't compared to everyone else.
  - 11y ago

adrr 11y ago

How do you stop people from scraping your site? Databrokers frequently will mine social sites to build profiles on people. There's legitimate reasons to block TOR for read-only content.

zzzcpan 11y ago
Scrapers visit many unique URLs. Such request patterns should be fairly easily distinguishable from the rest of the visitors. To do it on a large scale in realtime something like loglog[1] counter could be used. [1] https://en.wikipedia.org/wiki/Flajolet%E2%80%93Martin_algorithm https://en.wikipedia.org/wiki/Flajolet%E2%80%93Martin_algori...
- eastdakota 11y ago
  How do you id the scraper from request to request? Cookie? Bots can just discard. IP? Now we're back to the same problem. Some sort of super-cookie browser fingerprinting? Sure, maybe possible, but it's super sketchy and sacrifices anonymity which is important.
  - zzzcpan 11y ago
    I don't think it is necessary to identify the scraper. Something simple should work, like counting how many unique URLs for a particular website got accessed in some period of time and comparing that to similar thing, but counted for known non-bot users (normalized). If one is much bigger than the other one - start requiring CAPTCHAs, but only for those requests, that are not part of known non-bot users set of our probabilistic counter of requests or even a separate bloom filter, depending on what is going to be more efficient and more accurate.
fabulist 11y ago
It is 2016. The web is public. People scrape sites. Don't show them things they shouldn't see. Don't pretend you're going to stop 80legs from scraping you by blocking Tor.