6 ms·

Any time a large portion of internet traffic is controlled by a single source it brings problems like this with it. All cloudflare has to do is arbitrarily deci

by zer8k 3y ago

Any time a large portion of internet traffic is controlled by a single source it brings problems like this with it. All cloudflare has to do is arbitrarily decide who and who can't use the internet and effectively their word becomes law. Like most things it starts with an innocent premise (e.g. "an easy way to stop bad actors") and ends up extended to any number of arbitrary things. Worse, the argument from privacy advocates rings hollow because defending privacy means you have to allow Bad People (TM). The average drooler using the internet cannot understand the nuance. Even in the most innocent of cases, a bad commit getting merged, can bring down the internet. It has happened before with Cloudflare. Companies like Cloudflare, Google, Meta, etc are the reason anti-trust law exists. Unfortunately, it appears there is no one with any power that is willing to use the laws for their purpose. The internet in 20 years will be nothing like we've seen before. That's not a good thing.

NicoJuicy 3y ago
Everyone forgets that websites become almost unusable because of crawlers and bots. Website owners specifically choose for cloudflare to protect against this, it's not forced upon them by cloudflare.
- realusername 3y ago
  And who defines is an allowed crawler? Is Google the only allowed company to crawl the web? Isn't that the definition of monopoly? Could anybody still create a new search engine nowadays?
  - NicoJuicy 3y ago
    A valid crawler is following robots.txt
    - realusername 3y ago
      That's not going to be enough to pass Cloudflare.
      - NicoJuicy 3y ago
        Based on what? https://developers.cloudflare.com/support/troubleshooting/general-troubleshooting/troubleshooting-crawl-errors/ https://developers.cloudflare.com/support/troubleshooting/ge...
        buzer 3y ago
        https://developers.cloudflare.com/bots/reference/verified-bots-policy/ https://developers.cloudflare.com/bots/reference/verified-bo... https://radar.cloudflare.com/traffic/verified-bots https://radar.cloudflare.com/traffic/verified-bots https://blog.cloudflare.com/friendly-bots/ https://blog.cloudflare.com/friendly-bots/ > At Cloudflare, we manually “verify” good bots, so they don’t get blocked.
        gochi 3y ago
        >and honor robots.txt.
        NicoJuicy 3y ago
        Nice links! But cloudflare references robots.txt a lot ( which i mentioned before => to respect robots.txt) Additionally, they solve the authentication problem here. As a website owner that got bad crawlers ( that copied user agents), i just whitelisted Google's IP's and blocked all the other crawlers. It seems that cloudflare is actually fixing this problem and making competition for Google possible here. I recall an article that Bing actually circumvented robots.txt a bit, because site owners were only allowing Google and blocked all the rest => Bing. Which gave Google an unfair advantage ( searched for it, couldn't find it) Similar article to highlight the issue: https://www.fastcompany.com/90709672/the-little-known-reason-why-competing-with-google-is-so-hard https://www.fastcompany.com/90709672/the-little-known-reason... Our opinions about this seem to differ severely. Cloudflare actually enables good bots to start competing ( while respecting robots.txt).
    - deleted 3y ago
      [deleted]
- delfinom 3y ago
  Hah, you know, most crawlers were fine. The only one that actively DDOSed websites was fucking Yandex. It doesn't respect robots.txt and it will actively fight against any rate limits by spawning connections on new IPs the moment one is blocked
  - kayodelycaon 3y ago
    Search engine crawlers are a subset of crawlers. Sometimes you’re dealing with aggressive screen-scraping from competitors or various marketing tools. I’ve had to deal with these things easily bringing sites down.
    - tracker1 3y ago
      Same, and when more than half your links are dynamic search results, it can pile on and really bring things to a crawl. I worked on a fairly popular auto classifieds website, and more than 90% of traffic was various scrapers, and some were definitely a burden. Worse, is that it doesn't show up in analytics as it's not running client-js... Ironically equally bad was when bing started scraping with JS and it skewed google analytics. If all we had to deal with were the users, wouldn't need nearly the spend on the site. Started manually blocking some of the worst offenders.
  - NicoJuicy 3y ago
    ? I personally experienced many unbehaving crawlers/bots not respecting robots.txt and behaving like another user agent. Feel free to prove me wrong and disrupt cloudflare by only handling that use-case
    - NicoJuicy 3y ago
      Found an old reference of mine what I did to block crawlers/bots https://news.ycombinator.com/item?id=34101988 https://news.ycombinator.com/item?id=34101988 > Had a lot of spammers with Russian language. Implemented expanding xml-bombs, Google Captcha, hidden input fields and a couple of other things against bots. But the block on the russian language was most effective ( and since I was dogfooding it, I didn't see the harm at the time. But it's out of scope at this very moment, yes).
- everybodyknows 3y ago
  Website owner complains elsewhere in this topic of blocked users (country-specific): https://news.ycombinator.com/item?id=37050774 https://news.ycombinator.com/item?id=37050774
  - NicoJuicy 3y ago
    That's not a lot of information for such a complex subject. Eg. If you live in a dictatorship and use a VPN. You're traffic is together with a lot of people. The website owner can disable cloudflare their checks and that will leave their site unprotected. The choice of that is up to the website owner, no?
    - mlyle 3y ago
      It seems like a lot of people are pretending that because there's a contractual relationship between Cloudflare and many websites, that Cloudflare can't cause outsized harms to other parties. It really sucks for a third party to claim you're a bot and as a result you lose access to resources. And the potential for harm is only increasing. Worse still, the false positive rate appears unacceptably high (look at all the people here with substantial issues) and there's no recourse unless you get a highly voted thread on hacker news.
      - NicoJuicy 3y ago
        Tbh. There's a lot of technical folks, worldwide, here. That's a lot of edge cases in comparison. And their technical skills probably skews perception ( exotic browsers, behavior, VPN, Tor, ... ) As such, can you say, by indication, what percentage of web surfers "a lot" is? ( = Including non technical) I don't disagree that people can have problems. I'm just wondering how representative "a lot" in reality is.
        mlyle 3y ago
        Even if it's .01%, 1 out of 10,000 people losing a substantial fraction of utility without recourse sucks.
- sschueller 3y ago
  I don't agree. Maybe don't run WordPress that makes 100+ dB queries before first page load on some cheap $2 vps. There are all kinds of tools that you can easily deal with bots and the large DDOS your ISP can handle for you if you are willing to pay for it.
  - crote 3y ago
    > the large DDOS your ISP can handle for you if you are willing to pay for it That's exactly what people are paying Cloudflare for, because contrary to your local ISP they are actually competent at blocking a DDOS attack. People use services like Cloudflare exactly because they don't want to spend a fortune on complex infrastructure just to deal with abuse. Even a mostly-static page running on a reasonably-specced server can easily be overwhelmed by an attack. Why spend $1000 / month on hardware when you can spend $100 / month on Cloudflare's protection?
    - thedaly 3y ago
      > That's exactly what people are paying Cloudflare for Cloudflare actually provides this service for free (for simple use cases at least). I don't know how to come down on this issue. On one hand, I am against the centralization of cloudflare and the risks that come with it. On the other hand, cloudflare allows almost anyone to set up a simple website and serve it to large numbers of people with very little resources/cost and advanced protection from DDOS attacks.
      - tracker1 3y ago
        Similar mindset... also really intrigued with their developer tools as well. Workers, pages, D1, KV, etc. I was playing with a static site generator that deploys directly to a Cloudflare Pages setup, and it's lighning fast everywhere.
        NicoJuicy 3y ago
        Psst https://ai.cloudflare.com/ https://ai.cloudflare.com/ Note: light use-cases ( or should I say shared models?). Not heavy GPU tasks
- account42 3y ago
  > Everyone forgets that websites become almost unusable because of crawlers and bots. No one "forgets" this because it isn't true.
robertlagrant 3y ago
> Companies like Cloudflare, Google, Meta, etc are the reason anti-trust law exists Only if Cloudflare stops you moving to a competitor.
- warrenm 3y ago
  They [effectively] do prevent you from moving - they're guilty of vendor lock-in and running a passive, public man-in-the-middle attack platform (how many folks are using Cloudflare for DNS - either directly, or because their ISP is?)
adrr 3y ago
You don’t pay Cloudflare money nor are you customer. The customers are the sites that pay them money to protect their infrastructure. The customers have many choices in CDN/WAF providers and Cloudflare isn’t even largest, Akamai is. Fastest growing CDN is cloudfront. There is healthy competition so why would anti trust laws apply?
- zer8k 3y ago
  Cloudflare controls about 19% of websites. Of the websites that use a CDN, 80% of them use Cloudflare [0] (in a note at the bottom of link). [0] https://community.cloudflare.com/t/statistically-speaking-whats-the-percentage-of-total-sites-that-use-cf/372054 https://community.cloudflare.com/t/statistically-speaking-wh...
  - adrr 3y ago
    Traffic by bandwidth/number of users. Akamai powers Apple, Adobe, Microsoft, US government, Walmart, bunch of the major streamers.
- doctor_radium 3y ago
  Maybe some kind of public nuisance law applies then? Akamai knows their place and doesn't tell me to go pound sand when I'm trying to read somebody's obituary. Incidentally: legacy.com Ray ID: 7f3e7bad3afbb731 That's the difference to me.
cortesoft 3y ago
Cloudflare does not have nearly enough market share to be an anti-trust concern.
- zer8k 3y ago
  Cloudflare controls about 19% of websites. Of the websites that use a CDN, 80% of them use Cloudflare [0] (in a note at the bottom of link). [0] https://community.cloudflare.com/t/statistically-speaking-whats-the-percentage-of-total-sites-that-use-cf/372054 https://community.cloudflare.com/t/statistically-speaking-wh...
  - antonvs 3y ago
    Afaik, neither of those numbers legally constitute a monopoly and wouldn't qualify for antitrust action.
    - warrenm 3y ago
      They should
    - dontupvoteme 3y ago
      It will likely be up to the EU to step up again.
- deleted 3y ago
  [deleted]
shadowgovt 3y ago
The current internet is nothing like the internet of 20 years ago.

by zer8k 3y ago

NicoJuicy 3y ago

Everyone forgets that websites become almost unusable because of crawlers and bots. Website owners specifically choose for cloudflare to protect against this, it's not forced upon them by cloudflare.

realusername 3y ago
And who defines is an allowed crawler? Is Google the only allowed company to crawl the web? Isn't that the definition of monopoly? Could anybody still create a new search engine nowadays?
- NicoJuicy 3y ago
  A valid crawler is following robots.txt
  - realusername 3y ago
    That's not going to be enough to pass Cloudflare.
    - NicoJuicy 3y ago
      Based on what? https://developers.cloudflare.com/support/troubleshooting/general-troubleshooting/troubleshooting-crawl-errors/ https://developers.cloudflare.com/support/troubleshooting/ge...
      - buzer 3y ago
        https://developers.cloudflare.com/bots/reference/verified-bots-policy/ https://developers.cloudflare.com/bots/reference/verified-bo... https://radar.cloudflare.com/traffic/verified-bots https://radar.cloudflare.com/traffic/verified-bots https://blog.cloudflare.com/friendly-bots/ https://blog.cloudflare.com/friendly-bots/ > At Cloudflare, we manually “verify” good bots, so they don’t get blocked.
        gochi 3y ago
        >and honor robots.txt.
        NicoJuicy 3y ago
        Nice links! But cloudflare references robots.txt a lot ( which i mentioned before => to respect robots.txt) Additionally, they solve the authentication problem here. As a website owner that got bad crawlers ( that copied user agents), i just whitelisted Google's IP's and blocked all the other crawlers. It seems that cloudflare is actually fixing this problem and making competition for Google possible here. I recall an article that Bing actually circumvented robots.txt a bit, because site owners were only allowing Google and blocked all the rest => Bing. Which gave Google an unfair advantage ( searched for it, couldn't find it) Similar article to highlight the issue: https://www.fastcompany.com/90709672/the-little-known-reason-why-competing-with-google-is-so-hard https://www.fastcompany.com/90709672/the-little-known-reason... Our opinions about this seem to differ severely. Cloudflare actually enables good bots to start competing ( while respecting robots.txt).
  - deleted 3y ago
    [deleted]
delfinom 3y ago
Hah, you know, most crawlers were fine. The only one that actively DDOSed websites was fucking Yandex. It doesn't respect robots.txt and it will actively fight against any rate limits by spawning connections on new IPs the moment one is blocked
- kayodelycaon 3y ago
  Search engine crawlers are a subset of crawlers. Sometimes you’re dealing with aggressive screen-scraping from competitors or various marketing tools. I’ve had to deal with these things easily bringing sites down.
  - tracker1 3y ago
    Same, and when more than half your links are dynamic search results, it can pile on and really bring things to a crawl. I worked on a fairly popular auto classifieds website, and more than 90% of traffic was various scrapers, and some were definitely a burden. Worse, is that it doesn't show up in analytics as it's not running client-js... Ironically equally bad was when bing started scraping with JS and it skewed google analytics. If all we had to deal with were the users, wouldn't need nearly the spend on the site. Started manually blocking some of the worst offenders.
- NicoJuicy 3y ago
  ? I personally experienced many unbehaving crawlers/bots not respecting robots.txt and behaving like another user agent. Feel free to prove me wrong and disrupt cloudflare by only handling that use-case
  - NicoJuicy 3y ago
    Found an old reference of mine what I did to block crawlers/bots https://news.ycombinator.com/item?id=34101988 https://news.ycombinator.com/item?id=34101988 > Had a lot of spammers with Russian language. Implemented expanding xml-bombs, Google Captcha, hidden input fields and a couple of other things against bots. But the block on the russian language was most effective ( and since I was dogfooding it, I didn't see the harm at the time. But it's out of scope at this very moment, yes).
everybodyknows 3y ago
Website owner complains elsewhere in this topic of blocked users (country-specific): https://news.ycombinator.com/item?id=37050774 https://news.ycombinator.com/item?id=37050774
- NicoJuicy 3y ago
  That's not a lot of information for such a complex subject. Eg. If you live in a dictatorship and use a VPN. You're traffic is together with a lot of people. The website owner can disable cloudflare their checks and that will leave their site unprotected. The choice of that is up to the website owner, no?
  - mlyle 3y ago
    It seems like a lot of people are pretending that because there's a contractual relationship between Cloudflare and many websites, that Cloudflare can't cause outsized harms to other parties. It really sucks for a third party to claim you're a bot and as a result you lose access to resources. And the potential for harm is only increasing. Worse still, the false positive rate appears unacceptably high (look at all the people here with substantial issues) and there's no recourse unless you get a highly voted thread on hacker news.
    - NicoJuicy 3y ago
      Tbh. There's a lot of technical folks, worldwide, here. That's a lot of edge cases in comparison. And their technical skills probably skews perception ( exotic browsers, behavior, VPN, Tor, ... ) As such, can you say, by indication, what percentage of web surfers "a lot" is? ( = Including non technical) I don't disagree that people can have problems. I'm just wondering how representative "a lot" in reality is.
      - mlyle 3y ago
        Even if it's .01%, 1 out of 10,000 people losing a substantial fraction of utility without recourse sucks.
sschueller 3y ago
I don't agree. Maybe don't run WordPress that makes 100+ dB queries before first page load on some cheap $2 vps. There are all kinds of tools that you can easily deal with bots and the large DDOS your ISP can handle for you if you are willing to pay for it.
- crote 3y ago
  > the large DDOS your ISP can handle for you if you are willing to pay for it That's exactly what people are paying Cloudflare for, because contrary to your local ISP they are actually competent at blocking a DDOS attack. People use services like Cloudflare exactly because they don't want to spend a fortune on complex infrastructure just to deal with abuse. Even a mostly-static page running on a reasonably-specced server can easily be overwhelmed by an attack. Why spend $1000 / month on hardware when you can spend $100 / month on Cloudflare's protection?
  - thedaly 3y ago
    > That's exactly what people are paying Cloudflare for Cloudflare actually provides this service for free (for simple use cases at least). I don't know how to come down on this issue. On one hand, I am against the centralization of cloudflare and the risks that come with it. On the other hand, cloudflare allows almost anyone to set up a simple website and serve it to large numbers of people with very little resources/cost and advanced protection from DDOS attacks.
    - tracker1 3y ago
      Similar mindset... also really intrigued with their developer tools as well. Workers, pages, D1, KV, etc. I was playing with a static site generator that deploys directly to a Cloudflare Pages setup, and it's lighning fast everywhere.
      - NicoJuicy 3y ago
        Psst https://ai.cloudflare.com/ https://ai.cloudflare.com/ Note: light use-cases ( or should I say shared models?). Not heavy GPU tasks
account42 3y ago
> Everyone forgets that websites become almost unusable because of crawlers and bots. No one "forgets" this because it isn't true.

robertlagrant 3y ago

> Companies like Cloudflare, Google, Meta, etc are the reason anti-trust law exists Only if Cloudflare stops you moving to a competitor.

warrenm 3y ago
They [effectively] do prevent you from moving - they're guilty of vendor lock-in and running a passive, public man-in-the-middle attack platform (how many folks are using Cloudflare for DNS - either directly, or because their ISP is?)

adrr 3y ago

You don’t pay Cloudflare money nor are you customer. The customers are the sites that pay them money to protect their infrastructure. The customers have many choices in CDN/WAF providers and Cloudflare isn’t even largest, Akamai is. Fastest growing CDN is cloudfront. There is healthy competition so why would anti trust laws apply?

zer8k 3y ago
Cloudflare controls about 19% of websites. Of the websites that use a CDN, 80% of them use Cloudflare [0] (in a note at the bottom of link). [0] https://community.cloudflare.com/t/statistically-speaking-whats-the-percentage-of-total-sites-that-use-cf/372054 https://community.cloudflare.com/t/statistically-speaking-wh...
- adrr 3y ago
  Traffic by bandwidth/number of users. Akamai powers Apple, Adobe, Microsoft, US government, Walmart, bunch of the major streamers.
doctor_radium 3y ago
Maybe some kind of public nuisance law applies then? Akamai knows their place and doesn't tell me to go pound sand when I'm trying to read somebody's obituary. Incidentally: legacy.com Ray ID: 7f3e7bad3afbb731 That's the difference to me.

cortesoft 3y ago

Cloudflare does not have nearly enough market share to be an anti-trust concern.

zer8k 3y ago
Cloudflare controls about 19% of websites. Of the websites that use a CDN, 80% of them use Cloudflare [0] (in a note at the bottom of link). [0] https://community.cloudflare.com/t/statistically-speaking-whats-the-percentage-of-total-sites-that-use-cf/372054 https://community.cloudflare.com/t/statistically-speaking-wh...
- antonvs 3y ago
  Afaik, neither of those numbers legally constitute a monopoly and wouldn't qualify for antitrust action.
  - warrenm 3y ago
    They should
  - dontupvoteme 3y ago
    It will likely be up to the EU to step up again.
deleted 3y ago
[deleted]

shadowgovt 3y ago

The current internet is nothing like the internet of 20 years ago.