8 ms·

Cloudflare is both a great thing and a terrible thing that has happened to the internet in recent years. Great in that they have a fantastic UI to add your sit

Cloudflare is both a great thing and a terrible thing that has happened to the internet in recent years. Great in that they have a fantastic UI to add your site in, basically shielding the average user from attacks. Bad from a standpoint of that now only Google, Bing, and maybe other big search engines have the capabilities to actually crawl the internet now. I don't see us getting a massive innovation in search on the internet now that Google has such a massive foothold, and companies like Cloudflare stop innovation from happening.

jgrahamc 5y ago
I don't see us getting a massive innovation in search on the internet now that Google has such a massive foothold, and companies like Cloudflare stop innovation from happening. How are we "stopping search innovation"?
- o-__-o 5y ago
  How does my startup crawl Cloudflare sites without paying a hefty fee to Cloudflare?
  - jgrahamc 5y ago
    https://news.ycombinator.com/item?id=27153635 https://news.ycombinator.com/item?id=27153635
    - deleted 5y ago
      [deleted]
    - o-__-o 5y ago
      This will scale wonderfully!
      - jgrahamc 5y ago
        No, what scales is us making our DDoS and bot detection not disrupt the crawling of legit search engines that respect robots.txt, don't crawl at ridiculous speeds, don't do dumb stuff like pretend they are the Googlebot. We have teams who work on that. You can read more here: https://blog.cloudflare.com/tag/bots/ https://blog.cloudflare.com/tag/bots/ But let's suppose someone is building a new cool search engine and our ML stuff is blocking them. Then... contact us/me.
        o-__-o 5y ago
        So for my startup to crawl sites I must now adhere to Cloudflare’s Requirements of the Web(TM) or reach out to individual engineer, who may leave at any moment. Gotcha (but Google is allowed because Google was first to market)
        midev 5y ago
        Why would you possibly think you can do whatever you want to someone else's site? Yes, you must adhere to the controls that site administrators put in place, like Cloudflare.... You don't get to blast my site with requests, just because you want to...
        o-__-o 5y ago
        (a) Who said I was blasting your site with requests? Cloudflare stops much more than just blasts (b) But you’re a-ok with Google doing this. Gated communities aren’t really good for anybody but I see what you are saying.
        midev 5y ago
        Gated communities are great. They lower the risk of crime significantly: https://www.sciencedaily.com/releases/2013/03/130320115113.htm https://www.sciencedaily.com/releases/2013/03/130320115113.h... The same is true online. Apple's walled garden has kept hundreds of millions of people safe on their device. It's why iOS malware isn't a thing. > Cloudflare stops much more than just blasts Exactly. There's even more benefit to Cloudflare than just DDoS. Captcha's for stopping credential stuffing, for example.
      - 77pt77 5y ago
        It seems to be by design.
- navanchauhan 5y ago
  Crawling a Cloudflare powered website is basically impossible without needing to do some bodges as to how to crawl it. How can you expect someone to crawl a bunch of websites if they are actively blocked from accessing it? Now, you might say users can whitelist bots in their robots.txt file but then again will the person creating the engine individually ask companies to allow them to crawl? Also, slightly unrelated but Cloudflare protected websites are almost impossible to access via tor, the captcha never succeeds.
  - jgrahamc 5y ago
    If you are building a search engine and getting blocked you can always contact me and I'll make sure that the teams that work on bot detection and DDoS are aware. We would like to know because we should not be blocking a legit crawler like this.
    - throwaway3699 5y ago
      For every developer that sees this message, a few dozen will have given up.
      - new_guy 5y ago
        Exactly this. Cloudflare actively blocks legit crawlers. It shouldn't be dependent on seeing some random hn comment from some random at cloudflare to get that fixed.
    - oefrha 5y ago
      Say I’m interested in building a small scale domain-specific search engine and only just started development. There’s no prototype yet and may never be. In this situation, how do you determine it’s a legit crawler? And what about crawlers with even more limited scopes (targeting only a handful of sites) that they can’t possibly be called search engines? Are they ever considered legit?
      - jgrahamc 5y ago
- SXX 5y ago
  For instance there is no way for distributed search engines to work with CloudFlare. No, "contact me and we'll help" is not always a solution.
  - jgrahamc 5y ago
    Please explain the problem (here or via email to me).
    - catillac 5y ago
      I’m really sorry, but you appear to be the CTO of Cloudflare, which makes your not knowing the ins and outs of the problem already and basic questioning of it seem like sealioning.[1] [1] https://en.m.wikipedia.org/wiki/Sealioning https://en.m.wikipedia.org/wiki/Sealioning
      - jgrahamc 5y ago
        I do not understand what the parent means by a "distributed search engine" and I do not know what problem they are facing.
        viraptor 5y ago
        https://yacy.net/ https://yacy.net/ for example. Each interested node does indexing and serving some chunk of the results. Or in practice - each node quickly runs into a CloudFlare captcha preventing it from indexing content for a few hours/days. Since CF fronts a lot of the useful internet these days, it means it's effectively working against distributed indexing with its current captcha solution.
        jgrahamc 5y ago
        Thanks. I'll bring this to the attention of the bots and DDoS teams.
        5y ago
- surround 5y ago
  I trust that cloudflare will act responsibly in allowing small search engines through, but I really, really would rather not have to trust cloudflare. I don't believe that any organization can or will always act responsibly, which is why it's concerning that cloudflare controls so much of the internet.
  - EvanAnderson 5y ago
    Yes. This. I believe that John Graham-Cumming is genuine in his statements in this thread re: "contact me if you're running afoul of our controls", for example. If he leaves Cloudflare, Cloudflare "turns evil", etc, then that's all out the window. Individual companies having so much power gives me the willies.
- thejosh 5y ago
  Hi! Thanks for taking the time to reply. You mentioned about "legit" crawlers, what defines a "legit" crawler in the eyes of Cloudflare, and what happens when Cloudflare suddenly decides it does not want to honour that "agreement"? What happens if/when Cloudflare is sold, or the contact who greenlit these smaller "legit" crawlers moves on and decides that it no longer agrees with said website anymore? Is a price comparison site a "legit" crawler? What defines a "bot" vs a "crawler" in the eyes of Cloudflare? Would you need to notify your customers that you now also allow additional crawlers access to their sites, or would they need to opt into it via the Cloudflare dashboard? What happens when you have a falling out with my said company (it happens, relationships sour) and suddenly we can't make contact, then suddenly customers websites aren't being scraped because we're bots?
  - dannyw 5y ago
    There are a lot of hypotheticals here. I think you'll convince CloudFlare, and their customers, if you could name names and mention specific examples? If you are a price comparison site getting blocked by cloudflare, site owners may be losing sales, and that's good feedback.
    - jopsen 5y ago
      Or site owners may actively want to block a price comparison site.. Depending on the industry, etc..
      - thejosh 5y ago
        Agreed on this point, and most companies who would want to do these sorts of things would restrict it even if Cloudflare didn't exist. I really can't think of a good solution. But that's the tricky position Cloudflare is in - how does it balance everything.
    - hysan 5y ago
      Price comparison example: https://shucks.top/ https://shucks.top/ sometimes gets blocked by cloudflare. Most recent was getting blocked from checking B&H.
- pdimitar 5y ago
  By being gatekeepers on which website crawling is okay and which is not. No such filters should exist. Is it really that awfully bad without anything but basic filters (ban an IP for flooding)? Are there like, operations that try and spam every single Cloudflare-hosted website 24/7? Legitimately curious if your anti-bot measures come from actual bad experience with the internet or is it just a liability limitation move? (Namely to reduce potential suing surface by angry data owners and/or three-letter agencies.) --- Basically, if I am experimenting with a basic crawling program and I hit websites A and B 20 times each in a space of one hour, is that really deserving of a captcha or extra auth methods? Not flaming but I am really curious. Do you have any data and rationale posted somewhere that go into deeper detail about why Cloudflare's bot detection is how it is?
  - fivre 5y ago
    Cloudflare's customers request and then enable those features. Cloudflare itself doesn't give a damn about that traffic; they have bandwidth to spare. They will, however, happily sell tools to people that do care. That isn't to say that the customers are savvy and have a good understanding of different types of automated traffic and which automated traffic is harmful and which is benign. Many have a quite naive understanding that doesn't extend beyond "bots = bad, unless it's Google" and dial protection settings to the max for no good reason.
nabla9 5y ago
There was once cs professor who claimed that the internet does not scale around 2000. Either things choke up or you need huge investments into networks. It turns out that he was right, sort of. The vanilla attach server into the internet, server-to-client IP-network is pretty much dead. It has been replaced with CDN's , private delivery networks, cache on top of cache. Cloudfare, Amazon, Google and MS are the connection points for the IP. Their internal network infrastructure transfer most of the data.
- 542458 5y ago
  Is it pretty much dead? Yeah, if you’re moving FAANG level traffic you need something more fancy than LAMP + an internet connection, but I’ve seen dozens and dozens of sites with a plain old no-cdn, no-pdn, LAMP tech stack. Working with startups might bias your view - lots of companies are running extremely boring setups and they work just great.
  - nabla9 5y ago
    Respectfully, I think you miss the point. The fact that 98% of traffic goes trough this new infrastructure, allows some still plug their server to the net raw and their traffic still gets trough.
    - starfallg 5y ago
      >The fact that 98% of traffic goes trough this new infrastructure Doesn't mean that 98% of the value of the Internet results from this traffic. Even if you discount all of the web, there still lots of applications using the federated model (e.g. SMTP) or peer-to-peer (e.g. Crypto, VOIP), that require end-to-end connectivity.
      - nabla9 5y ago
        You expand this discussion into completely new direction. My original point is about capacity and if the old internet could work today. It seems like I'm correct, but I'm not so sure. I would like to see other opinions. Every response so far is "there exists". The real issue is if the internet could do everything without caching data near the edge.
ex_amazon_sde 5y ago
...also Cloudflare has been a disaster for Tor. It really harms Tor's usability.
Beaver117 5y ago
Maybe I'm cynical, but I don't see any innovation to be done in search. Google results have become much less useful over the past few years. If they cannot solve search with basically unlimited resources, how is a tiny company going to? 1. Filtering ever increasing trillions of spam/clickbait pages 2. Figuring out which results are useful information vs corporates trying to sell something. Those problems are not solveable by a couple guys in a garage
- spiderfarmer 5y ago
  Maybe not, but the garage guys should look into becoming the best search engine in a niche and expand from there.
- JeremyBanks 5y ago
  It's not clear to me that Google still gives a fuck about solving search/organizing the world's information and make it useful. The mess they've incentivized the web to become is very profitable for them.
- SXX 5y ago
  > If they cannot solve search with basically unlimited resources, how is a tiny company going to? Reminder: Google Is An Ad Company. Are you sure they actually want to solve search? Their primary interest is to be that corporation selling you something.
- janeroe 5y ago
  Maybe I'm cynical, but I don't see any innovation to be done in {AREANAME}. {PRODUCTNAME} has become much less useful over the past few years. If {MONOPOLISTNAME} cannot solve {AREANAME} with basically unlimited resources, how a tiny company going to? Nice pasta, can't believe someone would use it unironically. Maybe I'm cynical, but I don't see any progress to be done in government transparency. USA's government has become way more overreaching and much less transparent over the decades. If USA cannot solve this issue, how a small country going to?
fierro 5y ago
This feature is enabled at the behest of the site owner. I feel like site owners and operators should decide who gets to visit their site. Am I missing something obvious here?

jgrahamc 5y ago

I don't see us getting a massive innovation in search on the internet now that Google has such a massive foothold, and companies like Cloudflare stop innovation from happening. How are we "stopping search innovation"?

o-__-o 5y ago
How does my startup crawl Cloudflare sites without paying a hefty fee to Cloudflare?
- jgrahamc 5y ago
  https://news.ycombinator.com/item?id=27153635 https://news.ycombinator.com/item?id=27153635
  - deleted 5y ago
    [deleted]
  - o-__-o 5y ago
    This will scale wonderfully!
    - jgrahamc 5y ago
      No, what scales is us making our DDoS and bot detection not disrupt the crawling of legit search engines that respect robots.txt, don't crawl at ridiculous speeds, don't do dumb stuff like pretend they are the Googlebot. We have teams who work on that. You can read more here: https://blog.cloudflare.com/tag/bots/ https://blog.cloudflare.com/tag/bots/ But let's suppose someone is building a new cool search engine and our ML stuff is blocking them. Then... contact us/me.
      - o-__-o 5y ago
        So for my startup to crawl sites I must now adhere to Cloudflare’s Requirements of the Web(TM) or reach out to individual engineer, who may leave at any moment. Gotcha (but Google is allowed because Google was first to market)
        midev 5y ago
        Why would you possibly think you can do whatever you want to someone else's site? Yes, you must adhere to the controls that site administrators put in place, like Cloudflare.... You don't get to blast my site with requests, just because you want to...
        o-__-o 5y ago
        (a) Who said I was blasting your site with requests? Cloudflare stops much more than just blasts (b) But you’re a-ok with Google doing this. Gated communities aren’t really good for anybody but I see what you are saying.
        midev 5y ago
        Gated communities are great. They lower the risk of crime significantly: https://www.sciencedaily.com/releases/2013/03/130320115113.htm https://www.sciencedaily.com/releases/2013/03/130320115113.h... The same is true online. Apple's walled garden has kept hundreds of millions of people safe on their device. It's why iOS malware isn't a thing. > Cloudflare stops much more than just blasts Exactly. There's even more benefit to Cloudflare than just DDoS. Captcha's for stopping credential stuffing, for example.
    - 77pt77 5y ago
      It seems to be by design.
navanchauhan 5y ago
Crawling a Cloudflare powered website is basically impossible without needing to do some bodges as to how to crawl it. How can you expect someone to crawl a bunch of websites if they are actively blocked from accessing it? Now, you might say users can whitelist bots in their robots.txt file but then again will the person creating the engine individually ask companies to allow them to crawl? Also, slightly unrelated but Cloudflare protected websites are almost impossible to access via tor, the captcha never succeeds.
- jgrahamc 5y ago
  If you are building a search engine and getting blocked you can always contact me and I'll make sure that the teams that work on bot detection and DDoS are aware. We would like to know because we should not be blocking a legit crawler like this.
  - throwaway3699 5y ago
    For every developer that sees this message, a few dozen will have given up.
    - new_guy 5y ago
      Exactly this. Cloudflare actively blocks legit crawlers. It shouldn't be dependent on seeing some random hn comment from some random at cloudflare to get that fixed.
  - oefrha 5y ago
    Say I’m interested in building a small scale domain-specific search engine and only just started development. There’s no prototype yet and may never be. In this situation, how do you determine it’s a legit crawler? And what about crawlers with even more limited scopes (targeting only a handful of sites) that they can’t possibly be called search engines? Are they ever considered legit?
    - jgrahamc 5y ago
SXX 5y ago
For instance there is no way for distributed search engines to work with CloudFlare. No, "contact me and we'll help" is not always a solution.
- jgrahamc 5y ago
  Please explain the problem (here or via email to me).
  - catillac 5y ago
    I’m really sorry, but you appear to be the CTO of Cloudflare, which makes your not knowing the ins and outs of the problem already and basic questioning of it seem like sealioning.[1] [1] https://en.m.wikipedia.org/wiki/Sealioning https://en.m.wikipedia.org/wiki/Sealioning
    - jgrahamc 5y ago
      I do not understand what the parent means by a "distributed search engine" and I do not know what problem they are facing.
      - viraptor 5y ago
        https://yacy.net/ https://yacy.net/ for example. Each interested node does indexing and serving some chunk of the results. Or in practice - each node quickly runs into a CloudFlare captcha preventing it from indexing content for a few hours/days. Since CF fronts a lot of the useful internet these days, it means it's effectively working against distributed indexing with its current captcha solution.
        jgrahamc 5y ago
        Thanks. I'll bring this to the attention of the bots and DDoS teams.
        5y ago
surround 5y ago
I trust that cloudflare will act responsibly in allowing small search engines through, but I really, really would rather not have to trust cloudflare. I don't believe that any organization can or will always act responsibly, which is why it's concerning that cloudflare controls so much of the internet.
- EvanAnderson 5y ago
  Yes. This. I believe that John Graham-Cumming is genuine in his statements in this thread re: "contact me if you're running afoul of our controls", for example. If he leaves Cloudflare, Cloudflare "turns evil", etc, then that's all out the window. Individual companies having so much power gives me the willies.
thejosh 5y ago
Hi! Thanks for taking the time to reply. You mentioned about "legit" crawlers, what defines a "legit" crawler in the eyes of Cloudflare, and what happens when Cloudflare suddenly decides it does not want to honour that "agreement"? What happens if/when Cloudflare is sold, or the contact who greenlit these smaller "legit" crawlers moves on and decides that it no longer agrees with said website anymore? Is a price comparison site a "legit" crawler? What defines a "bot" vs a "crawler" in the eyes of Cloudflare? Would you need to notify your customers that you now also allow additional crawlers access to their sites, or would they need to opt into it via the Cloudflare dashboard? What happens when you have a falling out with my said company (it happens, relationships sour) and suddenly we can't make contact, then suddenly customers websites aren't being scraped because we're bots?
- dannyw 5y ago
  There are a lot of hypotheticals here. I think you'll convince CloudFlare, and their customers, if you could name names and mention specific examples? If you are a price comparison site getting blocked by cloudflare, site owners may be losing sales, and that's good feedback.
  - jopsen 5y ago
    Or site owners may actively want to block a price comparison site.. Depending on the industry, etc..
    - thejosh 5y ago
      Agreed on this point, and most companies who would want to do these sorts of things would restrict it even if Cloudflare didn't exist. I really can't think of a good solution. But that's the tricky position Cloudflare is in - how does it balance everything.
  - hysan 5y ago
    Price comparison example: https://shucks.top/ https://shucks.top/ sometimes gets blocked by cloudflare. Most recent was getting blocked from checking B&H.
pdimitar 5y ago
By being gatekeepers on which website crawling is okay and which is not. No such filters should exist. Is it really that awfully bad without anything but basic filters (ban an IP for flooding)? Are there like, operations that try and spam every single Cloudflare-hosted website 24/7? Legitimately curious if your anti-bot measures come from actual bad experience with the internet or is it just a liability limitation move? (Namely to reduce potential suing surface by angry data owners and/or three-letter agencies.) --- Basically, if I am experimenting with a basic crawling program and I hit websites A and B 20 times each in a space of one hour, is that really deserving of a captcha or extra auth methods? Not flaming but I am really curious. Do you have any data and rationale posted somewhere that go into deeper detail about why Cloudflare's bot detection is how it is?
- fivre 5y ago
  Cloudflare's customers request and then enable those features. Cloudflare itself doesn't give a damn about that traffic; they have bandwidth to spare. They will, however, happily sell tools to people that do care. That isn't to say that the customers are savvy and have a good understanding of different types of automated traffic and which automated traffic is harmful and which is benign. Many have a quite naive understanding that doesn't extend beyond "bots = bad, unless it's Google" and dial protection settings to the max for no good reason.

nabla9 5y ago

There was once cs professor who claimed that the internet does not scale around 2000. Either things choke up or you need huge investments into networks. It turns out that he was right, sort of. The vanilla attach server into the internet, server-to-client IP-network is pretty much dead. It has been replaced with CDN's , private delivery networks, cache on top of cache. Cloudfare, Amazon, Google and MS are the connection points for the IP. Their internal network infrastructure transfer most of the data.

542458 5y ago
Is it pretty much dead? Yeah, if you’re moving FAANG level traffic you need something more fancy than LAMP + an internet connection, but I’ve seen dozens and dozens of sites with a plain old no-cdn, no-pdn, LAMP tech stack. Working with startups might bias your view - lots of companies are running extremely boring setups and they work just great.
- nabla9 5y ago
  Respectfully, I think you miss the point. The fact that 98% of traffic goes trough this new infrastructure, allows some still plug their server to the net raw and their traffic still gets trough.
  - starfallg 5y ago
    >The fact that 98% of traffic goes trough this new infrastructure Doesn't mean that 98% of the value of the Internet results from this traffic. Even if you discount all of the web, there still lots of applications using the federated model (e.g. SMTP) or peer-to-peer (e.g. Crypto, VOIP), that require end-to-end connectivity.
    - nabla9 5y ago
      You expand this discussion into completely new direction. My original point is about capacity and if the old internet could work today. It seems like I'm correct, but I'm not so sure. I would like to see other opinions. Every response so far is "there exists". The real issue is if the internet could do everything without caching data near the edge.

ex_amazon_sde 5y ago

...also Cloudflare has been a disaster for Tor. It really harms Tor's usability.

Beaver117 5y ago

Maybe I'm cynical, but I don't see any innovation to be done in search. Google results have become much less useful over the past few years. If they cannot solve search with basically unlimited resources, how is a tiny company going to? 1. Filtering ever increasing trillions of spam/clickbait pages 2. Figuring out which results are useful information vs corporates trying to sell something. Those problems are not solveable by a couple guys in a garage

spiderfarmer 5y ago
Maybe not, but the garage guys should look into becoming the best search engine in a niche and expand from there.
JeremyBanks 5y ago
It's not clear to me that Google still gives a fuck about solving search/organizing the world's information and make it useful. The mess they've incentivized the web to become is very profitable for them.
SXX 5y ago
> If they cannot solve search with basically unlimited resources, how is a tiny company going to? Reminder: Google Is An Ad Company. Are you sure they actually want to solve search? Their primary interest is to be that corporation selling you something.
janeroe 5y ago
Maybe I'm cynical, but I don't see any innovation to be done in {AREANAME}. {PRODUCTNAME} has become much less useful over the past few years. If {MONOPOLISTNAME} cannot solve {AREANAME} with basically unlimited resources, how a tiny company going to? Nice pasta, can't believe someone would use it unironically. Maybe I'm cynical, but I don't see any progress to be done in government transparency. USA's government has become way more overreaching and much less transparent over the decades. If USA cannot solve this issue, how a small country going to?

fierro 5y ago

This feature is enabled at the behest of the site owner. I feel like site owners and operators should decide who gets to visit their site. Am I missing something obvious here?