7 ms·

I don't see us getting a massive innovation in search on the internet now that Google has such a massive foothold, and companies like Cloudflare stop innovation

I don't see us getting a massive innovation in search on the internet now that Google has such a massive foothold, and companies like Cloudflare stop innovation from happening. How are we "stopping search innovation"?

o-__-o 5y ago
How does my startup crawl Cloudflare sites without paying a hefty fee to Cloudflare?
- jgrahamc 5y ago
  https://news.ycombinator.com/item?id=27153635 https://news.ycombinator.com/item?id=27153635
  - deleted 5y ago
    [deleted]
  - o-__-o 5y ago
    This will scale wonderfully!
    - jgrahamc 5y ago
      No, what scales is us making our DDoS and bot detection not disrupt the crawling of legit search engines that respect robots.txt, don't crawl at ridiculous speeds, don't do dumb stuff like pretend they are the Googlebot. We have teams who work on that. You can read more here: https://blog.cloudflare.com/tag/bots/ https://blog.cloudflare.com/tag/bots/ But let's suppose someone is building a new cool search engine and our ML stuff is blocking them. Then... contact us/me.
      - o-__-o 5y ago
        So for my startup to crawl sites I must now adhere to Cloudflare’s Requirements of the Web(TM) or reach out to individual engineer, who may leave at any moment. Gotcha (but Google is allowed because Google was first to market)
        midev 5y ago
        Why would you possibly think you can do whatever you want to someone else's site? Yes, you must adhere to the controls that site administrators put in place, like Cloudflare.... You don't get to blast my site with requests, just because you want to...
        o-__-o 5y ago
        (a) Who said I was blasting your site with requests? Cloudflare stops much more than just blasts (b) But you’re a-ok with Google doing this. Gated communities aren’t really good for anybody but I see what you are saying.
        midev 5y ago
        Gated communities are great. They lower the risk of crime significantly: https://www.sciencedaily.com/releases/2013/03/130320115113.htm https://www.sciencedaily.com/releases/2013/03/130320115113.h... The same is true online. Apple's walled garden has kept hundreds of millions of people safe on their device. It's why iOS malware isn't a thing. > Cloudflare stops much more than just blasts Exactly. There's even more benefit to Cloudflare than just DDoS. Captcha's for stopping credential stuffing, for example.
        o-__-o 5y ago
        ..Didn’t realize my startup search engine stuffed credentials :( But hey if I pay Cloudflare enough, then I’ll get to blast your site and possibly stuff creds at the same time :/
      - timlardner 5y ago
        That doesn't sound unreasonable. Out of interest, what would you consider a ridiculous speed to be crawling at?
        LinuxBender 5y ago
        I can't speak for Cloudflare, but crawling speed should be dictated by the site owner via the robots.txt crawl-delay. [1] A site owner could also rate-limit unauthenticated requests by IP via the cloudflare header using a 429 too many requests error page. [1] - https://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive https://en.wikipedia.org/wiki/Robots_exclusion_standard#Craw...
        o-__-o 5y ago
        This here is the problem. It’s a new time no one wants to be Rfc compliant, just go behind a service and problem is solved. So no problem, time to move on web search is no longer exciting
    - 77pt77 5y ago
      It seems to be by design.
navanchauhan 5y ago
Crawling a Cloudflare powered website is basically impossible without needing to do some bodges as to how to crawl it. How can you expect someone to crawl a bunch of websites if they are actively blocked from accessing it? Now, you might say users can whitelist bots in their robots.txt file but then again will the person creating the engine individually ask companies to allow them to crawl? Also, slightly unrelated but Cloudflare protected websites are almost impossible to access via tor, the captcha never succeeds.
- jgrahamc 5y ago
  If you are building a search engine and getting blocked you can always contact me and I'll make sure that the teams that work on bot detection and DDoS are aware. We would like to know because we should not be blocking a legit crawler like this.
  - throwaway3699 5y ago
    For every developer that sees this message, a few dozen will have given up.
    - new_guy 5y ago
      Exactly this. Cloudflare actively blocks legit crawlers. It shouldn't be dependent on seeing some random hn comment from some random at cloudflare to get that fixed.
  - oefrha 5y ago
    Say I’m interested in building a small scale domain-specific search engine and only just started development. There’s no prototype yet and may never be. In this situation, how do you determine it’s a legit crawler? And what about crawlers with even more limited scopes (targeting only a handful of sites) that they can’t possibly be called search engines? Are they ever considered legit?
    - jgrahamc 5y ago
SXX 5y ago
For instance there is no way for distributed search engines to work with CloudFlare. No, "contact me and we'll help" is not always a solution.
- jgrahamc 5y ago
  Please explain the problem (here or via email to me).
  - catillac 5y ago
    I’m really sorry, but you appear to be the CTO of Cloudflare, which makes your not knowing the ins and outs of the problem already and basic questioning of it seem like sealioning.[1] [1] https://en.m.wikipedia.org/wiki/Sealioning https://en.m.wikipedia.org/wiki/Sealioning
    - jgrahamc 5y ago
      I do not understand what the parent means by a "distributed search engine" and I do not know what problem they are facing.
      - viraptor 5y ago
        https://yacy.net/ https://yacy.net/ for example. Each interested node does indexing and serving some chunk of the results. Or in practice - each node quickly runs into a CloudFlare captcha preventing it from indexing content for a few hours/days. Since CF fronts a lot of the useful internet these days, it means it's effectively working against distributed indexing with its current captcha solution.
        jgrahamc 5y ago
        Thanks. I'll bring this to the attention of the bots and DDoS teams.
        5y ago
surround 5y ago
I trust that cloudflare will act responsibly in allowing small search engines through, but I really, really would rather not have to trust cloudflare. I don't believe that any organization can or will always act responsibly, which is why it's concerning that cloudflare controls so much of the internet.
- EvanAnderson 5y ago
  Yes. This. I believe that John Graham-Cumming is genuine in his statements in this thread re: "contact me if you're running afoul of our controls", for example. If he leaves Cloudflare, Cloudflare "turns evil", etc, then that's all out the window. Individual companies having so much power gives me the willies.
thejosh 5y ago
Hi! Thanks for taking the time to reply. You mentioned about "legit" crawlers, what defines a "legit" crawler in the eyes of Cloudflare, and what happens when Cloudflare suddenly decides it does not want to honour that "agreement"? What happens if/when Cloudflare is sold, or the contact who greenlit these smaller "legit" crawlers moves on and decides that it no longer agrees with said website anymore? Is a price comparison site a "legit" crawler? What defines a "bot" vs a "crawler" in the eyes of Cloudflare? Would you need to notify your customers that you now also allow additional crawlers access to their sites, or would they need to opt into it via the Cloudflare dashboard? What happens when you have a falling out with my said company (it happens, relationships sour) and suddenly we can't make contact, then suddenly customers websites aren't being scraped because we're bots?
- dannyw 5y ago
  There are a lot of hypotheticals here. I think you'll convince CloudFlare, and their customers, if you could name names and mention specific examples? If you are a price comparison site getting blocked by cloudflare, site owners may be losing sales, and that's good feedback.
  - jopsen 5y ago
    Or site owners may actively want to block a price comparison site.. Depending on the industry, etc..
    - thejosh 5y ago
      Agreed on this point, and most companies who would want to do these sorts of things would restrict it even if Cloudflare didn't exist. I really can't think of a good solution. But that's the tricky position Cloudflare is in - how does it balance everything.
  - hysan 5y ago
    Price comparison example: https://shucks.top/ https://shucks.top/ sometimes gets blocked by cloudflare. Most recent was getting blocked from checking B&H.
pdimitar 5y ago
By being gatekeepers on which website crawling is okay and which is not. No such filters should exist. Is it really that awfully bad without anything but basic filters (ban an IP for flooding)? Are there like, operations that try and spam every single Cloudflare-hosted website 24/7? Legitimately curious if your anti-bot measures come from actual bad experience with the internet or is it just a liability limitation move? (Namely to reduce potential suing surface by angry data owners and/or three-letter agencies.) --- Basically, if I am experimenting with a basic crawling program and I hit websites A and B 20 times each in a space of one hour, is that really deserving of a captcha or extra auth methods? Not flaming but I am really curious. Do you have any data and rationale posted somewhere that go into deeper detail about why Cloudflare's bot detection is how it is?
- fivre 5y ago
  Cloudflare's customers request and then enable those features. Cloudflare itself doesn't give a damn about that traffic; they have bandwidth to spare. They will, however, happily sell tools to people that do care. That isn't to say that the customers are savvy and have a good understanding of different types of automated traffic and which automated traffic is harmful and which is benign. Many have a quite naive understanding that doesn't extend beyond "bots = bad, unless it's Google" and dial protection settings to the max for no good reason.

o-__-o 5y ago

How does my startup crawl Cloudflare sites without paying a hefty fee to Cloudflare?

jgrahamc 5y ago
https://news.ycombinator.com/item?id=27153635 https://news.ycombinator.com/item?id=27153635
- deleted 5y ago
  [deleted]
- o-__-o 5y ago
  This will scale wonderfully!
  - jgrahamc 5y ago
    No, what scales is us making our DDoS and bot detection not disrupt the crawling of legit search engines that respect robots.txt, don't crawl at ridiculous speeds, don't do dumb stuff like pretend they are the Googlebot. We have teams who work on that. You can read more here: https://blog.cloudflare.com/tag/bots/ https://blog.cloudflare.com/tag/bots/ But let's suppose someone is building a new cool search engine and our ML stuff is blocking them. Then... contact us/me.
    - o-__-o 5y ago
      So for my startup to crawl sites I must now adhere to Cloudflare’s Requirements of the Web(TM) or reach out to individual engineer, who may leave at any moment. Gotcha (but Google is allowed because Google was first to market)
      - midev 5y ago
        Why would you possibly think you can do whatever you want to someone else's site? Yes, you must adhere to the controls that site administrators put in place, like Cloudflare.... You don't get to blast my site with requests, just because you want to...
        o-__-o 5y ago
        (a) Who said I was blasting your site with requests? Cloudflare stops much more than just blasts (b) But you’re a-ok with Google doing this. Gated communities aren’t really good for anybody but I see what you are saying.
        midev 5y ago
        Gated communities are great. They lower the risk of crime significantly: https://www.sciencedaily.com/releases/2013/03/130320115113.htm https://www.sciencedaily.com/releases/2013/03/130320115113.h... The same is true online. Apple's walled garden has kept hundreds of millions of people safe on their device. It's why iOS malware isn't a thing. > Cloudflare stops much more than just blasts Exactly. There's even more benefit to Cloudflare than just DDoS. Captcha's for stopping credential stuffing, for example.
        o-__-o 5y ago
        ..Didn’t realize my startup search engine stuffed credentials :( But hey if I pay Cloudflare enough, then I’ll get to blast your site and possibly stuff creds at the same time :/
    - timlardner 5y ago
      That doesn't sound unreasonable. Out of interest, what would you consider a ridiculous speed to be crawling at?
      - LinuxBender 5y ago
        I can't speak for Cloudflare, but crawling speed should be dictated by the site owner via the robots.txt crawl-delay. [1] A site owner could also rate-limit unauthenticated requests by IP via the cloudflare header using a 429 too many requests error page. [1] - https://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive https://en.wikipedia.org/wiki/Robots_exclusion_standard#Craw...
        o-__-o 5y ago
        This here is the problem. It’s a new time no one wants to be Rfc compliant, just go behind a service and problem is solved. So no problem, time to move on web search is no longer exciting
  - 77pt77 5y ago
    It seems to be by design.

navanchauhan 5y ago

Crawling a Cloudflare powered website is basically impossible without needing to do some bodges as to how to crawl it. How can you expect someone to crawl a bunch of websites if they are actively blocked from accessing it? Now, you might say users can whitelist bots in their robots.txt file but then again will the person creating the engine individually ask companies to allow them to crawl? Also, slightly unrelated but Cloudflare protected websites are almost impossible to access via tor, the captcha never succeeds.

jgrahamc 5y ago
If you are building a search engine and getting blocked you can always contact me and I'll make sure that the teams that work on bot detection and DDoS are aware. We would like to know because we should not be blocking a legit crawler like this.
- throwaway3699 5y ago
  For every developer that sees this message, a few dozen will have given up.
  - new_guy 5y ago
    Exactly this. Cloudflare actively blocks legit crawlers. It shouldn't be dependent on seeing some random hn comment from some random at cloudflare to get that fixed.
- oefrha 5y ago
  Say I’m interested in building a small scale domain-specific search engine and only just started development. There’s no prototype yet and may never be. In this situation, how do you determine it’s a legit crawler? And what about crawlers with even more limited scopes (targeting only a handful of sites) that they can’t possibly be called search engines? Are they ever considered legit?
  - jgrahamc 5y ago

SXX 5y ago

For instance there is no way for distributed search engines to work with CloudFlare. No, "contact me and we'll help" is not always a solution.

jgrahamc 5y ago
Please explain the problem (here or via email to me).
- catillac 5y ago
  I’m really sorry, but you appear to be the CTO of Cloudflare, which makes your not knowing the ins and outs of the problem already and basic questioning of it seem like sealioning.[1] [1] https://en.m.wikipedia.org/wiki/Sealioning https://en.m.wikipedia.org/wiki/Sealioning
  - jgrahamc 5y ago
    I do not understand what the parent means by a "distributed search engine" and I do not know what problem they are facing.
    - viraptor 5y ago
      https://yacy.net/ https://yacy.net/ for example. Each interested node does indexing and serving some chunk of the results. Or in practice - each node quickly runs into a CloudFlare captcha preventing it from indexing content for a few hours/days. Since CF fronts a lot of the useful internet these days, it means it's effectively working against distributed indexing with its current captcha solution.
      - jgrahamc 5y ago
        Thanks. I'll bring this to the attention of the bots and DDoS teams.
        5y ago

surround 5y ago

I trust that cloudflare will act responsibly in allowing small search engines through, but I really, really would rather not have to trust cloudflare. I don't believe that any organization can or will always act responsibly, which is why it's concerning that cloudflare controls so much of the internet.

EvanAnderson 5y ago
Yes. This. I believe that John Graham-Cumming is genuine in his statements in this thread re: "contact me if you're running afoul of our controls", for example. If he leaves Cloudflare, Cloudflare "turns evil", etc, then that's all out the window. Individual companies having so much power gives me the willies.

thejosh 5y ago

Hi! Thanks for taking the time to reply. You mentioned about "legit" crawlers, what defines a "legit" crawler in the eyes of Cloudflare, and what happens when Cloudflare suddenly decides it does not want to honour that "agreement"? What happens if/when Cloudflare is sold, or the contact who greenlit these smaller "legit" crawlers moves on and decides that it no longer agrees with said website anymore? Is a price comparison site a "legit" crawler? What defines a "bot" vs a "crawler" in the eyes of Cloudflare? Would you need to notify your customers that you now also allow additional crawlers access to their sites, or would they need to opt into it via the Cloudflare dashboard? What happens when you have a falling out with my said company (it happens, relationships sour) and suddenly we can't make contact, then suddenly customers websites aren't being scraped because we're bots?

dannyw 5y ago
There are a lot of hypotheticals here. I think you'll convince CloudFlare, and their customers, if you could name names and mention specific examples? If you are a price comparison site getting blocked by cloudflare, site owners may be losing sales, and that's good feedback.
- jopsen 5y ago
  Or site owners may actively want to block a price comparison site.. Depending on the industry, etc..
  - thejosh 5y ago
    Agreed on this point, and most companies who would want to do these sorts of things would restrict it even if Cloudflare didn't exist. I really can't think of a good solution. But that's the tricky position Cloudflare is in - how does it balance everything.
- hysan 5y ago
  Price comparison example: https://shucks.top/ https://shucks.top/ sometimes gets blocked by cloudflare. Most recent was getting blocked from checking B&H.

pdimitar 5y ago

By being gatekeepers on which website crawling is okay and which is not. No such filters should exist. Is it really that awfully bad without anything but basic filters (ban an IP for flooding)? Are there like, operations that try and spam every single Cloudflare-hosted website 24/7? Legitimately curious if your anti-bot measures come from actual bad experience with the internet or is it just a liability limitation move? (Namely to reduce potential suing surface by angry data owners and/or three-letter agencies.) --- Basically, if I am experimenting with a basic crawling program and I hit websites A and B 20 times each in a space of one hour, is that really deserving of a captcha or extra auth methods? Not flaming but I am really curious. Do you have any data and rationale posted somewhere that go into deeper detail about why Cloudflare's bot detection is how it is?

fivre 5y ago
Cloudflare's customers request and then enable those features. Cloudflare itself doesn't give a damn about that traffic; they have bandwidth to spare. They will, however, happily sell tools to people that do care. That isn't to say that the customers are savvy and have a good understanding of different types of automated traffic and which automated traffic is harmful and which is benign. Many have a quite naive understanding that doesn't extend beyond "bots = bad, unless it's Google" and dial protection settings to the max for no good reason.