4 ms·

Crawling a Cloudflare powered website is basically impossible without needing to do some bodges as to how to crawl it. How can you expect someone to crawl a b

Crawling a Cloudflare powered website is basically impossible without needing to do some bodges as to how to crawl it. How can you expect someone to crawl a bunch of websites if they are actively blocked from accessing it? Now, you might say users can whitelist bots in their robots.txt file but then again will the person creating the engine individually ask companies to allow them to crawl? Also, slightly unrelated but Cloudflare protected websites are almost impossible to access via tor, the captcha never succeeds.

jgrahamc 5y ago
If you are building a search engine and getting blocked you can always contact me and I'll make sure that the teams that work on bot detection and DDoS are aware. We would like to know because we should not be blocking a legit crawler like this.
- throwaway3699 5y ago
  For every developer that sees this message, a few dozen will have given up.
  - new_guy 5y ago
    Exactly this. Cloudflare actively blocks legit crawlers. It shouldn't be dependent on seeing some random hn comment from some random at cloudflare to get that fixed.
- oefrha 5y ago
  Say I’m interested in building a small scale domain-specific search engine and only just started development. There’s no prototype yet and may never be. In this situation, how do you determine it’s a legit crawler? And what about crawlers with even more limited scopes (targeting only a handful of sites) that they can’t possibly be called search engines? Are they ever considered legit?
  - jgrahamc 5y ago
    Be a good netizen? Respect robots.txt. Don't lie in your User-Agent. Don't crawl at a ridiculous rate. All those are a good starting point.
    - joepie91_ 5y ago
      Do you not realize the more fundamental problem with you, as a company, essentially being the one who gatekeeps crawler access to the web?
      - dannyw 5y ago
        I use cloudflare out of my free will because there's malicious traffic out there, and I have enough control over everything. They're only a gatekeeper because sites voluntarily enter into commercial agreements with them. There's no coercion or manipulation like Google AMP.
      - midev 5y ago
        Customers pay for this as a feature. Why would they feel it's a fundamental problem? There's nothing that says admins need to let you crawl their site.
        vidarh 5y ago
        If people intend this to happen, sure . But how many people who put their sites behind Cloudflare is aware that this might be a side effect?
        midev 5y ago
        I would wager that most people that purchase Cloudflare are probably aware of the features it offers
        vidarh 5y ago
        Note the distinction. I'd wager that the vast majority of sites behind Cloudflare are not paying customers, and have not paid much attention beyond "hides my server IP slightly and stops DDOS's", without having thought more - or at all - about the wider implications.
- PaulHoule 5y ago
  What makes a web crawler "legit?" When I had a site that had millions of pages, I found that sites like Baidu would crawl my site as often, if not more often than Google. I already felt the relationship with Google was parasitic, but I looked through my logs and never found a single hit that came from Baidu and many of the other search engines that would overload my site. I was looking at a substantial part of the site running costs going to supporting web crawlers that were not doing anything (1) to help me, or (2) to help end users (if they don't want to send Chinese users to an English-speaking web site, why crawl the site?) So like it or not I am inclined to only allow Google and Bing in the robots.txt because Google is the only site that sends a significant amount of traffic and because Bing sends some, and Google needs some competition. There are web crawler behaviors that are annoying: harvesting email addresses, overloading your site, etc. But how do you know who is doing something wrong with the data and who is just collecting it do do nothing with it? (Probably 95% of web crawling ex. Google.)
  - acdha 5y ago
    > So like it or not I am inclined to only allow Google and Bing in the robots.txt because Google is the only site that sends a significant amount of traffic and because Bing sends some, and Google needs some competition. This sounds like you're onto a reasonable “legit” factor: does the crawler honor robots.txt? Baidu would be legit because they don't lie about their identity and if you put a rule in your robots.txt file they'll honor it.
- jmg03 5y ago
  What's the best way to contact you?
  - jgrahamc 5y ago
    jgc @ cloudflare
freshair 5y ago
> Also, slightly unrelated but Cloudflare protected websites are almost impossible to access via tor, the captcha never succeeds. Yes, I've never understood why it's seemingly so important to CAPTCHA me before serving me less than 100kb of read only plain jane HTML. What sort of "attack" is this stopping? I'm pretty sure the CAPTCHA itself is bigger than half the sites it blocks me from reading.

jgrahamc 5y ago

If you are building a search engine and getting blocked you can always contact me and I'll make sure that the teams that work on bot detection and DDoS are aware. We would like to know because we should not be blocking a legit crawler like this.

throwaway3699 5y ago
For every developer that sees this message, a few dozen will have given up.
- new_guy 5y ago
  Exactly this. Cloudflare actively blocks legit crawlers. It shouldn't be dependent on seeing some random hn comment from some random at cloudflare to get that fixed.
oefrha 5y ago
Say I’m interested in building a small scale domain-specific search engine and only just started development. There’s no prototype yet and may never be. In this situation, how do you determine it’s a legit crawler? And what about crawlers with even more limited scopes (targeting only a handful of sites) that they can’t possibly be called search engines? Are they ever considered legit?
- jgrahamc 5y ago
  Be a good netizen? Respect robots.txt. Don't lie in your User-Agent. Don't crawl at a ridiculous rate. All those are a good starting point.
  - joepie91_ 5y ago
    Do you not realize the more fundamental problem with you, as a company, essentially being the one who gatekeeps crawler access to the web?
    - dannyw 5y ago
      I use cloudflare out of my free will because there's malicious traffic out there, and I have enough control over everything. They're only a gatekeeper because sites voluntarily enter into commercial agreements with them. There's no coercion or manipulation like Google AMP.
    - midev 5y ago
      Customers pay for this as a feature. Why would they feel it's a fundamental problem? There's nothing that says admins need to let you crawl their site.
      - vidarh 5y ago
        If people intend this to happen, sure . But how many people who put their sites behind Cloudflare is aware that this might be a side effect?
        midev 5y ago
        I would wager that most people that purchase Cloudflare are probably aware of the features it offers
        vidarh 5y ago
        Note the distinction. I'd wager that the vast majority of sites behind Cloudflare are not paying customers, and have not paid much attention beyond "hides my server IP slightly and stops DDOS's", without having thought more - or at all - about the wider implications.
PaulHoule 5y ago
What makes a web crawler "legit?" When I had a site that had millions of pages, I found that sites like Baidu would crawl my site as often, if not more often than Google. I already felt the relationship with Google was parasitic, but I looked through my logs and never found a single hit that came from Baidu and many of the other search engines that would overload my site. I was looking at a substantial part of the site running costs going to supporting web crawlers that were not doing anything (1) to help me, or (2) to help end users (if they don't want to send Chinese users to an English-speaking web site, why crawl the site?) So like it or not I am inclined to only allow Google and Bing in the robots.txt because Google is the only site that sends a significant amount of traffic and because Bing sends some, and Google needs some competition. There are web crawler behaviors that are annoying: harvesting email addresses, overloading your site, etc. But how do you know who is doing something wrong with the data and who is just collecting it do do nothing with it? (Probably 95% of web crawling ex. Google.)
- acdha 5y ago
  > So like it or not I am inclined to only allow Google and Bing in the robots.txt because Google is the only site that sends a significant amount of traffic and because Bing sends some, and Google needs some competition. This sounds like you're onto a reasonable “legit” factor: does the crawler honor robots.txt? Baidu would be legit because they don't lie about their identity and if you put a rule in your robots.txt file they'll honor it.
jmg03 5y ago
What's the best way to contact you?
- jgrahamc 5y ago
  jgc @ cloudflare

freshair 5y ago

> Also, slightly unrelated but Cloudflare protected websites are almost impossible to access via tor, the captcha never succeeds. Yes, I've never understood why it's seemingly so important to CAPTCHA me before serving me less than 100kb of read only plain jane HTML. What sort of "attack" is this stopping? I'm pretty sure the CAPTCHA itself is bigger than half the sites it blocks me from reading.