5 ms·

Be a good netizen? Respect robots.txt. Don't lie in your User-Agent. Don't crawl at a ridiculous rate. All those are a good starting point.

joepie91_ 5y ago
Do you not realize the more fundamental problem with you, as a company, essentially being the one who gatekeeps crawler access to the web?
- dannyw 5y ago
  I use cloudflare out of my free will because there's malicious traffic out there, and I have enough control over everything. They're only a gatekeeper because sites voluntarily enter into commercial agreements with them. There's no coercion or manipulation like Google AMP.
- midev 5y ago
  Customers pay for this as a feature. Why would they feel it's a fundamental problem? There's nothing that says admins need to let you crawl their site.
  - vidarh 5y ago
    If people intend this to happen, sure . But how many people who put their sites behind Cloudflare is aware that this might be a side effect?
    - midev 5y ago
      I would wager that most people that purchase Cloudflare are probably aware of the features it offers
      - vidarh 5y ago
        Note the distinction. I'd wager that the vast majority of sites behind Cloudflare are not paying customers, and have not paid much attention beyond "hides my server IP slightly and stops DDOS's", without having thought more - or at all - about the wider implications.
        joepie91_ 5y ago
        This is exactly the case.
oefrha 5y ago
I think the problem is some IPs just straight-up always get CAPTCHAs from Cloudflare even if one’s a good netizen, respect robots.txt, not crawl at ridiculous rate, and not lie in the user agent. One reason is shared IP, which disproportionally affects people from third world countries as their ISPs don’t have enough IPv4 for everyone; but it also happened mysteriously to at least one dedicated IP I used in the past. Your confrontational tone is rather unfortunate, and the problem of course is that you don’t guarantee anything even if the user has done nothing wrong, as is manifest from the choice of the phrase “starting point”.
- ehutch79 5y ago
  Then the problem has nothing to do with your crawling.
foobiekr 5y ago
What is a “ridiculous rate”? Where is it documented?
Xamayon 5y ago
Even then, I've run into issues with scraping several sites for the reverse image search engine I operate. Luckily, in most cases I have been able to get in touch with the people running those sites to get a rule added for my IPs to allow them through. That's not scalable though, and limits where I can scrape/crawl from. Even something as simple as checking a site for updates every hour or two tends to get blocked after a few times. TBH, one of the only things I have found which helps is lying in the user agent and copying CF cookies. Luckily, I haven't had to play with that for a few months due to whitelisting, so not sure it it would still help. Things change rapidly.

joepie91_ 5y ago

Do you not realize the more fundamental problem with you, as a company, essentially being the one who gatekeeps crawler access to the web?

dannyw 5y ago
I use cloudflare out of my free will because there's malicious traffic out there, and I have enough control over everything. They're only a gatekeeper because sites voluntarily enter into commercial agreements with them. There's no coercion or manipulation like Google AMP.
midev 5y ago
Customers pay for this as a feature. Why would they feel it's a fundamental problem? There's nothing that says admins need to let you crawl their site.
- vidarh 5y ago
  If people intend this to happen, sure . But how many people who put their sites behind Cloudflare is aware that this might be a side effect?
  - midev 5y ago
    I would wager that most people that purchase Cloudflare are probably aware of the features it offers
    - vidarh 5y ago
      Note the distinction. I'd wager that the vast majority of sites behind Cloudflare are not paying customers, and have not paid much attention beyond "hides my server IP slightly and stops DDOS's", without having thought more - or at all - about the wider implications.
      - joepie91_ 5y ago
        This is exactly the case.

oefrha 5y ago

I think the problem is some IPs just straight-up always get CAPTCHAs from Cloudflare even if one’s a good netizen, respect robots.txt, not crawl at ridiculous rate, and not lie in the user agent. One reason is shared IP, which disproportionally affects people from third world countries as their ISPs don’t have enough IPv4 for everyone; but it also happened mysteriously to at least one dedicated IP I used in the past. Your confrontational tone is rather unfortunate, and the problem of course is that you don’t guarantee anything even if the user has done nothing wrong, as is manifest from the choice of the phrase “starting point”.

ehutch79 5y ago
Then the problem has nothing to do with your crawling.

foobiekr 5y ago

What is a “ridiculous rate”? Where is it documented?

Xamayon 5y ago

Even then, I've run into issues with scraping several sites for the reverse image search engine I operate. Luckily, in most cases I have been able to get in touch with the people running those sites to get a rule added for my IPs to allow them through. That's not scalable though, and limits where I can scrape/crawl from. Even something as simple as checking a site for updates every hour or two tends to get blocked after a few times. TBH, one of the only things I have found which helps is lying in the user agent and copying CF cookies. Luckily, I haven't had to play with that for a few months due to whitelisting, so not sure it it would still help. Things change rapidly.