5 ms·

> Our bot detection breaks down into four large components: > - Identification of well known legitimate bots; What about non-well-known legitimate bots? If I

by comex 7y ago

> Our bot detection breaks down into four large components: > - Identification of well known legitimate bots; What about non-well-known legitimate bots? If I run my own web crawler, am I at risk of falling into the tarpit (and having my IP address reported)?

avip 7y ago
What makes your crawler legitimate?
- AznHisoka 7y ago
  This builds up to a bigger question, with anti-competitive implications. What if you want to build a Google competitor? Is Google the only one that's allowed to crawl web pages now, and anyone else is automatically deemed a bad bot? Is CloudFlare the one that decides who can be a search engine, and who can't?
  - RcouF1uZ4gsC 7y ago
    >What if you want to build a Google competitor? Use CommonCrawl https://commoncrawl.org/ https://commoncrawl.org/ Then, after you can use that corpus and provide useful results and have people coming to your search site, then you can have a discussion with CloudFlare that you will be creating your own crawler.
    - 4gotunameagain 7y ago
      What if for whatever reason the innovation is on the crawler itself
      - kordlessagain 7y ago
        An AI crawler.
    - AznHisoka 7y ago
      Not sure how it is now, but when I looked into CommonCrawl, they had a significant delay in storing newer pages. It took at least a month before a web page made it into their index. Which makes creating any type of Google-like search engine very tough.
- Crosseye_Jack 7y ago
  What makes Google's legitimate over OP's? A big part of the net neutrality fight was "allowing the little guy a chance to play with the big boys". Part of the service says:- for Bandwidth Alliance partners, we’re going to hand the IP of the bot to the partner and get the bot kicked offline... If the infrastructure provider hosting the bot is part of the Bandwidth Alliance, we’ll share the bot’s IP address so they can shutdown the bot completely. The Bandwidth Alliance allows us to reduce transit costs with partners and, with this launch, also helps us work together with them to make the Internet safer for legitimate users. My reading of that is if CF decide your IP is bad, they can leverage the providers Bandwidth Alliance status to shutdown the providers customer. What if CF's systems misfire? Will there be a grace period? Will there be an appeals process? Will CF compensate anyone effected and have their hosting withdrawn from a misfire? No one likes bad bots, but I'm feeling more and more uneasy allowing CF decide which is which.
  - slig 7y ago
    > What makes Google's legitimate over OP's? They bring real visitors and customers. Random bots running on AWS or TOR bring me headaches and bigger bills.
    - mfontani 7y ago
      Random bots _using common "real browser" user-agent strings_ piss me off to no end.
      - slig 7y ago
        And that's exactly what random bots do.
    - Crosseye_Jack 7y ago
      > They bring real visitors and customers. if this is to be believed [0] less then 50% of google searches result in a click though to the site, so one could argue that because of things like snippets in search results Google don't always drive traffic to your site. [0] https://sparktoro.com/blog/less-than-half-of-google-searches-now-result-in-a-click/ https://sparktoro.com/blog/less-than-half-of-google-searches...
- andrerm 7y ago
  What makes it not legit? It's not clear at all.
mfontani 7y ago
If you crawl sites without a distinctive user-agent; do not obey robots.txt; crawl at an unnecessarily high rate, I _will_ ban you, regardless of your intent.
- cft 7y ago
  Banning and kicking offline is different. I would love to sue them if say I attempted to build a search engine and they attacked me.
- andrerm 7y ago
  That's not what they said. For the record, they said: > Identification of well known legitimate bots; > Hand written rules for simple bots that, however simple, get used day in, day out; > Our Bot Activity Detector model that spots the behavior of bots based on past traffic and blocks them; and > Our Trusted Client model that spots whether an HTTP User-Agent is what it says it is.
  - aussieguy1234 7y ago
    > Our Trusted Client model that spots whether an HTTP User-Agent is what it says it is. Well, instead of using curl scripts, crawlers will switch to Selenium and use a real scripted browser
hyperpape 7y ago
We all know the answer: https://danluu.com/googlebot-monopoly/ https://danluu.com/googlebot-monopoly/ In my dream world, Cloudflare would make this a two way street: you get protection against malicious bots, but you have to allow equitable treatment of all well behaved search engines (i.e. you can use robots.txt to ban lesser search engines, but doing that requires you to ban Google as well). Disclaimer: I have no current economic interest, but kinda want to run a personal search engine.

by comex 7y ago

avip 7y ago

What makes your crawler legitimate?

AznHisoka 7y ago
This builds up to a bigger question, with anti-competitive implications. What if you want to build a Google competitor? Is Google the only one that's allowed to crawl web pages now, and anyone else is automatically deemed a bad bot? Is CloudFlare the one that decides who can be a search engine, and who can't?
- RcouF1uZ4gsC 7y ago
  >What if you want to build a Google competitor? Use CommonCrawl https://commoncrawl.org/ https://commoncrawl.org/ Then, after you can use that corpus and provide useful results and have people coming to your search site, then you can have a discussion with CloudFlare that you will be creating your own crawler.
  - 4gotunameagain 7y ago
    What if for whatever reason the innovation is on the crawler itself
    - kordlessagain 7y ago
      An AI crawler.
  - AznHisoka 7y ago
    Not sure how it is now, but when I looked into CommonCrawl, they had a significant delay in storing newer pages. It took at least a month before a web page made it into their index. Which makes creating any type of Google-like search engine very tough.
Crosseye_Jack 7y ago
What makes Google's legitimate over OP's? A big part of the net neutrality fight was "allowing the little guy a chance to play with the big boys". Part of the service says:- for Bandwidth Alliance partners, we’re going to hand the IP of the bot to the partner and get the bot kicked offline... If the infrastructure provider hosting the bot is part of the Bandwidth Alliance, we’ll share the bot’s IP address so they can shutdown the bot completely. The Bandwidth Alliance allows us to reduce transit costs with partners and, with this launch, also helps us work together with them to make the Internet safer for legitimate users. My reading of that is if CF decide your IP is bad, they can leverage the providers Bandwidth Alliance status to shutdown the providers customer. What if CF's systems misfire? Will there be a grace period? Will there be an appeals process? Will CF compensate anyone effected and have their hosting withdrawn from a misfire? No one likes bad bots, but I'm feeling more and more uneasy allowing CF decide which is which.
- slig 7y ago
  > What makes Google's legitimate over OP's? They bring real visitors and customers. Random bots running on AWS or TOR bring me headaches and bigger bills.
  - mfontani 7y ago
    Random bots _using common "real browser" user-agent strings_ piss me off to no end.
    - slig 7y ago
      And that's exactly what random bots do.
  - Crosseye_Jack 7y ago
    > They bring real visitors and customers. if this is to be believed [0] less then 50% of google searches result in a click though to the site, so one could argue that because of things like snippets in search results Google don't always drive traffic to your site. [0] https://sparktoro.com/blog/less-than-half-of-google-searches-now-result-in-a-click/ https://sparktoro.com/blog/less-than-half-of-google-searches...
andrerm 7y ago
What makes it not legit? It's not clear at all.

mfontani 7y ago

If you crawl sites without a distinctive user-agent; do not obey robots.txt; crawl at an unnecessarily high rate, I _will_ ban you, regardless of your intent.

cft 7y ago
Banning and kicking offline is different. I would love to sue them if say I attempted to build a search engine and they attacked me.
andrerm 7y ago
That's not what they said. For the record, they said: > Identification of well known legitimate bots; > Hand written rules for simple bots that, however simple, get used day in, day out; > Our Bot Activity Detector model that spots the behavior of bots based on past traffic and blocks them; and > Our Trusted Client model that spots whether an HTTP User-Agent is what it says it is.
- aussieguy1234 7y ago
  > Our Trusted Client model that spots whether an HTTP User-Agent is what it says it is. Well, instead of using curl scripts, crawlers will switch to Selenium and use a real scripted browser

hyperpape 7y ago

We all know the answer: https://danluu.com/googlebot-monopoly/ https://danluu.com/googlebot-monopoly/ In my dream world, Cloudflare would make this a two way street: you get protection against malicious bots, but you have to allow equitable treatment of all well behaved search engines (i.e. you can use robots.txt to ban lesser search engines, but doing that requires you to ban Google as well). Disclaimer: I have no current economic interest, but kinda want to run a personal search engine.