7 ms·

What makes your crawler legitimate?

• AznHisoka 7y ago
  This builds up to a bigger question, with anti-competitive implications. What if you want to build a Google competitor? Is Google the only one that's allowed to crawl web pages now, and anyone else is automatically deemed a bad bot? Is CloudFlare the one that decides who can be a search engine, and who can't?
  • RcouF1uZ4gsC 7y ago
    >What if you want to build a Google competitor? Use CommonCrawl https://commoncrawl.org/ https://commoncrawl.org/ Then, after you can use that corpus and provide useful results and have people coming to your search site, then you can have a discussion with CloudFlare that you will be creating your own crawler.
    • 4gotunameagain 7y ago
      What if for whatever reason the innovation is on the crawler itself
      • kordlessagain 7y ago
        An AI crawler.
    • AznHisoka 7y ago
      Not sure how it is now, but when I looked into CommonCrawl, they had a significant delay in storing newer pages. It took at least a month before a web page made it into their index. Which makes creating any type of Google-like search engine very tough.
• Crosseye_Jack 7y ago
  What makes Google's legitimate over OP's? A big part of the net neutrality fight was "allowing the little guy a chance to play with the big boys". Part of the service says:- for Bandwidth Alliance partners, we’re going to hand the IP of the bot to the partner and get the bot kicked offline... If the infrastructure provider hosting the bot is part of the Bandwidth Alliance, we’ll share the bot’s IP address so they can shutdown the bot completely. The Bandwidth Alliance allows us to reduce transit costs with partners and, with this launch, also helps us work together with them to make the Internet safer for legitimate users. My reading of that is if CF decide your IP is bad, they can leverage the providers Bandwidth Alliance status to shutdown the providers customer. What if CF's systems misfire? Will there be a grace period? Will there be an appeals process? Will CF compensate anyone effected and have their hosting withdrawn from a misfire? No one likes bad bots, but I'm feeling more and more uneasy allowing CF decide which is which.
  • slig 7y ago
    > What makes Google's legitimate over OP's? They bring real visitors and customers. Random bots running on AWS or TOR bring me headaches and bigger bills.
    • mfontani 7y ago
      Random bots _using common "real browser" user-agent strings_ piss me off to no end.
      • slig 7y ago
        And that's exactly what random bots do.
    • Crosseye_Jack 7y ago
      > They bring real visitors and customers. if this is to be believed [0] less then 50% of google searches result in a click though to the site, so one could argue that because of things like snippets in search results Google don't always drive traffic to your site. [0] https://sparktoro.com/blog/less-than-half-of-google-searches-now-result-in-a-click/ https://sparktoro.com/blog/less-than-half-of-google-searches...
      • slig 7y ago
        I think that's fair. If the user was able to get what they're looking for without having to click, good for them. I'm betting that Google knows when that happen and uses it as a positive signal towards that site.
  • avip 7y ago
    >What makes Google's legitimate over OP's? Respecting robots.txt, using a well-known ua, coming from publicly declared, google owned IP block.
    • AznHisoka 7y ago
      "well-known ua" So if you're a small business that wants to build the next Google, you can't? Because obviously, your User Agent won't be well known when you start.
• andrerm 7y ago
  What makes it not legit? It's not clear at all.