4 ms·

One of the potential remedies is to require Google to open source their crawling data. This should make it easier for others who want to build a competing searc

• jsheard 2y ago
  Surely the crawling is the easiest part? The AI goldrush has seen dozens of new players gobbling up the entire internet. The hard part is turning that staggering amount of unstructured data into a useful search engine, and in particular fighting off black-hat SEO.
  • macNchz 2y ago
    Websites these days are often pretty hostile to crawling aside from the Googlebot. Other crawlers are asked not to by the robots.txt, fully blocked, or put through captchas. https://knuckleheads.club/ https://knuckleheads.club/ https://www.nytimes.com/2020/12/14/technology/how-google-dominates.html https://www.nytimes.com/2020/12/14/technology/how-google-dom...
    • jsheard 2y ago
      Good point. Now I think about it Google is actively contributing to that with their Reddit deal which has them blocking all scrapers besides Googlebot, even known legit search engine crawlers like Bingbot.
  • account42 2y ago
    > Surely the crawling is the easiest part? The AI goldrush has seen dozens of new players gobbling up the entire internet. The AI goldrush also has sites scrambling to block any crawlers other than a short list of approved search engines. Even before this mess Googlebot running on Google IPs often received special treatment. Search itself is still a hard problem but one that can be solved independently. Crawling on the other hand requires websites to cooperate and due to Google's size enough won't care to bother with anyone else.
• throwadobe 2y ago
  That's a pipe dream and really achieves nothing other than hurting Google.