3 ms·

If a crawler offered enough money they could be allowed too. It's not like Google has exclusive crawling rights.

• Nextgrid 10mo ago
  There is a logistics problem here - even if you had enough money to pay, how would you get in touch with every single site to even let them know you're happy to pay? It's not like site operators routinely scan their error logs to see your failed crawling attempts and your offer in the user-agent. Even if they see it, it's a classic chicken & egg problem: it's not worth the time of the site operator to engage with your offer until your search engine popular enough to matter, but your search engine will never become popular enough to matter if it doesn't have a critical mass of sites to begin with.
  • charcircuit 10mo ago
    Realistically you don't need every single site on board before you index becomes valuable. You can get in touch with sites via social media, email, discord, or even visiting them face to face.
    • stavros 10mo ago
      You really do need every single site, as search is a long tail problem. All the interesting stuff is in the fringes, if you only have a few big sites you'll have a search engine of spam.
      • charcircuit 10mo ago
        I think that is only needed for a small subset of queries. Seriously think of the last time you did a search and went to a fringe site as opposed to a well known brand or social media. Ranking quality is much more important than coverage over the whole internet.
        • stavros 10mo ago
          > Seriously think of the last time you did a search and went to a fringe site as opposed to a well known brand or social media. Oh, almost never. That's exactly why search sucks now.