4 ms·

Well that's to be expected, but if a lot of search engines start using it, it's likely that websites will start allowing it to crawl and index their pages. So t

Well that's to be expected, but if a lot of search engines start using it, it's likely that websites will start allowing it to crawl and index their pages. So there might be potential there.

Grimm1 6y ago
It's not an issue of being allowed to crawl.
- ziftface 6y ago
  Oh I guess I misunderstood then. Why don't they crawl at Google's scale?
  - Grimm1 6y ago
    Mostly it comes down to their crawler itself and the cost of storing all of that data, it's sufficiently prohibitive.