4 ms·

Pretty sure that only Google and Microsoft have the money and resources to crawl the entire internet. Or perhaps the only that can AND are willing to. Correct

by gdcbe 4y ago

Pretty sure that only Google and Microsoft have the money and resources to crawl the entire internet. Or perhaps the only that can AND are willing to. Correct me if I’m wrong though, but I’m pretty certain that all other search engines in the same category use one of these as their backend. Eg I’m pretty certain that counts for duckduckgo as well.

Santosh83 4y ago
I think Yandex have their own index and some others too like Marginalia, but the latter couldn't be called "in the same category" as the other three.
- kkielhofner 4y ago
  Yandex also suffered a security breach and their source code is available[0] although utilizing it in any way is ethically and legally dubious (at best). [0] - https://arstechnica.com/information-technology/2023/01/massive-yandex-code-leak-reveals-russian-search-engines-ranking-factors/amp/ https://arstechnica.com/information-technology/2023/01/massi...
antonok 4y ago
Brave Search has its own independent index too - https://brave.com/brave-search-beta/ https://brave.com/brave-search-beta/
quectophoton 4y ago
> Pretty sure that only Google and Microsoft have the money and resources to crawl the entire internet. Or perhaps the only that can AND are willing to. Money and resources and a dominant-enough position so that your crawlers are not blocked by websites. Unfortunately.
- rezonant 4y ago
  There's definitely gatekeeping on websites but having done a bunch of crawler work I can say you'd be surprised how rarely a site will outright block you if you just do the right things: have an identifiable user agent with a working URL that explains what your crawler does, respect robots.txt, implement polite crawling. As for actually being able to crawl the whole thing though, yeah it's stupidly expensive :-/
daoudc 4y ago
Mwmbl has its own index but it's orders of magnitude smaller than commercial search engines.
schemescape 4y ago
What about Common Crawl? https://commoncrawl.org/ https://commoncrawl.org/

by gdcbe 4y ago

Santosh83 4y ago

I think Yandex have their own index and some others too like Marginalia, but the latter couldn't be called "in the same category" as the other three.

kkielhofner 4y ago
Yandex also suffered a security breach and their source code is available[0] although utilizing it in any way is ethically and legally dubious (at best). [0] - https://arstechnica.com/information-technology/2023/01/massive-yandex-code-leak-reveals-russian-search-engines-ranking-factors/amp/ https://arstechnica.com/information-technology/2023/01/massi...

antonok 4y ago

Brave Search has its own independent index too - https://brave.com/brave-search-beta/ https://brave.com/brave-search-beta/

quectophoton 4y ago

> Pretty sure that only Google and Microsoft have the money and resources to crawl the entire internet. Or perhaps the only that can AND are willing to. Money and resources and a dominant-enough position so that your crawlers are not blocked by websites. Unfortunately.

rezonant 4y ago
There's definitely gatekeeping on websites but having done a bunch of crawler work I can say you'd be surprised how rarely a site will outright block you if you just do the right things: have an identifiable user agent with a working URL that explains what your crawler does, respect robots.txt, implement polite crawling. As for actually being able to crawl the whole thing though, yeah it's stupidly expensive :-/

daoudc 4y ago

Mwmbl has its own index but it's orders of magnitude smaller than commercial search engines.

schemescape 4y ago

What about Common Crawl? https://commoncrawl.org/ https://commoncrawl.org/