3 ms·

Indexing the web and doing a initial evaluation is hard and costy. Because of this only a few search engines do so, if we exclude Asia the only starch engines

Indexing the web and doing a initial evaluation is hard and costy. Because of this only a few search engines do so, if we exclude Asia the only starch engines I know which do so are Google, bing and Yandex(?). All other defer to one of thos search engines (most times bing) while decoupling, post and pre processing queries to a varying degree.

RGamma 5y ago
This situation almost screams for web indexing as a service. Would love to see some search engine innovation, especially advanced curation, classification, archival and filtering(!) abilities.
machiaweliczny 5y ago
There is index of most sites to download at https://commoncrawl.org/ https://commoncrawl.org/
allanrbo 5y ago
Brave say they have their own index actually, but also sometimes use other providers (maybe for when their own index didn't return enough results?). https://brave.com/brave-search-beta/ https://brave.com/brave-search-beta/
- dathinab 5y ago
  As far as I know Brave brought some Index startup but doesn't actually use it. Instead they use Bing while pretending to have their own index. Generally I would avoid Brave as the company behind is is quite doggy.
  - lopis 5y ago
    Ironically, Brave bough a German search engine...
  - rsj_hn 5y ago
    Brave search results are quite different from Bing's (they are better) -- where are you getting your info?
jhgb 5y ago
Just because it's hard doesn't mean you can't do it. > if we exclude Asia the only starch engines I know which do so are Google, bing and Yandex(?). Asia...and Europe? I'm pretty sure that for example Seznam.cz still has its own indexing (https://blog.seznam.cz/en/history-of-web-search/ https://blog.seznam.cz/en/history-of-web-search/). At the very least there used to be several more indexers as well. A version of one them is even accessible as source code: https://www.ucw.cz/holmes/ https://www.ucw.cz/holmes/
- jll29 5y ago
  There are more. For example, Syria's Web search engine Shamra has its own index according to its founder Shadi Saleh, whom I have interviewed here: https://irsg.bcs.org/informer/2019/07/syrias-first-web-search-engine-an-interview-with-shadi-saleh-2/ https://irsg.bcs.org/informer/2019/07/syrias-first-web-searc...
ColinHayhurst 5y ago
Mojeek and Gigablast. Both do not engage in tracking unlike all others. https://blog.mojeek.com/2021/05/no-tracking-search-how-does-it-work.html https://blog.mojeek.com/2021/05/no-tracking-search-how-does-...

RGamma 5y ago

This situation almost screams for web indexing as a service. Would love to see some search engine innovation, especially advanced curation, classification, archival and filtering(!) abilities.

machiaweliczny 5y ago

There is index of most sites to download at https://commoncrawl.org/ https://commoncrawl.org/

allanrbo 5y ago

Brave say they have their own index actually, but also sometimes use other providers (maybe for when their own index didn't return enough results?). https://brave.com/brave-search-beta/ https://brave.com/brave-search-beta/

dathinab 5y ago
As far as I know Brave brought some Index startup but doesn't actually use it. Instead they use Bing while pretending to have their own index. Generally I would avoid Brave as the company behind is is quite doggy.
- lopis 5y ago
  Ironically, Brave bough a German search engine...
- rsj_hn 5y ago
  Brave search results are quite different from Bing's (they are better) -- where are you getting your info?

jhgb 5y ago

Just because it's hard doesn't mean you can't do it. > if we exclude Asia the only starch engines I know which do so are Google, bing and Yandex(?). Asia...and Europe? I'm pretty sure that for example Seznam.cz still has its own indexing (https://blog.seznam.cz/en/history-of-web-search/ https://blog.seznam.cz/en/history-of-web-search/). At the very least there used to be several more indexers as well. A version of one them is even accessible as source code: https://www.ucw.cz/holmes/ https://www.ucw.cz/holmes/

jll29 5y ago
There are more. For example, Syria's Web search engine Shamra has its own index according to its founder Shadi Saleh, whom I have interviewed here: https://irsg.bcs.org/informer/2019/07/syrias-first-web-search-engine-an-interview-with-shadi-saleh-2/ https://irsg.bcs.org/informer/2019/07/syrias-first-web-searc...

ColinHayhurst 5y ago

Mojeek and Gigablast. Both do not engage in tracking unlike all others. https://blog.mojeek.com/2021/05/no-tracking-search-how-does-it-work.html https://blog.mojeek.com/2021/05/no-tracking-search-how-does-...