4 ms·

Kagi uses at least Google and Mojeek edit: > Realistically, there are only 2 search engines now. https://seirdy.one/posts/2021/03/10/search-engines-with-own-

Kagi uses at least Google and Mojeek edit: > Realistically, there are only 2 search engines now. https://seirdy.one/posts/2021/03/10/search-engines-with-own-indexes/ https://seirdy.one/posts/2021/03/10/search-engines-with-own-...

WarOnPrivacy 2y ago
> Realistically, there are only 2 search engines now. From the article: Many alternatives to GBY [Google, Bing, and Yandex] exist, but almost none of them have their own results; This seems to assert that ~0 other search providers do any crawling at all. Ever. Are we sure that's the case? (they could crawl but never ever return those results == more odd).
- ColinHayhurst 2y ago
  It's a very long article so understandable that you did not read on and learn about other search engines crawling beyond GBY. Still there are indeed very few that are crawling at web scale, and internationally. We are at 8 billion pages and totally independent [0], hence expressing our concerns to 404 media after being blanked by Reddit. [0] https://www.mojeek.com/about/why-mojeek https://www.mojeek.com/about/why-mojeek
  - WarOnPrivacy 2y ago
    > did not read on and learn about other search engines crawling beyond GBY. Still there are indeed very few that are crawling at web scale, and internationally That's helpful clarification. In criticism of the article, you might agree that none of them have their own results is a fairly absolute statement. It signals: Final word on the matter; no nuance to follow.
    - topaz0 2y ago
      Omitting the "almost" from "almost none" makes it sound disingenuously more absolute than it actually is.
      - WarOnPrivacy 2y ago
        > Omitting the "almost" from "almost none" makes it sound disingenuously more absolute than it actually is. Except that isn't what happened. Within the context, 'almost none' referred to size of the group, not to the amount of crawling. I was discussing how much crawling is being done - outside of the 'almost none' sized group. The 'almost none' sized group has 'their own results', so they do crawl. Based on that, the rest do not. Ergo, they do not crawl. A search engine that never crawls seems non-intuitive.
  - shadowgovt 2y ago
    I mean, I didn't read on because it's paid. I'm not taking their reporting without compensation, but that also means I didn't have the whole story. Such is life in this era of the internet.
    - deleted 2y ago
      [deleted]
- MichaelZuo 2y ago
  Bing provides far fewer verbatim results for pretty much all search queries that I've tested. And Yandex isn't much better for non cyrillic search, Baidu is only for the Chinese web effectively. And all other search engines either don't even attempt to do full web crawls anymore and/or buy from one of the four above. So realistically there's just one search engine for the full web that actually does the work.
  - WarOnPrivacy 2y ago
    > And Yandex isn't much better for non cyrillic search, I like Yandex when I'm rabbit-holing after obscure musicians/music. I routinely have a better experience than I do with DDG or Kagi or Goog.
    - freediver 2y ago
      Kagi also uses Yandex index so it would be unusual that something exists in Yandex and not in Kagi results.
      - WarOnPrivacy 2y ago
        > Kagi also uses Yandex index so it would be unusual that something exists in Yandex and not in Kagi results. I've no reason to doubt this so I'll attribute it to which results get pushed higher in the rankings. Different entities hold power in the E & W hemispheres and that can influence search results to a degree. Past that, there are some things I get at Yandex because Kagi seems incapable of searching for them. ex: Long strings like MD5 and larger hashes. edit:30 seconds after posting I searched a hash and Kagi sent back a ton of results. As recently as last week I got zero results (diff hash). That's the 2nd time in a week Kagi disproved a complaint of mine. I hope they continue to do so.
    - MichaelZuo 2y ago
      It's also vastly better for finding livejournal blogs.
  - dev1ycan 2y ago
- culi 2y ago
  I believe Brave Search is also starting their own index. There are some tiny independent indexes too: https://www.crawlson.com/ https://www.crawlson.com/ https://search.marginalia.nu/ https://search.marginalia.nu/ https://wiby.me/ https://wiby.me/ https://searchmysite.net/ https://searchmysite.net/
- darreninthenet 2y ago
  I believe Kagi has its own crawler as well and it merges all the results and does whatever Kagi does behind the scenes to show the mix
- tdeck 2y ago
  Aside: Does anyone know how the GBY term became a thing and why it includes Yandex but not Baidu?
Yawrehto 2y ago
Doesn't it list three major ones, Google, Bing, and Yandex, plus Mojeek and a few other small ones? That's a bit more than two.

WarOnPrivacy 2y ago

> Realistically, there are only 2 search engines now. From the article: Many alternatives to GBY [Google, Bing, and Yandex] exist, but almost none of them have their own results; This seems to assert that ~0 other search providers do any crawling at all. Ever. Are we sure that's the case? (they could crawl but never ever return those results == more odd).

ColinHayhurst 2y ago
It's a very long article so understandable that you did not read on and learn about other search engines crawling beyond GBY. Still there are indeed very few that are crawling at web scale, and internationally. We are at 8 billion pages and totally independent [0], hence expressing our concerns to 404 media after being blanked by Reddit. [0] https://www.mojeek.com/about/why-mojeek https://www.mojeek.com/about/why-mojeek
- WarOnPrivacy 2y ago
  > did not read on and learn about other search engines crawling beyond GBY. Still there are indeed very few that are crawling at web scale, and internationally That's helpful clarification. In criticism of the article, you might agree that none of them have their own results is a fairly absolute statement. It signals: Final word on the matter; no nuance to follow.
  - topaz0 2y ago
    Omitting the "almost" from "almost none" makes it sound disingenuously more absolute than it actually is.
    - WarOnPrivacy 2y ago
      > Omitting the "almost" from "almost none" makes it sound disingenuously more absolute than it actually is. Except that isn't what happened. Within the context, 'almost none' referred to size of the group, not to the amount of crawling. I was discussing how much crawling is being done - outside of the 'almost none' sized group. The 'almost none' sized group has 'their own results', so they do crawl. Based on that, the rest do not. Ergo, they do not crawl. A search engine that never crawls seems non-intuitive.
- shadowgovt 2y ago
  I mean, I didn't read on because it's paid. I'm not taking their reporting without compensation, but that also means I didn't have the whole story. Such is life in this era of the internet.
  - deleted 2y ago
    [deleted]
MichaelZuo 2y ago
Bing provides far fewer verbatim results for pretty much all search queries that I've tested. And Yandex isn't much better for non cyrillic search, Baidu is only for the Chinese web effectively. And all other search engines either don't even attempt to do full web crawls anymore and/or buy from one of the four above. So realistically there's just one search engine for the full web that actually does the work.
- WarOnPrivacy 2y ago
  > And Yandex isn't much better for non cyrillic search, I like Yandex when I'm rabbit-holing after obscure musicians/music. I routinely have a better experience than I do with DDG or Kagi or Goog.
  - freediver 2y ago
    Kagi also uses Yandex index so it would be unusual that something exists in Yandex and not in Kagi results.
    - WarOnPrivacy 2y ago
      > Kagi also uses Yandex index so it would be unusual that something exists in Yandex and not in Kagi results. I've no reason to doubt this so I'll attribute it to which results get pushed higher in the rankings. Different entities hold power in the E & W hemispheres and that can influence search results to a degree. Past that, there are some things I get at Yandex because Kagi seems incapable of searching for them. ex: Long strings like MD5 and larger hashes. edit:30 seconds after posting I searched a hash and Kagi sent back a ton of results. As recently as last week I got zero results (diff hash). That's the 2nd time in a week Kagi disproved a complaint of mine. I hope they continue to do so.
  - MichaelZuo 2y ago
    It's also vastly better for finding livejournal blogs.
- dev1ycan 2y ago
culi 2y ago
I believe Brave Search is also starting their own index. There are some tiny independent indexes too: https://www.crawlson.com/ https://www.crawlson.com/ https://search.marginalia.nu/ https://search.marginalia.nu/ https://wiby.me/ https://wiby.me/ https://searchmysite.net/ https://searchmysite.net/
darreninthenet 2y ago
I believe Kagi has its own crawler as well and it merges all the results and does whatever Kagi does behind the scenes to show the mix
tdeck 2y ago
Aside: Does anyone know how the GBY term became a thing and why it includes Yandex but not Baidu?

Yawrehto 2y ago

Doesn't it list three major ones, Google, Bing, and Yandex, plus Mojeek and a few other small ones? That's a bit more than two.