5 ms·

Google started becoming an answer engine from ~2015 with the introduction of "People Also Ask", and arguably earlier than that [0]. Mojeek.com is a (information

Google started becoming an answer engine from ~2015 with the introduction of "People Also Ask", and arguably earlier than that [0]. Mojeek.com is a (information retrieval) search engine, and we resist the temptation to also become an answer engine. So you might say we do not compete with Google. afte all we have a very different business model and proposition. As for the index; this underpins mojeek.com and our API; which customers use for search and/or AI. Common Crawl is ~3.5 billion pages and underpins LLMs. Our index is ~7 billion. Who knows what (else) Google, and Bing, do with their index ;) ? [0] https://blog.mojeek.com/2023/05/generative-ai-threatens-diversity-and-hyperlinks.html https://blog.mojeek.com/2023/05/generative-ai-threatens-dive...

fauigerzigerk 3y ago
>Google started becoming an answer engine from ~2015 with the introduction of "People Also Ask", and arguably earlier than that [0]. Mojeek.com is a (information retrieval) search engine, and we resist the temptation to also become an answer engine. So you might say we do not compete with Google. I'm sure you know your users well after so many years in the search engine business, but having read your article I must say I find your approach risky. You seem to be betting on search engines and answer engines continuing to be complementary rather than substitutes. But we are not the ones making this decision. Users will be making the decision in light of the newly available AI capabilities, and they will be making it with complete disregard for the health of the web, as is their nature :) The "funny" thing is that big publishers are as happy right now as I haven't seen them in the past 25 years, because it is so completely obvious that chat AIs will destroy the web unless big tech starts making big payments to big publishers. As you rightly say, small businesses and publishers will be collateral damage. But how do you make sure you're not collateral damage as well?
- marginalia_nu 3y ago
  It's basically the safest position you could be in. An LLM to digest results of a classic search index is greater than the sum of its parts. An LLMs that is not permitted to brush up on the relevant literature before answering a question generally doesn't produce very good answers, is prone to hallucinations etc. A pure LLM design isn't even a serious contender in the answer engine space.
  - fauigerzigerk 3y ago
    That would mean it's safe if "you" are Google or Microsoft+OpenAI as no one else has both a search index and an LLM.
    - marginalia_nu 3y ago
      You don't need to have both to sell search index access to anyone with an LLM, which seems like just about anyone these days.
      - fauigerzigerk 3y ago
        Why would publishers allow you to crawl their sites if you're not sending them any traffic? The big publishers certainly won't let you do that as they are selling their data to Google, Microsoft, Facebook and whoever else has the money to train a fully fledged LLM, which is certainly not everyone.
        marginalia_nu 3y ago
        Because it lets them sell data to other parties than Google and Facebook? That's actually pretty great. Only having a 1 or 2 customers kinda sucks. A search engine partnering with an answer engine may not send traffic, but the answer engine is a potential customer for the websites the search engines direct them to.
        fauigerzigerk 3y ago
        >Because it lets them sell data to other parties than Google and Facebook? Only indirectly by charging search engines for access to content. It would be an entirely different business model that requires a complex set of agreements between publishers, search engines and LLM providers. Granted it's not impossible and certainly worth considering if you have search engine expertise but no money to train an LLM.

fauigerzigerk 3y ago

>Google started becoming an answer engine from ~2015 with the introduction of "People Also Ask", and arguably earlier than that [0]. Mojeek.com is a (information retrieval) search engine, and we resist the temptation to also become an answer engine. So you might say we do not compete with Google. I'm sure you know your users well after so many years in the search engine business, but having read your article I must say I find your approach risky. You seem to be betting on search engines and answer engines continuing to be complementary rather than substitutes. But we are not the ones making this decision. Users will be making the decision in light of the newly available AI capabilities, and they will be making it with complete disregard for the health of the web, as is their nature :) The "funny" thing is that big publishers are as happy right now as I haven't seen them in the past 25 years, because it is so completely obvious that chat AIs will destroy the web unless big tech starts making big payments to big publishers. As you rightly say, small businesses and publishers will be collateral damage. But how do you make sure you're not collateral damage as well?

marginalia_nu 3y ago
It's basically the safest position you could be in. An LLM to digest results of a classic search index is greater than the sum of its parts. An LLMs that is not permitted to brush up on the relevant literature before answering a question generally doesn't produce very good answers, is prone to hallucinations etc. A pure LLM design isn't even a serious contender in the answer engine space.
- fauigerzigerk 3y ago
  That would mean it's safe if "you" are Google or Microsoft+OpenAI as no one else has both a search index and an LLM.
  - marginalia_nu 3y ago
    You don't need to have both to sell search index access to anyone with an LLM, which seems like just about anyone these days.
    - fauigerzigerk 3y ago
      Why would publishers allow you to crawl their sites if you're not sending them any traffic? The big publishers certainly won't let you do that as they are selling their data to Google, Microsoft, Facebook and whoever else has the money to train a fully fledged LLM, which is certainly not everyone.
      - marginalia_nu 3y ago
        Because it lets them sell data to other parties than Google and Facebook? That's actually pretty great. Only having a 1 or 2 customers kinda sucks. A search engine partnering with an answer engine may not send traffic, but the answer engine is a potential customer for the websites the search engines direct them to.
        fauigerzigerk 3y ago
        >Because it lets them sell data to other parties than Google and Facebook? Only indirectly by charging search engines for access to content. It would be an entirely different business model that requires a complex set of agreements between publishers, search engines and LLM providers. Granted it's not impossible and certainly worth considering if you have search engine expertise but no money to train an LLM.