4 ms·

I work on alternative search engines, my first was something called Whize which flared for a bit, hit the front page of product hunt and then died after a few m

I work on alternative search engines, my first was something called Whize which flared for a bit, hit the front page of product hunt and then died after a few months of trying to support a search experience that we've all become used to ala Google. We had thousands of users stopping through over the months but that's not enough for an AD supported model and ultimately indexing the web with a significant enough amount of coverage so that results weren't lacking and then working on a novel enough ranking methodology that combined vector search signals with classical search signals is a massive undertaking that was running us thousands of dollars a month. It's possible but you need to basically be prepared to burn a lot of money, more than I and my friend were willing to support. That's why most of these alternative search engines serve custom Google results combined with other aggregator content or license Bing and they produce good results but at the end of the day your still beholden to the index and to some degree the ranking signals of the engines your building over. So far in the landscape Kagi is doing a good job on establishing itself with it's use of lens and subscriber based support for it's boutique engine. Neeva and You.com are also both in that space and have the benefit of being supported by gobs of funding (Neeva in particular due to the background of the founding team.) All of these engines are attempting to provide the curated experience this article talks about in the form of programmatic boutique like searches over Google and Bing's existing index. From a technical standpoint this makes obvious sense. But. From the year+ of working on that search engine and my continued interest in alternative search engines here's the thing. Google and Bing index web pages, but there are so many alternative forms of curated information we've created over the last decade and a lot of that information is in semi-public spaces like slack, discord, whatsapp, telegram, facebook groups, newsletters etc. As more and more spam has propagated over the internet real useful information and discourse has retreated into these semi guarded spaces that the existing incumbents do not index and do not serve which means all of these respective engines are missing that information which means in turn you're missing that information regardless of where you search. Want to buy a car? The best review of what car to buy is probably happening in a car enthusiast discord or slack maybe an e-mail newsletter. But you won't see that. So my opinion? I think the next real killer search application is going to figure out how to index and make that content searchable not an alternative take on searching web pages.

fumar 4y ago
I agree with your last point. How would a search engine crawl closed spaces like discord or subscription newsletters? I find most of the reviews generated by bloggers, journalists, or influencers less valuable than an aggregate review on an HN post. I can read through a hundred comments before making a judgement.
- ianbutler 4y ago
  Well remember, discord isn't entirely closed. The benefit of these communities wanting grow is that they're for the most part a simple as finding the link to the discord and joining it to get your information. I have a lot of thoughts on how to capitalize on that and especially after the latest HiQ ruling and consulting with a lawyer who specializes in this type of thing on how to go about indexing that type of information. Happy to talk more over e-mail if that piques your interest. One of the e-mails I check is in my profile.
mrkramer 4y ago
>Google and Bing index web pages, but there are so many alternative forms of curated information we've created over the last decade and a lot of that information is in semi-public spaces like slack, discord, whatsapp, telegram, facebook groups, newsletters etc. That's called Deep Web("semi-public spaces like slack, discord, whatsapp, telegram, facebook groups, newsletters etc.") and you can't crawl and index it unless you partner with them and they allow you to do it. But the biggest question that keeps haunting me is Does the world need another internet search engine? Casual users just don't care if your new internet search engine is 10 or 20% better than Google, what I'm trying to say it needs to be something totally different and significantly better in order to catch the attention and the engagement of the users.
- ianbutler 4y ago
  So to the question of does the world need another search engine? I mean I think the fact that so many people think yes enough to both create new search engines and in some cases fund them, especially when google for a lot of people myself included really doesn't produce useful results outside of basic questions, is yes. It also makes more sense if users aren't the product and you can have a business doing 10's-100's of millions in revenue based on serving some niche without having the be exploitative via subscription or some b2b offering you make. Google is so big it's almost a certainly they are more than suboptimal at a profitable enough niche. DuckDuckGo proved that with it's focus on privacy which is most certainly a niche and the 100million in revenue they do. Personally I think there's a lot of B2B opportunity for a lot of this information where the search can be a multi billion dollar business just not on the backs of consumers.

fumar 4y ago

I agree with your last point. How would a search engine crawl closed spaces like discord or subscription newsletters? I find most of the reviews generated by bloggers, journalists, or influencers less valuable than an aggregate review on an HN post. I can read through a hundred comments before making a judgement.

ianbutler 4y ago
Well remember, discord isn't entirely closed. The benefit of these communities wanting grow is that they're for the most part a simple as finding the link to the discord and joining it to get your information. I have a lot of thoughts on how to capitalize on that and especially after the latest HiQ ruling and consulting with a lawyer who specializes in this type of thing on how to go about indexing that type of information. Happy to talk more over e-mail if that piques your interest. One of the e-mails I check is in my profile.

mrkramer 4y ago

>Google and Bing index web pages, but there are so many alternative forms of curated information we've created over the last decade and a lot of that information is in semi-public spaces like slack, discord, whatsapp, telegram, facebook groups, newsletters etc. That's called Deep Web("semi-public spaces like slack, discord, whatsapp, telegram, facebook groups, newsletters etc.") and you can't crawl and index it unless you partner with them and they allow you to do it. But the biggest question that keeps haunting me is Does the world need another internet search engine? Casual users just don't care if your new internet search engine is 10 or 20% better than Google, what I'm trying to say it needs to be something totally different and significantly better in order to catch the attention and the engagement of the users.

ianbutler 4y ago
So to the question of does the world need another search engine? I mean I think the fact that so many people think yes enough to both create new search engines and in some cases fund them, especially when google for a lot of people myself included really doesn't produce useful results outside of basic questions, is yes. It also makes more sense if users aren't the product and you can have a business doing 10's-100's of millions in revenue based on serving some niche without having the be exploitative via subscription or some b2b offering you make. Google is so big it's almost a certainly they are more than suboptimal at a profitable enough niche. DuckDuckGo proved that with it's focus on privacy which is most certainly a niche and the 100million in revenue they do. Personally I think there's a lot of B2B opportunity for a lot of this information where the search can be a multi billion dollar business just not on the backs of consumers.