5 ms·

Ask HN: Would you build a new search engine today?

We've recently seen the launch of a new search engine funded by Marc Benioff. I think that Search Engines have been mostly commoditized, and that the company with the biggest budget pretty much wins the game. However, maybe there's space for a different kind of search engine. Would you build a new one today? If so, what would make it different from anything else?

fsflover 5y ago
You don't need to build new search engines, because YaCy exists: https://yacy.net https://yacy.net (FLOSS and peer-to-peer). But you can think of a clever way to restrict the search index, like https://wiby.me https://wiby.me.
freediver 5y ago
Not only I would, but spent last three years building one. Kagi Search is today in private beta, with hundreds of users testing it. What makes it different is business model that aligns incentives and obsession with the user experience.
PaulHoule 5y ago
Google's search results really aren't that good today. Google keeps competition away because it is able to generate much more revenue per search than competitors. Worse results translate to worse revenue (e.g. if the #1 result was always the best, why look at the ads?) If somebody else could make the money work they could definitely give better search results.
- YXNjaGVyZWdlbgo 5y ago
  Do you really think the search results got worse or isn't it just a fact that there is much more lower quality content now? I think 60% of my searches are now done with "site:" on google and the results are way better than most of the onsite searches.
  - PaulHoule 5y ago
    It's true that the very existence of Google has motivated people to make limitless amounts of spam content that could get found in Google. For your second point what I know is that there has been 4 decades of research into search relevance ranking and about 0% of that knowledge has made it into real life products. I went through a phase where I interviewed people at 15 or so search vendors (e.g. Open Text) and not one of them told me that they or their customers thought relevance was a priority. One of them participated in the TREC competition because they thought it would give them some credibility but then they decided it didn't. If you look at any of those products you find they have hundreds of low-quality "integrations" with various data sources because they believe that's what sells the product.
tedyoung 5y ago
Like any business that I'd start, I'd find a niche where general-purpose search is not very helpful. For example, code search, or drug information. Fine-tune the querying, synonym detection, stop-words, stemming, etc., for that particular niche.
new_guy 5y ago
Googles mistake besides giving ads relevant to them and not what the user searched for is that they indexed everything (although not onion sites or anything like that) so the issue is garbage in, garbage out. A new search engine would just drop the top x sites from its index, and be aggressive with dropping sites. Blog spam? Link farm? Quora? Pinterest? Wikihow? Don't index them at all. A Shitty wordpress site that's just copying stuff verbatim from other sites? Drop them. Over engineered SEO? News sites? Fake news? Satire sites? Don't bother with them etc. (or maybe put a clear disclaimer in the case of satire sites) Make search about what it's supposed to be, discoverability. There are so many golden sites out there to be discovered under the absolute cruft Google and the like bury us in. And the new 'you.com' isn't any better because they're just getting their results from Bing, and they've been caught taking their results from Google!
- 1270018080 5y ago
  There should be a browser extension that completely removes any reference to Pinterest. Like it never existed.
  - 0des 5y ago
    "Unpinterested!", if you use Chrome - https://chrome.google.com/webstore/detail/unpinterested/gefaihkenmchjmcpcbpdijpoknfjpbfe https://chrome.google.com/webstore/detail/unpinterested/gefa... "Unpinterested!", if you use Firefox - https://addons.mozilla.org/en-GB/firefox/addon/unpinterested/ https://addons.mozilla.org/en-GB/firefox/addon/unpinterested...
- endisneigh 5y ago
  What you're describing isn't really that simple. Let's say you only take the top 200 to 500 sites, for example. For one, these top 200 to 500 are always changing. Suppose you keep them static. The sort of things like "spam", "fake news" and so forth are highly opinionated categories. Even if you know objectively that a site is spam, it's possible that it contains the information the user is looking for. Ultimately it would devolve to full curation, which has been tried and doesn't really work to the extent necessary to justify its own existence.
breckenedge 5y ago
Yes. Google indexes everything. This isn’t great for most of my workflows. I typically want to search a small network of reliable resources (StackOverflow, GitHub, Gitlab, Apple Developer documentation, etc.). You can narrow results somewhat, but the behavior feels janky. Combined technical docs and code search at scale is something I keep looking for.
- tedyoung 5y ago
  A programmable search engine for code that is limited to your trusted or preferred sites would be very useful. Minimal requirement is that it understands code and doesn't strip out operators, parens, etc. Would help if it was smart enough to show only latest information for your search. For example, no more XML configuration results for your question about modern Spring Boot applications.
marginalia_nu 5y ago
Frustrated with the limitations of what was available, I did just that. Turns out they are cheap to build and operate, and and an endless font of interesting engineering challenges.
- momofuku 5y ago
  Hey! I'm actually really interested to work on a personal project along this front. About a year back I organzied some files from research into a massive repository and I'd like to build a search engine to make it easier to query. Can you send me some pointers from where you started? Thank you for your time!
  - marginalia_nu 5y ago
    Assuming you can't or don't want to use something like Lucene or Solr (which are quite resource hungry), here's how to build a search engine. It's takes a diverse mix of computer science and low level programming to get working, but nothing is exorbitantly difficult. One side of the problem is to extract keywords from the documents, probably a good idea to start with something along the lines of OpenNLP and use that for TextRank. Well that's roughly what I'm doing now anyway, after having tried a few other approaches. It depends a bit on the nature of the documents I guess. Proper NLP is slow, but dumb TF-IDF goes a long way. (just google these things and you'll figure it out). If you want full text search, you'll probably want to look into constructing a suffix tree as well. Or just fake it using N-grams, but you absolutely need some rules limiting which N-grams are valid search terms, or their numbers grow astronomically. POS-tagging is useful here if you can afford it. As for the index: First you need a component that maps keywords to integers. This only needs to be one way, I use a custom memory mapped hashtable, but depending on how much data you've got, there are probably easier alternatives. Next you need to set up a reverse index that maps words-id to document-id. Again, this is a bunch of memory mapped binary data, I use a mix of hash tables and sorted arrays for binary searches. That's sort of it. Then you draw the rest of the owl.
  - kordlessagain 5y ago
    I’ve build something that uses Solr for indexing and some old ideas I had about crawling with Grub back in the day. The solution uses either the browser or a Selinum box to image the website and OCR to extract the text and bounding boxes. It will save the image in Google cloud and provides full text search and other Solr facet functionality for ranking. It’s not designed to crawl recursively, but it allows users to save both URLs and images they want to search for later. Menus work great and with some more work in extracting images from pages with OpenCV, it should allow for searching labels coming in from machine learning models. Happy to share with who ever wants to take a look. The repo for deploying Solr on Google Cloud is here: https://gitHub.com/kordless/grub-2.0 https://gitHub.com/kordless/grub-2.0
huma 5y ago
I don't, but I really appreciate those brave souls who try. A new, refreshing, honest search, like once Google was, is long overdue.
EZ-Cheeze 5y ago
I want a search engine for people who think like me and want the same things I do. There must be at least thousands of them and we'd be unstoppable together.
MrLeap 5y ago
Sure! I'd design it the same way I designed the text editor I'm making ( https://twitter.com/LeapJosh/status/1455511983512436737 https://twitter.com/LeapJosh/status/1455511983512436737 ) Hit the minimums. Be irreverent, elevate the ordinary. Manifest intrigue. If you can't do that, at least make your users feel weird.
Grayhornet0101 5y ago
Yeah, because Google don't list some searches(popular searches) for users so that they might love to explore what others love, so I might build one that does that.
reflect 5y ago
There may be some room for innovation in the way the results are handled, such as secondary searches and more powerful filters to make it easier to get where you wanted to without having to run a new opening search or using advanced search. Honestly I’d be happy with a google that just let me quickly hide results from certain sites.
mrkramer 5y ago
If it's not better than Google I wouldn't bother. Better search engine needs to focus more on language and linguistics to give better search results. Niche search engines or metasearch engines like Tripadvisor and Skyscanner are welcome because they come in handy where Bing or Google do not or in another words they are more specialized for certain niches than general internet search engines.
literallyaduck 5y ago
Search engines return millions of results and most users never get past page 1. Curating content for by subject matter and then aggregation and spidering the resulting content would yield better quality rather than quantity. Decentralization with cooperative subscriptions would provide a good search population rather than the unfiltered firehose of the current web.
- endisneigh 5y ago
  there are a lot of buzzwords here but how exactly would this work?
aaron695 5y ago
Google is the best and spends billions keeping the spam bots at bay. Lots of people claim it's worse than before. Don't trust them, they are idiots. Google doesn't index a few things. Scihub for instance. They also don't return results outside your language. This is a huge part of the internet they keep in walled gardens. Picture and video search is limited. And they have not continued with the facial recognition in their search. You could also improve on their filters. A few of these things you'd spend billions on to do from scratch, then Google would add.
muzani 5y ago
I contracted for a startup that was doing this. Google is huge and it has a huge flank, and it. There's a niche for certain topics, for example, where to buy fresh chicken. Eventually these things morph into their own products like Pinterest, imdb, or TripAdvisor.

fsflover 5y ago

You don't need to build new search engines, because YaCy exists: https://yacy.net https://yacy.net (FLOSS and peer-to-peer). But you can think of a clever way to restrict the search index, like https://wiby.me https://wiby.me.

freediver 5y ago

Not only I would, but spent last three years building one. Kagi Search is today in private beta, with hundreds of users testing it. What makes it different is business model that aligns incentives and obsession with the user experience.

PaulHoule 5y ago

Google's search results really aren't that good today. Google keeps competition away because it is able to generate much more revenue per search than competitors. Worse results translate to worse revenue (e.g. if the #1 result was always the best, why look at the ads?) If somebody else could make the money work they could definitely give better search results.

YXNjaGVyZWdlbgo 5y ago
Do you really think the search results got worse or isn't it just a fact that there is much more lower quality content now? I think 60% of my searches are now done with "site:" on google and the results are way better than most of the onsite searches.
- PaulHoule 5y ago
  It's true that the very existence of Google has motivated people to make limitless amounts of spam content that could get found in Google. For your second point what I know is that there has been 4 decades of research into search relevance ranking and about 0% of that knowledge has made it into real life products. I went through a phase where I interviewed people at 15 or so search vendors (e.g. Open Text) and not one of them told me that they or their customers thought relevance was a priority. One of them participated in the TREC competition because they thought it would give them some credibility but then they decided it didn't. If you look at any of those products you find they have hundreds of low-quality "integrations" with various data sources because they believe that's what sells the product.

tedyoung 5y ago

Like any business that I'd start, I'd find a niche where general-purpose search is not very helpful. For example, code search, or drug information. Fine-tune the querying, synonym detection, stop-words, stemming, etc., for that particular niche.

new_guy 5y ago

Googles mistake besides giving ads relevant to them and not what the user searched for is that they indexed everything (although not onion sites or anything like that) so the issue is garbage in, garbage out. A new search engine would just drop the top x sites from its index, and be aggressive with dropping sites. Blog spam? Link farm? Quora? Pinterest? Wikihow? Don't index them at all. A Shitty wordpress site that's just copying stuff verbatim from other sites? Drop them. Over engineered SEO? News sites? Fake news? Satire sites? Don't bother with them etc. (or maybe put a clear disclaimer in the case of satire sites) Make search about what it's supposed to be, discoverability. There are so many golden sites out there to be discovered under the absolute cruft Google and the like bury us in. And the new 'you.com' isn't any better because they're just getting their results from Bing, and they've been caught taking their results from Google!

1270018080 5y ago
There should be a browser extension that completely removes any reference to Pinterest. Like it never existed.
- 0des 5y ago
  "Unpinterested!", if you use Chrome - https://chrome.google.com/webstore/detail/unpinterested/gefaihkenmchjmcpcbpdijpoknfjpbfe https://chrome.google.com/webstore/detail/unpinterested/gefa... "Unpinterested!", if you use Firefox - https://addons.mozilla.org/en-GB/firefox/addon/unpinterested/ https://addons.mozilla.org/en-GB/firefox/addon/unpinterested...
endisneigh 5y ago
What you're describing isn't really that simple. Let's say you only take the top 200 to 500 sites, for example. For one, these top 200 to 500 are always changing. Suppose you keep them static. The sort of things like "spam", "fake news" and so forth are highly opinionated categories. Even if you know objectively that a site is spam, it's possible that it contains the information the user is looking for. Ultimately it would devolve to full curation, which has been tried and doesn't really work to the extent necessary to justify its own existence.

breckenedge 5y ago

Yes. Google indexes everything. This isn’t great for most of my workflows. I typically want to search a small network of reliable resources (StackOverflow, GitHub, Gitlab, Apple Developer documentation, etc.). You can narrow results somewhat, but the behavior feels janky. Combined technical docs and code search at scale is something I keep looking for.

tedyoung 5y ago
A programmable search engine for code that is limited to your trusted or preferred sites would be very useful. Minimal requirement is that it understands code and doesn't strip out operators, parens, etc. Would help if it was smart enough to show only latest information for your search. For example, no more XML configuration results for your question about modern Spring Boot applications.

marginalia_nu 5y ago

Frustrated with the limitations of what was available, I did just that. Turns out they are cheap to build and operate, and and an endless font of interesting engineering challenges.

momofuku 5y ago
Hey! I'm actually really interested to work on a personal project along this front. About a year back I organzied some files from research into a massive repository and I'd like to build a search engine to make it easier to query. Can you send me some pointers from where you started? Thank you for your time!
- marginalia_nu 5y ago
  Assuming you can't or don't want to use something like Lucene or Solr (which are quite resource hungry), here's how to build a search engine. It's takes a diverse mix of computer science and low level programming to get working, but nothing is exorbitantly difficult. One side of the problem is to extract keywords from the documents, probably a good idea to start with something along the lines of OpenNLP and use that for TextRank. Well that's roughly what I'm doing now anyway, after having tried a few other approaches. It depends a bit on the nature of the documents I guess. Proper NLP is slow, but dumb TF-IDF goes a long way. (just google these things and you'll figure it out). If you want full text search, you'll probably want to look into constructing a suffix tree as well. Or just fake it using N-grams, but you absolutely need some rules limiting which N-grams are valid search terms, or their numbers grow astronomically. POS-tagging is useful here if you can afford it. As for the index: First you need a component that maps keywords to integers. This only needs to be one way, I use a custom memory mapped hashtable, but depending on how much data you've got, there are probably easier alternatives. Next you need to set up a reverse index that maps words-id to document-id. Again, this is a bunch of memory mapped binary data, I use a mix of hash tables and sorted arrays for binary searches. That's sort of it. Then you draw the rest of the owl.
- kordlessagain 5y ago
  I’ve build something that uses Solr for indexing and some old ideas I had about crawling with Grub back in the day. The solution uses either the browser or a Selinum box to image the website and OCR to extract the text and bounding boxes. It will save the image in Google cloud and provides full text search and other Solr facet functionality for ranking. It’s not designed to crawl recursively, but it allows users to save both URLs and images they want to search for later. Menus work great and with some more work in extracting images from pages with OpenCV, it should allow for searching labels coming in from machine learning models. Happy to share with who ever wants to take a look. The repo for deploying Solr on Google Cloud is here: https://gitHub.com/kordless/grub-2.0 https://gitHub.com/kordless/grub-2.0

huma 5y ago

I don't, but I really appreciate those brave souls who try. A new, refreshing, honest search, like once Google was, is long overdue.

EZ-Cheeze 5y ago

I want a search engine for people who think like me and want the same things I do. There must be at least thousands of them and we'd be unstoppable together.

MrLeap 5y ago

Sure! I'd design it the same way I designed the text editor I'm making ( https://twitter.com/LeapJosh/status/1455511983512436737 https://twitter.com/LeapJosh/status/1455511983512436737 ) Hit the minimums. Be irreverent, elevate the ordinary. Manifest intrigue. If you can't do that, at least make your users feel weird.

Grayhornet0101 5y ago

Yeah, because Google don't list some searches(popular searches) for users so that they might love to explore what others love, so I might build one that does that.

reflect 5y ago

There may be some room for innovation in the way the results are handled, such as secondary searches and more powerful filters to make it easier to get where you wanted to without having to run a new opening search or using advanced search. Honestly I’d be happy with a google that just let me quickly hide results from certain sites.

mrkramer 5y ago

If it's not better than Google I wouldn't bother. Better search engine needs to focus more on language and linguistics to give better search results. Niche search engines or metasearch engines like Tripadvisor and Skyscanner are welcome because they come in handy where Bing or Google do not or in another words they are more specialized for certain niches than general internet search engines.

literallyaduck 5y ago

Search engines return millions of results and most users never get past page 1. Curating content for by subject matter and then aggregation and spidering the resulting content would yield better quality rather than quantity. Decentralization with cooperative subscriptions would provide a good search population rather than the unfiltered firehose of the current web.

endisneigh 5y ago
there are a lot of buzzwords here but how exactly would this work?

aaron695 5y ago

Google is the best and spends billions keeping the spam bots at bay. Lots of people claim it's worse than before. Don't trust them, they are idiots. Google doesn't index a few things. Scihub for instance. They also don't return results outside your language. This is a huge part of the internet they keep in walled gardens. Picture and video search is limited. And they have not continued with the facial recognition in their search. You could also improve on their filters. A few of these things you'd spend billions on to do from scratch, then Google would add.

muzani 5y ago

I contracted for a startup that was doing this. Google is huge and it has a huge flank, and it. There's a niche for certain topics, for example, where to buy fresh chicken. Eventually these things morph into their own products like Pinterest, imdb, or TripAdvisor.