5 ms·

I think 'general purpose' search engines are often doomed to fail, and this extends to 'web search engines' (which, in reality, do not search the full web and a

I think 'general purpose' search engines are often doomed to fail, and this extends to 'web search engines' (which, in reality, do not search the full web and are NOT general purpose.) This list makes a great list of attempts at general purpose search engines (I would include DDG and Bing) which ultimately fail and are not what the masses want. There's also precedence for how only domain-specific search engines are valuable / make sense these days: Google itself is shifting to ML-based answers (attempting to build the holy-grail of general-purpose search engines), and whatever you do isn't going to beat Google at that - even Google hasn't been successful at it yet. Examples of real-world domain-specific search engines today: * Image, YouTube, Maps search * Amazon search * Code search (Sourcegraph, cs.github.com, cs.opensource.google, etc.) * Facebook, Twitter, Reddit, TikTok, LinkedIn search * Documents search (iCloud, Drive, etc.) * Messenger search (Discord, Slack, Messenger, etc.) It's really telling how Google has largely only achieved general-purpose search in their domains of data (a Google search turns up YouTube videos, Maps locations) but isn't even complete in that area (Drive, images, etc. don't show up in Google results) General-purpose search is simply non-viable these days due to (a) data silos and (b) you need to have domain specific search to provide an edge (e.g. regex for code search, or drag-n-drop an image for image similarity search)

PestoDiRucola 4y ago
The issue is that some of these domain-specific search engines kinda suck on their own, for example reddit search is notoriously bad. This is often overcome by searching what you want in a general-purpose search engine and appending "reddit" to your search query.
exmadscientist 4y ago
So, I'm an electrical engineer (these days). I work with parts. Parts have part numbers. I need information about parts. Some of my parts are weird and rareish. It is shocking how many searches I do which are well described by "show me every single thing on the entire Internet which matches this string". (I'd love to match a simple regex, since part numbers have stupid trivial variations, but I'll take what's actually feasible.) Or, I often want to match only PDFs, but that hurdle seems well solved these days. There aren't that many results for a few of these things (and those are often the ones I need information about the most), so I sometimes really do want to see every result. I don't see any other way to do that than a full-text, full-web index. I just don't. And my job gets a lot harder if that class of search isn't possible anymore. So I'm desperately hoping these indices remain at least somewhat viable. And yes, I am noticing that Google is lovingly correcting my long part numbers that do actually match a fair bit of results to crap that is useless. Fuck that bullshit, Google.
- tintedfireglass 4y ago
  Google is running behind NLP(Natural Language Processing) causing problems like this. I feel we should learn to use search engines and not search-engines trying to guess what we want to find using some weird Black Box AI
  - imafish 4y ago
    I have had this problem for years in modern user interfaces. Most of the times, when a system is trying to guess what I want to do or find, it guesses wrong.
    - danuker 4y ago
      Indeed. Search got along just fine before NLP. You should at least have the option of a "raw" text search.
  - danuker 4y ago
    I'm guessing the following: to index a long tail of infrequently-used keywords (such as part numbers), it takes a lot of storage costs, with little if any financial return.
  - carschno 4y ago
    This is exactly the problem of "general purpose" search engines as mentioned by GP. In classic information retrieval theory, a search query is a textual representation of the a user's mental search intent. A search engine's task is to interpret that textual representation in order to serve the user's need. Basic NLP starts with simple techniques like stemming, synonyms, typo correction etc. But many users expect more, e.g. contextual disambiguation or broader synonym/similarity resolution. These users would see a search engine as dysfunctional if it could not even find pages in which a plural form or a spelling variation of a query term occurs. However, there is an inherent contradiction to those users who want to find a specific string. Google, Bing and others apply complex algorithms ("weird Black Box AI") in order to interpret the user intent and the meaning of web pages in order to optimize towards the majority of internet users. To some degree, they do try to also serve users of (large enough) minority use cases in which the user feels like they knew exactly what words (strings) they were looking for. But again, the different use cases pose contradictory interests that are probably not resolvable in a "general purpose" approach.
blowski 4y ago
I do a lot of those specific searches with hash bangs through DuckDuckGo. I would love a general purpose search engine. Most websites / web applications have bad search, and behave inconsistently. So I use site specific searches through DuckDuckGo.
april_22 4y ago
This is what I like so much about you.com They show results from specific, popular sites/apps (I listed some examples below) plus allow you to set preferred sources to see sites/apps you find useful (for me it's reddit) higher/on top of your search results. I find it especially nice for coding cause sites like GitHub and StackOverflow are supported. https://you.com/search?q=python+pandas+concatenate+two+dataframes https://you.com/search?q=python+pandas+concatenate+two+dataf... https://you.com/search?q=How+to+care+for+orchids https://you.com/search?q=How+to+care+for+orchids some supported sites: - GitHub - Reddit - StackOverflow - Arxiv.org - TikTok - LinkedIn - W3 Schools - Twitter
- rawoke083600 4y ago
  Is this the one where you had to install an extension first to search ? And the CTO/Owner had a fight with everyone on HN ??
  - ffhhj 4y ago
    link or didn't happen
jefftk 4y ago
> whatever you do isn't going to beat Google at that - even Google hasn't been successful at it yet Why not? This seems exactly like the sort of situation where an upstart could be very successful: an industry transitioning to a new approach.

PestoDiRucola 4y ago

The issue is that some of these domain-specific search engines kinda suck on their own, for example reddit search is notoriously bad. This is often overcome by searching what you want in a general-purpose search engine and appending "reddit" to your search query.

exmadscientist 4y ago

So, I'm an electrical engineer (these days). I work with parts. Parts have part numbers. I need information about parts. Some of my parts are weird and rareish. It is shocking how many searches I do which are well described by "show me every single thing on the entire Internet which matches this string". (I'd love to match a simple regex, since part numbers have stupid trivial variations, but I'll take what's actually feasible.) Or, I often want to match only PDFs, but that hurdle seems well solved these days. There aren't that many results for a few of these things (and those are often the ones I need information about the most), so I sometimes really do want to see every result. I don't see any other way to do that than a full-text, full-web index. I just don't. And my job gets a lot harder if that class of search isn't possible anymore. So I'm desperately hoping these indices remain at least somewhat viable. And yes, I am noticing that Google is lovingly correcting my long part numbers that do actually match a fair bit of results to crap that is useless. Fuck that bullshit, Google.

tintedfireglass 4y ago
Google is running behind NLP(Natural Language Processing) causing problems like this. I feel we should learn to use search engines and not search-engines trying to guess what we want to find using some weird Black Box AI
- imafish 4y ago
  I have had this problem for years in modern user interfaces. Most of the times, when a system is trying to guess what I want to do or find, it guesses wrong.
  - danuker 4y ago
    Indeed. Search got along just fine before NLP. You should at least have the option of a "raw" text search.
- danuker 4y ago
  I'm guessing the following: to index a long tail of infrequently-used keywords (such as part numbers), it takes a lot of storage costs, with little if any financial return.
- carschno 4y ago
  This is exactly the problem of "general purpose" search engines as mentioned by GP. In classic information retrieval theory, a search query is a textual representation of the a user's mental search intent. A search engine's task is to interpret that textual representation in order to serve the user's need. Basic NLP starts with simple techniques like stemming, synonyms, typo correction etc. But many users expect more, e.g. contextual disambiguation or broader synonym/similarity resolution. These users would see a search engine as dysfunctional if it could not even find pages in which a plural form or a spelling variation of a query term occurs. However, there is an inherent contradiction to those users who want to find a specific string. Google, Bing and others apply complex algorithms ("weird Black Box AI") in order to interpret the user intent and the meaning of web pages in order to optimize towards the majority of internet users. To some degree, they do try to also serve users of (large enough) minority use cases in which the user feels like they knew exactly what words (strings) they were looking for. But again, the different use cases pose contradictory interests that are probably not resolvable in a "general purpose" approach.

blowski 4y ago

I do a lot of those specific searches with hash bangs through DuckDuckGo. I would love a general purpose search engine. Most websites / web applications have bad search, and behave inconsistently. So I use site specific searches through DuckDuckGo.

april_22 4y ago

This is what I like so much about you.com They show results from specific, popular sites/apps (I listed some examples below) plus allow you to set preferred sources to see sites/apps you find useful (for me it's reddit) higher/on top of your search results. I find it especially nice for coding cause sites like GitHub and StackOverflow are supported. https://you.com/search?q=python+pandas+concatenate+two+dataframes https://you.com/search?q=python+pandas+concatenate+two+dataf... https://you.com/search?q=How+to+care+for+orchids https://you.com/search?q=How+to+care+for+orchids some supported sites: - GitHub - Reddit - StackOverflow - Arxiv.org - TikTok - LinkedIn - W3 Schools - Twitter

rawoke083600 4y ago
Is this the one where you had to install an extension first to search ? And the CTO/Owner had a fight with everyone on HN ??
- ffhhj 4y ago
  link or didn't happen

jefftk 4y ago

> whatever you do isn't going to beat Google at that - even Google hasn't been successful at it yet Why not? This seems exactly like the sort of situation where an upstart could be very successful: an industry transitioning to a new approach.