4 ms·

So, I'm an electrical engineer (these days). I work with parts. Parts have part numbers. I need information about parts. Some of my parts are weird and rareish.

So, I'm an electrical engineer (these days). I work with parts. Parts have part numbers. I need information about parts. Some of my parts are weird and rareish. It is shocking how many searches I do which are well described by "show me every single thing on the entire Internet which matches this string". (I'd love to match a simple regex, since part numbers have stupid trivial variations, but I'll take what's actually feasible.) Or, I often want to match only PDFs, but that hurdle seems well solved these days. There aren't that many results for a few of these things (and those are often the ones I need information about the most), so I sometimes really do want to see every result. I don't see any other way to do that than a full-text, full-web index. I just don't. And my job gets a lot harder if that class of search isn't possible anymore. So I'm desperately hoping these indices remain at least somewhat viable. And yes, I am noticing that Google is lovingly correcting my long part numbers that do actually match a fair bit of results to crap that is useless. Fuck that bullshit, Google.

tintedfireglass 4y ago
Google is running behind NLP(Natural Language Processing) causing problems like this. I feel we should learn to use search engines and not search-engines trying to guess what we want to find using some weird Black Box AI
- imafish 4y ago
  I have had this problem for years in modern user interfaces. Most of the times, when a system is trying to guess what I want to do or find, it guesses wrong.
  - danuker 4y ago
    Indeed. Search got along just fine before NLP. You should at least have the option of a "raw" text search.
- danuker 4y ago
  I'm guessing the following: to index a long tail of infrequently-used keywords (such as part numbers), it takes a lot of storage costs, with little if any financial return.
- carschno 4y ago
  This is exactly the problem of "general purpose" search engines as mentioned by GP. In classic information retrieval theory, a search query is a textual representation of the a user's mental search intent. A search engine's task is to interpret that textual representation in order to serve the user's need. Basic NLP starts with simple techniques like stemming, synonyms, typo correction etc. But many users expect more, e.g. contextual disambiguation or broader synonym/similarity resolution. These users would see a search engine as dysfunctional if it could not even find pages in which a plural form or a spelling variation of a query term occurs. However, there is an inherent contradiction to those users who want to find a specific string. Google, Bing and others apply complex algorithms ("weird Black Box AI") in order to interpret the user intent and the meaning of web pages in order to optimize towards the majority of internet users. To some degree, they do try to also serve users of (large enough) minority use cases in which the user feels like they knew exactly what words (strings) they were looking for. But again, the different use cases pose contradictory interests that are probably not resolvable in a "general purpose" approach.
  - RugnirViking 4y ago
    I follow that, but I still have to ask: Why does it often ignore the quote marks it suggests to me? It's supposed to search for that exact string, but it never does and shows the same rubbish regardless. The quote marks should resolve the contradition between synonym searching and specific strings and they don't for some bizzare reason
    - horsawlarway 4y ago
      Because their products are a complete fucking MESS. Quotes used to be an exact search, but it now overrides that basically everywhere and continues to search for what it thinks you mean (and has for like a decade now). Even though the "Missing: ___ ‎| Must include: ___" popup just wraps that word in quotes in the search. Which is infuriating, since they know damn well adding quotes won't actually search for that word. In reality, you need to click "tools" -> "all results" and change it to "verbatim" which does at least still appear to work... for now.
jve 4y ago
> actually match a fair bit of results to crap Do you use advanced search features? Like quote things that MUST match. ext:pdf etc. You can have this interface to help you: https://www.google.com/advanced_search https://www.google.com/advanced_search
- exmadscientist 4y ago
  I mean, I'm able to get to where I'm going, between Google and other engines. It's just annoying as fuck to see "Did you mean 'apple orchards in Siberia'?" for querying "ADS124S06IPBSR". Because that shit IS NOT the same. (Note for the pedantic: I made up that example because I can't remember a real one. But it gives you an idea.)
capitalsigma 4y ago
Full text search on the entire web index is not feasible in terms of latency and compute cost
- throwamon 4y ago
  Is it feasible to roughly categorize websites (or parts of websites, e.g. subreddits) and then perform full text search on them only, or at least establish priorities? For instance, it's extremely unlikely OP would find anything useful about part numbers on *.reddit.com/r/funny. Does anything like that exist today in usable form?
  - AndroidKitKat 4y ago
    It's interesting you ask that, because the now-paid Kagi Search has a feature like this called Lenses: https://blog.kagi.com/kagi-features#:~:text=and%20Google.-,Lenses!,-Lenses%20are%20Kagi%E2%80%99s https://blog.kagi.com/kagi-features#:~:text=and%20Google.-,L... I've found them somewhat helpful in getting rid of some of the garbage that pops up when I'm searching for a specific topic. But often, I find myself leaving them off since I sometimes want all the results on anything even slightly related because it might be useful.
    - throwamon 4y ago
      This looks awesome. Unfortunately the pricing is like an order of magnitude higher than what my broke ass can pay for, but I'm truly rooting for them.
  - capitalsigma 4y ago
    Sure -- you can restrict the search to websites that seem to be relevant to the query. Maybe you do it by hand at first, then you switch over to an ML model. You optimize the ML model for a few years, adding in more signals, and allowing it to guide other parts of the query to keep costs in check and raise quality (as measured by your preferred metrics)... and now you have reinvented fuzzy search that behaves in unintuitive ways, and people on HN complain that your service was better when it did raw string matching.

tintedfireglass 4y ago

Google is running behind NLP(Natural Language Processing) causing problems like this. I feel we should learn to use search engines and not search-engines trying to guess what we want to find using some weird Black Box AI

imafish 4y ago
I have had this problem for years in modern user interfaces. Most of the times, when a system is trying to guess what I want to do or find, it guesses wrong.
- danuker 4y ago
  Indeed. Search got along just fine before NLP. You should at least have the option of a "raw" text search.
danuker 4y ago
I'm guessing the following: to index a long tail of infrequently-used keywords (such as part numbers), it takes a lot of storage costs, with little if any financial return.
carschno 4y ago
This is exactly the problem of "general purpose" search engines as mentioned by GP. In classic information retrieval theory, a search query is a textual representation of the a user's mental search intent. A search engine's task is to interpret that textual representation in order to serve the user's need. Basic NLP starts with simple techniques like stemming, synonyms, typo correction etc. But many users expect more, e.g. contextual disambiguation or broader synonym/similarity resolution. These users would see a search engine as dysfunctional if it could not even find pages in which a plural form or a spelling variation of a query term occurs. However, there is an inherent contradiction to those users who want to find a specific string. Google, Bing and others apply complex algorithms ("weird Black Box AI") in order to interpret the user intent and the meaning of web pages in order to optimize towards the majority of internet users. To some degree, they do try to also serve users of (large enough) minority use cases in which the user feels like they knew exactly what words (strings) they were looking for. But again, the different use cases pose contradictory interests that are probably not resolvable in a "general purpose" approach.
- RugnirViking 4y ago
  I follow that, but I still have to ask: Why does it often ignore the quote marks it suggests to me? It's supposed to search for that exact string, but it never does and shows the same rubbish regardless. The quote marks should resolve the contradition between synonym searching and specific strings and they don't for some bizzare reason
  - horsawlarway 4y ago
    Because their products are a complete fucking MESS. Quotes used to be an exact search, but it now overrides that basically everywhere and continues to search for what it thinks you mean (and has for like a decade now). Even though the "Missing: ___ ‎| Must include: ___" popup just wraps that word in quotes in the search. Which is infuriating, since they know damn well adding quotes won't actually search for that word. In reality, you need to click "tools" -> "all results" and change it to "verbatim" which does at least still appear to work... for now.

jve 4y ago

> actually match a fair bit of results to crap Do you use advanced search features? Like quote things that MUST match. ext:pdf etc. You can have this interface to help you: https://www.google.com/advanced_search https://www.google.com/advanced_search

exmadscientist 4y ago
I mean, I'm able to get to where I'm going, between Google and other engines. It's just annoying as fuck to see "Did you mean 'apple orchards in Siberia'?" for querying "ADS124S06IPBSR". Because that shit IS NOT the same. (Note for the pedantic: I made up that example because I can't remember a real one. But it gives you an idea.)

capitalsigma 4y ago

Full text search on the entire web index is not feasible in terms of latency and compute cost

throwamon 4y ago
Is it feasible to roughly categorize websites (or parts of websites, e.g. subreddits) and then perform full text search on them only, or at least establish priorities? For instance, it's extremely unlikely OP would find anything useful about part numbers on *.reddit.com/r/funny. Does anything like that exist today in usable form?
- AndroidKitKat 4y ago
  It's interesting you ask that, because the now-paid Kagi Search has a feature like this called Lenses: https://blog.kagi.com/kagi-features#:~:text=and%20Google.-,Lenses!,-Lenses%20are%20Kagi%E2%80%99s https://blog.kagi.com/kagi-features#:~:text=and%20Google.-,L... I've found them somewhat helpful in getting rid of some of the garbage that pops up when I'm searching for a specific topic. But often, I find myself leaving them off since I sometimes want all the results on anything even slightly related because it might be useful.
  - throwamon 4y ago
    This looks awesome. Unfortunately the pricing is like an order of magnitude higher than what my broke ass can pay for, but I'm truly rooting for them.
- capitalsigma 4y ago
  Sure -- you can restrict the search to websites that seem to be relevant to the query. Maybe you do it by hand at first, then you switch over to an ML model. You optimize the ML model for a few years, adding in more signals, and allowing it to guide other parts of the query to keep costs in check and raise quality (as measured by your preferred metrics)... and now you have reinvented fuzzy search that behaves in unintuitive ways, and people on HN complain that your service was better when it did raw string matching.