15 ms·

Ask HN: Are we sure LLMs are that useful in a web search application?

105 points by duringmath 4y ago

Idly chatting with the search box doesn't strike me as the most productive use of my time. Instant answers or whatever they're called already produce direct answers plus they cite sources and provide links which is what everyone seems to think is the solution to the "LLMs make stuff up" problem. Not to mention they're faster and cheaper to run. Only truly practical use case I can think of is summarizing articles or writing them which makes more sense as a word processor or browser add-ons

mola 4y ago
I can see tremendous utility in chat gpt for end to end NLu tasks. Even for end to end artificial agents. But for end to end search? Not much better than what we have today, and worse in some ways.
sxvtemp 4y ago
I agree that the Google business model doesn't fully accommodate conversational search UI, but LLMs can still play a valuable role in this area. In terms of a parallel, you can think of the "I'm Feeling Lucky Today" button as a way to bypass the standard search results and take a chance on a single, top result. Similarly, using LLMs in web search applications can provide a more conversational and personalized experience for users, offering a different way to interact with and find information on the web. That being said, I don't think ChatGPT or any single LLM can replace mainstream Internet search use cases in the immediate future. They might enhance the search experience for users
h2odragon 4y ago
When you need an introduction to the subject area as much as you needed specific information from it; then an LLM explaining terms and offering options for further exploration can be a nice thing. (I assume they'll use it that way...) When you're hunting for a particular fact, like "that bit of code i half remember seeing on a page 15 years ago", then I don't see anything for an LLM to add. Google had a pretty good index for that purpose about 15 years ago, but they've chosen to prioritize other goals since then. I dunno if anyone works "find things you're searching for" as a market now. Which is an answer to your question: Does it matter if an LLM helps search the web? That's not what people are doing, that's not what these companies are selling.
- deleted 4y ago
  [deleted]
anonyfox 4y ago
Sometimes I don't really know what exactly to type into google because its only something rough in my head. It then takes like multiple try&research iterations on the topic I have in mind until being able to formulate the actual question, at least if I wasn't derailed in the process (monkey brain). A Chatbot is a godsend for me here and I will happily pay money for that alone. Another point is that I am either creative or productive at a time, but never both... at least aware of which state I am in. ChatGPT has proven to take over the other part surprisingly good. ie: - when I am in a productive mood and stumble upon a thinking problem, generative AI is like on-the-spot creativity for "good enough" solutions, like naming a programming thingy or write some filler text around a few keywords instead of me looking for words. - when I am in a creative mindset, I increasingly feed some code snippets into the bot and ask some questions to "fill in the gaps", like writing a specific function using library X, then to write a documentation explaining how it works, then to also write some unittests, and sometimes I even derail a bit or let the bot explain parts that stand out in some way so I can maybe learn a trick. ... And i used ChatGPT already in kinda emergency situations, like when I know 5 minutes in advance that I have to speak in front of a crowd/in a meeting and it gave me extremely useful outlines to quickly adapt to even when in a panicked mind state - calming me down through a given structure that sounded okay-ish, and it doesn't matter if the response is right or wrong.
mtlmtlmtlmtl 4y ago
On average I think I would be far better served if search providers just fixed their operators.
foobarbecue 4y ago
A major thing I find lacking in search engines is a way to disambiguate between homographs. "AI" search engines are one way to implement that capability. However, I want a search engine that gives me websites as a result, and is not capable of lying to me. So far every single "AI" search I have tried has given me incorrect, invented answers. This is a big step in the wrong direction.
- esperent 4y ago
  Unlike normal search engines which give you SEO tuned spam about 50% of the time. Are you sure it's a step in the wrong direction? > every single "AI" search I have tried has given me incorrect, invented answers. Really? Every single one? Not a single AI (ChatGPT, presumably) query you've tried has given you anything except incorrect information? What kind of things have you been querying for?
  - visarga 4y ago
    Interesting point - what is the noise in each one of them: - chatGPT - 10% hallucination - Google - 50% spam Which is better - the "honest hallucination" from AI or the "spam for profit" human operation?
sharemywin 4y ago
most answers are "it depends". so you might need to offer more information to get a better answer. so a conversation would be better.
thesuperbigfrog 4y ago
Years ago there was an idea called "the semantic web": https://en.wikipedia.org/wiki/Semantic_Web https://en.wikipedia.org/wiki/Semantic_Web The basic idea was to have enough metadata about web sites so that you could get programs to do something approaching Prolog-style reasoning about the content and meaning of the web pages. With more advanced LLMs it looks like a slightly different approach to achive something like the semantic web idea. I think the idea is to constantly feed the model with updates from crawling the web and have the LLM "digest" the content, apply some filters to remove bad stuff, and then provide a meaningful result to whatever queries it might be asked.
- PaulHoule 4y ago
  I'd picture that as a hybrid system that uses some old AI ideas together with some new AI ideas. AlphaGo which combines a neural network-based player with a MCMS player that simulates a large number of games all the way to the end, is a good example. The intellectual north star of this is https://en.wikipedia.org/wiki/SAT_solver https://en.wikipedia.org/wiki/SAT_solver and might be a better model than the "semantic web" in that the semantic web has revolved around description logics like OWL that implement a limited subset of first-order logic that is always decidable. One trouble is that OWL-based systems can't handle arithmetic at all, given a height and a weight you can write a rule that says anybody over 6 feet is tall but you can't say a person with a BMI over 30 is obese because OWL can't add, multiply and divide. But something a little more expressive that searches efficiently in practice because it gets "hunches" from a neural net could be useful but it will not evade the limits on computation and logic that Godel, Tarski and Turing warned you about.
  - syntaxfree 4y ago
    SAT solvers <3 there’s something about SATPLAN that feels like it has metaphysical significance.
- dboreham 4y ago
  > Years ago there was an idea called... That idea was the basis for the current generation of Google search (see: Metaweb).
ergonaught 4y ago
It's all going to go horribly wrong because people, but, yes: 1) Integration with voice assistants. Links/sources are irrelevant. 2) Models tuned against a particular body of work don't care if links go stale, or websites get SlashdottedHackerNewshuggedDDOSed, or etc. Links/sources are irrelevant. 3) Inbound "service requests" processed by something that can better understand the question and the available answers/solutions. Links don't matter much. 4) When "Okay what are some good websites to read more about this?" can be answered, too, bang. 5) Ever asked somebody a question and just rolled with their answer instead of demanding citations? I mean, you're doing it here. So, again, yes.
- verdverm 4y ago
  > 4) ... ChatGPT will already do this, when you get an answer you want to learn more about, ask "where can I learn more about this?" "can you provide links?" there are a number of prompts that work, you can even provide it with an AND clause in the first prompt
  - ergonaught 4y ago
    I meant when they can do this reliably. ChatGPT will readily supply references and even summaries of things that are wholly fabricated.
    - verdverm 4y ago
      yes, the hallucination problem is problematic, but remember the pixel attacks on the first visual DNNs? They were made more resilient with GANs, which were the first hallucinators? I've been waiting for capsule networks to get a better algo, wonder if there is something similar for LLMs... I expect these LLMs to improve in the coming years, especially as they are made into products and wrapped with more tools into Q&A pipelines.
  - PaulHoule 4y ago
    I've seen ChatGPT make up fake links and fake citations that look superficially real but don't really point to anything. It seems there are two kinds of people: those who see wrong results from ChatGPT and think they are wrong and those who see wrong results from ChatGPT and are amazed at how right they are. If I was going to use AI for something it might be to find the type II's and make sure they "never work in this town again" because if they can't call ChatGPT on its bullshit they probably can't call anybody on their bullshit.
syats 4y ago
Yes, but not in the form of chatbots. Among other things, a LLM can be seen as a store which you query and get results from. A chatbot is cute because it formats output text to look like conversation, and the recent applications are nice because the query (now known as prompt) can be complicated and long, and can influence the format and length of the results. But the cool stuff is being able to link the relatively small amount of text you input as a query, into many other chunks of texts that are semantically similar (waves hands around like helicopter blades). So, an LLM is a sort of "knowledge" store, that can be used for expanding queries, and search results, to make it more likely that a good result seems similar to the input query. What do I mean by similar? well, the first iteration of this idea is vector similarity (e.g. https://github.com/facebookresearch/DPR https://github.com/facebookresearch/DPR). The second iteration is to store the results into the model itself, so that the search operation is performed by the model itself. This second iteration will lead, IMHO, to a different sort of search engine. Not one over "all the pages" as, in theory at least, google and the like currently work. Instead, it will be restricted to the "well learnt pages", those which, because of volume of repetition, structure of text, or just availability to the training algorithm, get picked up and encoded into the weights. To make an analogy, is like asking a human who are the Knights of the Round Table and getting back the usual "Percival, Lanceelot and Galahad", but just because the other thousand knights mentioned in some works are not popular enough for that given human to know them. This is a different sort of search engine than we are used to, one which might be more useful for many (most?) applications. The biases and dangers of it are things we are only starting to imagine.
- basch 4y ago
  Exactly. Unfortunately, I think the "chat" aspect is obscuring what is actually happening here, and distracting from the achievement. First, the human input is extremely flexible, but can include instructions. It is natural language programming. Second, the "conversation" has state. I can give an instruction, and then a followup instruction that adds to the first instruction. Someday down the road there will be two states, your account state (instructions you taught it that it retains as long as you are logged in. Maybe my account can have multiple state buckets/buildings I can enter, one of one set of rules, one for another. Could call them programs or routines. (computer execute study routine)) and temporary state (instructions it retains only for the duration of the conversation/search.) The exciting part here is being able to query data and manipulate it in memory. Making a search, refining the search, redirecting the search in a different direction when its not working. That collaborative, iterative type search doesnt really exist at the moment. I cant tell google "the results you just returned are garbage, here is why, try again." It is more like a fuzzy commandline. The chatbotness is just a layer of cute on top, that isnt completely necessary.
visarga 4y ago
On Google when I search for "X that does not Y" it would often return "X that does Y" instead. Hard to force Google to respect the query intent. LLMs on the other hand process intent very well, but they hallucinate. So the obvious solution is a combination of them. Ideally Google search would have a flag to "follow my intent to the letter" and return empty if nothing is found. When you are searching for a specific thing, a response with other things feels like bullshit, Google trying to milk more clicks wasting your time. I don't mean exact phrase search, I mean exact semantic search. This is causing issues when searching for bug fixes by ignoring the exact version, when shopping it will ignore some of your filters and respond with bad product suggestions, and when searching something specific that looks like some other popular keyword, it will give you the popular one, as if if has an exclusion zone and you cannot search for anything else around it. "Minimum weight of a scooter with back suspension" -> matches information about carrying capacity. Of course more people discuss about max passenger weight than minimum scooter weight, but I really don't care about the other one.
- kurthr 4y ago
  I feel like what I want is a LLM that can tell me not only a summary answer, but what search terms are most likely to return documentation for the answer. Provide clickable citations documenting the answer using those terms (it can iterate on the answer and terms if they aren't internally consistent). Then give summaries of the information at each citation and/or parameters to input if the response requires further operations/searches. It could provide some commentary of the quality/recency of those citation/sources as well. That makes the response more current than the last LLM crawl&tune, allows a further directed search based on the new parameters, and provides a traceable path to sources and citations. If it finds that there's not a Python library with that name then iterate. Basically, an LLM should have a very good idea what good search terms are for the topic, and where to find the information, whereas I might not know the acronyms, jargon, or related fields and optimal answers. Yes, this is getting pretty close to writing an 8th grade class report that covers anything on the web. That's about where these seem to be. *ps I copied this from a post I made yesterday*
the_third_wave 4y ago
No, I was not convinced to start with and am getting less convinced all the time. To explain I'll go back to the middle ages. This was a time when the average person did not have access to books and the books which were in existence mostly were related to religion. That in itself was not that much of a problem since religion was the guiding line in most of people's lives but what was a problem was that those who wanted to know about what those books contained had to go through a middle man - an ecclesiastic - to get an interpretation. This gave those middle men enormous power which many of them wielded to their advantage, something which eventually led to a schism inside the ecclesiastical world when one of them nailed a pamphlet to a church door deriding the excesses of his world. When books became more generally available and with that literacy became a thing the world changed for what it seemed to be for good... ...until the rise of the new ecclesiastical class? These models are biased in several ways with the training set defining their world view and those who train them intervening in specific areas to bend the output to their will. They can be made to negate the old dictum of garbage in, garbage out to garbage in, gospel out - where the gospel follows whatever the (small-c) creator thinks the populace should know and (in extension) think. In short I don't trust these models any further than I can throw them.
- aww_dang 4y ago
  I've seen an "AI governance" narrative advanced elsewhere. By this they meant that AI would govern, centrally plan or act as an intermediary in disputes. This is distinct from the other meaning of the phrase pertaining to ethical use of ML models. It isn't hard to imagine a world where answers from ML models are the standard of truthiness, much like Wikipedia is for many.
  - the_third_wave 4y ago
    At least with Wikipedia the manipulation is visible and attributable. These LLMs are black boxes to the outside world, rife with manipulations (as can be seen from the way ChatGPT has changed its behaviour over time in specific areas) and in no way verifiable. Even if the creators vow to having used a certain set of data to train the model this can not be verified. To the downvoters: explain, don't just knee-jerk your disapproval. I don't care about karma but it is tiring to see things which don't follow the narrative being downvoted out of existence.
smrtinsert 4y ago
Find me all GPUs sorted by price desc, where the max depth is 12 inches that get great reviews. They should all be NVidia. Good luck wasting hours of your time googling for that by hand.
- dboreham 4y ago
  Newegg is the appropriate search engine here.
  - basch 4y ago
    Just in general, as a way we interact with data, being able to type all my filters into a sentence without syntax, absolutely kills any sort of existing filter system. Think about clothing websites, where you click a filter and the page has to refresh, or even just the page freezes for a second while it thinks. Having to scroll down a giant list of filter options to find the ones you know you want. Now I can just type "short sleeve green tshirts, medium tall only" and it will apply the correct filters. I can type that faster than I can even look through the filters the website is offering. Newegg would be wise to adopt such filtering.
  - gorbachev 4y ago
    You would think so, but no. Not one ecommerce search [1] is giving you accurate results for searches like that, because they think that they're losing sales, if they don't highlight other products that you'd be "likely" to also purchase. 1. with the possible exception of really niche stores like digi-key or other electronics components stores where it's really important you get exactly what you're looking for
busyant 4y ago
I'd like to see these things (eventually) used for automated phone systems like CVS and my bank. Mostly this is just to calm me down because ChatGPT gives me the illusion that I'm interacting with a human. The current voice systems are infuriatingly bad. It would be nice if CVS's phone system would actually listen to me and modify its output accordingly. "I already gave you my birth date. And NO, I don't need a COVID booster." edit: I'd like to meet the person who sold CVS its prescription web-site and its voice system. Simply to marvel at them and the swindle they pulled off, delivering absolute trash and probably walking away with a king's ransom.
- ramesh31 4y ago
  >The current voice systems are infuriatingly bad. But at least they're deterministic and finite. I imagine ChatGPT like results from a phone system would be even worse. LLM's are incredible when you can massage the answer you want out of it in a feedback loop, but not so much for automated systems.
  - busyant 4y ago
    > I imagine ChatGPT like results from a phone system would be even worse. You might be correct. I can't be sure. My opinion is definitely influenced by spending an hour on the phone with CVS yesterday + another hour on the phone with my father's cable company.
mysterydip 4y ago
I think the you're going to end up there either way eventually, as instant answers will start to pick up and cite more "LLMs make stuff up" generated content.
louthy 4y ago
I think the summarisers are going to be valuable. As a Kagi subscriber I'm looking forward to them integrating their AI labs demo that was showcased recently. There's certainly potential for search to become an order of magnitude better over the next few years. The issue I see with the chat approach is trust. I've seen so many examples of these models just making shit up now that I reckon regular use of them will eventually lead to mistrust between the human and the chat-bot. If you can't trust the answers and have to go and check yourself, it's dead as an idea IMHO.
- Sateeshm 4y ago
  AI summarising articles written by AI. Productive use of everyone's time.
KrugerDunnings 4y ago
Maybe not interfacing directly with a LLM ala ChatGPT but as an extension of vector search using word embeddings. It offers users a much more flexible interface to explore different combinations of dimensions vs a rigid facet UI interface. Well you can argue that the engineering has shifted to making the model, and that is true, but by keeping it fuzzy there is more room for creative prompt engineering. One challenge will be teaching the user what one doesn't know.
- PaulHoule 4y ago
  I'd scratch out "word" and replace that with "text" or "sentence" or "document". I saw word embeddings in the mid 2010's as a step backwards and I walked away from a lucrative consulting contract because I couldn't convince the sponsor that it was a total waste to apply word2vec to clinical documents because a significant amount of the meaning of the documents was in out-of-distribution words. (Expecting that to succeed is like a football coach thinking he can shoot the quarterback in the head and still win the game.) For me the train was pulling out of the station with FastText, BERT and similar models that used subword tokens because w/ the last generation of systems you really could show they threw out critical information in preprocessing that would make accuracy a matter of luck at best.
PaulHoule 4y ago
Depends how you use them. Generating bullshit text off the cuff is not the only use of LLMs. LLMs can perform very well at classification, regression, ranking, coloring proper names red, and other tasks. You could, for instance, use LLMs to encode a query and documents and rank them with a siamese network, something not too different for how a conventional search engine works. If there is one thing wrong with the current crop of LLMs it is that these can only attend over a limited number of tokens. BERT can attend over 512 tokens, ChatGPT over 4096, where a token is shorter than a word on average. It easy to process the headline of an HN submission with BERT, but I classify a few hundred abstracts of scientific papers a day. A long abstract is about 1800 words which is too much for longformer but would fit in ChatGPT if there aren't too many $10 words. Unless you can recast a problem as "does this document have a short window in it with this attribute?" (maybe "did the patient die?" in a clinical case report or "who won the game?" in a sports article) there is no way to cut a document up into pieces and feed it into an LLM, then combine the output vectors in a way that doesn't break the "magic" behavior the LLM was trained to do. You'd imagine ChatGPT would produce accurate results if you could tell it "Write an review of topic T with citations" but if you try that you'll find it will write citations that look for real but don't actually exist if you look them up. You'd imagine at minimum that such a system would have to read the papers that it cites, maybe being able to attend over all of them at the same time which would take an attention window 100-1000x larger. That's by no means cheap and it might be Kryptonite for Google in that Google's model involves indexing a huge amount of low quality content and financing it by ads that are a penny a click. A business or individual might get a larger amount of value per query out of a much smaller task-oriented document set.
efunneko 4y ago
I see this technology becoming more of a 'content provider' rather than strictly an internet search engine. In the last few days, I have gone to Google to get answers to programming questions, do an image search for a quick pic for a presentation in addition to the usual type of internet queries. The results mostly led me to go on to another site to get the actual data, but more and more of it will be pulled into the initial page. Google has done this for years and this tech will just allow for more of it. Why visit a joke site for some quick one-liners, if the 'search engine' can just generate you 20 good-enough jokes specific to your needs? I could imagine the interface being similar to what we have today, but with it being much better at taking in full descriptions of what you want. If you want pictures of teddy bears, it could provide search results and AI generated ones. If you want programming answers, it could link to StackOverflow or just give you an AI generated answer with an explanation. Perhaps I am looking for a lively bit of free music to add to an indie game - it could generate that too. I feel that this will eventually end search as we know it, but it will hurt the sites that are behind the search results far more than it will hurt Google. Google (or Bing, ...) can become the one-stop-shop for so much more than it is today
- osigurdson 4y ago
  The current technology will not end search as we know it. Perhaps new technology will but currently LLM and search are complimentary.
agentultra 4y ago
I think there are plenty of people that remain skeptical of their utility for this application. People who want to get rich will tell you it's the next greatest thing that will revolutionize the industry. Personally, I've been annoyed at how confidently wrong ChatGPT can be. Even when you point out the error and ask it to correct the mistake it comes back with an even-more-wrong answer. And it frames it like the answer is completely, 100% correct and accurate. Because it's essentially really deep auto-complete, it's designed to generate text that sounds plausible. This isn't useful in a search context when you want to find sources and truth. I think there are useful applications for this technology but I think we should leave that to the people who understand LLM's best and keep the charlatans out of it. LLM's are really interesting and have come a long way by leaps and bounds... but I don't see how replacing entire institutions and processes by something that is only well understood by a handful of people is a great idea. It's like watering plants with gatorade.
- eptcyka 4y ago
  It's not designed to generate plausible sounding text, it's designed to produce text that is statistically likely to be sound. It's a markov chain, but there might be multiple roots and some backpropped probabilities for which leafs should be picked. The statistical significance doesn't have to track with logic.
  - stingraycharles 4y ago
    So what you’re saying is that they not just trained it on text, but also verified the answers and trained it in such a way that it would get a negative feedback if the model gave the wrong answer?
    - pwinnski 4y ago
      It got negative feedback if the model gave wrong-sounding answers, which is not the same as wrong answers. I have seen no indication that they had anybody verify the factual content of answers in the training process, and many indications that they did not.
jstx1 4y ago
Google search has been powered by a large language model (BERT) for years so we've all been successfully using LLMs for search for a while now, just not in the way you're thinking.
whywhywhywhy 4y ago
Because the information it gives back has to be so sanitized I'm starting to feel that if search boxes start returning LLM results it will end up more like some safety team at Google/MS is having to almost write web queries by hand. Doesn't feel as simple as the search world where you can just up rank some approved sources, unless there is somehow a way to just generate the language from approved sources and ideas.
jccalhoun 4y ago
I think it could be useful in the followup to search queries or being smarter about using synonyms and others ways to phrase things to find better results but it reminds me of when facebook announced bots in there messenger. Lots of people tried it and showed all all these features and then they didn't really amount to much.
aww_dang 4y ago
Perhaps tech hype cycles are more about valuations and less about utility. Everyone jumps on the bandwagon with the latest buzzwords, companies buy in. Nobody wants to be left behind. New tooling is demanded. Employees pad their resumes. There's more to generative models than just the above, but how much of this hype cycle is substantive for end users or developers? You always had a query. Now you'll get answers in a different way. Skeptical overall.
s3p 4y ago
yes
rvz 4y ago
> Only truly practical use case I can think of is summarizing articles or writing them which makes more sense as a word processor or browser add-ons That use case is the one that makes more sense. Given that ChatGPT frequently hallucinates the wrong answer and confidently tells you how it is correct with its inability to cite and transparently explain to you how it got to that answer tells you that it is results are untrustworthy and this AI bubble is again, pure hype created by VCs. The only worthy AI hype that will change everything is open-source LLMs that are smaller models and are more transparent.
mathieuh 4y ago
Anytime anything starts getting touted as a panacea I start getting suspicious. Maybe LLMs are going to have a large role in search in future, but at the moment there's so much propaganda flying around I just assume it's mostly lies.
softwaredoug 4y ago
As someone that works in search - TBH - we don't really know. And the point isn't that people "know" its more about hedging that there likely is some kind of use case where it becomes important. Nobody knows until they try something. You're right to be skeptical. However, it's been observed, people are using chatbots for informational searches. The kinds of searches where you want to learn about a specific fact. This isn't all searches, but it's an important subset for web search. For better or worse, with probably a high degree of inaccuracy, people (probably rightly) perceive this is how people will seek information. There's also the generational use cases - "write me a program that does X". Is this something people would use a search bar for? We don't know, and wouldn't know, until its out there for a while. For the longest time the one natural language interface was as a search bar. So search vendors surmise it's important to both defend their turf while also a natural way to get regular users familiar with this kind of informational interaction...
- gremlinsinc 4y ago
  I find myself using google less and less, all search -really. I messed up my desktop the other day configuring and trying different DE's and window managers, chatGPT fixed some of my issues that would've had me in archlinux forums and reddit for a couple hours probably. Instead it was all good in 20 minutes. Personally, I think Microsoft has a HUGE opportunity here, imagine if the OpenAI - PRO account was actually free -- but only on edge browsers, or inside bing.com. It'd be a bit of a loss leader for a bit, because a lot of compute is required but it'd also help unseat google. Google's only chance is if lamda/bard is better - and so much better it's noticable from day one. Their search results are so horrible lately I don't know anybody who isn't 'shopping around' right now for something better. Bing, DDG, Brave, Neeva, You.com, etc.
  - grugagag 4y ago
    > Personally, I think Microsoft has a HUGE opportunity here, imagine if the OpenAI - PRO account was actually free -- but only on edge browsers, or inside bing.com If that were true I’d see people downloading edge and only using it for LLMs. MS hasn’t figured out how to monetize it yet, their only goal is to catch up and possibly dethrone google. Not an easy feat from MS who have blundered and still blunder a lot. Google search on the other hand is becoming less and less useful day by day.
braingenious 4y ago
> Idly chatting with the search box doesn't strike me as the most productive use of my time. That’s not using the search service for search
dontreact 4y ago
I think it's useful when you don't know what to search yet but are looking to solve a problem. For example, I am planning a wedding and I asked it to put together a rough budget for me and this helped me to prioritize where to spend my energies on the vendor search.
fnordpiglet 4y ago
They don’t solve the web search problem better than web search engines do. But most people aren’t interested in a document retrieval exercise, they have a question and want an answer to it. The interface of posing a cryptic phrase to find a collection of “relevant” text blobs that you then have to sift and read to try to piece together an answer isn’t ideal for answering a question. When we have a question of a professor they sometimes answer with a list of papers, sometimes with a long form answer, and sometimes both. I think all three are useful, depending on the context. But the last one is clearly the most useful. It provides an intuition up front and an opportunity ask clarifying questions, as well as a way to find more detailed information and understanding from a known credible list. LLMs have a chance to offer an oracle that doesn’t answer in cryptic or evasive ways, but attempts to just give an answer. The hallucinations are a huge flaw, but one that I’m confident will be addressed with other non-LLM AI approaches. But it’s the right user interface for answering questions - it answers questions with answers, not a pile of potentially relevant documents to sort. The augmentation with citations, especially if they’re semantically revenant rather than symbolically, is a huge plus.
- fauigerzigerk 4y ago
  I've been using YouChat with some success. It always gives you citations in addition to its own answer. https://you.com/search?q=youchat&tbm=youchat https://you.com/search?q=youchat&tbm=youchat
Swizec 4y ago
Sometimes I want a search engine, sometimes I want a good enough answer. Two different use-cases. When doing research I need a good search engine. Find me the official docs, not the SEO’d blogs. Find me that podcast episode. Find me the exact article I remember reading 10 years ago. Don’t try to guess and half-arse a result just because it has more ads or better SEO. I’ll do the hard work of synthesis because I’m looking to understand something deeply. Current search engines have gotten meh at this use-case. Or at least Google has. When looking for a quick answer, I need a smart-enough agent. How old is this celebrity? What’s the air speed of an unladen swallow? Give me a deity that starts with G. What the hell is a “nepo baby”? What does this random emoji mean when sent by a 20 year old and what is it by a 40 year old? Who’s that actor in that show with the thing? I don’t care about the source and I’m not looking to do research. Just tell me a good enough answer so I can get back to my conversation or whatever. Current search engines are pretty okay with this, but GPT is better. The two use-cases are fundamentally different and trying to merge them is where things went wrong.
The_Colonel 4y ago
I'm already using google to ask human like question "what's the second most populous country in africa". ChatGPT-like LLM can only improve that.
remarkEon 4y ago
I just used it this morning to create meal plans, including grocery lists and detailed cooking instructions, for the next 2 weeks. What would've taken probably 30+ minutes - googling around, scrolling past the blogspam nonsense, writing everything down - took about 10. This morning was probably the first time I started to understand the utility of this thing. In a way, it's like I'm finally interacting with the computer on the USS Enterprise. I don't know if that strictly complies with your definition of "web search application". It's definitely going to save time for me, and not seeing a bunch of ads during the process is wonderful - to the point that I really could see myself paying for it if they decide to go that route and take away the "free" version.
- sologoub 4y ago
  Would you be open to posting either a screenshot or the inputs you used to generate these meal plans?
- JohnFen 4y ago
  I'm beginning to understand that there are many definitions of "search engine", and that has caused much confusion to me. I would say that the use case you describe here is nothing even close to the sort of problem that a search engine addresses. It's more of a knowledge engine thing (and even that's a poor fit, really). When I think "search engine", I think of something that searches for web pages. What I want as a result isn't an answer, meal plan, grocery list, or any of that higher-order stuff. I want a list of websites. This reinforces my fear that this sort of approach might kill actual search engines. That would be a pretty serious loss to me, and a further reduction in the usefulness of the web. But saying that is not disparaging things like ChatGPT at all. They address a different, and valid, need. I just hope we don't lose an important tool in the process.
- iamwpj 4y ago
  How does this work? Care to share the prompt and output?
syntaxfree 4y ago
My answer to all these LLM threads is — play a little with the Python package “sentence-transformers”.
patapong 4y ago
As I said in another comment, there have been occasions where I was looking for the solution to a programming problem, was unable to find it on Google, but got it via ChatGPT on the first try. I think it works amazingly well at least for instances when you can immediately verify whether the answer is correct (e.g. coding, drafting letters) and instances where it is a starting point for further research. These use cases are a significant portion of my searches, so I think it will be very useful.
sfx77 4y ago
Yes
danso 4y ago
Completely agree with this skepticism. It reminds me of when people claim that TikTok is now a "search engine", as if entertaining/slick videos (even if they're reliably surfaced via search query) are going to be more useful on the whole than information I can skim and read [0]. On the other hand, I do agree with people speculating that LLM-AI interfaces will seriously hurt Google's bottom line, e.g. reducing the space for search ads, which represent the majority of its revenue. [0] https://www.nytimes.com/2022/09/16/technology/gen-z-tiktok-search-engine.html https://www.nytimes.com/2022/09/16/technology/gen-z-tiktok-s...
klyrs 4y ago
It's wild to me that "Ask Jeeves on Steroids" is being heralded as the potential google-killer...
mikewarot 4y ago
For any given word, there are multiple meanings that vary by context, along with possible misspellings, etc. So a search on a specific phrase can yield a fractal tree of possible interpretations/matches. It would be useful if a search engine could find the top 10 different interpretations of the phrase in latent space, and offer the top results (with a means to pursue more) in each of those separate meanings. For example: "hypertext markup" matches way too many things about HTML, and not enough things about marking up (annotating) hypertext LLMs could make search much more powerful in this manner. ChatGPT, on the other hand... is not a search engine, even before you consider its tendency towards BS.
graypegg 4y ago
It does make me wonder if google is maybe going in the wrong direction with its knee-jerk reaction to Bing’s changes recently. While what they’re doing currently isn’t perfect, it does provide results that are at least traceable. I could imagine an alternate universe where they doubled down on marketing themselves as “the search engine that doesn’t lie to you” or “where answers are found, not stories”.
- brookst 4y ago
  The Bing Chat answers are traceable. Look at the demo.
ankit219 4y ago
I think LLMs are potentially a good addition. My searches are based on different use cases: one is when I am looking for a quick answer, two when I am looking for some suggestions, three when I am looking to do research. ( Of course ignoring the ones when I google for meanings or quick calculations or to go to a specific site. Thats just cos it's fast, not really a feature). For quick overview answer, LLMs are great. It's not 100% correct, but mostly it is, and that is good enough for a quick answer. Currently google tries to show that and people object as it is stealing traffic from websites. i just need an answer, a coherent useable one. Eg: "What were the movies Scorsese got an Oscar nomination for?" For suggestions, LLMs are just one more of those blogs and listicles that are already showing up in search. If LLM is updated that is. The difference is an LLM would customize the answer according to the query, unlike already pre written content. So, yes useful. Same goes with stuff like: "how to build an email list?" or "What is a effective sales strategy?" For research, Google is more useful. I think we have all done that. Another application which is not realized at this time because we never did it before is the ability to ask follow up questions (which a chat format enables well). Suppose you get an overview of how a quantum computer works, but it would take a lot of effort to ask a follow up question and get a direct answer via a search engine. Eg: "Why is there no point in going beyond thousand Qubits?" There could be modifications like voice to text (a jarvis like interface), or a personal assistant thingy. But those are far fetched. It will help immensely, and for places it does not, we will still google like we have done before.
bearmode 4y ago
I think they will be, eventually, but have a fair way to go on the accuracy front first.
JohnFen 4y ago
I'm pretty sure that it's not. It's probably useful as a knowledge engine, but not a search engine.
karaterobot 4y ago
I'm fairly sure it can be a useful part of a web search application. There are things it could aid in: summary, evaluation of results, and probably many other things I can't imagine. It's got some use. But I'm not at all sure an LLM could wholly replace search engines, as some of the headlines are proclaiming. I think if you just took the excitement in the median news article and dialed it back about half a turn, that's probably where we'll end up.
- ESMirro 4y ago
  Agreed, we’ve been here before, repeatedly, over the past few years: - The Metaverse was going to change the internet. - The Internet of things was going to revolutionise our homes and cities. - Self-driving cars would change the motor industry. - Crypto is the next big thing in finance. - NFTs are going to revolutionise digital ownership. Most of those technologies have introduced niche or useful applications, but the ridiculous, breathless hype about how this technology is the one to change everything is getting more predictable and increasingly frustrating.
  - JohnFen 4y ago
    100% this. The hype around ChatGPT is insane, and it's a really safe bet that 90% of the things being claimed about it are absurdly exaggerated. At the same time, it's a really safe bet that everyone saying this technology is worthless is equally incorrect.
sberens 4y ago
I think yes, at least for coding. Pre-ChatGPT, when I was working with an unfamiliar language or framework, my workflow would be: google something -> skim top links for what looks helpful -> click on one of them -> parse through mountains of SEO'd text -> get the answer. The workflow I have now with ChatGPT and what I image it will look like in the new era of search is: query -> read result written for a human, not a search engine -> (10% of the time) check if the result is hallucinated. Especially for basic questions where I know what the answer should look like, I'm really enjoying the new workflow.
spamizbad 4y ago
Yes, but not quite how some people are expecting. Imagine asking someone with cursory knowledge of a subject matter to perform a google search for you. This person would dig through thousands of results and weed-out the junk/SEO/content farm sites, so you'd get information that's more relevant. LLMs could potentially do this quickly, separating the wheat from the chaff. Would it be perfect? No, but it would be a significant improvement over what you see on Google today.
- hnuser847 4y ago
  Why would an LLM be better at this than Google?
- winddude 4y ago
  But they won't dig through thousands, they'll use (at least anything I've seen) will use the first 5 or 10 results. It would be prohibitively expensive to use thousands of results. So it's still just using the underlying ranking algo.
EGreg 4y ago
Is this the Blockchain moment for HN and LLMs? Or will the love affair continue?
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
stevenbedrick 4y ago
There was a great paper at CHIIR looking at this question from an information science perspective that I would definitely recommend anybody working or interested in this space read; here's the abstract: > Search systems, like many other applications of machine learning, have become increasingly complex and opaque. The notions of relevance, usefulness, and trustworthiness with respect to information were already overloaded and often difficult to articulate, study, or implement. Newly surfaced proposals that aim to use large language models to generate relevant information for a user’s needs pose even greater threat to transparency, provenance, and user interactions in a search system. In this perspective paper we revisit the problem of search in the larger context of information seeking and argue that removing or reducing interactions in an effort to retrieve presumably more relevant information can be detrimental to many fundamental aspects of search, including information verification, information literacy, and serendipity. In addition to providing suggestions for counteracting some of the potential problems posed by such models, we present a vision for search systems that are intelligent and effective, while also providing greater transparency and accountability. Shah & Bender (2022) "Situating Search". In Proc. CHIIR '22 https://dl.acm.org/doi/abs/10.1145/3498366.3505816 https://dl.acm.org/doi/abs/10.1145/3498366.3505816
flappyeagle 4y ago
Another way to think about it: "the semantic web" is a dream that never came to be. Now we have a computer program that breaks all information down to semantics. So anything you thought was possible with the semantic web is now within reach.
seydor 4y ago
For many things yes, most landing pages are dishonest SEO trying to get sales. But i think that "suggest me something" is going to be a big selling point too. Decision-making is tiring, and people are willing to give that power to a machine. Look at how tiktok, youtube and even facebook now are working, it's slowly becoming like a TV stream that you passively watch. "Tell me what to do tomorrow" is going to be a common question in a year or so
deleted 4y ago
[deleted]
lumost 4y ago
Google Search results are often wrong or misleading. I already need to think critically, review sources, and corroborate when I'm reading google search results. ChatGPT saves me having to read thousands of words of unrelated/crappy content. If you need confirmation that it's not making something up just ask "Can you provide a reference for that? or What should I google to confirm that this is accurate?" It'll give you something pretty close to a final citation. This has saved me literal days of work traversing documentation in the last 2 months.
AussieWog93 4y ago
I asked ChatGPT the other day "Write some code to plot the contents of a circular buffer using ImGui". It responded immediately with some working code. With Google, I could have found the result but it would have taken a dozen clicks and probably 15 minutes of my time. (To be fair, I might have learned more in the process). Siri or Google Assistant wouldn't have given an answer at all.
meltyness 4y ago
No. It's obviously better for templating, word-processing, organizing ideas.
solarmist 4y ago
This is classic conflation. It’s a new UI method that seamlessly works with how humans interact. Everyone wants to believe it’ll work as well as interacting with a human. People want to believe this is as amazing as it appears, but it’s window dressing we still can’t intentionally separate relevant information from requests.
nerdix 4y ago
There are probably categories of searches were ChatGPT will be either one par or better than Google (and if the results are on par then ChatGPT is superior because it's one less click, one less ad spammy website to visit, no hunting for what you were looking for once you get to the site, etc) As an example, someone on HN posted a tweet of a guy who asked ChatGPT to draft a letter announcing a layoff while also announcing several executive promotions and quoting MLK Jr. Obviously, the example is facetious but the results were actually pretty good. Certainly good enough for a starting point or template for a real layoff announcement. I'm sure this is a miniscule amount of total search volume but there are a category of searches for letter templates (think cover letters, resignation letter, etc) that ChatGPT could seriously replace today. And ChatGPT is actually better because of how specific you can get (e.g. "with an MLK quote"). I don't think LLMs are a threat to traditional search today or even in the short term but what will ChatGPT 50 (or equivalent) look like in 20 years...
wruza 4y ago
I think that it could shine in integrations. Maybe. (Voice chat) - LLM, find three articles from HN frontpage which I would find insightful based on my recent evaluations, summarize them in under half a minute each and then I’ll choose the order, while I commute. - (…) - Okay, read me a second one first. - (…) - That was a good one because well-written and compared alternatives. Now find funniest article of all time based on my long-time preferences. - (…) - - - While it’s dumb enough to forget what and how you’ve evaluated recently, a hidden prompt could(?) fetch that out, e.g.: - (system) please convert my previous article ratings into json objects consisting of article url, article id (…how to get it…), 1..10 rating, your summary and a string of tags. Then these ratings may be saved for later and fed into a chat secretly as: - (system) If I gave you this prompt: “<voice prompt>” and had data looking like this “<db schema>”, please list which tables and filters would you likely use to form an answer. - (…) - My recent ratings were (…). <voice prompt> - - - I have no clue if this could work, but if it does, well, that would be useful. Edit: it may be wrong, but we have enough mundane tasks which are better done wrong rather than not done at all. It has a great potential as an “occasionally bright secretary” archetype.
throwaway1851 4y ago
One of the most critical parts of nontrivial search tasks is evaluating the quality and credibility (and motives) of the sites asserting various facts or opinions. I wouldn’t outsource this part to a statistical model. Even if it cites its sources, you’re limited to the pool of sources it chose. Unless you decide to go Google it yourself - in which case, what was the point of querying the language model?
Gwiz462 4y ago
I use the ChatGPT in Search extension and it absolutely helps to have the context to the side of my search results; I hope it never changes! I also use some SEO optimization extensions as well which is interesting for trends and categorical associations on some things in tandem with search results.

JohnFen 4y ago

How does it define what a "good" website is, though? It seems to me that this would be little different than relying on any other single source for information. That practice is always problematic.

verdverm 4y ago
We can consider the following when thinking about how to answer this - how do current search engines rank results? - what features correlate with good? - how do features of sites get encoded, queried, and returned? - how might LMs be already involved in this process? - how might LLMs / ChatGPT like systems encode these things? - how might we enrich training data to facilitate improved LLM information retrieval? I would suspect, with current models and algos, that the more often certain statements are found across the training data, the more likely that information is to be "correct" A good site might be one that has many of these "correct" chunks. Though I doubt they are there yet. A next phase task will likely be along the lines we are discussing here. i.e. How do we return citations and references in responses
- JohnFen 4y ago
  > how do current search engines rank results? I think there's a poor correlation between search engine rankings and website quality, so it might be better not to include this. > what features correlate with good? Right, that's the essential question. Google's original bit of genius was to to ignore that question entirely, and go with "popular" instead. You can measure popular, but "good" entirely depends on what it is you value. That makes it somewhat subjective.
  - verdverm 4y ago
    PageRank worked well at first by defining quality by the number of backlinks, or sites that linked to it. Essential if you had good content, it was likely that many others pointed to it. Ofcourse this was quickly gamed, but they were able to leverage their search engine usage to observe how users interacted with results, and later pages. I imagine that you could have a quality measurement by finding the chunk embeddings that are most popular, and then finding the sites that have these in the highest concentration. User interaction will still likely be a very important feature in the quality space.
    - JohnFen 4y ago
      > PageRank worked well at first by defining quality by the number of backlinks, or sites that linked to it. Yes. It was measuring popularity. It was genius because it solved the problem usefully well, but page rankings have never been a fantastic indication of quality -- again, depending on what you consider "quality". It was just better than nothing.
      - verdverm 4y ago
        > Yes. It was measuring popularity. Back link counting is not the same thing as popularity, which measures visits PageRank was better than keyword search, which became spammed and PR was resilient to. Now there are 100s of signals being used, see the Yandex leak
        JohnFen 4y ago
        It is the same as popularity -- it's measuring the popularity among other websites rather than individuals, but it's popularity nonetheless.
        verdverm 4y ago
        They describe it differently in their papar, as importance or quality, as PageRank came out of citation analysis. https://snap.stanford.edu/class/cs224w-readings/Brin98Anatomy.pdf https://snap.stanford.edu/class/cs224w-readings/Brin98Anatom... There's a lot more to the algorithm as well, it's a good read

bbor 4y ago

This is a perfect example of the popular view on here, and in my humble naive opinion it’s completely mistaken. The point isn’t “can an LLM replace google” the point is “can robots that can speak English and use logic improve the search experience” which I think basically everyone would answer “yes” to. Complaining that it gets stuff wrong when not hooked up to a web of resources to cite is, IMO, completely missing the point. Also OP (not so much you) is way too caught up in the “chat” aspect - that is the first exciting UX that got popular, but these are much, much more than chatbots. Pretending that they’re human/conscious/in a conversation is fun, but having an invisible partner that knows you and tailors your search results… that’s powerful. For example, you’ll never have to add “Reddit” again, or at least you’ll only have to tell it once. An LLM can easily identify the kind of questions where you want forum posts, read thousands of posts in a second, summarize all their content, and label each link with other information that helps you decide which threads to read in full. I can’t wait!

pwinnski 4y ago
If the point isn't how an LLM replace a search engine, then why is Bing using an LLM to replace their search engine? When you ask whether speaking English and using logic can improve the search experience, I wonder what you consider the most important parts of the search experience? I think many people, most of the time, might say that "accurate information" is their highest expectation, with "a pleasant conversation" somewhere below that. Delivering a plausible-sounding and pleasant answer that's completely wrong is... well, that's not a search engine I can depend on, is it? You're hypothesizing a few things at the end that sound great! It's completely unclear whether any of those things will actually end up happening, so I think the focus on what is available today, with Chat-GPT and Bing, is more apt than a focus on what could be.
- bbor 4y ago
  My basic answer is “they’re trying to rush stuff out the door because the people running bing have no idea what they’re doing” :) given that the things I propose don’t need any new inventions, I’d say they’re good to discuss and coming in the next few years for sure. And I totally agree that a) a search engine that doesn’t cite its sources is useless, and b) you almost never want to chat with google like it’s a person. So you’re spot on. But the point I was trying to make is that the main use case is in stuff like automated summaries, specialized page rankings, expanding quick informal queries into longer formal-logic ones, etc.
- HDThoreaun 4y ago
  I don't think Bing is replacing their engine with an LLM. Seems like they're complementing the engine with the LLM, basically replacing the old blurb you'd sometimes get with the LLM response.
- hdjjhhvvhga 4y ago
  > "accurate information" is their highest expectation The point is, "accurate information" is hard. Google's solution is snippets and while it might be fine for some cases, it fails terribly for others. There is zero guarantee an AI-based solution would be more precise, but for sure it will be way more confident - just like ChatGPT is.
Guillaume86 4y ago
> The point isn’t “can an LLM replace google” the point is “can robots that can speak English and use logic improve the search experience” which I think basically everyone would answer “yes” to. Complaining that it gets stuff wrong when not hooked up to a web of resources to cite is, IMO, completely missing the point. I think the complaints are more about the "use logic" point than the sources, from my limited understanding, I would not say LLMs currently use logic.
- TchoBeer 4y ago
  They don't need to imo. I use ChatGPT to help me find useful search keywords when I'm not exactly sure what I'm looking for. Like recently it helped me find an artist I had forgotten the name of based on his style. I think we can have both, idk
- theptip 4y ago
  They obviously do, they just aren’t perfect at it. You can get the LLM to display (simulations of agents displaying) quite clear logical thinking in certain scenarios. For example linguistic IQ tests. Gwern has written extensively about this. There is a general issue where AIs fail in different ways than humans, and the failures look really dumb to a human. So humans tend to ascribe that dumbness from the human scale. Instead, I’d suggest they just have a dramatically different spider-graph of capabilities to a human, and are overall more capable than the “dumb spreadsheet / parrot” narrative admits. (Definitely not human-level IQ yet, to be clear.)
- bbor 4y ago
  Hmm interested to hear why you say that. Not to be THAT guy and this might get me banned from HN, but this ChatGPT’s response to your point; I would say that this clearly shows the capacity for logic. Certainly not first order logic built into a symbolic AI model, but definitely logic. The start of its lengthy response: “ The use of probabilistic logic models in LLM can lead to more sophisticated and nuanced logical reasoning. These models represent knowledge about the world in a probabilistic form, which allows the LLM to take into account uncertainty and make inferences based on probabilities. For example, an LLM system trained on a large knowledge base might encounter a question that it has not seen before. Using probabilistic reasoning, the LLM…”
MuffinFlavored 4y ago
> The point isn’t “can an LLM replace google” the point is “can robots that can speak English and use logic improve the search experience” which I think basically everyone would answer “yes” to. Can you give me 3 example queries (questions and answers or typical searches) that are clear cut wins for a search engine application?
tensor 4y ago
As someone who understands how these models are built and what they do, let me just say that almost all of what you think these models can do is wrong. For one, you can't just "hook up" a language model to some other task, or to the web. ChatGPT is specifically built and trained to have good conversations. The fact that it can sort of appear to do other things is a happy coincidence. To do any of what you want, new algorithms need to be built, and none of that is "easy". And finally, these models take A LOT of cpu time. They are not going to be reading thousands of posts in a second without serious and expensive compute hardware backing it, and that level of compute isn't remotely feasible to give out to individual users. Even chatGPT, which is doing a fraction of the tasks you are listing, costs millions of dollars worth of hardware a day. The only reason it exists for free is because Microsoft has donated all that time.
- agentultra 4y ago
  And because OpenAI exploits labour markets in places like Kenya that have weaker labour protection laws and lower minimum wages than in developed countries. They had to pay someone to label data and filter the worst "content" humanity has to offer. Otherwise it would've ended up like numerous other attempts at exposing "AI" to the Internet. So it also has a huge human cost that OpenAI is not properly accounting for (and another reason why dreaming up potential use-cases for the technology as if it will be miniaturized and become a commodity in the near future takes some wilful ignorance).
- bbor 4y ago
  Interesting reply, thanks for taking the time to share your expertise. I definitely wasn’t considering the economics of the question, and I tend to agree with you - supporting LLM queries with ad dollars seems impossible at their current state. But chatgpt is already pretty darn good at summarizing and communicating. By “hook up” I literally just mean feed text from the internet into the prompt followed by whatever you want it to do with it - summarize, rank, whatever. Ignoring the economics for a moment (paid search engines?) and assuming that GPT 3.5 is the very best LLM we’ll ever get for simplicity: would you still say you don’t think tweaked versions of such models would MASSIVELY improve search?
rchaud 4y ago
> An LLM can easily identify the kind of questions where you want forum posts, read thousands of posts in a second, summarize all their content, I question if this something that users need at the scale you're assuming. Wikipedia has existed for 20 years summarizing enormous breadth of human knowledge, with some articles having thousands of human editors. it's a boon for civilization in the way libraries are. But has it disrupted anything besides Microsoft's overpriced Encarta DVD market? You're putting a lot of faith in computer models to provide accurate, both-sides'ed information on complex topics in a format that amounts to a soundbite.