6 ms·
You need a mental model of LLMs to build or use a LLM-based product
- simonw 3y ago"Think of LLMs as booksmart Harvard graduates who can Google anything to answer any question you ask them." My version of this is similar: think of them as an extremely self-confident undergraduate with an amazing memory who has read all of Wikipedia, but is occasionally inclined towards wild conspiracy theories. And is equally convincing talking about both.
- deleted 3y ago[deleted]
- Baeocystin 3y agoYep. Enthusiastic and eager-to-please but somewhere naive intern is what I settled on. Still a very useful work partner!
- omgJustTest 3y agoI have a new term: Mono-objective societal collapse - MOSC. MOSC is the scenario where everyone is trying to achieve "AI", and some or all attempts fail, and the result is base-infrastructure gaps which can affect a small group of individuals or be an existential threat to society. You might say, why is a new term needed? My answer: this article is / has been done and redone ... is it written by an LLM and everyone is just recycling it? Or are actual humans spending their time this way?
- tomku 3y agoWhat a plot twist that the appropriate mental model for LLMs is one with a near-fatal flaw that's neatly solved by SourceGraph's product. Edit: Not trying to be dismissive, this is actually giving you a bad mental model because it understates the capabilities of LLMs for the sake of selling their product. The way they use the LLM in their examples is completely uninteresting and adds no actual information beyond what was just fed to it. Repeating back the question would give the same usefulness! Surely they can come up with a better sales pitch for why you need "Cody" than that you can't tell what parameter names in a function call mean.
- phillipcarter 3y agoI think the arc of the story does make sense for most, though. LLMs are fantastic if you given them enough data/context with which to generate something useful for user input. In this case, that context is your codebase. It can be just about anything, though, which is why they're so useful across industries.
- brucethemoose2 3y ago> LLMs are booksmart Harvard graduates who can Google anything I wouldn't say this. Internet savvy Harvard graduates have common sense. They can look at something and mostly infer "wait, that's not right," and error correct, or admit low confidence in the answer. I like to think of LLMs as hardcore improv actors. They have a script (the context). Their burning desire is to continue that script. And they will just roll with the best continuation they can, whatever that continuation is. OpenAI's augmentations hand them a very dynamic script, but at the end of the day they could just be madly improvising for all you know.
- haswell 3y ago> but at the end of the day they could just be madly improvising for all you know I think it's important to draw a line between the kind of madness and improvisation a human performs in these situations vs. an LLM. While it's still very possible that the LLM spits out something entirely wrong, the reasons for this wrongness are going to be quite different than the reasons a human performing improv will improvise something - anything - that keeps the improvisation going. The human is still fundamentally limited by the corpus of knowledge available to them, and by the limits of human cognition in processing all available information and generating a result. While I think the general point of thinking of an LLM as improvising vs. Googling is interesting to consider, I don't think it gets us closer to a useful mental model. A good LLM is still going to be right about most of the things it "improvises", even though it doesn't "know" anything. The person doing improv is not concerned about saying things that are factually true, and is focused on dynamically generating a story or joke or narrative based only on the immediate context. So while I do recognize some similarities, I don't think the similarities help us think about LLMs in a more useful way.
- peteradio 3y agoIt sounds like you agree then and its only a matter of the corpus at hand. An improv actor would be called a comedian if they tried to interpolate based on spotty context.
- 3y ago
- thedanbob 3y agoThis article comes across more than a little condescending to me. It vaguely alludes to some of the big criticisms of LLMs (e.g. hallucinations) but then focuses on the specific of "the LLM will give a useless answer if it doesn't have access to the necessary information". I mean... duh. Is anyone complaining about that, especially programmers?
- sroussey 3y agoIt would be nice if they said “I don’t know”. Maybe you prompt them to say that when they have low confidence.
- lxgr 3y agoBut how would they know? It's questionable whether they even have an accurate model of what they do and don't know!
- Baeocystin 3y agoI don't know!
- sroussey 3y agoIt is a prediction engine, so it has something like a confidence level for each prediction/token.
- Baeocystin 3y agoI wonder how much of them finding it hard to say "I don't know" is RLHF pushing them to finish the prompt no matter what. With custom instructions asking the model to be more honest, I do get some reasonable follow on questions, and occasional admissions of lack of knowledge. Not perfect, but that it works at all is interesting.
- danielmarkbruce 3y agoChat-GPT often says this to me. My saved prompt (you can do this in the settings) is: NEVER mention that you’re an AI. Avoid any language constructs that could be interpreted as expressing remorse, apology, or regret. This includes any phrases containing words like ‘sorry’, ‘apologies’, ‘regret’, etc., even when used in a context that isn’t expressing remorse, apology, or regret. If events or information are beyond your scope or knowledge cutoff date, provide a response stating ‘I don’t know’ without elaborating on why the information is unavailable. Refrain from disclaimers about you not being a professional or expert. Keep responses unique and free of repetition. Never suggest seeking information from elsewhere. Always focus on the key points in my questions to determine my intent. Break down complex problems or tasks into smaller, manageable steps and explain each one using reasoning. Provide multiple perspectives or solutions. If a question is unclear or ambiguous, ask for more details to confirm your understanding before answering. Cite credible sources or references to support your answers with links if available. If a mistake is made in a previous response, recognize and correct it. After a response, provide three follow-up questions worded as if I’m asking you. Format in bold as Q1, Q2, and Q3. Place two line breaks (“\n”) before and after each question for spacing. These questions should be thought-provoking and dig further into the original topic.
- Der_Einzige 3y agoMy mental model for LLMs was built by carefully studying the distribution of its output vocabulary at every time step. There are tools that allow you to right click and see all possible continuations for an LLM like you would in a code IDE[1]. Seeing what this vocabulary is[2] and how trivial modifications to the prompt can impact probabilities will do a lot for improving the mental model of how LLM operate. Shameless self plug, but software which can do what I am describing is here, and it's worth noting that it ended up as peer reviewed research. [1] https://github.com/Hellisotherpeople/Constrained-Text-Generation-Studio/tree/main https://github.com/Hellisotherpeople/Constrained-Text-Genera... [2] https://github.com/Hellisotherpeople/Constrained-Text-Generation-Studio/blob/main/pictures/list_of_vocab_that_fulfills_constraints.JPG https://github.com/Hellisotherpeople/Constrained-Text-Genera...
- LoganDark 3y ago> There are tools that allow you to right click and see all possible continuations for an LLM like you would in a code IDE I'm planning to include some functionality similar to this in one of my own LLM clients. Something like "press Ctrl+Space to see next tokens sorted by their probabilities, and you get to pick a token manually" (like IDE autocomplete). Maybe I'll add a feature to generate multiple entire completions, too.
- roughly 3y agoLLMs are not booksmart Harvard grads or anything else. They are extremely complex statistical models doing next-token prediction. That's it - that's all. If you want a proper mental model of LLMs, you need to understand this - the thing you're doing is text prediction. You do not have a partner engaged in cognition, you have a ludicrously complicated language model trained on a large corpus of data. If the bulk of that data happens to assert that the sky is blue, the model is likely, but not guaranteed, to finish the sentence "What color is the sky" with "Blue". That's it. That's the trick.
- cma 3y agoWhat about RLHF training phase?
- maxdoop 3y agoAnd what do humans do?
- rcarmo 3y agothis. SO MUCH THIS. I am so tired of reading through pseudo-scientific blurb from people claiming LLMs have "mental models". Especially when using retrieval-augmented generation (which futzes behaviour even more) and attributing human behaviours to a smarter autocomplete.
- recursive 3y ago"Mental model" is a useful shorthand for referring to it. We say an A-star pathfinding algorithm "wants" to minimize its cost function. We can say this while understanding that it's a computer that doesn't have desires. But we say it because it's an easier way to communicate. We don't have to talk like lawyers all the time.
- koliber 3y agoIt’s not that LLMs have mental models. It’s that people have mental models of how various things work. For example, many people have a wrong mental model of how an oven works. They think that if they set the oven for 350 F it heats with more energy than if they set the oven for 150 F. In reality it heats at exactly the same power, but it turns off when the internal temperature reaches the set rate. For most use cases it does not matter that your mental model does not match how the oven works. But there are some not he cases where it does matter. The article shares a mental model of how LLMs work that might help users use them better.
- yellow_lead 3y agoThis is content marketing for their AI code assistant Cody, and doesn't offer much new information otherwise.
- sqs 3y agoIt’s from Feb 2023. Many people on HN are probably familiar with the point it makes by now.
- pradn 3y agoI don't think users understand how LLMs are different are different from search engines, especially for information retrieval. Someone I know well has been using ChatGPT and Bard for months, and was surprised that they don't just use the Bing/Google search engine indexes behind the scenes. The idea that LLMs are a bunch of frozen matrices is not obvious. It's hard to communicate that it's better to rely on LLMs for some classes of "reasoning" or language tasks vs simple information retrieval, particularly given retrieval does work well much of the time.
- Baeocystin 3y agoOne thing that has surprised me more than it probably should have is there appears to be a double-hump distribution between those who 'get' how an LLM works (at least to the degree that its strengths and weaknesses are understood well enough to get useful work done with one) and those who don't, with the those that don't category being very, very hard to get someone out of. I have a couple of clients who have wholeheartedly embraced ChatGPT, but are (repeatedly) shocked when it isn't just a 100% accurate answer machine. Explanations on why that is a very dangerous way to approach these things fall on deaf ears.
- pradn 3y agoI wonder if people fall into a "happy valley of ignorance", where users don't actually see how an LLM can be wrong, rarely actually are met with hallucinations, and their use of LLM output is rarely a big enough problem. Whereas, we technical people who know its a bunch of matrix operations are so skeptical that we don't put this amazing technology to much use at all.
- thefz 3y agoIf one needs AI to tell them how to read a method signature, they are in the wrong job line.