16 ms·

Legal models hallucinate in 1 out of 6 (or more) benchmarking queries

• bithive123 2y ago
  For anyone who still doesn't understand why more and more folks are pointing out that LLMs "hallucinate" 100% of the time, let me put it to you this way: what is the LLM doing differently when it generates tokens that are "wrong" compared to when the tokens are "right"? If there is a difference, where does that exist? In the mechanism of the LLM, or in your mind? Bonus question 1: Why do humans produce speech? Is it a representation of text? Why then do humans produce text? Is there an intent to communicate, or merely to construct a "correct" symbolic representation? Bonus question 2: It's been said that every possible phrase is encoded in the constant pi. Do we think pi has intelligence? Intent?
  • throw46365 2y ago
    > If there is a difference, where does that exist? In the mechanism of the LLM, or in your mind? Thank you for this sentence: it is hard to get across how often Gen-AI proponents are actually projecting perceived success onto LLMs while downplaying error.
    • empath75 2y ago
      I mostly see the reverse.
      • throw46365 2y ago
        You mostly see people projecting perceived error onto LLMs? I don't think I've seen a single article about an AI getting things wrong, recently, where there was a nuanced notion about whether it was actually wrong. I don't think we're anywhere close to "nuanced mistakes are the main problem" yet.
        • empath75 2y ago
          I mostly see people ignoring successes and playing up every error.
          • throw46365 2y ago
            But the errors are fundamental, and the successes actually subjective as a result. That is, it appears to get things right, really a lot, but the conclusions people draw about why it gets things right are undermined by the nature of the errors. Like, it must have a world model, it must understand the meaning of... etc.; the nature of the errors they are downplaying fundamentally undermines the certainty of these projections.
  • sigmoid10 2y ago
    >what is the LLM doing differently when it generates tokens that are "wrong" compared to when the tokens are "right"? When an LLM is trained, it essentially compresses the knowledge of the training data corpus into a world model. "Right" and "false" are thereby only emergent when you have a different world model for yourself that tells you a different answer, most likely because the LLM was undertrained on the target domain. But to the LLM, the token with the highest probability will be the most likely correct answer, similarly to how you might have a "gut feeling" when asked about something which you clearly don't understand and have no education in. And you will be wrong just as often. The perceived overconfidence of wrong answers likely stems from human behaviour in the training data as well. LLMs are not better than humans, but they are also not worse. They are just a distilled encoding of human behaviour, which in turn might be all that the human mind is in the end.
    • smogcutter 2y ago
      > But to the LLM, the token with the highest probability will be the most likely correct answer This is precisely what people are identifying as problematic
    • swatcoder 2y ago
      No. LLM's become fluent in constructing coherent, sophisticated text in natural language from training on obscene amounts of coherent, sophisticated text in natural language. Importantly, there is no such corpus of text that contains only accurate knowledge, let alone knowledge as it unambiguously applies to some specific domain. It's unclear that any such corpus could exist (a millennias old discussion in philosophy with no possible resolution), but even if you take for granted that such a corpus could, we don't have one. So what happens is that after learning how to construct coherent, sophisticated text in natural language from all the bullshit-adled general text that includes truth and fiction and lies and fantasy and bullshit and garbage and old text and new text, there is a subsequent effort to sort of tune the model in on some generating useful text towards some purpose. And here, again, it's important to distinguish that this subsequent training is about utility ("you're a helpful chatbot", "this will trigger a function call that will supplement results", etc) and so still can't focus strictly on knowledge. LLM's can produce intelligent output that may be correct and may be verifiable, but the way they work and the way they need to be trained prevents them from ever actually representing knowledge itself. The best they can do is create text that is more or less fluent and more or less useful. It's awesome and has lots and lots of potential, but it's a radically different thing than a material individual that's composed of countless disparate linguistic and non-linguistic systems that have never yet been technologically replicated or modeled.
  • temporarely 2y ago
    The terminology is wrong but your point is valid. There is no internal criteria or mechanism for statement verification. As the mind likely is also in part a high dimensional construct and LLMs to an extent represent our collective jumble of 'notions' it is natural that their emits resonate with human users. Q1: A ""correct" symbolic representation" of x. What is x? Your "Is there an intent to communicate, or" choice construct is problematic. Why would one require a "symbolic representation" of x, x likely being a 'meaningful thought'. So this is a hot debate whether semantics is primary or merely application. I believe it is primary in which case "symbolic representation" is 'an aid' to gaining a concrete sense of what is 'somehow' 'understood'. You observe a phenomena, and understand its dynamics. You may even anticipate it while observing. To formalize that understanding is the beginning of 'expression'. Q2: because while there is a function LLM(encodings, q) that emits 'plausible' responses, an equivalent function for Pi does not exist outside of 'pure inexpressible realm of understanding' :)
    • salawat 2y ago
      >I believe it is primary in which case "symbolic representation" is 'an aid' to gaining a concrete sense of what is 'somehow' 'understood'. There is nothing magic about perception to distinguish it meaningfully from symbolic representation; in point of fact, that which you experience is in and of itself a symbolic representation of the world around you. You do not sense the frequencies outside the physical transduction capabilities of the human ear, or the wavelengths similarly beyond the capability to transduce of the human eye, or feel distinct contact beyond the density of haptic transduction of somatic nerves. Nevertheless, those phenomena are still there, and despite their insensible nature, have an effect on you. Your entire perception is a map, which one would be well advised to not mistake for the territory. To dismiss symbolic representation as something that only occurs on communication after perception is to "lose sight" of the fact that all the data your mind integrates into a perception is itself, symbolic. Communication, and symbolic representation is all there is, and it happens long before we even get to the partnof reality where I'm trying to select words to converse about it with you.
      • Gormo 2y ago
  • Drakim 2y ago
    What is the difference between a google map style application that shows pixels that are "right" for a road, and pixels that are "wrong" for a road? A pixel is a pixel, colors cannot be true or false. The only way we can say a pixel is right or wrong in google maps is whether the act of the human utilizing that pixel to understand geographical information results in the human correctly navigating the road. In the same way, an LLM can tell me all kinds of things, and those things are just words, which are just characters, which are just bytes, which are just electrons. There is no truth or false value to an electron flowing in a specific pattern in my CPU. The real question is what I, the human, get out of reading those words, if I end up correctly navigating and understanding something about the world based on what I read.
    • deleted 2y ago
      [deleted]
    • _boffin_ 2y ago
      But then again, that road, according to a neighboring country who thinks that it’s their land and it isn’t a road. Depending on what country you’re in, it makes a difference
    • tines 2y ago
      > The real question is what I, the human, get out of reading those words So then you agree with the OP that an LLM is not intelligent in the same way that Google Maps is not intelligent? That seems to be where your argument leads but you're replying in a way that makes me think you are disagreeing with the op.
      • contravariant 2y ago
        Any reply will always sound like someone is disagreeing, even if they claim not to. Though in this case I'm not even sure what the comment they're supposedly disagreeing with is even claiming. Is it even claiming anything?
        • tines 2y ago
  • dangerwill 2y ago
    Do we actually think this way though? When I am talking with someone I am cognating about what information and emotion I want to impart to the person / thinking about how they are perceiving me and the sentence construction flows from these intents. Only the sentence construction is even analogous to token generation, and even then, we edit our sentences in our heads all the time before or while talking. Instead of just a constant forward stream of tokens from the LLM.
  • jellicle 2y ago
    If I make a machine that makes random sounds in approximately the human vocal range, and occasionally humans who listen to it hear "words" (in their language, naturally), then is that machine telling the truth when words are heard and "hallucinating" the rest of the time?
    • bombcar 2y ago
      Now I'm imagining a device that takes as input the roaring of a furnace, and only outputs when it recognizes words.
  • mcguire 2y ago
    (An aside: Writing is a representation of speech, not the other way around.)
  • visarga 2y ago
    > what is the LLM doing differently when it generates tokens that are "wrong" compared to when the tokens are "right"? When they don't recall correctly, it is hallucination. When they recall perfectly, it is regurgitation/copyright infringement. We find issue either way. May I remind you that we also hallucinate, memory plays tricks on us. We often google stuff just to be sure. It is not the hallucination part that is a real difference between humans and LLMs. > Why do humans produce speech? We produce language to solve body/social/environment related problems. LLMs don't have bodies but they do have environments, such as a chat room, where the user is the environment for the model. In fact chat rooms produce trillions of tokens per month worth of interaction and immediate feedback. If you look at what happens with those trillions of tokens - they go into the heads of hundreds of millions of people, who use the LLM assistance to solve their problems, and of course produce real world effects. Then it will reflect in the next training set, creating a second feedback loop between LLM and environment. By the way, humans don't produce speech individually, if left alone, without humanity as support. We only learn speech when we get together. Language is social. Human brain is not so smart on its own, but language collects experience across generations. We rely on language for intelligence to a larger degree than we like to admit. Isn't it a mystery how LLMs learned so many language skills purely from imitating us without their own experience? It shows just how powerful language is on its own. And it shows it can be independent on substrate.
    • jgalt212 2y ago
      > When they don't recall correctly, it is hallucination. When they recall perfectly, it is regurgitation/copyright infringement. We find issue either way. You nailed it right there.
  • gwern 2y ago
    > let me put it to you this way: what is the LLM doing differently when it generates tokens that are "wrong" compared to when the tokens are "right"? It is conditioning on latents about truth, falsity, reliability, and calibration. All of these inferred latents have been shown to exist inside LLMs, as they need to exist for LLMs to do their jobs in accurately predicting the next token. (Imagine trying to predict tokens in, say, discussions about people arguing or critiques of fictional stories, or discussing mistakes made by people, and not having latents like that!) LLMs also model other things: for example, you can use them to predict demographic information about the authors of texts (https://arxiv.org/abs/2310.07298 https://arxiv.org/abs/2310.07298), even though this is something that pretty much never exists IRL, a piece of text with a demographic label like "written by a 28yo"; it is simply a latent that the LLM has learned for its usefulness, and can be tapped into. This is why a LLM can generate text that it thinks was written by a Valley girl in the 1980s, or text which is 'wrong', or text which is 'right', and this is why you see things like in Codex, they found that if the prompted code had subtle bugs, the completions tend to have subtle bugs - because the model knows there's 'good' and 'bad' code, and 'bad' code will be followed by more bad code, and so on. This should all come as no surprise - what else did you think would happen? - but pointing out that for it to be possible, the LLM has to be inferring hidden properties of the text like the nature of its author, seems to surprise people.
    • timr 2y ago
      > It is conditioning on latents about truth, falsity, reliability, and calibration. All of these inferred latents have been shown to exist inside LLMs, as they need to exist for LLMs to do their jobs in accurately predicting the next token. No, it isn't, and no, they haven't [1], and no, they don't. The only thing that "needs to exist" for an LLM to generate the next token is a whole bunch of training data containing that token, so that it can condition based on context. You can stare at your navel and claim that these higher-level concepts end up encoded in the bajillions of free parameters of the model -- and hey, maybe they do -- but that's not the same thing as "conditioning on latents". There's no explicit representation of "truth" in an LLM, just like there's no explicit representation of a dog in Stable Diffusion. Do the thought exercise: if you trained an LLM on nothing but nonsense text, would it produce "truth"? LLMs "hallucinate" precisely because they have no idea what truth means. It's just a weird emergent outcome that when you train them on the entire internet, they generate something close to enough to truthy, most of the time. But it's all tokens to the model. [1] I have no idea how you could make the claim that something like a latent conceptualization of truth is "proven" to exist, given that proving any non-trivial statement true or false is basically impossible. How would you even evaluate this capability?
  • Lerc 2y ago
    >what is the LLM doing differently when it generates tokens that are "wrong" compared to when the tokens are "right"? If there is a difference, where does that exist? In the mechanism of the LLM, or in your mind? If there were a detectable difference within the mechanism, the problem of hallucinations would be easy to fix. There may be ways to analyze logits to find patterns of uncertainty characteristics related to hallucinations. Perhaps deeper introspection of weights might turn up patterns. The difference isn't really in your mind either. The difference is simply that the one answer correlates with reality and the other does not. The point of AI models is to generalize from the training data, that implicitly means generating output that it hasn't seen as input. Perhaps the issue is not so much that it is generalizing/guessing but the degree to which making a guess is the right call is dependent on context.
  • yarg 2y ago
    Bonus question 2 is the most ridiculous straw man I've seen in a very long time. The existence of arbitrary string encodings in transcendental numbers proves absolutely nothing about the processing capabilities of adaptive algorithms.
    • pennomi 2y ago
      Exactly. Reading digits of pi doesn’t converge toward anything. (And neither do infinite typewriter monkeys.) Any correct value they get is random, and exceedingly rare. LLMs corral a similar randomness to statistically answer things correctly more often than not.
  • ToValueFunfetti 2y ago
    A witness to a crime thinks that there were 6 shots fired; in fact there were only 2. They remember correctly the gender of the criminal, the color of their jacket, the street corner where it happened, and the time. There is no difference in their mind between the true memories and the false memory. I write six pieces of code that I believe have no bugs. One has an off-by-one error. I didn't have any different experience writing the buggy code than I did writing the functional code, and I must execute the code to understand that anything different occurred. Shall I conclude that myself and the witness were hallucinating when we got the right answers? That intelligence is not the thing that got us there?
    • sangnoir 2y ago
      People who are consistently unable to distinguish fiction from reality make terribly witnesses; if an obviously high crackhead would fare better than an LLM on the witness stand.
    • Gormo 2y ago
      > Shall I conclude that myself and the witness were hallucinating when we got the right answers? If you were recalling stored memories of experiences that were actual interactions with external reality, and some of those memories were subsequently corrupted, then no, you were not hallucinating. If you were engaging in a completely endogenous stochastic process to generate information independently of any interaction with external reality, then yes, you were hallucinating. > That intelligence is not the thing that got us there? It's not. In both cases, the information you are recalling is stored data generated by external input. The storage medium happens to be imperfect, however, and occasionally flips bits, so later reads might not exactly match what was written. But in neither case was the original data generated via a procedural algorithm independently of external input.
  • poniexprsns 2y ago
    [dead]
  • apantel 2y ago
    Here’s the issue: humans do the same thing: the brain builds up a model of the world but the model is not the world. It is a virtual approximation or interpretation based on training data: passed experiences, perceptions, etc. A human can tell you the sky is blue based on its model. So can any LLM. The sky is blue. So the output from both models is truthy.
    • Gormo 2y ago
      > A human can tell you the sky is blue based on its model. So can any LLM. The sky is blue. So the output from both models is truthy. But a human can also tell you that the sky is blue based looking at the sky, without engaging in any model-based inference. An LLM cannot do that, and can only rely on its model. Humans can engage in both empirical observation and stochastic inference. An LLM can only engage in stochastic inference. So while both can be truthy, only humans currently have the capacity to be truthful. It's also worth pointing out that even if human minds worked the same way as LLMs, our training data consists of an aggregation of exactly those empirical observation -- we are tokenizing and correlating our actual experiences of reality, and only subsequently representing the output of our inferences with words. The LLM, on the other hand, is trained only on that second-order data -- the words -- without having access to the much more thorough primary data that it represents.
• spdustin 2y ago
  I'm going to go out on a limb and say that LexisNexis and Thompson Reuters didn't do nearly enough (if any) taxonomical engineering of the corpus before deciding to just do some sliding window chunking. Without that, the whole "think/plan" part of a natural language query pipeline is all but useless. I just want to bang a marching bass drum while walking through their office and continually shout, "you still have to do the messy part!"
• dukeofdoom 2y ago
  Are laws even logically consistent. They seem arbitrary at times, and even product of their time, like fashion.
  • meepmorp 2y ago
    What does that have to do with whether or not LLMs spit out bullshit or not?
    • dukeofdoom 2y ago
      Wouldn't it increase the probability of bullshit, if they're trained on bullshit.
• happypumpkin 2y ago
  I have an uncle who is an attorney in X state. I had him try, using GPT4, a bunch of prompts about X state law in his specialty and the rate was of hallucination was much higher than 1 in 6. Probably half or more were incorrect. Often the answers would be correct for other states, but not for X state. Alternatively, they were correct for X state at a certain point in time, but no longer are.
• 1970-01-01 2y ago
  Yes, legal analysis is probably the most consistent scenario where I've seen it completely fail. I'm glad this is finally getting more attention. https://news.ycombinator.com/item?id=40366267 https://news.ycombinator.com/item?id=40366267 https://news.ycombinator.com/item?id=40348947#40362183 https://news.ycombinator.com/item?id=40348947#40362183
• elicksaur 2y ago
  Let someone stake their license on LLMs giving legal advice. We’ve seen this happen, and it seems to typically result in sanctions. Until that’s not the typical outcome, LLM performance is all just talk.
• tiahura 2y ago
  The solution is simple - put the relevant law in the prompt. The larger context size of Claude lets you shove in half-a-dozen cases, your petition, and a few other docs. When you do that, the quality is more than passable, and instantaneous.
• scotty79 2y ago
  How does that compare to human lawyers? How often human lawyers produce plausibly sounding wrong bs?
• rayiner 2y ago
  The article’s cited examples of errors, such as not understanding that a precedent has been overruled or inventing a provision of the bankruptcy code, gives a fascinating insight into how LLMs work and their limitations.
• lionkor 2y ago
  I feel like LLMs just show how utterly stupid we all are, fundamentally. All it takes is a machine using full sentences, and suddenly we assume it must be smart. This is why humans fall for scams, and this is why humans get into cults. I guess thats something LLMs can teach us; how flawed we are in evaluating whether someone is speaking the truth without researching anything.
• cushpush 2y ago
  Geoffrey Hinton says the term "confabulate" is more apt to describe the mutex/mutating memory recall operation of the brain as it summons "known facts" and "gets them wrong." confabulation is a feature, not a bug, of the brain, and also a feature, not a bug. If the goal is to reach brainlike computing, we are getting closer.
• he0001 2y ago
  When talking to LLMs their responses are always a bit off. The best way I can describe it, it’s like they are speaking a specific dialect but you know they are using expressions that wouldn’t be used in that dialect. Or, it’s like they are way over their head in the specific matter and reiterate things they really don’t understand. Like there’s no substance in what they are saying. And if they do this for a simple or a matter that you know of, how can you trust their answers in a matter if you can’t evaluate their answer? It doesn’t get better when they are totally mixing things up but say it with certainty, instead of just saying that they don’t know or admit that they have gaps in what they know. They don’t just get it wrong like humans do, it’s just weirdly wrong. Humans usually get it consistently wrong. Perhaps it’s about what they are not saying. Would I always be able to tell it’s not a human I talk with, probably not. But chances are that I will know, the longer I talk to them.
• RecycledEle 2y ago
  How would a random attorney do? I wish someone would get free consultations with 100 attorneys and record the meetings. Then grade their answers in the same way the LLMs were graded. My guess (and it is only a guess) is that the AIs outperform real attorneys during free consultations by hallucinating less and giving truthful answers more often.
• GiorgioG 2y ago
  60% of the time it works every time.
• TheBlight 2y ago
  Does it really matter?
  • jimbokun 2y ago
    Does it matter if there's a one in six chance following your lawyer's advice ends up with you in jail?
  • newprint 2y ago
    yes, it does. There is already at least one cases where law firm messed-up writing docs using AI.
    • deleted 2y ago
      [deleted]
  • swarnie 2y ago
    Only if you want a factual answer that can be depended on. If you just want a text bot that makes statements one word at a time we can stop throwing 100bs at this sector and some marketing departments need to come in this weekend and redo a pitch or two.
  • deleted 2y ago
    [deleted]
  • swatcoder 2y ago
    For commercializing generative AI tools, which needs to happen in order for all the investment to pay off, it absolutely does. In an adversarial industry like law, where there's often another party actively invested in challenging your work, high error rates in source material translate directly to high risk of failure/rejection in a case or claim. Putting numbers on the failure rate for current products helps the industry understand whether those products are reliable enough yet and what degree of oversight they require.
• roughly 2y ago
  LLMs hallucinate 100% of the time - hallucinating is what they Do. Often they’re hallucinating things that also occur in reality, but it sounds like the legal products are only doing so 66-83% of the time.
  • soulofmischief 2y ago
    Words are meaningless if we don't adhere to some common definition. "Hallucination" and "generation" are two separate concepts, and it is unproductive to conflate them.
    • mycall 2y ago
      Indeed. Humans hallucinate under these definitions.
      • roughly 2y ago
        Which, indeed, we do. The current understanding of perception is that sense inputs serve to correct the brain's existing model of the world - you do not look at something and then perceive it, you generate an image of it and then use your vision to correct it.
      • kevindamm 2y ago
        Actual hallucination is quite eye-opening in part because of the realization that perception lenses all our senses _all the time_. So while I agree that it's over-broadening the definition of hallucination, I think it ironically appropriate too.
      • jedbrown 2y ago
        The point is that LLMs are never right for the right reason. Humans who understand the subject matter can make mistakes, but they are mistakes of a different nature. The issue reminds me of this from Terry Tao (LLMs being not-even pre-rigorous, but adept at forging the style of rigorous exposition): It is perhaps worth noting that mathematicians at all three of the above stages of mathematical development can still make formal mistakes in their mathematical writing. However, the nature of these mistakes tends to be rather different, depending on what stage one is at: 1. Mathematicians at the pre-rigorous stage of development often make formal errors because they are unable to understand how the rigorous mathematical formalism actually works, and are instead applying formal rules or heuristics blindly. It can often be quite difficult for such mathematicians to appreciate and correct these errors even when those errors are explicitly pointed out to them. 2. Mathematicians at the rigorous stage of development can still make formal errors because they have not yet perfected their formal understanding, or are unable to perform enough “sanity checks” against intuition or other rules of thumb to catch, say, a sign error, or a failure to correctly verify a crucial hypothesis in a tool. However, such errors can usually be detected (and often repaired) once they are pointed out to them. 3. Mathematicians at the post-rigorous stage of development are not infallible, and are still capable of making formal errors in their writing. But this is often because they no longer need the formalism in order to perform high-level mathematical reasoning, and are actually proceeding largely through intuition, which is then translated (possibly incorrectly) into formal mathematical language. The distinction between the three types of errors can lead to the phenomenon (which can often be quite puzzling to readers at earlier stages of mathematical development) of a mathematical argument by a post-rigorous mathematician which locally contains a number of typos and other formal errors, but is globally quite sound, with the local errors propagating for a while before being cancelled out by other local errors. (In contrast, when unchecked by a solid intuition, once an error is introduced in an argument by a pre-rigorous or rigorous mathematician, it is possible for the error to propagate out of control until one is left with complete nonsense at the end of the argument.) https://terrytao.wordpress.com/career-advice/theres-more-to-mathematics-than-rigour-and-proofs/ https://terrytao.wordpress.com/career-advice/theres-more-to-...
• betaby 2y ago
  AI tools hallucinate 100% of the time for me for the security related topics, in particular encryption.
  • kromem 2y ago
    Which models and how are you prompting/setting the context? In some cases hallucinations are unavoidable, but 100% sounds like it might be a usage issue.
    • betaby 2y ago
      Yes, I'm holding it wrong.
• dcchambers 2y ago
  This is the (near term) future of white collar jobs: AI does most of the work, but real people will still be required to audit and verify basically everything. I don't know how much net gain productivity there will be...in certain professions an error rate that high is unacceptable and dangerous.
  • __loam 2y ago
    Sounds like an awesome way for an entire professional class to slowly lose their skills.
    • dcchambers 2y ago
      Yup...that's my fear. That and the fact that we're no longer training and leveling up juniors. Companies want to hire just one senior employee and let them use AI as they would a team of junior employees. But since we're no longer training juniors, we're no longer creating seniors. What happens when there's no one skilled enough to replace those senior employees when they leave?
      • __loam 2y ago
        Companies continuing the pattern of offloading training costs while complaining that it's hard to find skilled workers. LLMs are just an excuse.
      • Workaccount2 2y ago
        This is only really an issue if AI plateaus right around now. I think the consensus though is still pretty strongly that these cutting edge models we have now are going to be laughably bad in a few years.
  • romeros 2y ago
    Similar to the self checkout process at Walmart/HEB etc. One person will be there for any issues and will intervene minimally.
    • 2y ago
• glorygut123 2y ago
  How much of the time do humans hallucinate?
  • neolefty 2y ago
    Well, we are always using our internal models for work like this. When our internal models and what we say don't match reality, we might call it "hallucination" or "getting it wrong" or even "lying". When we get it right — our internal model matches reality, and we also say something that matches reality — we call it "truthfulness", and it's super valuable. But I think there are two really different sources of hallucination / inaccuracy / lies. One is that the internal model is wrong — "Oops, I was wrong. The CVS isn't on Main St." The other is when we decide to deceive. "Haha, I sent you to Main St. CVS is really on Center Rd." Two very different internal processes with the same outcome.
    • Gormo 2y ago
      "All models are wrong, but some are useful". If we were only engaging in model-based inference, then we, too, would always be hallucinating. But the very thing you're pointing out -- that we act differently when our internal model is wrong vs. when it is right -- is the crux of the difference. We use models, but then we have the ability to immediately test the output of those models for correctness, because we have semantic, not just syntactic, awareness of both the input and output data. We have criteria for determining the accuracy of what our model is producing. LLMs don't, and are only capable of engaging in stochastic inference from the pre-defined model, which solely represents syntactic patterns, and have no ability to determine whether anything they output is semantically correct.
  • raydev 2y ago
    If I’m paying them, not much, and I can hold them accountable if they’re too wrong.
  • yoyohello13 2y ago
    Who cares? Humans are accountable. This is about LLMs.
  • 2y ago
• parhamn 2y ago
  I try to read a few scotus rulings a year and I always wonder if anyone actually reads all the citations, notes, references, etc. It just feels like doing it right would take hundreds of hours. Sometimes I wonder if the judges have read their full final opinions or if its just the clerks. Im genuinely curuous, how does this work in regular law? Does anyone ever check all the citations, etc? Does a judge really deep read all 200 pages of filings and arguments? Do they just scan them efficiently as a lot of it is familiar?
  • kemitchell 2y ago
    I practice law in California. Lawyers absolutely read every word of court decisions—citations, notes, appendices, concurrences, and dissents. Usually multiple times. But that tends to happen when looking to cite, criticize, or argue about a particular decision that matters. In other words, when you might have to defend your interpretation against another lawyer. And when you have the time. Which is money. Lawyers reading for other reasons may skim, hunt for specific passages, or even just glance any provided syllabus, depending on why they are reading. When you've read enough of these, by the same judges, living and dead, over and over, you get a sense not just of formatting and structure but written and thinking style, as well. You get better at determining whether and when it's worth a full, deep read. One of the things American law schools teach by experience in the first year is that the level of attention, organization, and critical thinking expected in reading is far higher, in its own peculiar way, than what most students are used to putting in, even from very strong academic backgrounds. Then they develop endurance for it, by assigning many cases to read for each class session, several times a week, for several classes at once. All in a competitive environment where your hiring prospects largely come down to grades and your grades come down to a single exam per subject, each awarded on strict statistical curves. Part of it's that you learn what you need to read. Part of it's that you just grind. When you've been grinding long enough, you don't even feel it anymore. It still hurts, but it's a long, slow abrasion on your mind and personality. Not a stitch in your side anymore. I'll never tell anyone off from reading Supreme Court opinions. It's your court! As long as you can keep it. But for most folks reading for interest, the syllabus is fine. If you want just a little more than that, read the intro and concluding sections of the "opinion of the court". Then read the intro of each dissent, if there are any.
• garbanz0 2y ago
  i get the sense that the solution to this problem is more use of LLMs (running critical feedback and review in a loop) rather than less use of LLMs. If you can build good tooling around current kinda dumb LLMs now to lower that number, we will be in a pretty good position as the foundational models continue to improve.
  • lyu07282 2y ago
    Yeah I'd imagine the problem is not verifying the output against retrieved documents. If it just hallucinates it would ignore the given context, something that can absolutely be verified by another LLM.
• duxup 2y ago
  I was wondering if these are fly by night kinda random AI startup type companies offering these services. >we put the claims of two providers, LexisNexis (creator of Lexis+ AI) and Thomson Reuters (creator of Westlaw AI-Assisted Research and Ask Practical Law AI)), to the test LexisNexis, and Thomson Reuters, those are established big companies offering these services.
  • __loam 2y ago
    I would be interested in how specialist companies like Harvey, which is backed by OpenAI, are doing in this space. They're pretty secretive though.
    • janalsncm 2y ago
      If I was a law firm, I would not trust my client’s information with Harvey or OpenAI. It honestly doesn’t matter what they promise if my reputation is on the line. No company can promise not to be hacked or leaked.
      • deleted 2y ago
        [deleted]
      • tiahura 2y ago
        But your firm can guarantee it won’t be hacked? Or broken into. Or social engineered. Or…
        • meepmorp 2y ago
          What's more likely - someone targeting your particular law firm or someone targeting OpenAI?
          • sjy 2y ago
            I’d say a small law firm. They don’t have a security team, they’re routinely targeted by phishers, and their data is easier to convert into money than the huge mass of (mostly completely uninteresting) data fed into ChatGPT.
• fitzgera1d 2y ago
  What's the hallucination rate of human lawyers? Spent $1,400/hour on a series of unusable drafts just the other day.
  • pbhjpbhj 2y ago
    Pretty low if they want to pass bar exams.
  • NoboruWataya 2y ago
    In the sense of just completely making up references out of nowhere? Very low. Humans make mistakes, of course, but they tend not to be so egregious. Grounding a submission on a made up reference is quite likely to be fatal to your case in a way that most human errors aren't.
  • echelon 2y ago
    Hence the excitement behind investing in AI legal tools.
    • fitzgera1d 2y ago
      Further, I now get emails from legal that regularly seem drafted by ChatGPT.
  • squigz 2y ago
    This is the important bit that's always missing in discussions about LLM/AI applications to existing industries - what is the rate of mistakes for a human worker?
• 101008 2y ago
  I know LLMs and genAI are trendy and they seem espectacular and work like magic, but maybe they are not the tool for every problem. I can see them working fine in a lot of aspects/problem, but that doesn't mean they will work fine in every aspect that involves text, even if they are refined, etc. I guess once the fashion passes (probably not soon, unfortunately), people will see the real value of LLMs and will start applying them to problem that are solvable by them, and not absolutely everything.
  • Workaccount2 2y ago
    No one knows what the real value of LLMs is though. Its a nascent technology. We could be discussing the internet circa 1994 or we could be discussing bitcoin circa 2011.
    • skydhash 2y ago
      > the internet circa 1994 or we could be discussing bitcoin circa 2011. People were clear about the value of both. SMTP by itself was already a clear value for business. The WWW itself was invented in 1989. Same for Bitcoin as the title of the original paper, "Bitcoin: A Peer-to-Peer Electronic Cash System ", explains it all. Some capabilities were not there yet, but it was mostly infrastructure and engineering issues (Acceptance was another issue for bitcoin). LLMs generate tokens. The generation is governed by the models, but the model itself has no concept of usefulness or truth. Books are not enough to train them. The WWW has a lot of garbage informations. And natural language is insufficient to describe exact processes without formalizing it. The fact is that the human mind can already do what the LLMs fans hope them to achieve. And we can transfer knowledge through media. Getting something done is mostly assembling the right people in a room, recursively. And we built tools to enhance productivity. I believe that Microsoft and others are hoping that the general public will be ok with wrong results. But the fact is that all the capabilities exposed so far have an alternative and more deterministic process to achieve the same result. But it often requires learning or hiring a specialist and it's sad to see how many people are balking at that.
  • __loam 2y ago
    Problems like auto-generating spam emails for Nigerian princes.
• elitan 2y ago
  What's the same rate for humans?
  • Gormo 2y ago
    LLMs are only capable of hallucinating, whereas humans are capable of hallucinating, but are also capable of empirically observing reality. So whatever the rate is, it's necessarily lower than those for LLMs.
• pmarreck 2y ago
  Lawyers and programmers are valuable because of the rigor of the precise wording (or code) they write, and this is also where LLM's can screw up. I wonder if tweaking the randomness factor might help here
• VancouverMan 2y ago
  That's roughly the range of nonsensical or incorrect answers I've gotten from various human lawyers, accountants, financial advisors, and other professionals over the years. I've come to expect about a fifth to a quarter of their advice to have problems with it. Being wrong a third of the time would be a notable improvement for the human medical professionals I've dealt with. It's basically a coin-flip when it comes to the correctness and quality of what they claim.
  • malicka 2y ago
    There is a key difference: When an AI is wrong, it’s “buyer beware; you knew AI isn’t infallible.” You have no recourse. When a human is incompetent, you can blame them or their employer. You have recourse.
    • zarathustreal 2y ago
      Wait, where do you live? You are able to afford legal fees to fight a hospital??
      • freejazz 2y ago
        Smarmy response, but the point about malpractice stands.
        • zarathustreal 2y ago
          I don’t think that word “smarmy” means what you think it means, and there is no valid point about malpractice - that’s the essence of the comment. Laws are only as effective as the means of enforcing them, and in this case not everyone has equal access to the justice system.
          • freejazz 2y ago
            No, they don't. Not everyone has equal access to AI either, or medicine for that matter. You can feel free to disagree. I did, after all.
• neolefty 2y ago
  We think of truthfulness as boolean — whether in humans or AI models — but it may be more useful to think of it as a skill. You can become more truthful by aligning your inner model with reality and then also by checking it with ground truth. A challenging and intrinsically imperfect process either way!
  • Gormo 2y ago
    LLMs don't have access to reality, though. They're trained on the symbolic output of other models.
• ImAnAmateur 2y ago
  I got an ad for an AI legal consultant and it made me upset. Someone is trying to make a cheap buck by outsourcing legal advice to an unreliable source. Now I'm more prepared to explain how it's a bad idea, instead of just why. I like that the article explains a few different ways hallucinations creep in, besides the obvious. Maybe what's most needed is a better retrieval/search system? If the AI can't fetch high quality data, then it's doomed before it tries.
  • mtlmtlmtlmtl 2y ago
    What really upsets me is the idea of making therapist LLMs. They may be able to regurgitate an impressive amount of psychology texts, but that doesn't give them the theory of mind, observational skill, experience or judgement needed to be a good psychotherapist.
    • lazide 2y ago
      That one is doomed to failure from a customer perspective IMO - not because people want, or therapists do, anything that you’re saying though. But often because the real reason why people are in therapy is to get someone else’s perspective on what they’re experiencing. And fundamentally, that is destined to fail when it’s not an actual human involved.
• RecycledEle 2y ago
  My guess (and this is only a guess) is that LLMs hallucinate on legal questions because they expect things to make sense, but the law does not make sense. Lawyers have a vested interest in keeping the law an indecipherable and self-contradicting mess so that people have to pay them to get answers and help. This is the same reason (corrupt) judges treat attorneys better than people who represent themselves and refuse to point out what the law says in court.
  • lyu07282 2y ago
    People's hatred for lawyers isn't going to distract them from their hatred of LLMs it seems, so you get downvoted. People's context window is as short as an LLMs lmao
  • pbhjpbhj 2y ago
    Ramble: It's probably best to take a step back and ask what you actually want an LLM to do for you. Using RAG and just having it raise references to consider is probably the best you can expect. I have sometimes asked legal question (to which I pretty much know the answer) of LLMs and my consideration is that a lot of the time the section of an Act (UK) is the principle node about which a piece of information is anchored, but the anchoring is very loose and many sections are often mentioned in proximity, leading to poor differentiation of the relevance of a document/sentence/word vector to a particular piece of legislation. It might be fixed by training NER to recognise treaties/acts/conventions and the jurisdiction and always using that to label references; I suspect the "1" in "section 1" or "17 USC 1", say, is not being tokenised as part of the "section" and this contributes to poor results. Maybe someone has worked on this? Also, context for the jurisdiction in which a discussion is taking place is often not really given - can the LLM tell that a law firm is talking about PCT, or EPC rather than USC when it discusses first-to-file for patent law and nothing in the document itself mentions the jurisdiction or any of those three initialisms? How about when the same law firm whose blog is mentioning these things represents the same clients at WIPO, EPO and USPTO? If you're going to fine-tune for that with human question-answer sessions you're going to need some really skilled workers who know the field well. You probably then need specific prompt templates for legal questions too. Then they need to layer in precedence and authority of different courts, recognise obiter statements from other statements, recognise summaries of positions of different parties aren't original statements by the speaker, disregard ultra vires statements, ... simples.
• Severian 2y ago
  JFC, this is why lawyers exist. If you trust AI LLMs to answer your legal questions accurately you are a special kind of fool.
  • scblock 2y ago
    These tools are designed for and marketed to lawyers to use. These are not generalist LLM products. Your "this is why lawyers exist" statement makes no sense in the context of these products. One of the models studied markets itself with "AI-Assisted Research on Westlaw Precision is the first generative AI offering from Thomson Reuters and will help legal professionals find the answers they need faster and with high confidence." Another says "Most attorneys know Practical Law as the trusted, up-to-date legal know-how platform that helps them get accurate answers across major practice areas. Its newest enhancement, Ask Practical Law AI is a generative AI search tool that dramatically improves the way you access the trusted expertise from Practical Law." A third says "Transform your legal work with Lexis+ AI"
  • throw46365 2y ago
    It's already actual lawyers themselves trusting ChatGPT to generate paperwork: https://www.theguardian.com/world/2024/feb/29/canada-lawyer-chatgpt-fake-cases-ai https://www.theguardian.com/world/2024/feb/29/canada-lawyer-... https://arstechnica.com/tech-policy/2023/06/lawyers-have-real-bad-day-in-court-after-citing-fake-cases-made-up-by-chatgpt/ https://arstechnica.com/tech-policy/2023/06/lawyers-have-rea... Also Michael Cohen used ChatGPT, sent cases to his attorney and then realised they were nonsense: https://www.npr.org/2023/12/30/1222273745/michael-cohen-ai-fake-legal-cases https://www.npr.org/2023/12/30/1222273745/michael-cohen-ai-f... I don't know how else to get this message across, but it does this all the time in all subjects. It doesn't just occasionally hallucinate mistakes. The mechanism by which it makes non-mistakes is identical and it can't tell the difference. There is no profession where you a) you shouldn't prefer an expert over ChatGPT and b) you won't find experts idiotically using ChatGPT to reduce their workloads. This is why it's a grotesquely inappropriately positioned and marketed set of products.
    • Severian 2y ago
• glenstein 2y ago
  Large language models are one particular implementation of machine intelligence optimized for generating language but not necessarily for being true or for being grounded in sourceable to reference material or facts. I think we're taking models that have been optimized for a particular thing and repurposing them towards new tasks and then assuming that the principles of large language models as we have developed them are just universally inherent properties of AI, which I think is not at all the case. Conceivably, you absolutely could train models where their own built-in metric of success involves a notion of appropriateness of citations. The models we have simply don't do that, but I don't think we should proceed to the assumption that we are witnessing the future of AI with hallucinations everywhere, I just think it's a lazy repurposing of systems that were not initially designed with these things in mind.
  • Terr_ 2y ago
    > optimized for a particular thing and repurposing them A lot of human exuberance (and thus investment) is riding on the questionable idea that a really good text fragments correlation specialist system can usefully impersonate a generalist AI. ("LLM, which rocks are the best to eat?") Even worse: Some people assume an exceptional text-specialist model will usefully meta-impersonate a generalist model impersonating a specific and different kind of specialist! ("LLM, do my taxes.")
  • thfuran 2y ago
    How many of the large systems in the world (civil infrastructure, bureaucracies, the internet, etc) don't involve a great deal of repurposing previously present systems for purposes beyond their original design intent? I don't think we're guaranteed to be stuck with hallucinating AI embedded in everything, but if something gets popular enough to work its way into a lot of tech stacks, it can easily persist well beyond the point where anyone designing things from the ground up would include it simply because it was already there.
  • duxup 2y ago
    Yeah as far as my personal use cases, code or basic Q & A that sort of chains together naturally. It's easy enough for me to quick test these ideas or code. I can typically eyeball it when it is wrong and if not then, I'll run it and see. Done. It works for me, but mostly because I can easily validate and I'm confidant in quick / painless validations on my end. Something in depth that involves a great deal more multi step reasoning to get from point A to B, I've got my doubts. If anything that's where I see AI wander off the path into a mess.
• resource_waste 2y ago
  I'm so good at AI, that I can beat this. Ask the same question 4 different times, on 4 different models. If any of the 4 contradict each other, have a human manually review. You can even change the question, change the seed, etc... Its interesting how few people know how to use LLMs effectively. Here is my most recent protip, after getting an answer say: "That was a 2/10 answer, give me a 10/10 answer"
  • myhf 2y ago
    > If any of the 4 contradict each other If you were able to answer this definitively, then you wouldn't need the 4 intermediate models.
  • pbhjpbhj 2y ago
    https://unify.ai/chat?default=true https://unify.ai/chat?default=true might help with that, was mentioned here a few days ago, https://news.ycombinator.com/item?id=40441945 https://news.ycombinator.com/item?id=40441945; it's a router to send prompts to multiple LLM.
• Phiwise_ 2y ago
  McCarthy warned us about this.
• localfirst 2y ago
  seeing a lot of bad takes in the comments but from enterprise sales point of view: companies aren't buying or trusting LLMs 1 out of 6 times the blackbox produces completely wonky/risky output. Yet it costs far far more to fix the changes introduced by the blackbox including the hallucinations. This is the blockchain argument all over again I see in the comments. Literally just swap it out for AI/LLM and it feels like the crypto bubble peak What AI hype is doing is giving companies to massively garnish white collar wages by making everybody think it can (when in reality AI cannot replace a sentient human interface completely) extrapolating to wild science fiction novels. A bit disappointing to see, thought reddit would be more for this type of speculation.
• russdpale 2y ago
  Knowing this, lawyers who use these tools should be disbarred. Period.