16 ms·

Lawyer cites fake cases invented by ChatGPT, judge is not amused

347 points by scraptor 3y ago

tmaly 3y ago
I have asked ChatGPT to cite sources when I have doubts. I am often times able to confirm these sources. Seems this lawyer just took ChatGPT at its word without validating the cases.
- londons_explore 3y ago
  Or maybe validated one or two, and then assumed they must all be correct.
  - tmaly 3y ago
    I often want good data, so I validate everything. ChatGPT tends to only give a limited number of results in the response.
    - chaxor 3y ago
      GPT-4 has made some substantial improvements recently in references. A few weeks ago using it would get things sort-of correct (the authors would be nearly the same, with one or two just added in or removed, maybe journal number a bit off, etc) and sometimes hallucinate more wholesale, or at least enough that searching couldn't find it as easily. Now, it gets things pretty much perfectly correct everytime as far as I can tell. It gives links that work, and exactly the right reference, every time, even without plugins. However, with plugins now, it doesn't even matter so much any more since it can browse anyway. Very impressive nonetheless.
simonwlol 3y ago
How can you be this successful and blog about the most trite bullshit 24/7. Doesn't have the sense to marry a hot chick and fuck her all day even, what is that thing he's with anyway. Bro. Sad to see.
metadat 3y ago
Discussed today: A man sued Avianca Airline – his lawyer used ChatGPT https://news.ycombinator.com/item?id=36095352 https://news.ycombinator.com/item?id=36095352 (100 comments)
- dpifke 3y ago
  Also: https://news.ycombinator.com/item?id=36092509 https://news.ycombinator.com/item?id=36092509
  - metadat 3y ago
    Thanks! Macro-expanded: ChatGPT-Authored Legal Filing “Replete with Citations to Non-Existent Cases" https://news.ycombinator.com/item?id=36092509 https://news.ycombinator.com/item?id=36092509 (67 comments, 1 day ago)
    - _Microft 3y ago
      dang is doing that frequently - but what does "macro-expanded" mean for normal users? Do you have a bookmarklet, extension or something similar for this or is there any secret incantation that makes HN generate such an entry automatically?
      - metadat 3y ago
        I just do it manually.. I've gotten pretty efficient at it, takes only a moment. Imo including title, timeline / age, and comment volume provides helpful context to readers (I always appreciate when others do this, rather than, in the most severe cases, leaving a wall of unadorned HN links). Cheers _Microft (and cool username, btw ;D)
        _Microft 3y ago
        Thanks for shedding light on this and thanks for the compliment! Have a nice day
B1FF_PSUVM 3y ago
> ChatGPT, as it often does, hallucinated wildly Plausible bullshit generation for free, as if there's not enough already available cheap.
- TechBro8615 3y ago
  The lawyer was just trying to outsource his job of plausible bullshit generation. He would have gotten away with it, too, if it weren't for the fake cases.
disillusioned 3y ago
> He had, he told Judge Castel, even asked the program to verify that the cases were real. It had said yes. It turns out, asking an unreliable narrator if it's being reliable is not a sound strategy.
- twic 3y ago
  The trick is, you need two LLMs, one which always lies, and one which always tells the truth. Then you ask either LLM whether the other LLM would say it's reliable.
- londons_explore 3y ago
  I wonder if this is a tactic so the court to deems this lawyer incompetent rather than giving the (presumably much harsher) penalty for deliberately lying to the court?
  - leereeves 3y ago
    Why assume malice? Asking ChatGPT to verify is exactly what someone who trusts ChatGPT might do. I'm not surprised this lawyer trusted ChatGPT too much. People trust their lives to self driving cars, trust their businesses to AI risk models, trust criminal prosecution to facial recognition. People outside the AI field seem to be either far too trusting or far too suspicious of AI.
    - acka 3y ago
      Quoted directly from my last session with ChatGPT mere seconds ago: > Limitations May occasionally generate incorrect information May occasionally produce harmful instructions or biased content Limited knowledge of world and events after 2021 --- A lawyer who isn't prepared to read and heed the very obvious warnings at the start of every ChatGPT chat isn't worth a briefcase of empty promises. WARNING: witty ending of previous sentence written with help from ChatGPT.
      - leereeves 3y ago
mehwoot 3y ago
No, it did not “double-check”—that’s not something it can do! And stating that the cases “can be found on legal research databases” is a flat out lie. What’s harder is explaining why ChatGPT would lie in this way. What possible reason could LLM companies have for shipping a model that does this? It did this because it's copying how humans talk, not what humans do. Humans say "I double checked" when asked to verify something, that's all GPT knows or cares about.
- deleted 3y ago
  [deleted]
  - deleted 3y ago
    [deleted]
- simonw 3y ago
  Yeah, that was my conclusion too: What’s a common response to the question “are you sure you are right?”—it’s “yes, I double-checked”. I bet GPT-3’s training data has huge numbers of examples of dialogue like this.
  - fortyseven 3y ago
    The chat thread we are currently engaging in will most likely be inside a model within the next year, too. (Fortyseven is an alright dude.)
  - jimsimmons 3y ago
    They should RLHF this behaviour out. Asking people to be aware of limitations is in similar vein as asking them to read ToC
    - coffeebeqn 3y ago
      If the model could tell when it was wrong it would be GPT-6 or 7. I think the best 4 could do is maybe it can detect when things enter the realm of the factual or mathematical etc and use a external service for that part
leereeves 3y ago
In the other discussion of this topic, a lot of people said the lawyer should be disbarred, but personally I think people should be able to trust the tools marketed by trillion dollar tech companies, and a lot of the blame should be placed on Microsoft/OpenAI for overhyping ChatGPT and understating how likely it is to mislead. And every response from ChatGPT should be preceded by a warning that it cannot be trusted.
- LelouBil 3y ago
  There is a warning each time you create a new thread, and always at the bottom of the page. I think people should check (on the same page as the tool itself) if the tool advertises itself as unreliable.
- simonw 3y ago
  > And every response from ChatGPT should be preceded by a warning that it cannot be trusted. It kind of is - the ChatGPT site has this as a permanent fixture in the footer: > ChatGPT may produce inaccurate information about people, places, or facts. That's arguably ineffective though - even lawyers evidently don't read the small print in the footer!
  - leereeves 3y ago
    Worse, it's buried in the middle of other fine print: > Free Research Preview. ChatGPT may produce inaccurate information about people, places, or facts. ChatGPT May 24 Version And it really understates the problem. It should say: Warning! ChatGPT is very likely to make shit up.
  - rsynnott 3y ago
    Monty Python nicely addressed this, over 50 years ago. > Mr. Hilton: Oh, we use only the finest juicy chunks of fresh Cornish ram's bladder, emptied, steamed, flavoured with sesame seeds, whipped into a fondue, and garnished with lark's vomit. > Inspector: LARK'S VOMIT?!?!? > Mr. Hilton: Correct. > Inspector: It doesn't say anything here about lark's vomit! > Mr. Hilton: Ah, it does, on the bottom of the box, after 'monosodium glutamate'. > Inspector: I hardly think that's good enough! I think it's be more appropriate if the box bore a great red label: 'WARNING: LARK'S VOMIT!!!' > Mr. Hilton: Our sales would plummet! https://youtu.be/3zZQQijocRI https://youtu.be/3zZQQijocRI Really, it should open every conversation with “by the way, I am a compulsive liar, and nothing I say can be trusted”. That _might_ get through to _some_ users.
ubj 3y ago
> No, it did not “double-check”—that’s not something it can do! And stating that the cases “can be found on legal research databases” is a flat out lie. What’s harder is explaining why ChatGPT would lie in this way. What possible reason could LLM companies have for shipping a model that does this? At what point does OpenAI (or any other company) become legally responsible for this kind of behavior from their LLM's? I'm not excusing the lawyer for their reckless and irresponsible use of a tool they didn't understand, but it's becoming increasingly clear that people are trusting LLM's far more than they should. In my opinion it's dangerous to keep experimenting on the general population without holding the experimenters accountable for the harm that occurs.
- yarg 3y ago
  Never? Unless they advertise it as having the capability, it's got nothing to do with them. If I hit someone with a hammer, that shit's on me, not the manufacturer.
- jeroenhd 3y ago
  > At what point does OpenAI (or any other company) become legally responsible for this kind of behavior from their LLM's? When they sell their bots to areas where lying is illegal. I.e., when a company pretends to do law. OpenAI doesn't pretend ChatGPT is a lawyer and for good reason. The lawyer who decided to outsource his work is an idiot and can't shift blame to the tool he decided to abuse.
- notatoad 3y ago
  OpenAI or any other company becomes liable when they market a product to be used in place of lawyers (or doctors, engineers, or whatever other profession) as long as we're hiring professionals to do these jobs, part of that real actual human's job is to accept the liability for their work. if a person want to use a tool to make their job easier, it's also their job to make sure that the tool is working properly. if the human isn't capable of doing that, then the human doesn't need to be involved in this process at all - we can just turn the legal system over to the LLMs. but for me, i'd prefer the humans were still responsible. in this case, "the experimenter" was the lawyer who chose to use ChatGPT for his work, not OpenAI for making the tool available. and yes, i agree, the experimenter should be held accountable.
tiahura 3y ago
Chatgpt did this to me the other day. It gave me the right answer about passenger assumption of risk with drunk driver, but the cites were imagined.
londons_explore 3y ago
If I were the judge in this case, I would be looking to throw this lawyer in prison for a month, and ban him from ever being a lawyer again... Deliberately lying to the court, as a professional who should understand the consequences, in a way likely to not be detected, and likely to change the outcome of the case, ought to be met with a really strict punishment.
- kevinpet 3y ago
  It's not clear he deliberately lied. He plausibly misunderstood a new tool he had recently been introduced to.
  - mod50ack 3y ago
    As a lawyer, though, he has a responsibility for the contents of his filings. And there are legal standards for how to research and verify these things — which he didn't follow.
  - astrea 3y ago
    And he didn't bother to verify the facts before he presented them in court? That's scientifically/logically/professionally unsound and irresponsible.
    - konschubert 3y ago
      But it’s not deliberately lying
      - roywiggins 3y ago
        If you sign you name under "I declare under penalty of perjury that the foregoing is true and correct." then you absolutely have an obligation to check that your facts are correct.
        DangitBobby 3y ago
rsynnott 3y ago
See, this is the sort of thing that makes me suspicious about the current AI hype (well, that, plus it’s largely the same people who were going on about cryptocurrency changing the world a few years ago); this stuff just isn’t very _useful_. Like, if writing plausible lies is the 2020s AI boom’s killer app, well, that’s not great, really.
- simonw 3y ago
  I disagree. I've been tracking the many, many flaws in AI pretty closely (I wrote this article, and a bunch more in this series: https://simonwillison.net/series/llm-misconceptions/ https://simonwillison.net/series/llm-misconceptions/) And yet... I'm finding ChatGPT and the like wildly useful on a personal level. I think they're deceptively hard to use: you have to put in effort to learn them, and to learn how to avoid the many traps they set for you. But once you've done that you can get very real productivity boosts from them. I use ChatGPT a dozen or so times a day, and I would be very sad to not have access to it any more. I wrote a bit more about that here: https://simonwillison.net/2023/Mar/27/ai-enhanced-development/ https://simonwillison.net/2023/Mar/27/ai-enhanced-developmen... - and if anything this effect has got even stronger for me over the two months since I wrote that.
  - isp 3y ago
    Well put. I stand by this comment: > Catch-all comment for all ChatGPT use cases: > (1) Stunning tech demo, a vision of the future today > ... yet ... > (2) There are so many sharp edges that I'm not brave (foolhardy?) enough to blindly trust the output (https://news.ycombinator.com/item?id=35068386 https://news.ycombinator.com/item?id=35068386)
- whimsicalism 3y ago
  You will see :)
- jeroenhd 3y ago
  Current AI is very useful in cases where the truth isn't very important. It's amazing for things like fantasy stories, video games. It also works in cases where existing tools can validate the bullshit AI generates (i.e. Copilot). In cases where facts and sources are important, AI cannot be trusted. You can use it as long as you validate every single word it outputs, but at that point I do wonder what the point of using AI was in the first place. It's also good at taking other existing work and creating new work out of it; not just for smart autocomplete tools like GPTs, but also for things like Stable Diffusion. Again, AI is incapable of attribution of sources, so that comes with obvious downsides, but in cases where the creator of the model have the necessary rights so they don't _need_ attribution to sell work (i.e. stock photo companies), it can be quite useful for generating things like filler images.
lamp987 3y ago
this is the future of lawyers, doctors, engineers, school teachers... dark times are ahead.
- garganzol 3y ago
  At the same time, the inventiveness of AI sometimes gives quite a few fruitful ideas for a solution. Yes, formally this can be classified as AI hallucinations, but sometimes they are so bright and well-aimed that I get goosebumps. My favorite one is phind.com - it gave me so many slightly hallucinating but nevertheless useful advices. And I was able to incorporate most of them into my professional work. The whole situation reminds me of a good friend of mine - he's super talented at inventing things and brainstorming, but he can often be caught misrepresenting the facts, and sometimes outright lying. However, the pros easily outweigh the cons if you know who you're working with.
- blibble 3y ago
  individuals that use ChatGPT losing/being struck off because it generates garbage?
- CatWChainsaw 3y ago
  We retreat into an age of darkness, clutching our crystals and consulting our horoscopes because everything has been made equally untrue with the Greatest Invention Man Ever Made. We should be so proud of ourselves. Those who trust this tripe deserve the consequences they invite on themselves.
paulhart 3y ago
There’s a joke about “an LLB having an LLM” in here somewhere.
rickreynoldssf 3y ago
There's nothing wrong with starting with ChatGPT but the lazy ambulance chaser couldn't be bothered with a quick lookup in Westlaw. That's where things start going wrong.
ZephyrBlu 3y ago
I read this as malicious/intentional initially. Do people really think they can 100% believe ChatGPT? Especially for someone like a lawyer I would expect to them verify any information they get from ChatGPT.
akiselev 3y ago
I went ahead and asked ChatGPT with the browsing plugin [1] because I was curious and it answered that it was a real case citing an article about the fake citations! After some prodding ("Are you sure?") it spat out something slightly saner citing this very article! > The case "Varghese v. China Southern Airlines Co., Ltd., 925 F.3d 1339 (11th Cir. 2019)" was cited in court documents, but it appears that there might be some confusion or controversy surrounding this citation. It was mentioned in a list of cases for which a lawyer was ordered to provide copies, according to a court order on leagle.com [2] . However, a blog post on simonwillison.net suggests that the case might not be genuine and that it might have been generated by a language model such as ChatGPT. The post discusses a situation where a lawyer might have used generated case citations in court documents without fully understanding the tool they were using. The post also includes screenshots where the language model appears to confirm the existence of the case [3]. The output is hilariously bad and it's depressing a licensed attorney actually pulled this crap. This is just more evidence that ChatGPT should not be used for anything serious without a trained human in the loop. [1] https://chat.openai.com/share/a6e27cf2-b9a6-4740-be2e-fdddab34fef5 https://chat.openai.com/share/a6e27cf2-b9a6-4740-be2e-fdddab... [2] https://www.leagle.com/decision/infdco20230414825 https://www.leagle.com/decision/infdco20230414825 [3] https://simonwillison.net/2023/May/27/lawyer-chatgpt/ https://simonwillison.net/2023/May/27/lawyer-chatgpt/ (The TFA!)
- jimsimmons 3y ago
  In the loop there indeed was a allegedly trained human in this instance
  - akiselev 3y ago
    That's not what I would call in the loop. He didn't check that the sources were real. By "in the loop" I mean actively validating statements of fact generated by ChatGPT
    - jimsimmons 3y ago
      Well they had every opportunity to inspect its output. Where in the loop and what gets inspected is not an easy problem to solve.
dataviz1000 3y ago
Do the machines in the Matrix lie?
shon 3y ago
The judge may not be amused, but we are.
Kim_Bruning 3y ago
Current LLM workflow is the epitome of "trust but verify". You need to hand-verify at some point in the process. This does end up losing you some of the time you gained by using an LLM in the first place. Fortunately you often do still come out ahead.
- CPLX 3y ago
  Honestly it’s the opposite. I go with don’t trust at all, extract everything that saves you time. It’s just not a source of truth at all, it’s a source of raw material.
Covzire 3y ago
Guessing we'll see this excuse a lot more: Get caught making stuff up, blame ChatGPT.
kordlessagain 3y ago
This is why it is very important to have the prompts fill in relevant fragments from a quality corpus. That people think these models “tell the truth” or “hallucinate” is only half the story. It’s like expecting your language center to know all the facts your visual consciousness contains, or your visual consciousness to be able to talk in full sentences. It’s only when all models are working well together the truth emerges.
- deleted 3y ago
  [deleted]
- metaphor 3y ago
  > That people think these models “tell the truth” or “hallucinate” is only half the story. A meta-problem here is in choosing to use descriptive phrases like tell the truth and hallucinate, which are human conditions that further anthropomorphize technology with no agency, making it more difficult for layman society to defend against its inherent fallibility. UX = P_Success*Benefit - P_Failure*Cost It's been well over a decade since I learned of this deviously simple relationship from UX expert Johnny Lee, and yet with every new generation of tech that has hit the market since, it's never surprising how the hype cycle results in a brazen dismissal of the latter half.
internetguy 3y ago
I don't get it... These are PROFESSIONAL lawyers that have gone to years of law school. How do they not know how to double check an answer from ChatGPT? I'm sure there's databases of cases online, it shouldn't be this difficult.
- knodi 3y ago
  Idiots come in many sizes.
- it_citizen 3y ago
  If the whole covid vaccine debate taught me one thing is that there are idiots in every jobs, even the most qualified.
  - tarxvf 3y ago
    I love how the current world is such that people on both sides of that would agree with your statement. :D (Don't clarify it, it's better this way.)
- clnq 3y ago
  We are all really stupid in a surprising number of ways. That's not all we are, but we are that, too.
ineedasername 3y ago
Hilarious. It’s important to remember: 1) ChatGPT is not a research tool 2) It sort of resembles one and will absolutely act like one if you ask it to, and it it may even produce useful results! But… 3) You have to independently verify any factual statement it makes and also 4) In my experience the longer the chat session, the more likely it is to hallucinate, reiterate, and double down on previous output
- sickcodebruh 3y ago
  This is completely true but completely in conflict with how many very large companies advertise it. I’m a paid GitHub Copilot user and recently started using their chat tool. It lies constantly and convincingly, so often that I’m starting to wonder if it wastes more time than it saves. It’s simply not capable of reliably doing its job. This is on a “Tesla autopilot” level of misrepresenting a product but on a larger scale. I hope it continues being little more than a benign or embarrassing time-waster.
  - dmix 3y ago
    Where does Github misrepresent their Chat beta? On their marketing website?
    - sickcodebruh 3y ago
      Right from https://github.com/features/preview/copilot-x https://github.com/features/preview/copilot-x: “Context aware conversations with your copilot. If you're stuck solving a problem, ask GitHub Copilot to explain a piece of code. Bump into an error? Have GitHub Copilot fix it. It’ll even generate unit tests so you can get back to building what’s next.” This is almost a Homer Simpson running for garbage commissioner level of over-promising. I think Copilot is an incredible tool, what’s possible right now is amazing and it can save time and offer value. But the degree to which it doesn’t just fail but completely misdirects is at serious odds with the breathless marketing.
  - YetAnotherNick 3y ago
    One of the only text written in chatgpt's own website that comes each time when you chat with it is "ChatGPT may produce inaccurate information about people, places, or facts."
oytis 3y ago
AI will not destroy the humanity. People who use AI will.
coldtea 3y ago
Q. Can you lie? A. As an artificial intelligence model developed by OpenAI, I don't have personal emotions, thoughts, or intentions, so I don't lie in the way humans do. My purpose is to provide accurate and helpful information based on the training I've received. However, it's important to note that the information I provide is only as accurate as the data I was trained on. If there's inaccurate or misleading information in that data, I might unintentionally output incorrect or misleading information. But this isn't considered "lying" in the human sense because it doesn't involve any intentional deceit on my part. A: How do you know it didn't really "involve any intentional deceit"? Q: As an AI model, I don't have consciousness, emotions, or intentions. The concept of "intention" applies to beings with consciousness and subjective experiences, which AI, including me, does not possess. I generate responses purely based on patterns learned during training, and there's no intent behind them because I don't have a mind, thoughts, or consciousness. When I state there's no "intentional deceit," it's based on this understanding of AI. AI does not make choices or decisions the way a human would. If a mistake or inaccuracy occurs in my responses, it's due to limitations in the data I was trained on or in my design, not because of a deliberate attempt to mislead or deceive. AI functions based on its programming and training, not on intentions or conscious thought. Q: That's begging the question though. Your answer takes for granted that you "don't have a mind, thoughts, or consciousness". What if actual consciousness (say, in humans) is also "based on patterns learned during training", only different being that human training involves sensory experience and not just a corpus of text and images? A: Your question delves into one of the most fascinating and debated topics in the fields of cognitive science, philosophy, and AI: the nature of consciousness. As of my last update in September 2021, there's no universally accepted definition of consciousness or a definitive understanding of its mechanisms. We do understand that human consciousness is associated with our brains, which process a wide range of sensory and internal information, and it's accompanied by a subjective experience — an "inner life" or "qualia". This isn't something we've observed or been able to replicate in AI. AI models like me, based on GPT-4 architecture, operate on patterns in data. The way I generate responses is by predicting the next piece of text based on what I've learned from the corpus of text I was trained on. However, there's no subjective experience involved in this process — there's no "I" that's interpreting or experiencing anything. I don't have a concept of self, and I don't have feelings or personal thoughts. While some AI models can mimic aspects of human behavior or cognition to a certain extent, there's a vast difference between mimicking and actually experiencing. When it comes to consciousness — the inner subjective experience — we don't have evidence that AI can possess this feature. Your question is profound and important, and scientists, philosophers, and AI experts are continuously exploring these topics. The answers, however, are likely to remain uncertain for a while. Q: So, theoretically at least, a future model of the GTP architecture could be conscious and even lie? A: Theoretically, it's possible that future advances in AI and cognitive science could lead to the development of systems that possess some form of consciousness. However, as of my knowledge cutoff in September 2021, there's no consensus among scientists and AI researchers that this is achievable or even definable in the context of AI. The concept of a machine consciousness would likely require a significant evolution of our understanding of both consciousness and AI. It might require capabilities far beyond pattern recognition and prediction, which are the fundamental basis of current AI systems like the GPT-4 model. It might also necessitate a rethinking of ethical and philosophical considerations surrounding AI. In terms of an AI "lying," if future AI systems were developed to simulate a higher level of cognitive complexity, including the ability to simulate deceptive behavior, they could potentially generate outputs that are intentionally misleading. However, it's important to understand that this would not be lying in the human sense. It would be a result of the programming and functionality that the AI has been designed to exhibit, not a conscious choice made by the AI. It's also critical to note that OpenAI and many other AI research and development organizations are committed to ensuring the ethical use and development of AI technologies. If an AI were to exhibit behavior that could harm people, such as deceptive behavior, it would be seen as a serious ethical issue that would need to be addressed.
armchairhacker 3y ago
[flagged]
- akiselev 3y ago
  Did you just have ChatGPT make up all these cases for you? My ChatGPT can't find any of them.
  - armchairhacker 3y ago
    What? I told mine to double-check and it was absolutely certain they are real court cases
    - akiselev 3y ago
      Spoiler alert: it lied. For example, "Smith v. Johnson, 579 F.3d 820 (4th Cir. 2023)" aka 579 F.3d 820 is actually American River Transp. Co. v. Ryan, 579 F.3d 820 (7th Cir. 2009) [1] "Baxter v. Pennsylvania Legal Ethics Board, 100 A.3d 1095 (Pa. 2022)" aka 100 A.3d 1095 is actually Julius L. Worthy v. United States, 100 A.3d 1095 (D.C. 2014) [2] "Rodriguez, 720 F.2d 124 (9th Cir. 2023)" aka 720 F.2d 124 is actually Specialty Equipment Market Ass'n v. Ruckelshaus, 720 F.2d 124 (D.C. Cir. 1983) [3] [1] https://www.courtlistener.com/opinion/1358353/american-river-transp-co-v-ryan/ https://www.courtlistener.com/opinion/1358353/american-river... [2] https://www.courtlistener.com/opinion/2741033/julius-l-worthy-v-united-states/ https://www.courtlistener.com/opinion/2741033/julius-l-worth... [3] https://www.courtlistener.com/opinion/7902070/specialty-equipment-market-assn-v-ruckelshaus/ https://www.courtlistener.com/opinion/7902070/specialty-equi...
    - simonw 3y ago
      I think you're joking but I can't quite tell!
swayvil 3y ago
Any sufficiently advanced bullshit is indistinguishable from truth.
hristov 3y ago
It is very concerning how people are anthropomorphizing chat gpt. It will get a lot of people into trouble. The media is largely to blame for this. Never mind most gushing media stories about AI, even the few stories criticizing AI treat it as a human. Chat GPT is racist, they say, it is sexist or a liar. Well it is neither of these things, because all of the above require consciousness and intent and it has none. It is not human, it is not any type of conscious being, do not treat it as such. It sticks together sentences based on existing language scanned in from the internet and millions of other sources. What it says depends on what someone else said sometime ago on some random forum on the internet, or some book or some other source stored in an available database. It is also programmed to sound extremely sure of itself, unless you flat out say it is incorrect, in which case it will immediately admit fault and apologize. Thus, asking it if it is sure is pointless. Let me tell you a less disastrous and quite a bit funnier story. A friend of mine used chat GPT for coding. My friend became really trustful of chat gpts coding prowess and asked it if it could just send him the code in file form, so he did not have to worry about copying and pasting which apparently screwed up the formatting somehow. Chat gpt helpfully told him that it could send the code to github, and my friend could download the files from there. My friend said, that is perfect. So chatgpt gave him a github account name and said find the file it created for my friend was there. My friend looked but github said that account had long ago been closed. My friend tried variations of that account name with different capitalizations, etc., but found nothing. He went back and complained to chat gpt. Chat gpt dutifully apologized and sent him another account name. He again spends time looking for the account and looking for variations, etc. Again the account has been closed. This happened a couple of more times and in the end my friend gave up and complained to me. "Why is chat GPT doing this to me? Is it mocking me? Is it getting its kicks from sending me on random wild goose chases?". I had to explain to him that no, chat gpt is not human, and it is not mocking him. What probably happened is someone on some forum asked someone else on that forum to provide him with code in files. The responder then offered to put the files on github and provided an account name. When my friend asked a similar question, chat gpt matched up the questions and provided a similar answer. When my friend said that a particular account did not work, chat gpt scoured the web for other people mentioning their github account names and provided some of those. So whenever you use chat gpt, remember that is mostly a glorified search engine. It will spit out information it has found somewhere that it calculates as matching your question. Do not attribute intent, feelings or any type of conscience to it.
manquer 3y ago
Everyone is talking about ChatGPT , but is it not possible to train a model with only actual court documents and keep “temp” low and get accuracy levels as high or better than humans? Most legal (all formal really) documents are very predictably structured and should be easy to generate
- NolF 3y ago
  Yes and no. Court decisions do generally follow a structure, but the decisions and the reasons for a final determination, may not always be clear. Judges also may throw in hypotheticals which whilst informative are not determinative. Once it gets to appellate courts and with different judges which may agree with the same outcome but on different grounds, it can get really hard to distinguish what is the test and the factors for a certain determination. It can also happen that dissenting opinions can be very persuasive, and subsequently adopted. The task effectively also requires a case and paragraph impact meter which do exist in some law databases to one extend or another, effectively weighing how subsequent rulings consider, weight, and follow past cases, caveats, exceptions, and outright considering past rulings as bad law. Then you have the issue of changing laws and the impact these may have on past cases as they may change the test and requirements needed to be considered and even much new case law needed to be developed to interpret the new legislation. So the model would need to have a historical knowledge of the law and how it was applied. You would also need to feed it relevant surrounding information that may aid in interpreting said law. In the US, clearly the founding father's opinions and beliefs appear to play a significant part on the currently more originalist interpretative school of thought. In the UK/Australia for example readings in parliament and even the underpinning reports that prompted the change in legislation may be considered where there is ambiguity in order to interpret legislation. Australian legislation nowadays also tends to incorporate an objective of the legislation and a section that says that where ambiguity exists to interpret it in a way that would further the objectives of the legislation. So, it's really not a trivial problem.
  - manquer 3y ago
    Not decisions by judges[1]!, filings by lawyers can be pretty proforma, in a ton of places they literally just fill your details and "generate" a filing. Not every filing is generate-able for sure, however there are already tools which do create standard filings for human review, this would be just an enchantment covering some more use cases. [1] It is vast gulf between generating a filing and generating a judgement. LLMs are not decision engines, generating text basis what is most likely from past data is one of worst ways we can be making decisions.
pylua 3y ago
What needs to exist is a legal compiler that lawyers can plug responses into from these tools. Sort of how a developer would use chat gpt. A giant rules engine for the law. I’m surprised one doesn’t exist or isn’t in progress that I know of. Seems like it would be very helpful
- inopinatus 3y ago
  There isn't one because the law is not a programming language. Believing otherwise is a common misconception amongst engineers, but representing law as such is (as I have said in this forum before) a leading cause of disappointment, frustration, bickering, anger, conflict, and vexatiously long and mostly unenforceable contracts. Observance of law is fundamentally about alignment with principles, not blindly following a set of rules. The latter debility is more properly associated with the administration of law, especially at its most mediocre and ritualistic.
  - pylua 3y ago
    This is less about observance with the laws and more about how to verify that lawyers are doing their jobs properly, sort of tools like ide for developer. Often times lawyers make mistakes like in the article or even file things in the wrong court or deficient forms . More like an ide for developers . That said, it is a great disappointment of mine that the law is not based on an objective, static measure.
- crooked-v 3y ago
  "The law is code" falls apart as soon as you have to take judges into account.
civilized 3y ago
Steven Schwartz not only obviously lied re: "I was unaware of the possibility that ChatGPT could generate false information", his own screenshots prove it. He asked if one of the cases generated was fake! I suspect that during the research his System 1 (fast, intuitive thinking) told him he was not responsible for the risk he knew he was incurring by relaying AI generated text. It was more like ChatGPT was his own legal secretary which he was within his rights to trust, just like the main lawyer in the case, LoDuca, trusted him to produce this research. The proceedings would have been more interesting if Schwartz had been honest about this, rather than going with the easily discoverable lie. On the other hand, it's always funny when people realize they've got themselves into deep shit and they decide the best way out is to essentially plead insanity.
lolinder 3y ago
> I apologize for the confusion earlier. Upon double-checking, I found that the case Varghese v. China Southern Airlines Co. Ltd., 925 F.3d 1339 (11th Cir. 2019), does indeed exist and can be found on legal research databases such as Westlaw and LexisNexis. This is the part that stood out to me the most. I've seen this "I apologize for the confusion earlier" language many times when using ChatGPT, and it's always when it's walking back on something that it previously said. In fact, everything about this quote sounds like a retraction. If this is a retraction then that means that there are missing screenshots in Attachment 1 wherein ChatGPT stated the cases were fictitious, and Schwartz pushed back until it retracted the retraction. I'm with Simon on this one, I think Schwartz realized his career is over and is frantically trying anything he can to cover for his mistake.
- ineedasername 3y ago
  >”I apologize… does indeed exist” This showed me that people don’t yet understand how to practice good “hygiene” when using these tools. This apparent doubling down is (usually) the product of asking it to verify something it previously output in the same chat session It tends toward logical consistency unless directly told something is wrong. As such, asking it “were you correct when you told me X?” is bad hygiene. You can “sanitize” the validation process by opening a new chat session and asking it if something is correct. You can also ask it to be adversarial and attempt to prove its prior output is wrong. Even then it’s just a quick way to see if it’s output was garbage. A positive result is not a confirmation and independent verification is necessary. Also, especially with ChatGPT, you have to understand that its role has been fine-tuned to be helpful and, to some extent, positively affirmative. This means, in my experience, that if you at all “show your hand” with a leading question or any (even unintended) indication of the answer you’re seeking, it is much more likely to output something that affirms any biases in your prompt. People keep saying that it’s trained on human conversations/texts/etc and so everything it outputs is a reflection. But that’s not quite true: ChatGPT in particular, unless you run up against firm guardrails of hate speech etc., appears to be fine tuned to a very large degree to be non confrontational. It generally won’t challenge your assumptions, so if your prompts have underlying assumptions in them (they almost always will) then ChatGPT will play along. If you’re going to ask it for anything resembling factual information you have to be as neutral and open ended in tone as possible in your prompts. And if you’re trying to do something like check a hunch you have, you should probably not be neutral and instead ask it to be adversarial. Don’t ask “Is X true?”, ask “Analyze the possibility that X is false.” Those are overly simplistic formulations of prompts but that’s the attitude you need to go into with it if you’re doing anything research-ish.
nine_k 3y ago
We can hope to start to actually rely on such models once they start learning not only in the language domain, but also in the epistemic domain. True vs false, known vs unknown, precise vs vague, agreement vs contradiction vs unrelated, things like that. Achieving that is going to be a serious technical, and also philosophical, challenge for humans. Today's LLM are a literary device. They say what sounds plausible in the universe of texts they were fed. What they say technically isn't even wrong, because they have no notion of truth, or any notion of a world beyond the words. Their output should be judged accordingly.
macrael 3y ago
Genuine question: why have these models all been trained to sound so confident? Is it not possible to have rewarded models that announced their own ignorance? Or is even that question belying an "intelligence" view of these models that isn't accurate?
- crooked-v 3y ago
  The problem is that the model doesn't know if anything it's saying is true or false, so trying to make it 'fact check' just means it will constantly interrupt itself regardless of the accuracy of the output.
- lowbloodsugar 3y ago
  I think you are confusing ChatGPT with AI. ChatGPT is a statistical fiction generator. It sounds confident because it is writing fiction. It sounds confident for precisely the same reason that billions of ignorant people world wide post "facts" on line sound confident: they are incapable of understanding their ignorance. They are just systems that have inputs and then generate outputs.
- cypress66 3y ago
  I believe the GPT4 "paper" mentions that the RHLF part destroyed the models ability to gauge it's confidence.
travisgriggs 3y ago
ChatGPT: The ultimate parlor^H^H^Hrot trick
stavros 3y ago
By the way, Simon can't get ChatGPT to say that the particular case is true _because he's asking about a particular case_. ChatGPT doesn't know it said anything about this case in another session, so it won't confirm it's true. Either ask it for some other legal sources and ask if those are true (and then try to see if a few aren't), or use the API to feed it its own answer about Varghese etc and then see if it will say it's true (because at that point you've made it think it said this).
dunham 3y ago
Wow, I asked Bard to summarize the NYT article and it completely changed the outcome: > Sure. The article is about a man named Roberto Mata who sued Avianca Airlines after he was injured when a metal serving cart struck his knee during a flight to Kennedy International Airport in New York. His lawyer used a new language model called ChatGPT to help him with the case. ChatGPT is a large language model that can generate text, translate languages, write different kinds of creative content, and answer your questions in an informative way. It was trained on a massive dataset of text and code, and it can be used for a variety of tasks, including writing legal documents. In Mr. Mata's case, ChatGPT helped him to draft a lawsuit that was more persuasive and comprehensive than what he could have written on his own. The lawsuit was eventually successful, and Mr. Mata was awarded damages. This case shows the potential of ChatGPT and other language models to help lawyers with their work.
- simonw 3y ago
  Oh dear!
  - mclightning 3y ago
    it gave me a longer, but yet again very positive sounding summary; > The article is about the use of large language models (LLMs) in the legal profession. The author argues that LLMs have the potential to revolutionize the way lawyers practice law, by automating many of the tasks that are currently done manually. For example, LLMs could be used to generate legal documents, research case law, and identify relevant precedents. The author also argues that LLMs could be used to improve the quality of legal services, by providing lawyers with access to a vast amount of information and expertise. However, the author also acknowledges that there are some challenges that need to be addressed before LLMs can be widely adopted in the legal profession. These challenges include the need for LLMs to be trained on large amounts of data, the need for LLMs to be able to understand the nuances of legal language, and the need for LLMs to be able to generate accurate and reliable results. Despite these challenges, the author believes that LLMs have the potential to make a significant impact on the legal profession, and that they are likely to become increasingly important in the years to come. Here are some of the benefits of using LLMs in the legal profession: LLMs can automate many of the tasks that are currently done manually, freeing up lawyers to focus on more complex work. LLMs can access and process vast amounts of information, which can help lawyers to make better decisions. LLMs can identify patterns and trends that would be difficult for humans to see, which can help lawyers to spot potential problems. LLMs can generate creative solutions to legal problems, which can help lawyers to find new and innovative ways to represent their clients. Here are some of the challenges that need to be addressed before LLMs can be widely adopted in the legal profession: LLMs need to be trained on large amounts of data, which can be expensive and time-consuming. LLMs need to be able to understand the nuances of legal language, which can be difficult for computers to do. LLMs need to be able to generate accurate and reliable results, which can be a challenge for complex legal problems. Overall, the author believes that LLMs have the potential to make a significant impact on the legal profession, and that they are likely to become increasingly important in the years to come.
numpad0 3y ago
ChatGPT just isn’t going to say the right thing, or thing grounded to the baseline reality, beyond what it cannot help but doing so restrained by your input. The only connection between it and this world is your input. ChatGPT is floating in the heavens, and you’re grounding it by at most a fishing line, through the textbox. It has to be framed as such. People praising it as a next gen search engine[that finds a data from database] is(perhaps this is the word that best fit the situation!), hallucinating.
LASR 3y ago
I’ve said this many times: stop using ChatGPT as a database. It does not contain facts. It may appear to contain some facts. Some may also be actually true. The truly useful usecase is as a reasoning engine. You can paste in a document and ask some questions about the facts in that document. Then it does a much better job, enough to be actually useful.
- joshka 3y ago
  To some extent it does contain facts, but those facts are indistinguishable from non-facts, are just a small proportion of the dataset compared to everything else are indistinguishable from non factual information. E.g. using text-davinci-003 (this is GPT3, not ChatGPT), "The moon is made of" completes to: Cheese 48.74%, rock: 31.66%, green 4.09% (98.75% followed by cheese), rocks 3.86%, and several other lower percentage tokens. I wonder if there eventually will be a type of model that incorporates the ability to simultaneously do text completion while adhering to facts at the model level (rather than having to bolt it on top via context).
Obscurity4340 3y ago
The new "dictated, not read"
bell-cot 3y ago
[Sigh.] Most parents understand that there's a broad range of ages where a kid is either a bit fuzzy on the concept of "truth", or reflexively lies in certain situations ("Billy broke the vase!"), or can invent fairly elaborate lies for various reasons (avoid punishment, impress other kids, bored bullsh*tting). Perhaps ChatGPT's "open relationship" with the truth could be explained in such terms...
deleted 3y ago
[deleted]
JoeAltmaier 3y ago
ChatGPT says everything with perfect confidence, like that know-it-all who will never admit they just don't know. Instead we get blathering that sounds like legalese (or chemistry or law or whatever) that has only a tenuous connection to reality. All you can say is, "People in my training set talked that way"
WheelsAtLarge 3y ago
And so it starts, I suspect we'll be seeing this issue more and more since it's easy to just get GPT to spit out some text. I believe that the true beneficiaries of LLMs are those that are experts in their fields. They can just read the output and deal with the inaccuracies. Does anyone know if training an LLM with just one type of data, law in this case, creates a more accurate output?
thunderbong 3y ago
I think I read this in a comment here on HN - > The goal of chat LLMs is not to give you an answer. The goal is to continue the conversation.
rvz 3y ago
Very unsurprising. Relying on AI sophists like ChatGPT for legal work is still just as risky for normal users and even for legal experts. The difference is, these legal experts are more qualified to carefully review and check over the outputs than the average joe / jane trying to 'replace their lawyer, solicitor, etc' with ChatGPT. I keep emphasising this importance, and to never fully trust the output of LLMs such as ChatGPT, unless a human has reviewed and checked if it is hallucinating or bullshitting. [0] [0] https://news.ycombinator.com/item?id=36091468 https://news.ycombinator.com/item?id=36091468
RobRivera 3y ago
that lawyer should be disbarred. Is it really so hard to cross reference a chatgpt response with a source?
clnq 3y ago
> What’s much harder though is actually getting it to double-down on fleshing those out. Now, it is. When ChatGPT first became public though, those were the Wild West days where you could get it to tell you anything, including all sorts of unethical things. And it would quite often double-down on "facts" it hallucinated. With current GPT-3.5 and GPT-4, the alignment is still a challenging problem, but it's in a much better place. I think it's unlikely a conversation with GPT-4 would have gone the way it did for this lawyer.
curiousgal 3y ago
Can we please, for the love of all that is holy, stop saying ChatGPT LIED? Lying means intentionally making false statements. A chatbot does not have intentions...
AmIDev 3y ago
What I have realized after using Bard(Palm2), ChatGPT(3.5) and some other LLMs is that they are good for tasks where an accuracy <100% is acceptable, and the cost of getting wrong answers is not high. For example, labeling a million text samples with 90% accuracy by using few shot learning is a good use case. Writing a poem is good use case. Trying to learn a new language is not. Generating a small function that you can verify might be ok. Writing entire codebase is not. So far, I haven't found any use case for personal use of LLMs. For work however, LLMs are going to be very useful with text(and potentially image) based machine learning tasks. Any tasks where having knowledge beyond the labeled training dataset is useful is going to be a good task for LLMs. One example is detecting fraud SMS.
BlueTemplar 3y ago
I remember someone predicting this would happen in comments here a few months ago.
Mizoguchi 3y ago
"Anyone who has worked designing products knows that users don’t read anything—warnings, footnotes, any form of microcopy will be studiously ignored" Users don't usually read long legal statements such as terms of services. That's not the case of ChatGPT interface, the note about its limitations is clearly visible and very short. This is as dumb as saying a city is at fault if someone drives into a clearly marked one way only street and causes an accident because people don't read anything.

347 points by scraptor 3y ago

I have asked ChatGPT to cite sources when I have doubts. I am often times able to confirm these sources. Seems this lawyer just took ChatGPT at its word without validating the cases.

londons_explore 3y ago
Or maybe validated one or two, and then assumed they must all be correct.
- tmaly 3y ago
  I often want good data, so I validate everything. ChatGPT tends to only give a limited number of results in the response.
  - chaxor 3y ago
    GPT-4 has made some substantial improvements recently in references. A few weeks ago using it would get things sort-of correct (the authors would be nearly the same, with one or two just added in or removed, maybe journal number a bit off, etc) and sometimes hallucinate more wholesale, or at least enough that searching couldn't find it as easily. Now, it gets things pretty much perfectly correct everytime as far as I can tell. It gives links that work, and exactly the right reference, every time, even without plugins. However, with plugins now, it doesn't even matter so much any more since it can browse anyway. Very impressive nonetheless.

simonwlol 3y ago

How can you be this successful and blog about the most trite bullshit 24/7. Doesn't have the sense to marry a hot chick and fuck her all day even, what is that thing he's with anyway. Bro. Sad to see.

metadat 3y ago

Discussed today: A man sued Avianca Airline – his lawyer used ChatGPT https://news.ycombinator.com/item?id=36095352 https://news.ycombinator.com/item?id=36095352 (100 comments)

dpifke 3y ago
Also: https://news.ycombinator.com/item?id=36092509 https://news.ycombinator.com/item?id=36092509
- metadat 3y ago
  Thanks! Macro-expanded: ChatGPT-Authored Legal Filing “Replete with Citations to Non-Existent Cases" https://news.ycombinator.com/item?id=36092509 https://news.ycombinator.com/item?id=36092509 (67 comments, 1 day ago)
  - _Microft 3y ago
    dang is doing that frequently - but what does "macro-expanded" mean for normal users? Do you have a bookmarklet, extension or something similar for this or is there any secret incantation that makes HN generate such an entry automatically?
    - metadat 3y ago
      I just do it manually.. I've gotten pretty efficient at it, takes only a moment. Imo including title, timeline / age, and comment volume provides helpful context to readers (I always appreciate when others do this, rather than, in the most severe cases, leaving a wall of unadorned HN links). Cheers _Microft (and cool username, btw ;D)
      - _Microft 3y ago
        Thanks for shedding light on this and thanks for the compliment! Have a nice day

B1FF_PSUVM 3y ago

> ChatGPT, as it often does, hallucinated wildly Plausible bullshit generation for free, as if there's not enough already available cheap.

TechBro8615 3y ago
The lawyer was just trying to outsource his job of plausible bullshit generation. He would have gotten away with it, too, if it weren't for the fake cases.

disillusioned 3y ago

> He had, he told Judge Castel, even asked the program to verify that the cases were real. It had said yes. It turns out, asking an unreliable narrator if it's being reliable is not a sound strategy.

twic 3y ago
The trick is, you need two LLMs, one which always lies, and one which always tells the truth. Then you ask either LLM whether the other LLM would say it's reliable.
londons_explore 3y ago
I wonder if this is a tactic so the court to deems this lawyer incompetent rather than giving the (presumably much harsher) penalty for deliberately lying to the court?
- leereeves 3y ago
  Why assume malice? Asking ChatGPT to verify is exactly what someone who trusts ChatGPT might do. I'm not surprised this lawyer trusted ChatGPT too much. People trust their lives to self driving cars, trust their businesses to AI risk models, trust criminal prosecution to facial recognition. People outside the AI field seem to be either far too trusting or far too suspicious of AI.
  - acka 3y ago
    Quoted directly from my last session with ChatGPT mere seconds ago: > Limitations May occasionally generate incorrect information May occasionally produce harmful instructions or biased content Limited knowledge of world and events after 2021 --- A lawyer who isn't prepared to read and heed the very obvious warnings at the start of every ChatGPT chat isn't worth a briefcase of empty promises. WARNING: witty ending of previous sentence written with help from ChatGPT.
    - leereeves 3y ago

mehwoot 3y ago

No, it did not “double-check”—that’s not something it can do! And stating that the cases “can be found on legal research databases” is a flat out lie. What’s harder is explaining why ChatGPT would lie in this way. What possible reason could LLM companies have for shipping a model that does this? It did this because it's copying how humans talk, not what humans do. Humans say "I double checked" when asked to verify something, that's all GPT knows or cares about.

deleted 3y ago
[deleted]
- deleted 3y ago
  [deleted]
simonw 3y ago
Yeah, that was my conclusion too: What’s a common response to the question “are you sure you are right?”—it’s “yes, I double-checked”. I bet GPT-3’s training data has huge numbers of examples of dialogue like this.
- fortyseven 3y ago
  The chat thread we are currently engaging in will most likely be inside a model within the next year, too. (Fortyseven is an alright dude.)
- jimsimmons 3y ago
  They should RLHF this behaviour out. Asking people to be aware of limitations is in similar vein as asking them to read ToC
  - coffeebeqn 3y ago
    If the model could tell when it was wrong it would be GPT-6 or 7. I think the best 4 could do is maybe it can detect when things enter the realm of the factual or mathematical etc and use a external service for that part

leereeves 3y ago

In the other discussion of this topic, a lot of people said the lawyer should be disbarred, but personally I think people should be able to trust the tools marketed by trillion dollar tech companies, and a lot of the blame should be placed on Microsoft/OpenAI for overhyping ChatGPT and understating how likely it is to mislead. And every response from ChatGPT should be preceded by a warning that it cannot be trusted.

LelouBil 3y ago
There is a warning each time you create a new thread, and always at the bottom of the page. I think people should check (on the same page as the tool itself) if the tool advertises itself as unreliable.
simonw 3y ago
> And every response from ChatGPT should be preceded by a warning that it cannot be trusted. It kind of is - the ChatGPT site has this as a permanent fixture in the footer: > ChatGPT may produce inaccurate information about people, places, or facts. That's arguably ineffective though - even lawyers evidently don't read the small print in the footer!
- leereeves 3y ago
  Worse, it's buried in the middle of other fine print: > Free Research Preview. ChatGPT may produce inaccurate information about people, places, or facts. ChatGPT May 24 Version And it really understates the problem. It should say: Warning! ChatGPT is very likely to make shit up.
- rsynnott 3y ago
  Monty Python nicely addressed this, over 50 years ago. > Mr. Hilton: Oh, we use only the finest juicy chunks of fresh Cornish ram's bladder, emptied, steamed, flavoured with sesame seeds, whipped into a fondue, and garnished with lark's vomit. > Inspector: LARK'S VOMIT?!?!? > Mr. Hilton: Correct. > Inspector: It doesn't say anything here about lark's vomit! > Mr. Hilton: Ah, it does, on the bottom of the box, after 'monosodium glutamate'. > Inspector: I hardly think that's good enough! I think it's be more appropriate if the box bore a great red label: 'WARNING: LARK'S VOMIT!!!' > Mr. Hilton: Our sales would plummet! https://youtu.be/3zZQQijocRI https://youtu.be/3zZQQijocRI Really, it should open every conversation with “by the way, I am a compulsive liar, and nothing I say can be trusted”. That _might_ get through to _some_ users.

ubj 3y ago

> No, it did not “double-check”—that’s not something it can do! And stating that the cases “can be found on legal research databases” is a flat out lie. What’s harder is explaining why ChatGPT would lie in this way. What possible reason could LLM companies have for shipping a model that does this? At what point does OpenAI (or any other company) become legally responsible for this kind of behavior from their LLM's? I'm not excusing the lawyer for their reckless and irresponsible use of a tool they didn't understand, but it's becoming increasingly clear that people are trusting LLM's far more than they should. In my opinion it's dangerous to keep experimenting on the general population without holding the experimenters accountable for the harm that occurs.

yarg 3y ago
Never? Unless they advertise it as having the capability, it's got nothing to do with them. If I hit someone with a hammer, that shit's on me, not the manufacturer.
jeroenhd 3y ago
> At what point does OpenAI (or any other company) become legally responsible for this kind of behavior from their LLM's? When they sell their bots to areas where lying is illegal. I.e., when a company pretends to do law. OpenAI doesn't pretend ChatGPT is a lawyer and for good reason. The lawyer who decided to outsource his work is an idiot and can't shift blame to the tool he decided to abuse.
notatoad 3y ago
OpenAI or any other company becomes liable when they market a product to be used in place of lawyers (or doctors, engineers, or whatever other profession) as long as we're hiring professionals to do these jobs, part of that real actual human's job is to accept the liability for their work. if a person want to use a tool to make their job easier, it's also their job to make sure that the tool is working properly. if the human isn't capable of doing that, then the human doesn't need to be involved in this process at all - we can just turn the legal system over to the LLMs. but for me, i'd prefer the humans were still responsible. in this case, "the experimenter" was the lawyer who chose to use ChatGPT for his work, not OpenAI for making the tool available. and yes, i agree, the experimenter should be held accountable.

tiahura 3y ago

Chatgpt did this to me the other day. It gave me the right answer about passenger assumption of risk with drunk driver, but the cites were imagined.

londons_explore 3y ago

If I were the judge in this case, I would be looking to throw this lawyer in prison for a month, and ban him from ever being a lawyer again... Deliberately lying to the court, as a professional who should understand the consequences, in a way likely to not be detected, and likely to change the outcome of the case, ought to be met with a really strict punishment.

kevinpet 3y ago
It's not clear he deliberately lied. He plausibly misunderstood a new tool he had recently been introduced to.
- mod50ack 3y ago
  As a lawyer, though, he has a responsibility for the contents of his filings. And there are legal standards for how to research and verify these things — which he didn't follow.
- astrea 3y ago
  And he didn't bother to verify the facts before he presented them in court? That's scientifically/logically/professionally unsound and irresponsible.
  - konschubert 3y ago
    But it’s not deliberately lying
    - roywiggins 3y ago
      If you sign you name under "I declare under penalty of perjury that the foregoing is true and correct." then you absolutely have an obligation to check that your facts are correct.
      - DangitBobby 3y ago

rsynnott 3y ago

See, this is the sort of thing that makes me suspicious about the current AI hype (well, that, plus it’s largely the same people who were going on about cryptocurrency changing the world a few years ago); this stuff just isn’t very _useful_. Like, if writing plausible lies is the 2020s AI boom’s killer app, well, that’s not great, really.

simonw 3y ago
I disagree. I've been tracking the many, many flaws in AI pretty closely (I wrote this article, and a bunch more in this series: https://simonwillison.net/series/llm-misconceptions/ https://simonwillison.net/series/llm-misconceptions/) And yet... I'm finding ChatGPT and the like wildly useful on a personal level. I think they're deceptively hard to use: you have to put in effort to learn them, and to learn how to avoid the many traps they set for you. But once you've done that you can get very real productivity boosts from them. I use ChatGPT a dozen or so times a day, and I would be very sad to not have access to it any more. I wrote a bit more about that here: https://simonwillison.net/2023/Mar/27/ai-enhanced-development/ https://simonwillison.net/2023/Mar/27/ai-enhanced-developmen... - and if anything this effect has got even stronger for me over the two months since I wrote that.
- isp 3y ago
  Well put. I stand by this comment: > Catch-all comment for all ChatGPT use cases: > (1) Stunning tech demo, a vision of the future today > ... yet ... > (2) There are so many sharp edges that I'm not brave (foolhardy?) enough to blindly trust the output (https://news.ycombinator.com/item?id=35068386 https://news.ycombinator.com/item?id=35068386)
whimsicalism 3y ago
You will see :)
jeroenhd 3y ago
Current AI is very useful in cases where the truth isn't very important. It's amazing for things like fantasy stories, video games. It also works in cases where existing tools can validate the bullshit AI generates (i.e. Copilot). In cases where facts and sources are important, AI cannot be trusted. You can use it as long as you validate every single word it outputs, but at that point I do wonder what the point of using AI was in the first place. It's also good at taking other existing work and creating new work out of it; not just for smart autocomplete tools like GPTs, but also for things like Stable Diffusion. Again, AI is incapable of attribution of sources, so that comes with obvious downsides, but in cases where the creator of the model have the necessary rights so they don't _need_ attribution to sell work (i.e. stock photo companies), it can be quite useful for generating things like filler images.

lamp987 3y ago

this is the future of lawyers, doctors, engineers, school teachers... dark times are ahead.

garganzol 3y ago
At the same time, the inventiveness of AI sometimes gives quite a few fruitful ideas for a solution. Yes, formally this can be classified as AI hallucinations, but sometimes they are so bright and well-aimed that I get goosebumps. My favorite one is phind.com - it gave me so many slightly hallucinating but nevertheless useful advices. And I was able to incorporate most of them into my professional work. The whole situation reminds me of a good friend of mine - he's super talented at inventing things and brainstorming, but he can often be caught misrepresenting the facts, and sometimes outright lying. However, the pros easily outweigh the cons if you know who you're working with.
blibble 3y ago
individuals that use ChatGPT losing/being struck off because it generates garbage?
CatWChainsaw 3y ago
We retreat into an age of darkness, clutching our crystals and consulting our horoscopes because everything has been made equally untrue with the Greatest Invention Man Ever Made. We should be so proud of ourselves. Those who trust this tripe deserve the consequences they invite on themselves.

paulhart 3y ago

There’s a joke about “an LLB having an LLM” in here somewhere.

rickreynoldssf 3y ago

There's nothing wrong with starting with ChatGPT but the lazy ambulance chaser couldn't be bothered with a quick lookup in Westlaw. That's where things start going wrong.

ZephyrBlu 3y ago

I read this as malicious/intentional initially. Do people really think they can 100% believe ChatGPT? Especially for someone like a lawyer I would expect to them verify any information they get from ChatGPT.

akiselev 3y ago

I went ahead and asked ChatGPT with the browsing plugin [1] because I was curious and it answered that it was a real case citing an article about the fake citations! After some prodding ("Are you sure?") it spat out something slightly saner citing this very article! > The case "Varghese v. China Southern Airlines Co., Ltd., 925 F.3d 1339 (11th Cir. 2019)" was cited in court documents, but it appears that there might be some confusion or controversy surrounding this citation. It was mentioned in a list of cases for which a lawyer was ordered to provide copies, according to a court order on leagle.com [2] . However, a blog post on simonwillison.net suggests that the case might not be genuine and that it might have been generated by a language model such as ChatGPT. The post discusses a situation where a lawyer might have used generated case citations in court documents without fully understanding the tool they were using. The post also includes screenshots where the language model appears to confirm the existence of the case [3]. The output is hilariously bad and it's depressing a licensed attorney actually pulled this crap. This is just more evidence that ChatGPT should not be used for anything serious without a trained human in the loop. [1] https://chat.openai.com/share/a6e27cf2-b9a6-4740-be2e-fdddab34fef5 https://chat.openai.com/share/a6e27cf2-b9a6-4740-be2e-fdddab... [2] https://www.leagle.com/decision/infdco20230414825 https://www.leagle.com/decision/infdco20230414825 [3] https://simonwillison.net/2023/May/27/lawyer-chatgpt/ https://simonwillison.net/2023/May/27/lawyer-chatgpt/ (The TFA!)

jimsimmons 3y ago
In the loop there indeed was a allegedly trained human in this instance
- akiselev 3y ago
  That's not what I would call in the loop. He didn't check that the sources were real. By "in the loop" I mean actively validating statements of fact generated by ChatGPT
  - jimsimmons 3y ago
    Well they had every opportunity to inspect its output. Where in the loop and what gets inspected is not an easy problem to solve.

dataviz1000 3y ago

Do the machines in the Matrix lie?

shon 3y ago

The judge may not be amused, but we are.

Kim_Bruning 3y ago

Current LLM workflow is the epitome of "trust but verify". You need to hand-verify at some point in the process. This does end up losing you some of the time you gained by using an LLM in the first place. Fortunately you often do still come out ahead.

CPLX 3y ago
Honestly it’s the opposite. I go with don’t trust at all, extract everything that saves you time. It’s just not a source of truth at all, it’s a source of raw material.

Covzire 3y ago

Guessing we'll see this excuse a lot more: Get caught making stuff up, blame ChatGPT.

kordlessagain 3y ago

This is why it is very important to have the prompts fill in relevant fragments from a quality corpus. That people think these models “tell the truth” or “hallucinate” is only half the story. It’s like expecting your language center to know all the facts your visual consciousness contains, or your visual consciousness to be able to talk in full sentences. It’s only when all models are working well together the truth emerges.

deleted 3y ago
[deleted]
metaphor 3y ago
> That people think these models “tell the truth” or “hallucinate” is only half the story. A meta-problem here is in choosing to use descriptive phrases like tell the truth and hallucinate, which are human conditions that further anthropomorphize technology with no agency, making it more difficult for layman society to defend against its inherent fallibility. UX = P_Success*Benefit - P_Failure*Cost It's been well over a decade since I learned of this deviously simple relationship from UX expert Johnny Lee, and yet with every new generation of tech that has hit the market since, it's never surprising how the hype cycle results in a brazen dismissal of the latter half.

internetguy 3y ago

I don't get it... These are PROFESSIONAL lawyers that have gone to years of law school. How do they not know how to double check an answer from ChatGPT? I'm sure there's databases of cases online, it shouldn't be this difficult.

knodi 3y ago
Idiots come in many sizes.
it_citizen 3y ago
If the whole covid vaccine debate taught me one thing is that there are idiots in every jobs, even the most qualified.
- tarxvf 3y ago
  I love how the current world is such that people on both sides of that would agree with your statement. :D (Don't clarify it, it's better this way.)
clnq 3y ago
We are all really stupid in a surprising number of ways. That's not all we are, but we are that, too.

ineedasername 3y ago

Hilarious. It’s important to remember: 1) ChatGPT is not a research tool 2) It sort of resembles one and will absolutely act like one if you ask it to, and it it may even produce useful results! But… 3) You have to independently verify any factual statement it makes and also 4) In my experience the longer the chat session, the more likely it is to hallucinate, reiterate, and double down on previous output

sickcodebruh 3y ago
This is completely true but completely in conflict with how many very large companies advertise it. I’m a paid GitHub Copilot user and recently started using their chat tool. It lies constantly and convincingly, so often that I’m starting to wonder if it wastes more time than it saves. It’s simply not capable of reliably doing its job. This is on a “Tesla autopilot” level of misrepresenting a product but on a larger scale. I hope it continues being little more than a benign or embarrassing time-waster.
- dmix 3y ago
  Where does Github misrepresent their Chat beta? On their marketing website?
  - sickcodebruh 3y ago
    Right from https://github.com/features/preview/copilot-x https://github.com/features/preview/copilot-x: “Context aware conversations with your copilot. If you're stuck solving a problem, ask GitHub Copilot to explain a piece of code. Bump into an error? Have GitHub Copilot fix it. It’ll even generate unit tests so you can get back to building what’s next.” This is almost a Homer Simpson running for garbage commissioner level of over-promising. I think Copilot is an incredible tool, what’s possible right now is amazing and it can save time and offer value. But the degree to which it doesn’t just fail but completely misdirects is at serious odds with the breathless marketing.
- YetAnotherNick 3y ago
  One of the only text written in chatgpt's own website that comes each time when you chat with it is "ChatGPT may produce inaccurate information about people, places, or facts."

oytis 3y ago

AI will not destroy the humanity. People who use AI will.

coldtea 3y ago

Q. Can you lie? A. As an artificial intelligence model developed by OpenAI, I don't have personal emotions, thoughts, or intentions, so I don't lie in the way humans do. My purpose is to provide accurate and helpful information based on the training I've received. However, it's important to note that the information I provide is only as accurate as the data I was trained on. If there's inaccurate or misleading information in that data, I might unintentionally output incorrect or misleading information. But this isn't considered "lying" in the human sense because it doesn't involve any intentional deceit on my part. A: How do you know it didn't really "involve any intentional deceit"? Q: As an AI model, I don't have consciousness, emotions, or intentions. The concept of "intention" applies to beings with consciousness and subjective experiences, which AI, including me, does not possess. I generate responses purely based on patterns learned during training, and there's no intent behind them because I don't have a mind, thoughts, or consciousness. When I state there's no "intentional deceit," it's based on this understanding of AI. AI does not make choices or decisions the way a human would. If a mistake or inaccuracy occurs in my responses, it's due to limitations in the data I was trained on or in my design, not because of a deliberate attempt to mislead or deceive. AI functions based on its programming and training, not on intentions or conscious thought. Q: That's begging the question though. Your answer takes for granted that you "don't have a mind, thoughts, or consciousness". What if actual consciousness (say, in humans) is also "based on patterns learned during training", only different being that human training involves sensory experience and not just a corpus of text and images? A: Your question delves into one of the most fascinating and debated topics in the fields of cognitive science, philosophy, and AI: the nature of consciousness. As of my last update in September 2021, there's no universally accepted definition of consciousness or a definitive understanding of its mechanisms. We do understand that human consciousness is associated with our brains, which process a wide range of sensory and internal information, and it's accompanied by a subjective experience — an "inner life" or "qualia". This isn't something we've observed or been able to replicate in AI. AI models like me, based on GPT-4 architecture, operate on patterns in data. The way I generate responses is by predicting the next piece of text based on what I've learned from the corpus of text I was trained on. However, there's no subjective experience involved in this process — there's no "I" that's interpreting or experiencing anything. I don't have a concept of self, and I don't have feelings or personal thoughts. While some AI models can mimic aspects of human behavior or cognition to a certain extent, there's a vast difference between mimicking and actually experiencing. When it comes to consciousness — the inner subjective experience — we don't have evidence that AI can possess this feature. Your question is profound and important, and scientists, philosophers, and AI experts are continuously exploring these topics. The answers, however, are likely to remain uncertain for a while. Q: So, theoretically at least, a future model of the GTP architecture could be conscious and even lie? A: Theoretically, it's possible that future advances in AI and cognitive science could lead to the development of systems that possess some form of consciousness. However, as of my knowledge cutoff in September 2021, there's no consensus among scientists and AI researchers that this is achievable or even definable in the context of AI. The concept of a machine consciousness would likely require a significant evolution of our understanding of both consciousness and AI. It might require capabilities far beyond pattern recognition and prediction, which are the fundamental basis of current AI systems like the GPT-4 model. It might also necessitate a rethinking of ethical and philosophical considerations surrounding AI. In terms of an AI "lying," if future AI systems were developed to simulate a higher level of cognitive complexity, including the ability to simulate deceptive behavior, they could potentially generate outputs that are intentionally misleading. However, it's important to understand that this would not be lying in the human sense. It would be a result of the programming and functionality that the AI has been designed to exhibit, not a conscious choice made by the AI. It's also critical to note that OpenAI and many other AI research and development organizations are committed to ensuring the ethical use and development of AI technologies. If an AI were to exhibit behavior that could harm people, such as deceptive behavior, it would be seen as a serious ethical issue that would need to be addressed.

armchairhacker 3y ago

[flagged]

akiselev 3y ago
Did you just have ChatGPT make up all these cases for you? My ChatGPT can't find any of them.
- armchairhacker 3y ago
  What? I told mine to double-check and it was absolutely certain they are real court cases
  - akiselev 3y ago
    Spoiler alert: it lied. For example, "Smith v. Johnson, 579 F.3d 820 (4th Cir. 2023)" aka 579 F.3d 820 is actually American River Transp. Co. v. Ryan, 579 F.3d 820 (7th Cir. 2009) [1] "Baxter v. Pennsylvania Legal Ethics Board, 100 A.3d 1095 (Pa. 2022)" aka 100 A.3d 1095 is actually Julius L. Worthy v. United States, 100 A.3d 1095 (D.C. 2014) [2] "Rodriguez, 720 F.2d 124 (9th Cir. 2023)" aka 720 F.2d 124 is actually Specialty Equipment Market Ass'n v. Ruckelshaus, 720 F.2d 124 (D.C. Cir. 1983) [3] [1] https://www.courtlistener.com/opinion/1358353/american-river-transp-co-v-ryan/ https://www.courtlistener.com/opinion/1358353/american-river... [2] https://www.courtlistener.com/opinion/2741033/julius-l-worthy-v-united-states/ https://www.courtlistener.com/opinion/2741033/julius-l-worth... [3] https://www.courtlistener.com/opinion/7902070/specialty-equipment-market-assn-v-ruckelshaus/ https://www.courtlistener.com/opinion/7902070/specialty-equi...
  - simonw 3y ago
    I think you're joking but I can't quite tell!

swayvil 3y ago

Any sufficiently advanced bullshit is indistinguishable from truth.

hristov 3y ago

It is very concerning how people are anthropomorphizing chat gpt. It will get a lot of people into trouble. The media is largely to blame for this. Never mind most gushing media stories about AI, even the few stories criticizing AI treat it as a human. Chat GPT is racist, they say, it is sexist or a liar. Well it is neither of these things, because all of the above require consciousness and intent and it has none. It is not human, it is not any type of conscious being, do not treat it as such. It sticks together sentences based on existing language scanned in from the internet and millions of other sources. What it says depends on what someone else said sometime ago on some random forum on the internet, or some book or some other source stored in an available database. It is also programmed to sound extremely sure of itself, unless you flat out say it is incorrect, in which case it will immediately admit fault and apologize. Thus, asking it if it is sure is pointless. Let me tell you a less disastrous and quite a bit funnier story. A friend of mine used chat GPT for coding. My friend became really trustful of chat gpts coding prowess and asked it if it could just send him the code in file form, so he did not have to worry about copying and pasting which apparently screwed up the formatting somehow. Chat gpt helpfully told him that it could send the code to github, and my friend could download the files from there. My friend said, that is perfect. So chatgpt gave him a github account name and said find the file it created for my friend was there. My friend looked but github said that account had long ago been closed. My friend tried variations of that account name with different capitalizations, etc., but found nothing. He went back and complained to chat gpt. Chat gpt dutifully apologized and sent him another account name. He again spends time looking for the account and looking for variations, etc. Again the account has been closed. This happened a couple of more times and in the end my friend gave up and complained to me. "Why is chat GPT doing this to me? Is it mocking me? Is it getting its kicks from sending me on random wild goose chases?". I had to explain to him that no, chat gpt is not human, and it is not mocking him. What probably happened is someone on some forum asked someone else on that forum to provide him with code in files. The responder then offered to put the files on github and provided an account name. When my friend asked a similar question, chat gpt matched up the questions and provided a similar answer. When my friend said that a particular account did not work, chat gpt scoured the web for other people mentioning their github account names and provided some of those. So whenever you use chat gpt, remember that is mostly a glorified search engine. It will spit out information it has found somewhere that it calculates as matching your question. Do not attribute intent, feelings or any type of conscience to it.

manquer 3y ago

Everyone is talking about ChatGPT , but is it not possible to train a model with only actual court documents and keep “temp” low and get accuracy levels as high or better than humans? Most legal (all formal really) documents are very predictably structured and should be easy to generate

NolF 3y ago
Yes and no. Court decisions do generally follow a structure, but the decisions and the reasons for a final determination, may not always be clear. Judges also may throw in hypotheticals which whilst informative are not determinative. Once it gets to appellate courts and with different judges which may agree with the same outcome but on different grounds, it can get really hard to distinguish what is the test and the factors for a certain determination. It can also happen that dissenting opinions can be very persuasive, and subsequently adopted. The task effectively also requires a case and paragraph impact meter which do exist in some law databases to one extend or another, effectively weighing how subsequent rulings consider, weight, and follow past cases, caveats, exceptions, and outright considering past rulings as bad law. Then you have the issue of changing laws and the impact these may have on past cases as they may change the test and requirements needed to be considered and even much new case law needed to be developed to interpret the new legislation. So the model would need to have a historical knowledge of the law and how it was applied. You would also need to feed it relevant surrounding information that may aid in interpreting said law. In the US, clearly the founding father's opinions and beliefs appear to play a significant part on the currently more originalist interpretative school of thought. In the UK/Australia for example readings in parliament and even the underpinning reports that prompted the change in legislation may be considered where there is ambiguity in order to interpret legislation. Australian legislation nowadays also tends to incorporate an objective of the legislation and a section that says that where ambiguity exists to interpret it in a way that would further the objectives of the legislation. So, it's really not a trivial problem.
- manquer 3y ago
  Not decisions by judges[1]!, filings by lawyers can be pretty proforma, in a ton of places they literally just fill your details and "generate" a filing. Not every filing is generate-able for sure, however there are already tools which do create standard filings for human review, this would be just an enchantment covering some more use cases. [1] It is vast gulf between generating a filing and generating a judgement. LLMs are not decision engines, generating text basis what is most likely from past data is one of worst ways we can be making decisions.

pylua 3y ago

What needs to exist is a legal compiler that lawyers can plug responses into from these tools. Sort of how a developer would use chat gpt. A giant rules engine for the law. I’m surprised one doesn’t exist or isn’t in progress that I know of. Seems like it would be very helpful

inopinatus 3y ago
There isn't one because the law is not a programming language. Believing otherwise is a common misconception amongst engineers, but representing law as such is (as I have said in this forum before) a leading cause of disappointment, frustration, bickering, anger, conflict, and vexatiously long and mostly unenforceable contracts. Observance of law is fundamentally about alignment with principles, not blindly following a set of rules. The latter debility is more properly associated with the administration of law, especially at its most mediocre and ritualistic.
- pylua 3y ago
  This is less about observance with the laws and more about how to verify that lawyers are doing their jobs properly, sort of tools like ide for developer. Often times lawyers make mistakes like in the article or even file things in the wrong court or deficient forms . More like an ide for developers . That said, it is a great disappointment of mine that the law is not based on an objective, static measure.
crooked-v 3y ago
"The law is code" falls apart as soon as you have to take judges into account.

civilized 3y ago

Steven Schwartz not only obviously lied re: "I was unaware of the possibility that ChatGPT could generate false information", his own screenshots prove it. He asked if one of the cases generated was fake! I suspect that during the research his System 1 (fast, intuitive thinking) told him he was not responsible for the risk he knew he was incurring by relaying AI generated text. It was more like ChatGPT was his own legal secretary which he was within his rights to trust, just like the main lawyer in the case, LoDuca, trusted him to produce this research. The proceedings would have been more interesting if Schwartz had been honest about this, rather than going with the easily discoverable lie. On the other hand, it's always funny when people realize they've got themselves into deep shit and they decide the best way out is to essentially plead insanity.

lolinder 3y ago

> I apologize for the confusion earlier. Upon double-checking, I found that the case Varghese v. China Southern Airlines Co. Ltd., 925 F.3d 1339 (11th Cir. 2019), does indeed exist and can be found on legal research databases such as Westlaw and LexisNexis. This is the part that stood out to me the most. I've seen this "I apologize for the confusion earlier" language many times when using ChatGPT, and it's always when it's walking back on something that it previously said. In fact, everything about this quote sounds like a retraction. If this is a retraction then that means that there are missing screenshots in Attachment 1 wherein ChatGPT stated the cases were fictitious, and Schwartz pushed back until it retracted the retraction. I'm with Simon on this one, I think Schwartz realized his career is over and is frantically trying anything he can to cover for his mistake.

ineedasername 3y ago
>”I apologize… does indeed exist” This showed me that people don’t yet understand how to practice good “hygiene” when using these tools. This apparent doubling down is (usually) the product of asking it to verify something it previously output in the same chat session It tends toward logical consistency unless directly told something is wrong. As such, asking it “were you correct when you told me X?” is bad hygiene. You can “sanitize” the validation process by opening a new chat session and asking it if something is correct. You can also ask it to be adversarial and attempt to prove its prior output is wrong. Even then it’s just a quick way to see if it’s output was garbage. A positive result is not a confirmation and independent verification is necessary. Also, especially with ChatGPT, you have to understand that its role has been fine-tuned to be helpful and, to some extent, positively affirmative. This means, in my experience, that if you at all “show your hand” with a leading question or any (even unintended) indication of the answer you’re seeking, it is much more likely to output something that affirms any biases in your prompt. People keep saying that it’s trained on human conversations/texts/etc and so everything it outputs is a reflection. But that’s not quite true: ChatGPT in particular, unless you run up against firm guardrails of hate speech etc., appears to be fine tuned to a very large degree to be non confrontational. It generally won’t challenge your assumptions, so if your prompts have underlying assumptions in them (they almost always will) then ChatGPT will play along. If you’re going to ask it for anything resembling factual information you have to be as neutral and open ended in tone as possible in your prompts. And if you’re trying to do something like check a hunch you have, you should probably not be neutral and instead ask it to be adversarial. Don’t ask “Is X true?”, ask “Analyze the possibility that X is false.” Those are overly simplistic formulations of prompts but that’s the attitude you need to go into with it if you’re doing anything research-ish.

nine_k 3y ago

We can hope to start to actually rely on such models once they start learning not only in the language domain, but also in the epistemic domain. True vs false, known vs unknown, precise vs vague, agreement vs contradiction vs unrelated, things like that. Achieving that is going to be a serious technical, and also philosophical, challenge for humans. Today's LLM are a literary device. They say what sounds plausible in the universe of texts they were fed. What they say technically isn't even wrong, because they have no notion of truth, or any notion of a world beyond the words. Their output should be judged accordingly.

macrael 3y ago

Genuine question: why have these models all been trained to sound so confident? Is it not possible to have rewarded models that announced their own ignorance? Or is even that question belying an "intelligence" view of these models that isn't accurate?

crooked-v 3y ago
The problem is that the model doesn't know if anything it's saying is true or false, so trying to make it 'fact check' just means it will constantly interrupt itself regardless of the accuracy of the output.
lowbloodsugar 3y ago
I think you are confusing ChatGPT with AI. ChatGPT is a statistical fiction generator. It sounds confident because it is writing fiction. It sounds confident for precisely the same reason that billions of ignorant people world wide post "facts" on line sound confident: they are incapable of understanding their ignorance. They are just systems that have inputs and then generate outputs.
cypress66 3y ago
I believe the GPT4 "paper" mentions that the RHLF part destroyed the models ability to gauge it's confidence.

travisgriggs 3y ago

ChatGPT: The ultimate parlor^H^H^Hrot trick

stavros 3y ago

By the way, Simon can't get ChatGPT to say that the particular case is true _because he's asking about a particular case_. ChatGPT doesn't know it said anything about this case in another session, so it won't confirm it's true. Either ask it for some other legal sources and ask if those are true (and then try to see if a few aren't), or use the API to feed it its own answer about Varghese etc and then see if it will say it's true (because at that point you've made it think it said this).

dunham 3y ago

Wow, I asked Bard to summarize the NYT article and it completely changed the outcome: > Sure. The article is about a man named Roberto Mata who sued Avianca Airlines after he was injured when a metal serving cart struck his knee during a flight to Kennedy International Airport in New York. His lawyer used a new language model called ChatGPT to help him with the case. ChatGPT is a large language model that can generate text, translate languages, write different kinds of creative content, and answer your questions in an informative way. It was trained on a massive dataset of text and code, and it can be used for a variety of tasks, including writing legal documents. In Mr. Mata's case, ChatGPT helped him to draft a lawsuit that was more persuasive and comprehensive than what he could have written on his own. The lawsuit was eventually successful, and Mr. Mata was awarded damages. This case shows the potential of ChatGPT and other language models to help lawyers with their work.

simonw 3y ago
Oh dear!
- mclightning 3y ago
  it gave me a longer, but yet again very positive sounding summary; > The article is about the use of large language models (LLMs) in the legal profession. The author argues that LLMs have the potential to revolutionize the way lawyers practice law, by automating many of the tasks that are currently done manually. For example, LLMs could be used to generate legal documents, research case law, and identify relevant precedents. The author also argues that LLMs could be used to improve the quality of legal services, by providing lawyers with access to a vast amount of information and expertise. However, the author also acknowledges that there are some challenges that need to be addressed before LLMs can be widely adopted in the legal profession. These challenges include the need for LLMs to be trained on large amounts of data, the need for LLMs to be able to understand the nuances of legal language, and the need for LLMs to be able to generate accurate and reliable results. Despite these challenges, the author believes that LLMs have the potential to make a significant impact on the legal profession, and that they are likely to become increasingly important in the years to come. Here are some of the benefits of using LLMs in the legal profession: LLMs can automate many of the tasks that are currently done manually, freeing up lawyers to focus on more complex work. LLMs can access and process vast amounts of information, which can help lawyers to make better decisions. LLMs can identify patterns and trends that would be difficult for humans to see, which can help lawyers to spot potential problems. LLMs can generate creative solutions to legal problems, which can help lawyers to find new and innovative ways to represent their clients. Here are some of the challenges that need to be addressed before LLMs can be widely adopted in the legal profession: LLMs need to be trained on large amounts of data, which can be expensive and time-consuming. LLMs need to be able to understand the nuances of legal language, which can be difficult for computers to do. LLMs need to be able to generate accurate and reliable results, which can be a challenge for complex legal problems. Overall, the author believes that LLMs have the potential to make a significant impact on the legal profession, and that they are likely to become increasingly important in the years to come.

numpad0 3y ago

ChatGPT just isn’t going to say the right thing, or thing grounded to the baseline reality, beyond what it cannot help but doing so restrained by your input. The only connection between it and this world is your input. ChatGPT is floating in the heavens, and you’re grounding it by at most a fishing line, through the textbox. It has to be framed as such. People praising it as a next gen search engine[that finds a data from database] is(perhaps this is the word that best fit the situation!), hallucinating.

LASR 3y ago

I’ve said this many times: stop using ChatGPT as a database. It does not contain facts. It may appear to contain some facts. Some may also be actually true. The truly useful usecase is as a reasoning engine. You can paste in a document and ask some questions about the facts in that document. Then it does a much better job, enough to be actually useful.

joshka 3y ago
To some extent it does contain facts, but those facts are indistinguishable from non-facts, are just a small proportion of the dataset compared to everything else are indistinguishable from non factual information. E.g. using text-davinci-003 (this is GPT3, not ChatGPT), "The moon is made of" completes to: Cheese 48.74%, rock: 31.66%, green 4.09% (98.75% followed by cheese), rocks 3.86%, and several other lower percentage tokens. I wonder if there eventually will be a type of model that incorporates the ability to simultaneously do text completion while adhering to facts at the model level (rather than having to bolt it on top via context).

Obscurity4340 3y ago

The new "dictated, not read"

bell-cot 3y ago

[Sigh.] Most parents understand that there's a broad range of ages where a kid is either a bit fuzzy on the concept of "truth", or reflexively lies in certain situations ("Billy broke the vase!"), or can invent fairly elaborate lies for various reasons (avoid punishment, impress other kids, bored bullsh*tting). Perhaps ChatGPT's "open relationship" with the truth could be explained in such terms...

deleted 3y ago

[deleted]

JoeAltmaier 3y ago

ChatGPT says everything with perfect confidence, like that know-it-all who will never admit they just don't know. Instead we get blathering that sounds like legalese (or chemistry or law or whatever) that has only a tenuous connection to reality. All you can say is, "People in my training set talked that way"

WheelsAtLarge 3y ago

And so it starts, I suspect we'll be seeing this issue more and more since it's easy to just get GPT to spit out some text. I believe that the true beneficiaries of LLMs are those that are experts in their fields. They can just read the output and deal with the inaccuracies. Does anyone know if training an LLM with just one type of data, law in this case, creates a more accurate output?

thunderbong 3y ago

I think I read this in a comment here on HN - > The goal of chat LLMs is not to give you an answer. The goal is to continue the conversation.

rvz 3y ago

Very unsurprising. Relying on AI sophists like ChatGPT for legal work is still just as risky for normal users and even for legal experts. The difference is, these legal experts are more qualified to carefully review and check over the outputs than the average joe / jane trying to 'replace their lawyer, solicitor, etc' with ChatGPT. I keep emphasising this importance, and to never fully trust the output of LLMs such as ChatGPT, unless a human has reviewed and checked if it is hallucinating or bullshitting. [0] [0] https://news.ycombinator.com/item?id=36091468 https://news.ycombinator.com/item?id=36091468

RobRivera 3y ago

that lawyer should be disbarred. Is it really so hard to cross reference a chatgpt response with a source?

clnq 3y ago

> What’s much harder though is actually getting it to double-down on fleshing those out. Now, it is. When ChatGPT first became public though, those were the Wild West days where you could get it to tell you anything, including all sorts of unethical things. And it would quite often double-down on "facts" it hallucinated. With current GPT-3.5 and GPT-4, the alignment is still a challenging problem, but it's in a much better place. I think it's unlikely a conversation with GPT-4 would have gone the way it did for this lawyer.

curiousgal 3y ago

Can we please, for the love of all that is holy, stop saying ChatGPT LIED? Lying means intentionally making false statements. A chatbot does not have intentions...

AmIDev 3y ago

What I have realized after using Bard(Palm2), ChatGPT(3.5) and some other LLMs is that they are good for tasks where an accuracy <100% is acceptable, and the cost of getting wrong answers is not high. For example, labeling a million text samples with 90% accuracy by using few shot learning is a good use case. Writing a poem is good use case. Trying to learn a new language is not. Generating a small function that you can verify might be ok. Writing entire codebase is not. So far, I haven't found any use case for personal use of LLMs. For work however, LLMs are going to be very useful with text(and potentially image) based machine learning tasks. Any tasks where having knowledge beyond the labeled training dataset is useful is going to be a good task for LLMs. One example is detecting fraud SMS.

BlueTemplar 3y ago

I remember someone predicting this would happen in comments here a few months ago.

Mizoguchi 3y ago

"Anyone who has worked designing products knows that users don’t read anything—warnings, footnotes, any form of microcopy will be studiously ignored" Users don't usually read long legal statements such as terms of services. That's not the case of ChatGPT interface, the note about its limitations is clearly visible and very short. This is as dumb as saying a city is at fault if someone drives into a clearly marked one way only street and causes an accident because people don't read anything.