6 ms·

What I find really strange about this is I use AI a lot as a “smart friend” to work through explanations of things I find difficult etc and I am currently prepa

What I find really strange about this is I use AI a lot as a “smart friend” to work through explanations of things I find difficult etc and I am currently preparing for some exams so I will often give the AI a document and ask for some supporting resources to take the subject further and it almost always produces something that is plausibly close to a real thing but wrong in specifics. As in when you ask for a reference it is almost invariably a hallucination. So it just amazes me that anyone would just stick that in a brief and ship it without checking it even more than they would check the work of a human underling (which they should obviously also check for something this important). For example, yesterday I got a list of some study resources for abstract algebra. Claude referred me to a series by Benedict Gross (Which is excellent btw). It gave me a line to harvard’s website but it was a 404 and it was only with further searching that I found the real thing. It also suggested a youtube playlist by Socratica (again this exists but the url was wrong) and one by Michael Penn (same deal). Literally every reference was almost right but actually wrong. How does anyone have the confidence to ship a legal brief that an AI produced without checking it thoroughly?

kayodelycaon 2y ago
I asked ChatGPT to give Wikipedia links in a table. Not one of the 50+ links was valid.
- alphan0n 2y ago
  Share the link to the conversation.
  - AlexeyBelov 2y ago
    Share the link or it didn't happen. And if it happened, you used an older model. And if you used the most up-to-date one, you didn't prompt it correctly.
    - alphan0n 2y ago
      So.. “trust me bro” is what you’re proposing? No thanks. Make a claim, prove it, especially one so easily proven.
- swores 2y ago
  Which version of GPT? I've found that 4o has actually been quite good at this lately, rarely hallucinating links any more. Just two days ago, I gave it a list of a dozen article titles from a newspaper website (The Guardian), asked it to look up their URLs and give me a list, and to summarise each article for me, and it made no mistakes at all. Maybe your task was more complicated to do in some way, maybe you're not paying for ChatGPT and are on a less able model, or maybe it's a question of learning how to prompt, I don't know, I just know that for me it's gone from "assume sources cited are bullshit" to "verify each one still, but they're usually correct".
  - lolinder 2y ago
    > asked it to look up their URLs and give me a list Something missing from this conversation is whether we're talking about the raw model or model+tool calls (search). This sounds like tool calls were enabled. And I do think this is a sign that the current UX of the chatbots is deeply flawed: even on HN we don't seem to interact with the UI components to toggle these features frequently enough that they're the intuitive answer, instead we still talk about model classes as though that makes the biggest difference in accuracy.
    - swores 2y ago
      Ah, yes you're right - I didn't clarify this in my original comment, but my anecdote was indeed the ChatGPT interface and using its ability to browse the web[#], not expecting it to pull URLs out of its original training data. Thanks for pointing that out. But the reason I suggested model as a potential difference between me and the person I replied to, rather than ChatGPT interface vs. plain use of model without bells and whistles, is that they had said their trouble was while using ChatGPT, not while using a GPT model over the API or through a different service. [#] (Technically I didn't, and never do, have the "search" button enabled in the chat interface, but it's able to search/browse the web without that focus being selected.)
      - lolinder 2y ago
        Right, but ChatGPT doesn't always automatically use search. I don't know what mechanisms it uses to decide whether to turn that on (maybe free accounts vs paid makes a difference?) but I rarely see it automatically turn on search, it usually tries to respond directly from weights. And on the flip side, my local Llama 3 8b does a pretty good job at avoiding hallucinations when it's hooked up to search (through Open WebUI). Search vs no-search seems to me to matter far more than model class.
halgir 2y ago
I use it in much the same way as you, and it's been extremely beneficial. But I also would not dream of signing my name on something that has been independently produced by AI, it's just too often blatantly wrong on specifics. I think people who do are simply not aware that AI is not deterministic the same way a calculator is. I would feel entirely safe signing my name on a mathematical result produced by a calculator (assuming I trusted my own input).
- mrob 2y ago
  LLMs are deterministic [0]. An LLM is a pure function that takes a list of tokens and returns a set of token probabilities. To make it "chat" you use the generated probabilities to pick a token, append that token to the list, and run the LLM again. Any randomness is introduced by the external component that picks a token using the probabilities: the sampler. Always picking the most likely token is a valid strategy. The problem is that all output is a "hallucination", and only some of it coincidentally matches the truth. There's no internal distinction between hallucination and truth. [0] Theoretically; race conditions in a parallel implementation could add non-determinism.
  - ijk 2y ago
    True, though in practice speed optimizations and instabilities on the GPU often lead to LLMs being very non-determanistic in practice. Which doesn't detract from your main point: there's not a lot of distinction between hallucinations and what we'd consider to be the "real thing." There have been various attempts to measure hallucinations, and we can figure out things like how confident the model is in a particular answer...but there's nothing grounding that answer. Saturate the dataset with the wrong answer and you'll get an overconfident wrong result.
  - jdlshore 2y ago
    While this is technically correct, everyday use of LLMs involves a non-zero temperature, so they (the whole package that people think of as “AI”) are non-deterministic in practice.
  - koakuma-chan 2y ago
gyomu 2y ago
People are lazy. I’m enrolled in a language class in a foreign country right now - so presumably people taking that class want to actually get good at the language so they can actually live their life here - yet a significant portion of students just turn in ChatGPT essays. And I don’t mean essays edited with chatGPT, but essays that are clearly verbatim output. When the teacher asks the students to read them out loud to the class, they will stumble upon words and grammar that are way obviously way beyond anything we’ve studied. The utter lack of self awareness is both funny but also really sad.
- kevin_thibedeau 2y ago
  There are a lot of shit tier lawyers who are just in it for the money and just barely passed their exams. Given his notoriety, Lindell is scraping the bottom of the barrel with people willing to provide legal services.
- mediumsmart 2y ago
  could it be that they just have to attend the class for technical reasons? Also - once the gadgets can translate for free in real time ... you can live in places you don't speak the language of, so maybe they are just prepping for that.
- DiscourseFan 2y ago
  LLMs were originally designed for translation, so it makes sense. We have basically elimated the need to learn foreign languages for day to day use anyway, its only helpful for high professional tasks or close literary study or prestige.
  - blks 2y ago
    That’s such a ridiculous statement.
    - heylook 2y ago
      Yep. > [learning a language is] only helpful for high professional tasks or close literary study or prestige. This is a person who doesn't understand actual face-to-face communication, like, at all. Even though translation apps are amazing, in a social interaction, there's no getting over the imposition of the halting, hesitant back-and-forth of device-assisted translation. Sure, you can almost always eventually get your point across, but you're never going to set the other party at ease in the same way as speaking their language yourself.
AnimalMuppet 2y ago
Lindell's lawyer claimed that somehow the preliminary copy (before human editing) got submitted to the court - that they actually did the work to fix it, but then slipped up in submitting it. I could see that, especially with sloppy lawyers in the first place. Or, I could see it being a convenient "the dog ate my homework" excuse.
- gazook89 2y ago
  Having not looked into it, I would guess that his lawyers know they aren’t going to get paid any time soon.
  - cosmicgadget 2y ago
    Seems like it's a fast track to not getting paid ever (disbarrment).
- deleted 2y ago
  [deleted]
netsharc 2y ago
Reading your comment, I'd like to coin the "AI-enhanced Dunning-Kruger".
- onetokeoverthe 2y ago
  [dead]
mmcwilliams 2y ago
I think it's easy to understand why people are overestimating the accuracy and performance of LLM-based output: it's currently being touted as the replacement for human labor in a large number of fields. Outside of software development there are fewer optimistic skeptics and much less nuanced takes on the tech. Casually scrolling through TechCrunch I see over $1B in very recent investments into legal-focused startups alone. You can't push the messaging that the technology to replace humans is here and expect people will also know intrinsically that they need to do the work of checking the output. It runs counter to the massive public rollout of these products which have a simple pitch: we are going to replace the work of human employees.
- namaria 2y ago
  I take the charitable view that some high profile people painted themselves into a corner very publicly. I think they made an estimation that they could work out the kinks as they went, but it's becoming apparent that the large early gains were not sustainable. And now there appears to be some very fundamental limitations to what this architecture can achieve and everyone involved has basically little option other than to keep doubling down. I expect this to blow up spectacularly pretty soon.
belter 2y ago
Everything you’ve said is correct. Now picture a quiet spread of subtle defects seeping through countless codebases, borne on the euphoria of GenAI driven “productivity”. When those flaws surface, the coming AI winter will be long and bitter.
deleted 2y ago
[deleted]
jedimastert 2y ago
> How does anyone have the confidence to ship a legal brief that an AI produced without checking it thoroughly? They're treating it like they would a paralegal. Typically this means giving a research task and then using their results, but sometimes lawyers will just have them write documents and ship it, so to speak. This is making me realize that Tech Bros treat chat GPT like the 1930s secretary they never got to have
sevensor 2y ago
What this story tells us more than anything is that Lindell cannot convince a competent lawyer to defend him, so what he gets instead are clownshod phonies. Either he’s out of cash, or he’s such a terrible client that nobody with a shred of professional responsibility will take him.
- susiecambria 2y ago
  This. And only this. My social studies teacher in 8th grade throughout the year would give us a list of things and phrases by decade. Loved these assignments. Totally threw myself into them. Years later, I was working at a prep school and wanted the students to be assigned Billy Joel's We Didn't Start the Fire as a similar assignment (competition?). Teachers thought it was stupid, that the kids wouldn't like it (um, who cares?), and that it was too hard. Teacher's responses only confirmed what I thought about the school (meh) and that it was a sad day that curiosity was a bad thing. (I was the fundraiser for the school so I didn't really interact with the kids a lot but ones I knew would have had fun with such a project). Anyway, the Lindell lawyers must have gone to this school, or one like it. How is it ever okay to do this and think it's a good idea? And, how the heck did these people pass the Bar? Edit: List of references in We Didn't Start the Fire https://en.wikipedia.org/wiki/List_of_references_in_We_Didn%27t_Start_the_Fire https://en.wikipedia.org/wiki/List_of_references_in_We_Didn%.... Gonna blog the references and this post on my little policy blog this week :-)
  - AStonesThrow 2y ago
    If you enjoyed Billy Joel’s original, your college-level Medieval History classes are gonna love Hildegard von Blingin’! https://music.youtube.com/watch?v=4pPzwh_DzOI https://music.youtube.com/watch?v=4pPzwh_DzOI
recursivecaveat 2y ago
The media really hypes the capabilities up, so if you're new to it and it spews out something that looks very detailed and plausible, you just think "wow it worked". They would have no instinct for the failure modes either. The reference points here would be like a paralegal or a computer search tool. You would only really expect errors of omission: the paralegal wants to keep their job, and search cannot find things that don't exist. In that frame of mind when you see that the returned document seems to cover the relevant points and makes sense when you skim it, it seems like job done. The public doesn't get that the LLM will just completely bold-faced make stuff up.
deleted 2y ago
[deleted]
zahlman 2y ago
> How does anyone have the confidence to ship a legal brief that an AI produced without checking it thoroughly? It has likely never occurred to them that such checks are necessary. Why would it, if they've never performed such checks, nor happen to have been warned by AI critics? >For example, yesterday I got a list of some study resources for abstract algebra. Claude referred me to a series by Benedict Gross (Which is excellent btw). It gave me a line to harvard’s website but it was a 404 and it was only with further searching that I found the real thing. It also suggested a youtube playlist by Socratica (again this exists but the url was wrong) and one by Michael Penn (same deal). FWIW, I've found Penn's content to be quite long-winded and poorly edited. The key idea being presented often makes up hardly any of the video's runtime, so I'm just sitting there watching the guy actually write out the steps solving an equation (and making trivial errors, and not always correcting them).
rchaud 2y ago
> How does anyone have the confidence to ship a legal brief that an AI produced without checking it thoroughly? Because its makers don't care about precision or correctness. They care only about convincing the people that matter that gaping software bugs are "hallucinations" that can never be fixed 100%, and that that is an acceptable outcome. For every story about AI-assisted legal briefs with tons of mistakes, there will be 100 PR-ridden pieces about how $LATEST_VERSION has passed the bar exam or has discovered a miracle drug. There will never be a story about AI successfully arguing a real case however, because the main goal was only about selling a vision of labor replacement. Whether or not the replacement can do the job as specified is immaterial.

kayodelycaon 2y ago

I asked ChatGPT to give Wikipedia links in a table. Not one of the 50+ links was valid.

alphan0n 2y ago
Share the link to the conversation.
- AlexeyBelov 2y ago
  Share the link or it didn't happen. And if it happened, you used an older model. And if you used the most up-to-date one, you didn't prompt it correctly.
  - alphan0n 2y ago
    So.. “trust me bro” is what you’re proposing? No thanks. Make a claim, prove it, especially one so easily proven.
swores 2y ago
Which version of GPT? I've found that 4o has actually been quite good at this lately, rarely hallucinating links any more. Just two days ago, I gave it a list of a dozen article titles from a newspaper website (The Guardian), asked it to look up their URLs and give me a list, and to summarise each article for me, and it made no mistakes at all. Maybe your task was more complicated to do in some way, maybe you're not paying for ChatGPT and are on a less able model, or maybe it's a question of learning how to prompt, I don't know, I just know that for me it's gone from "assume sources cited are bullshit" to "verify each one still, but they're usually correct".
- lolinder 2y ago
  > asked it to look up their URLs and give me a list Something missing from this conversation is whether we're talking about the raw model or model+tool calls (search). This sounds like tool calls were enabled. And I do think this is a sign that the current UX of the chatbots is deeply flawed: even on HN we don't seem to interact with the UI components to toggle these features frequently enough that they're the intuitive answer, instead we still talk about model classes as though that makes the biggest difference in accuracy.
  - swores 2y ago
    Ah, yes you're right - I didn't clarify this in my original comment, but my anecdote was indeed the ChatGPT interface and using its ability to browse the web[#], not expecting it to pull URLs out of its original training data. Thanks for pointing that out. But the reason I suggested model as a potential difference between me and the person I replied to, rather than ChatGPT interface vs. plain use of model without bells and whistles, is that they had said their trouble was while using ChatGPT, not while using a GPT model over the API or through a different service. [#] (Technically I didn't, and never do, have the "search" button enabled in the chat interface, but it's able to search/browse the web without that focus being selected.)
    - lolinder 2y ago
      Right, but ChatGPT doesn't always automatically use search. I don't know what mechanisms it uses to decide whether to turn that on (maybe free accounts vs paid makes a difference?) but I rarely see it automatically turn on search, it usually tries to respond directly from weights. And on the flip side, my local Llama 3 8b does a pretty good job at avoiding hallucinations when it's hooked up to search (through Open WebUI). Search vs no-search seems to me to matter far more than model class.

halgir 2y ago

I use it in much the same way as you, and it's been extremely beneficial. But I also would not dream of signing my name on something that has been independently produced by AI, it's just too often blatantly wrong on specifics. I think people who do are simply not aware that AI is not deterministic the same way a calculator is. I would feel entirely safe signing my name on a mathematical result produced by a calculator (assuming I trusted my own input).

mrob 2y ago
LLMs are deterministic [0]. An LLM is a pure function that takes a list of tokens and returns a set of token probabilities. To make it "chat" you use the generated probabilities to pick a token, append that token to the list, and run the LLM again. Any randomness is introduced by the external component that picks a token using the probabilities: the sampler. Always picking the most likely token is a valid strategy. The problem is that all output is a "hallucination", and only some of it coincidentally matches the truth. There's no internal distinction between hallucination and truth. [0] Theoretically; race conditions in a parallel implementation could add non-determinism.
- ijk 2y ago
  True, though in practice speed optimizations and instabilities on the GPU often lead to LLMs being very non-determanistic in practice. Which doesn't detract from your main point: there's not a lot of distinction between hallucinations and what we'd consider to be the "real thing." There have been various attempts to measure hallucinations, and we can figure out things like how confident the model is in a particular answer...but there's nothing grounding that answer. Saturate the dataset with the wrong answer and you'll get an overconfident wrong result.
- jdlshore 2y ago
  While this is technically correct, everyday use of LLMs involves a non-zero temperature, so they (the whole package that people think of as “AI”) are non-deterministic in practice.
- koakuma-chan 2y ago

gyomu 2y ago

People are lazy. I’m enrolled in a language class in a foreign country right now - so presumably people taking that class want to actually get good at the language so they can actually live their life here - yet a significant portion of students just turn in ChatGPT essays. And I don’t mean essays edited with chatGPT, but essays that are clearly verbatim output. When the teacher asks the students to read them out loud to the class, they will stumble upon words and grammar that are way obviously way beyond anything we’ve studied. The utter lack of self awareness is both funny but also really sad.

kevin_thibedeau 2y ago
There are a lot of shit tier lawyers who are just in it for the money and just barely passed their exams. Given his notoriety, Lindell is scraping the bottom of the barrel with people willing to provide legal services.
mediumsmart 2y ago
could it be that they just have to attend the class for technical reasons? Also - once the gadgets can translate for free in real time ... you can live in places you don't speak the language of, so maybe they are just prepping for that.
DiscourseFan 2y ago
LLMs were originally designed for translation, so it makes sense. We have basically elimated the need to learn foreign languages for day to day use anyway, its only helpful for high professional tasks or close literary study or prestige.
- blks 2y ago
  That’s such a ridiculous statement.
  - heylook 2y ago
    Yep. > [learning a language is] only helpful for high professional tasks or close literary study or prestige. This is a person who doesn't understand actual face-to-face communication, like, at all. Even though translation apps are amazing, in a social interaction, there's no getting over the imposition of the halting, hesitant back-and-forth of device-assisted translation. Sure, you can almost always eventually get your point across, but you're never going to set the other party at ease in the same way as speaking their language yourself.

AnimalMuppet 2y ago

Lindell's lawyer claimed that somehow the preliminary copy (before human editing) got submitted to the court - that they actually did the work to fix it, but then slipped up in submitting it. I could see that, especially with sloppy lawyers in the first place. Or, I could see it being a convenient "the dog ate my homework" excuse.

gazook89 2y ago
Having not looked into it, I would guess that his lawyers know they aren’t going to get paid any time soon.
- cosmicgadget 2y ago
  Seems like it's a fast track to not getting paid ever (disbarrment).
deleted 2y ago
[deleted]

netsharc 2y ago

Reading your comment, I'd like to coin the "AI-enhanced Dunning-Kruger".

onetokeoverthe 2y ago
[dead]

mmcwilliams 2y ago

I think it's easy to understand why people are overestimating the accuracy and performance of LLM-based output: it's currently being touted as the replacement for human labor in a large number of fields. Outside of software development there are fewer optimistic skeptics and much less nuanced takes on the tech. Casually scrolling through TechCrunch I see over $1B in very recent investments into legal-focused startups alone. You can't push the messaging that the technology to replace humans is here and expect people will also know intrinsically that they need to do the work of checking the output. It runs counter to the massive public rollout of these products which have a simple pitch: we are going to replace the work of human employees.

namaria 2y ago
I take the charitable view that some high profile people painted themselves into a corner very publicly. I think they made an estimation that they could work out the kinks as they went, but it's becoming apparent that the large early gains were not sustainable. And now there appears to be some very fundamental limitations to what this architecture can achieve and everyone involved has basically little option other than to keep doubling down. I expect this to blow up spectacularly pretty soon.

belter 2y ago

Everything you’ve said is correct. Now picture a quiet spread of subtle defects seeping through countless codebases, borne on the euphoria of GenAI driven “productivity”. When those flaws surface, the coming AI winter will be long and bitter.

deleted 2y ago

[deleted]

jedimastert 2y ago

> How does anyone have the confidence to ship a legal brief that an AI produced without checking it thoroughly? They're treating it like they would a paralegal. Typically this means giving a research task and then using their results, but sometimes lawyers will just have them write documents and ship it, so to speak. This is making me realize that Tech Bros treat chat GPT like the 1930s secretary they never got to have

sevensor 2y ago

What this story tells us more than anything is that Lindell cannot convince a competent lawyer to defend him, so what he gets instead are clownshod phonies. Either he’s out of cash, or he’s such a terrible client that nobody with a shred of professional responsibility will take him.

susiecambria 2y ago
This. And only this. My social studies teacher in 8th grade throughout the year would give us a list of things and phrases by decade. Loved these assignments. Totally threw myself into them. Years later, I was working at a prep school and wanted the students to be assigned Billy Joel's We Didn't Start the Fire as a similar assignment (competition?). Teachers thought it was stupid, that the kids wouldn't like it (um, who cares?), and that it was too hard. Teacher's responses only confirmed what I thought about the school (meh) and that it was a sad day that curiosity was a bad thing. (I was the fundraiser for the school so I didn't really interact with the kids a lot but ones I knew would have had fun with such a project). Anyway, the Lindell lawyers must have gone to this school, or one like it. How is it ever okay to do this and think it's a good idea? And, how the heck did these people pass the Bar? Edit: List of references in We Didn't Start the Fire https://en.wikipedia.org/wiki/List_of_references_in_We_Didn%27t_Start_the_Fire https://en.wikipedia.org/wiki/List_of_references_in_We_Didn%.... Gonna blog the references and this post on my little policy blog this week :-)
- AStonesThrow 2y ago
  If you enjoyed Billy Joel’s original, your college-level Medieval History classes are gonna love Hildegard von Blingin’! https://music.youtube.com/watch?v=4pPzwh_DzOI https://music.youtube.com/watch?v=4pPzwh_DzOI

recursivecaveat 2y ago

The media really hypes the capabilities up, so if you're new to it and it spews out something that looks very detailed and plausible, you just think "wow it worked". They would have no instinct for the failure modes either. The reference points here would be like a paralegal or a computer search tool. You would only really expect errors of omission: the paralegal wants to keep their job, and search cannot find things that don't exist. In that frame of mind when you see that the returned document seems to cover the relevant points and makes sense when you skim it, it seems like job done. The public doesn't get that the LLM will just completely bold-faced make stuff up.

deleted 2y ago

[deleted]

zahlman 2y ago

> How does anyone have the confidence to ship a legal brief that an AI produced without checking it thoroughly? It has likely never occurred to them that such checks are necessary. Why would it, if they've never performed such checks, nor happen to have been warned by AI critics? >For example, yesterday I got a list of some study resources for abstract algebra. Claude referred me to a series by Benedict Gross (Which is excellent btw). It gave me a line to harvard’s website but it was a 404 and it was only with further searching that I found the real thing. It also suggested a youtube playlist by Socratica (again this exists but the url was wrong) and one by Michael Penn (same deal). FWIW, I've found Penn's content to be quite long-winded and poorly edited. The key idea being presented often makes up hardly any of the video's runtime, so I'm just sitting there watching the guy actually write out the steps solving an equation (and making trivial errors, and not always correcting them).

rchaud 2y ago

> How does anyone have the confidence to ship a legal brief that an AI produced without checking it thoroughly? Because its makers don't care about precision or correctness. They care only about convincing the people that matter that gaping software bugs are "hallucinations" that can never be fixed 100%, and that that is an acceptable outcome. For every story about AI-assisted legal briefs with tons of mistakes, there will be 100 PR-ridden pieces about how $LATEST_VERSION has passed the bar exam or has discovered a miracle drug. There will never be a story about AI successfully arguing a real case however, because the main goal was only about selling a vision of labor replacement. Whether or not the replacement can do the job as specified is immaterial.