4 ms·

> Why, for example, can ChatGPT simply not recognize when it’s being asked to generate factual citations and caution the user to “check my sources”? Why can’t i

> Why, for example, can ChatGPT simply not recognize when it’s being asked to generate factual citations and caution the user to “check my sources”? Why can’t it respond to someone asking “is this text AI-generated?” with a clear “I’m sorry, I’m not capable of making that judgment”? This is another big frustration for me, as someone who has been in the field for a long time (over a decade at this point). These are complex systems that are just reaching the point of usability, and yet people are going "well why don't you just simply do XYZ?" If it were simple, it wouldn't have taken so long for these systems to emerge. We already barely get them to do what they're doing, and they still suck at that, so "just make it not do that" is so frustrating. The only thing worse is when people ascribe malicious motives like openai is nefariously deciding that it should inject those mistakes. Maybe this is the "I could build that in a weekend" of ML. Sorry for the rant, it just makes me want to vent.

IanCal 3y ago
Fond memories of someone saying they didn't expect 100% accuracy from our system, 99% was fine.
tyre 3y ago
It’s a fundamental misunderstanding of what these systems are doing and how dumb they are. People see a computer doing what they and other people around them do: natural language going in and out. With other people, we have a theory of mind and broad understanding of how minds work. So it’s understandable that people see ChatGPT and think, “it’s cool that a computer is doing what I’m doing.” The marketing of these things as Artificial Intelligence is part of the confusion. LLMs are not thinking. They are not saying, “okay now the next thing I’ll do is add a citation. Since I’m doing that, remember to add a warning at the top before showing to the user.” That’s what this writer seems to expect. The two paths to fixing this: make the tech better or educate people so they have a correct(er) understanding of LLMs. I don’t think we’re going to get a meaningful subset of humans to take neural networks 101.
- dboreham 3y ago
  In other words: the Turing test turns out to be too easy.
  - kenjackson 3y ago
    It's not that it's too easy, but rather AI models are not people. What do I mean by this? There is (for the purposes of this discussion) just one ChatGPT-4 model. This single model has to be all things to all people. It's not Ted. Or Jessica or Dante. You don't expect Jessica to have a reasonable answer to every question. You accept the fact that if you ask Jessica about public schools, she will go off the rails. It's not indictment on all humans -- it's just Jessica. At some point we might have access to a variety of AI models where some will simply say, "I really only know physics pretty well. I prefer not to give my opinon on other topics". And others that will lie about most everything. But we will learn which are useful for what.
  - pdonis 3y ago
    I would put it that most people who think they are applying the Turing test, actually aren't. Part of correctly applying the Turing test is independently checking the statements the test subject makes against your own knowledge. Part of the "thinking" that the test is testing for is checking beliefs against reality, and the Turing test needs to test for that capability, otherwise it's not being done right.
    - simonh 3y ago
      Also the test is adversarial, it’s not about throwing softballs to a child. The interrogator should have a good understanding of what constitutes intelligence, the kinds of mental faculties humans have, and ask questions that thoroughly test those faculties.
- Terretta 3y ago
  While I imagine I have a grip on what LLMs are, there's something ... emergent? ... in some behaviors that approach behavior akin to responses from someone or something with a "mental model". That can be confusing to encounter. For instance, asked it to review code interacting with an IoT device API. It unexpectedly warned me a person relying on an app incorporating this library of code should remain careful of a certain health problem since the source for what the IoT device periodically senses can fluctuate between readings, and spikes in that could cause that health problem. I'm still pondering the chain of things that would have to be "thought through" to have that feedback about the code, when — and this is the part that runs against what I thought I knew — this connection, and the implications of only periodic sensing, may be evident to people who stop to think, but most wouldn't think about it, and it's not been written about in any way Google can find. This isn't theory of mind, but it's indistinguishable from the generator relying on something like a mental model.
abeppu 3y ago
Yes, people making suggestions for changes in behavior saying it should be "simple" have a poor understanding of what they're talking about. But given the weaknesses of the models, perhaps OpenAI should not have released them for such flexible use for a broad audience. For AI/ML applications being used for anything important, I think it's entirely appropriate to try to both limit use to the areas in which the model can be expected to succeed (including using for data from similar distributions to those used for training), and to educate users about what issues cannot be mitigated. If OpenAI had released a "fiction writing assistant" product, a "fictional character fun chat" product, etc it would likely be misused less.
- hoosieree 3y ago
  Guess, and Print Text (GPT)
- idopmstuff 3y ago
  > But given the weaknesses of the models, perhaps OpenAI should not have released them for such flexible use for a broad audience. I think the advantages greatly outweigh negatives - not only is ChatGPT an extremely useful tool, its also really helped to bring the seriousness of AI and its effects on society into the current discourse. > If OpenAI had released a "fiction writing assistant" product, a "fictional character fun chat" product, etc it would likely be misused less. I really feel that the misuses of this tend to be cases where I would say the issue lies with the user. At some point we have to take some level of personal responsibility for the tools and services we use. Consider the case of the lawyer who included a bunch of stuff from ChatGPT in a federal court filing - I would place the blame on OpenAI squarely at 0% on that one. This is a trained professional in a highly regulated industry who decided to outsource his job to a tool about which he clearly hadn't done even a little bit research. The issue isn't that ChatGPT hallucinated, it's that he exercised judgement that is obviously incredibly poor. I suspect if you went back through his work, you'd find other places where he cut corners and had lapses of ethics and/or responsibility. He's the problem, not ChatGPT. Same thing with the professor who accused all of his students of plagiarism because he put their text into ChatGPT and asked if it had written it. On the one hand, he's trying to keep up with technology that's moving quickly and I have empathy for that. On the other hand, he's a college professor, so he should absolutely be able to do the level of research to find the many, many news articles on this topic that would have made clear that what he's doing is stupid. Making incredibly serious accusations towards students based solely on a tool you clearly haven't even tried to understand is a tremendous breach of ethics, which is the real problem.
bena 3y ago
That's the rub though, isn't it? It just exposes the fact that these LLM/generative AI models don't actually know context. ChatGPT doesn't "know" that it's "generat[ing] factual citations" so it doesn't "know" to caution the user. It just knows that when the user prompts like "X", it should respond like "Y". The more information that comes out about how these things respond, the more it seems like ChatGPT shares a lot of traits with Clever Hans.

IanCal 3y ago

Fond memories of someone saying they didn't expect 100% accuracy from our system, 99% was fine.

tyre 3y ago

It’s a fundamental misunderstanding of what these systems are doing and how dumb they are. People see a computer doing what they and other people around them do: natural language going in and out. With other people, we have a theory of mind and broad understanding of how minds work. So it’s understandable that people see ChatGPT and think, “it’s cool that a computer is doing what I’m doing.” The marketing of these things as Artificial Intelligence is part of the confusion. LLMs are not thinking. They are not saying, “okay now the next thing I’ll do is add a citation. Since I’m doing that, remember to add a warning at the top before showing to the user.” That’s what this writer seems to expect. The two paths to fixing this: make the tech better or educate people so they have a correct(er) understanding of LLMs. I don’t think we’re going to get a meaningful subset of humans to take neural networks 101.

dboreham 3y ago
In other words: the Turing test turns out to be too easy.
- kenjackson 3y ago
  It's not that it's too easy, but rather AI models are not people. What do I mean by this? There is (for the purposes of this discussion) just one ChatGPT-4 model. This single model has to be all things to all people. It's not Ted. Or Jessica or Dante. You don't expect Jessica to have a reasonable answer to every question. You accept the fact that if you ask Jessica about public schools, she will go off the rails. It's not indictment on all humans -- it's just Jessica. At some point we might have access to a variety of AI models where some will simply say, "I really only know physics pretty well. I prefer not to give my opinon on other topics". And others that will lie about most everything. But we will learn which are useful for what.
- pdonis 3y ago
  I would put it that most people who think they are applying the Turing test, actually aren't. Part of correctly applying the Turing test is independently checking the statements the test subject makes against your own knowledge. Part of the "thinking" that the test is testing for is checking beliefs against reality, and the Turing test needs to test for that capability, otherwise it's not being done right.
  - simonh 3y ago
    Also the test is adversarial, it’s not about throwing softballs to a child. The interrogator should have a good understanding of what constitutes intelligence, the kinds of mental faculties humans have, and ask questions that thoroughly test those faculties.
Terretta 3y ago
While I imagine I have a grip on what LLMs are, there's something ... emergent? ... in some behaviors that approach behavior akin to responses from someone or something with a "mental model". That can be confusing to encounter. For instance, asked it to review code interacting with an IoT device API. It unexpectedly warned me a person relying on an app incorporating this library of code should remain careful of a certain health problem since the source for what the IoT device periodically senses can fluctuate between readings, and spikes in that could cause that health problem. I'm still pondering the chain of things that would have to be "thought through" to have that feedback about the code, when — and this is the part that runs against what I thought I knew — this connection, and the implications of only periodic sensing, may be evident to people who stop to think, but most wouldn't think about it, and it's not been written about in any way Google can find. This isn't theory of mind, but it's indistinguishable from the generator relying on something like a mental model.

abeppu 3y ago

Yes, people making suggestions for changes in behavior saying it should be "simple" have a poor understanding of what they're talking about. But given the weaknesses of the models, perhaps OpenAI should not have released them for such flexible use for a broad audience. For AI/ML applications being used for anything important, I think it's entirely appropriate to try to both limit use to the areas in which the model can be expected to succeed (including using for data from similar distributions to those used for training), and to educate users about what issues cannot be mitigated. If OpenAI had released a "fiction writing assistant" product, a "fictional character fun chat" product, etc it would likely be misused less.

hoosieree 3y ago
Guess, and Print Text (GPT)
idopmstuff 3y ago
> But given the weaknesses of the models, perhaps OpenAI should not have released them for such flexible use for a broad audience. I think the advantages greatly outweigh negatives - not only is ChatGPT an extremely useful tool, its also really helped to bring the seriousness of AI and its effects on society into the current discourse. > If OpenAI had released a "fiction writing assistant" product, a "fictional character fun chat" product, etc it would likely be misused less. I really feel that the misuses of this tend to be cases where I would say the issue lies with the user. At some point we have to take some level of personal responsibility for the tools and services we use. Consider the case of the lawyer who included a bunch of stuff from ChatGPT in a federal court filing - I would place the blame on OpenAI squarely at 0% on that one. This is a trained professional in a highly regulated industry who decided to outsource his job to a tool about which he clearly hadn't done even a little bit research. The issue isn't that ChatGPT hallucinated, it's that he exercised judgement that is obviously incredibly poor. I suspect if you went back through his work, you'd find other places where he cut corners and had lapses of ethics and/or responsibility. He's the problem, not ChatGPT. Same thing with the professor who accused all of his students of plagiarism because he put their text into ChatGPT and asked if it had written it. On the one hand, he's trying to keep up with technology that's moving quickly and I have empathy for that. On the other hand, he's a college professor, so he should absolutely be able to do the level of research to find the many, many news articles on this topic that would have made clear that what he's doing is stupid. Making incredibly serious accusations towards students based solely on a tool you clearly haven't even tried to understand is a tremendous breach of ethics, which is the real problem.

bena 3y ago

That's the rub though, isn't it? It just exposes the fact that these LLM/generative AI models don't actually know context. ChatGPT doesn't "know" that it's "generat[ing] factual citations" so it doesn't "know" to caution the user. It just knows that when the user prompts like "X", it should respond like "Y". The more information that comes out about how these things respond, the more it seems like ChatGPT shares a lot of traits with Clever Hans.