7 ms·

There are degrees of "wrongness". OpenAI sometimes gives answers that are laughably wrong. It's exactly this degree of wrongness that google can't afford. Examp

There are degrees of "wrongness". OpenAI sometimes gives answers that are laughably wrong. It's exactly this degree of wrongness that google can't afford. Example: > What is the weight of 1 kilogram of nails? ChatGpt> The weight of 1 kilogram of nails will depend on the size and type of nails being used. On average, a kilogram of nails will weigh between 2.2 and 4 pounds (1-2 kg), depending on the size and type of nails. For example, a kilogram of small finishing nails may weigh less than a kilogram of large framing nails. The weight of the nails can also vary depending on the material they are made of, with steel nails being heavier than aluminum or plastic nails. BTW, when I ask the same question in Russian, the response is ... 7kg.

lumost 4y ago
These examples of wrongness seem cherry picked. I recently had a discussion with chatGPT where it succinctly clarified how functions of differential operators are defined and their properties. I didn’t know operator valued functions existed at the start of the conversation.
- LastTrain 4y ago
  It is the very correct answers that are cherry picked.
  - jjcon 4y ago
    Have you had many conversations with it? For me it took an hour before I found it saying anything particularly wrong and even then it was more subtle than the above.
    - grogenaut 4y ago
      It often doesn't seem wrong but it's also not right, it's very vague in a lot of places, when you get down to specifics it starts getting really wrong or flip flopping a lot. I had issues with this almost off the bat. It's like Dunning Kruger as a service really.
    - dumbfounder 4y ago
      It can’t do haikus. It very confidently puts them together with wrong syllable counts over and over even though you correct it many times. Then you ask it why it is so bad at counting syllables and it gives a great answer about how it is trained by text and that it doesn’t hear the words so it is hard to count syllables. But it doesn’t explain this when it is putting the haikus together or when you correct it over and over. It is humble when you directly challenge it, but it needs to be more transparent when it is feeding you garbage.
    - LastTrain 4y ago
      In my experience it takes a lot of leading to get anything interesting - it is very dependent on my prompts. I've 'learned' how to get better output from it, because lets face it, it is boring to try and speak with it naturally and experience the junk it responds with. And the 'very correct' class of which I spoke really does seem to be the exception not the rule.
- tgsovlerkhgsel 4y ago
  The problem is that these bots are extremely good at generating valid-sounding bullshit. Human-generated bullshit and bullshit generated by previous iterations of spam blogs used to be relatively easy to identify as bullshit. These models will confidently give you an answer, sounding perfectly plausible, even if it is completely wrong.
  - godelski 4y ago
    I think the biggest lesson to learn from all this is that just because things sound convincing doesn't mean it is accurate. We should probably incorporate this same skepticism when talking to people as we have when talking to machines (but that doesn't mean we should abandon good faith).
  - WalterBright 4y ago
    Hmm, sounds like our favorite politicians.
- godelski 4y ago
  What you're talking about is what we in the ML world call a stochastic parrot. You may have also heard the term "gullibility gap." A lot of language and conversations can be held that don't require any actual understanding of the subject matter, but rather because it follows certain patterns. People and LLMs can trick you into thinking they are highly intelligent because they can speak eloquently but that doesn't mean they are intelligent themselves. These LLMs can't understand inference or extrapolation, things that humans do easily (though we all know plenty of people that are idiots and can't do this). The same can be said about programming, which includes a lot more patterns. People joke that modern programming is slapping together APIs and it would be unsurprising that a (albeit really sophisticated) stochastic parrot can do this. But I've also seen it hand me code that looks correct but has major issues upon investigation. Don't let something fool you just because it appears intelligent. Human or machine we must handle information with care.
  - lumost 4y ago
    As a fellow participant in the ML world, I think there is compelling evidence to disagree with this take. ChatGPT’s responses on operator valued functions were accurate and valid, however ages of time on google failed to turn up this topic previously. On coding tasks, chatGPT can ask clarifying questions on requirements and determine if it has enough information to write the code correctly. Unfortunately you need to direct it to ask questions as needed and include appropriate tests to get the right answer. ChatGPT also fails in cases where it needs feedback from an external environment/documentation search. It will produce close enough code, but may need correction based on observed errors etc.
    - adriand 4y ago
      I’ve been using it regularly for programming assistance over the past two weeks and it’s extremely helpful. Others have pointed out that it sometimes produces inaccurate results, but if you already have domain expertise, as I do for programming, that is easy to identify. But it’s still a massive timesaver! I’ve been doing things like saying, “what follows is the database schema for entities X and Y, write a function that…” and then pasting in the schema, and it generates code good enough to copy and paste. It can also be instructed to modify results in various ways, for instance, I can ask it to provide the same code but in a different language, or to avoid using a certain framework feature, or to use a certain framework feature. It’s remarkable. Between ChatGPT and Copilot my workflow today is different in a way I couldn’t have begun to contemplate just a few weeks ago. Once they figure out additional ways to ensure correctness, I think it’s a totally new world we live in.
- resource0x 4y ago
  Examples of wrongness include most of arithmetic and logical inference (like in the example above). If you ask about the mass of 1 kilogram of nails, it gives the correct answer. The problem is that when the answer is wrong, it's not a "bug" that can be "fixed". It's just happens that, based on training data, the parameters of the resultant Rube Goldberg device are such that the weight of 1 kilogram of nails depends on the type of nails. It doesn't make sense even to ask why.
  - AnimalMuppet 4y ago
    So it fails in situations where there are precisely correct answers, and thrives in vagueness. I suppose that shouldn't surprise me. You could think about coupling it with an inference engine, and letting the inference engine win if it can generate a result, and otherwise going with the ChatGPT output. That might fix it to some degree.
Aperocky 4y ago
Russian nails are clearly superior in mass per mass scenario.
- resource0x 4y ago
  It's just in Russia, every reported number gets exaggerated by at least a factor of 7. :-)
rerdavies 4y ago
> What is the weight of 1 kilogram of nails? Correct answer: Depends on your current acceleration and/or the current force of gravity, puny human. :-P
Izkata 4y ago
Well to be pedantic, kilograms are mass, not weight. So the AI has the correct answer buried in the English version, assuming how we use "pounds" in the US: 2.2 pounds [on Earth]. Also there's an old riddle: What weighs more, a pound of feathers or a pound of gold? A pound of feathers is 16 ounces, but gold (at least at one point in the past, wikipedia indicates this isn't used anymore) is measured on a different scale and is only 12 ounces, so the pound of feathers is actually heavier.

lumost 4y ago

These examples of wrongness seem cherry picked. I recently had a discussion with chatGPT where it succinctly clarified how functions of differential operators are defined and their properties. I didn’t know operator valued functions existed at the start of the conversation.

LastTrain 4y ago
It is the very correct answers that are cherry picked.
- jjcon 4y ago
  Have you had many conversations with it? For me it took an hour before I found it saying anything particularly wrong and even then it was more subtle than the above.
  - grogenaut 4y ago
    It often doesn't seem wrong but it's also not right, it's very vague in a lot of places, when you get down to specifics it starts getting really wrong or flip flopping a lot. I had issues with this almost off the bat. It's like Dunning Kruger as a service really.
  - dumbfounder 4y ago
    It can’t do haikus. It very confidently puts them together with wrong syllable counts over and over even though you correct it many times. Then you ask it why it is so bad at counting syllables and it gives a great answer about how it is trained by text and that it doesn’t hear the words so it is hard to count syllables. But it doesn’t explain this when it is putting the haikus together or when you correct it over and over. It is humble when you directly challenge it, but it needs to be more transparent when it is feeding you garbage.
  - LastTrain 4y ago
    In my experience it takes a lot of leading to get anything interesting - it is very dependent on my prompts. I've 'learned' how to get better output from it, because lets face it, it is boring to try and speak with it naturally and experience the junk it responds with. And the 'very correct' class of which I spoke really does seem to be the exception not the rule.
tgsovlerkhgsel 4y ago
The problem is that these bots are extremely good at generating valid-sounding bullshit. Human-generated bullshit and bullshit generated by previous iterations of spam blogs used to be relatively easy to identify as bullshit. These models will confidently give you an answer, sounding perfectly plausible, even if it is completely wrong.
- godelski 4y ago
  I think the biggest lesson to learn from all this is that just because things sound convincing doesn't mean it is accurate. We should probably incorporate this same skepticism when talking to people as we have when talking to machines (but that doesn't mean we should abandon good faith).
- WalterBright 4y ago
  Hmm, sounds like our favorite politicians.
godelski 4y ago
What you're talking about is what we in the ML world call a stochastic parrot. You may have also heard the term "gullibility gap." A lot of language and conversations can be held that don't require any actual understanding of the subject matter, but rather because it follows certain patterns. People and LLMs can trick you into thinking they are highly intelligent because they can speak eloquently but that doesn't mean they are intelligent themselves. These LLMs can't understand inference or extrapolation, things that humans do easily (though we all know plenty of people that are idiots and can't do this). The same can be said about programming, which includes a lot more patterns. People joke that modern programming is slapping together APIs and it would be unsurprising that a (albeit really sophisticated) stochastic parrot can do this. But I've also seen it hand me code that looks correct but has major issues upon investigation. Don't let something fool you just because it appears intelligent. Human or machine we must handle information with care.
- lumost 4y ago
  As a fellow participant in the ML world, I think there is compelling evidence to disagree with this take. ChatGPT’s responses on operator valued functions were accurate and valid, however ages of time on google failed to turn up this topic previously. On coding tasks, chatGPT can ask clarifying questions on requirements and determine if it has enough information to write the code correctly. Unfortunately you need to direct it to ask questions as needed and include appropriate tests to get the right answer. ChatGPT also fails in cases where it needs feedback from an external environment/documentation search. It will produce close enough code, but may need correction based on observed errors etc.
  - adriand 4y ago
    I’ve been using it regularly for programming assistance over the past two weeks and it’s extremely helpful. Others have pointed out that it sometimes produces inaccurate results, but if you already have domain expertise, as I do for programming, that is easy to identify. But it’s still a massive timesaver! I’ve been doing things like saying, “what follows is the database schema for entities X and Y, write a function that…” and then pasting in the schema, and it generates code good enough to copy and paste. It can also be instructed to modify results in various ways, for instance, I can ask it to provide the same code but in a different language, or to avoid using a certain framework feature, or to use a certain framework feature. It’s remarkable. Between ChatGPT and Copilot my workflow today is different in a way I couldn’t have begun to contemplate just a few weeks ago. Once they figure out additional ways to ensure correctness, I think it’s a totally new world we live in.
resource0x 4y ago
Examples of wrongness include most of arithmetic and logical inference (like in the example above). If you ask about the mass of 1 kilogram of nails, it gives the correct answer. The problem is that when the answer is wrong, it's not a "bug" that can be "fixed". It's just happens that, based on training data, the parameters of the resultant Rube Goldberg device are such that the weight of 1 kilogram of nails depends on the type of nails. It doesn't make sense even to ask why.
- AnimalMuppet 4y ago
  So it fails in situations where there are precisely correct answers, and thrives in vagueness. I suppose that shouldn't surprise me. You could think about coupling it with an inference engine, and letting the inference engine win if it can generate a result, and otherwise going with the ChatGPT output. That might fix it to some degree.

Aperocky 4y ago

Russian nails are clearly superior in mass per mass scenario.

resource0x 4y ago
It's just in Russia, every reported number gets exaggerated by at least a factor of 7. :-)

rerdavies 4y ago

> What is the weight of 1 kilogram of nails? Correct answer: Depends on your current acceleration and/or the current force of gravity, puny human. :-P

Izkata 4y ago

Well to be pedantic, kilograms are mass, not weight. So the AI has the correct answer buried in the English version, assuming how we use "pounds" in the US: 2.2 pounds [on Earth]. Also there's an old riddle: What weighs more, a pound of feathers or a pound of gold? A pound of feathers is 16 ounces, but gold (at least at one point in the past, wikipedia indicates this isn't used anymore) is measured on a different scale and is only 12 ounces, so the pound of feathers is actually heavier.