5 ms·

This happens literally every time. Someone always says "ChatGPT can't do this!", but then when someone actually runs the example, chatGPT gets it right. Now wha

This happens literally every time. Someone always says "ChatGPT can't do this!", but then when someone actually runs the example, chatGPT gets it right. Now what the OP is going to do next is proceed to move goalposts and say like "but umm I just asked chatgpt this, so clearly they modified the code in realtime to get the answer right"

whimsicalism 2y ago
yeah… that’s hn for you. that said, i don’t think this is a good test - i’ve seen it circling on twitter for months and it is almost certainly trained on similar tasks
anamexis 2y ago
How is that moving the goalposts?
JumpCrisscross 2y ago
> Someone always says "ChatGPT can't do this!", but then when someone actually runs the example, chatGPT gets it right I mean, if I had OpenAI’s resources I’d have a team tasked with monitoring social to debug trending fuck-ups. (Before that: add compute time to frequently-asked novel queries.)
- jprete 2y ago
  I was thinking something very similar. Posting about a problem adds information back to the system, and every company selling model time for money has a vested interest in patching publicly visible holes. This could even be automated; LLMs can sentiment-analyze social media posts to surface ones that are critical of LLM outputs, then automatically extract features of the post to change things about the running model to improve similar results with no intervention.
jorvi 2y ago
Prompt: In the Netherlands, in terms of drinks, is there a particular spirit that represents the country? > Yes, in the Netherlands, jenever (also known as genever) is the traditional spirit that represents the country. Jenever is a type of Dutch gin that has a distinctive flavor, often made from malt wine and flavored with juniper berries. It has a long history in the Netherlands, dating back to the 16th century, and is considered the precursor to modern gin. > Jenever comes in two main types: oude (old) jenever, which has a richer, maltier flavor and is often aged in wooden casks, and jonge (young) jenever, which is lighter and has a more neutral taste. Both types can be enjoyed straight or used in cocktails. > In addition to jenever, the Netherlands is also known for its liqueurs, such as advocaat, a rich and creamy drink made from eggs, sugar, and brandy, often enjoyed as a dessert or in cocktails. However, jenever remains the most iconic spirit associated with Dutch culture. This is completely wrong. Jenever certainly is very Dutch, but no one would say it is iconic as the Dutch spirit. For example, if you asked up north in Friesland, they would say Berenburg. This happens literally every time. Someone always says "ChatGPT can do this!", but then within one or two prompts, its gets it wrong.
- ludwik 2y ago
  But what does this have to do with reasoning? Yes, LLMs are not knowledge bases, and seeing people treat them as such absolutely terrifies me. However, I don’t see how the fact that LLMs often hallucinate “facts” is relevant to a discussion about their reasoning capabilities.
  - zahlman 2y ago
    "Hallucinating a fact" that isn't in the training set and is also illogical, is exactly what a failure to reason correctly looks like.
    - ludwik 2y ago
      Reasoning involves making accurate inferences based on the information provided in the current context, rather than recalling arbitrary facts from the training data.
      - zahlman 2y ago
        Yes, that's what I said. The whole point of hallucinations is that they aren't "arbitrary facts recalled from the training data". They represent attempts to synthesize (i.e., infer) new facts. But because the inferences are not accurate, and because the synthesis process is not sound, the attempt cannot be called reasoning. It is equally possible to "reason" about things you already know, as about things you've just been told. In fact, the capacity to speculatively, without prompting attempt such reasoning is a big part of cognition.
- elif 2y ago
  'Berenberg is made by adding herbs to jenever' From your comment it would seem that you are disputing jenever's popularity by saying jenever is more popular... Perhaps it was a good faith mistake? If so, that would imply that the AI knows more about jenever than you?
  - 2y ago
stocknoob 2y ago
Similarly, in every thread there’s an AI skeptic who says LLMs are “useless” for coding, and never provides an example query for what they were trying.
- mu53 2y ago
  If you ask about more niche language features or libraries, chatgpt will make up libraries or functions to fill the gap. When asking an LLM to write a script for you, I would say 10 to 30 % of the time that it completely fails. Again, making up an API or just getting things straight up wrong. Its very helpful, especially when starting from 0 with the beginner questions, but it fails in many scenarios.
- zahlman 2y ago
  Because the argument isn't based on individual query results. See for example my comment on a previous post https://news.ycombinator.com/item?id=42563715 https://news.ycombinator.com/item?id=42563715 .

whimsicalism 2y ago

yeah… that’s hn for you. that said, i don’t think this is a good test - i’ve seen it circling on twitter for months and it is almost certainly trained on similar tasks

anamexis 2y ago

How is that moving the goalposts?

JumpCrisscross 2y ago

> Someone always says "ChatGPT can't do this!", but then when someone actually runs the example, chatGPT gets it right I mean, if I had OpenAI’s resources I’d have a team tasked with monitoring social to debug trending fuck-ups. (Before that: add compute time to frequently-asked novel queries.)

jprete 2y ago
I was thinking something very similar. Posting about a problem adds information back to the system, and every company selling model time for money has a vested interest in patching publicly visible holes. This could even be automated; LLMs can sentiment-analyze social media posts to surface ones that are critical of LLM outputs, then automatically extract features of the post to change things about the running model to improve similar results with no intervention.

jorvi 2y ago

Prompt: In the Netherlands, in terms of drinks, is there a particular spirit that represents the country? > Yes, in the Netherlands, jenever (also known as genever) is the traditional spirit that represents the country. Jenever is a type of Dutch gin that has a distinctive flavor, often made from malt wine and flavored with juniper berries. It has a long history in the Netherlands, dating back to the 16th century, and is considered the precursor to modern gin. > Jenever comes in two main types: oude (old) jenever, which has a richer, maltier flavor and is often aged in wooden casks, and jonge (young) jenever, which is lighter and has a more neutral taste. Both types can be enjoyed straight or used in cocktails. > In addition to jenever, the Netherlands is also known for its liqueurs, such as advocaat, a rich and creamy drink made from eggs, sugar, and brandy, often enjoyed as a dessert or in cocktails. However, jenever remains the most iconic spirit associated with Dutch culture. This is completely wrong. Jenever certainly is very Dutch, but no one would say it is iconic as the Dutch spirit. For example, if you asked up north in Friesland, they would say Berenburg. This happens literally every time. Someone always says "ChatGPT can do this!", but then within one or two prompts, its gets it wrong.

ludwik 2y ago
But what does this have to do with reasoning? Yes, LLMs are not knowledge bases, and seeing people treat them as such absolutely terrifies me. However, I don’t see how the fact that LLMs often hallucinate “facts” is relevant to a discussion about their reasoning capabilities.
- zahlman 2y ago
  "Hallucinating a fact" that isn't in the training set and is also illogical, is exactly what a failure to reason correctly looks like.
  - ludwik 2y ago
    Reasoning involves making accurate inferences based on the information provided in the current context, rather than recalling arbitrary facts from the training data.
    - zahlman 2y ago
      Yes, that's what I said. The whole point of hallucinations is that they aren't "arbitrary facts recalled from the training data". They represent attempts to synthesize (i.e., infer) new facts. But because the inferences are not accurate, and because the synthesis process is not sound, the attempt cannot be called reasoning. It is equally possible to "reason" about things you already know, as about things you've just been told. In fact, the capacity to speculatively, without prompting attempt such reasoning is a big part of cognition.
elif 2y ago
'Berenberg is made by adding herbs to jenever' From your comment it would seem that you are disputing jenever's popularity by saying jenever is more popular... Perhaps it was a good faith mistake? If so, that would imply that the AI knows more about jenever than you?
- 2y ago

stocknoob 2y ago

Similarly, in every thread there’s an AI skeptic who says LLMs are “useless” for coding, and never provides an example query for what they were trying.

mu53 2y ago
If you ask about more niche language features or libraries, chatgpt will make up libraries or functions to fill the gap. When asking an LLM to write a script for you, I would say 10 to 30 % of the time that it completely fails. Again, making up an API or just getting things straight up wrong. Its very helpful, especially when starting from 0 with the beginner questions, but it fails in many scenarios.
zahlman 2y ago
Because the argument isn't based on individual query results. See for example my comment on a previous post https://news.ycombinator.com/item?id=42563715 https://news.ycombinator.com/item?id=42563715 .