5 ms·

It’s fundamentally same technology with same limitations. You can throw a lot of money at it to try to fix short comings, but they’ll be just bandaids. LLMs ar

It’s fundamentally same technology with same limitations. You can throw a lot of money at it to try to fix short comings, but they’ll be just bandaids. LLMs are extremely amazing autocomplete mechanisms. But that’s all they are at the core - there’s no intelligence involved.

rowanG077 2y ago
That's a pretty rich statement considering they perform pretty good at tests which we have designed to measure intelligence. I don't see how you can say there is no intelligence.
- whythre 2y ago
  Sounds like a problem with the tests being administered. There is a lot of woo around LLMs, a lot of people have a vested interest in hyping and selling AI; heck, even referring to LLMs as AI is a form of branding.
  - rowanG077 2y ago
    So turn it around then. When is an AI intelligent? Since when it performs at human standard in intelligence tests doesn't seem enough for you.
    - blooalien 2y ago
      https://arstechnica.com/science/2023/07/a-jargon-free-explanation-of-how-ai-large-language-models-work/ https://arstechnica.com/science/2023/07/a-jargon-free-explan... There's no intelligence involved in "artificial intelligence" as it currently stands. It's all marketing hype around a really fancy statistical completion engine. Intelligence would require thought and reasoning, which current "AI" does not do, no matter how convincingly it fakes it.
- hunter2_ 2y ago
  Every computing task that was previously the domain of humans, even simple arithmetic back when that was astonishingly rare for a machine, could be described as intelligence if you want to anthropomorphize it or as something less if you don't.
  - rowanG077 2y ago
    I'm simply applying "We use this test to measure intelligence in humans what does this AI do.". We have a priori here that it measures intelligence, before the AI existed. Now the AI scores high on this measurement. There is nothing else but t conclude the AI is intelligent.
    - genocidicbunny 2y ago
      Yes there is. I could conclude that the test wasn't actually measuring intelligence, but just one component that when summed with other components displays intelligence; That is, if the test was purported to alone measure intelligence, it was a flawed assumption. We used to measure intelligence with IQ tests, those are now known to largely be bunk. What's to say our other intelligence tests aren't similarly flawed?
      - rowanG077 2y ago
        Huh? IQ tests are by far the best measure of intelligence we have. Where did you read it is bunk?
    - knotthebest 2y ago
      You’re not understanding the tests. They were designed for humans. They were also designed to be taken once. LLMs have been trained on these tests numerous times. LLMs are also not humans. We can’t conceivably compare the two
      - rowanG077 2y ago
        Just saying these tests were designed for humans doesn't mean anything. You have to specify why exactly it doesn't work for an AI. Or rather let me pose this question. What is the intellectual test you envisions that proves an AI is intelligent that any non-disabled human can easily pass. I'm willing to bet 500 USD it will pass that hurdle in the coming 10 years if you are willing to put your money where your mouth is.
- ben_w 2y ago
  They are indeed — every test, even every informal measure, that I saw before about a decade ago —there's some AI which can do that now; but I think this is revealing that what used to be a distinction without a difference in humans is now suddenly very important indeed. I used to see people getting criticised for being "book-smart" and lacking practical experience… but someone who was able to learn from books can quickly learn from real life, too. AI need a lot of examples to learn from, and make up for this by being on hardware that beats biological neurones by the same degree to which marathon runners beat continental drift, so it can go through those examples much faster, leading to fantastic performance. But the shape of that performance graph is very un-human — you never see a human that's approximately 80-90% accurate at every level of mathematics from basic algebra to helping Terence Tao: https://pandaily.com/mathematician-terence-tao-comments-on-chatgpt/ https://pandaily.com/mathematician-terence-tao-comments-on-c...
- eaglefield 2y ago
  IMO the word intelligence doesn't seem a good description for the thing LLMs possess, they perform pretty well on some tests measuring it. But they're also sometimes wrong in ways intelligent beings (humans) never are. Like this[0] riddle about boats and farmers i stumbled over recently: > A farmer stands on the side of a river with a sheep. There is a boat on the riverbank that has room for exactly one person and one sheep. How can the farmer get across with the sheep in the fewest number of trips? It's obviously riffing on the classic wolf sheep lettuce riddle, but I don't think that's gonna fool any humans into answering anything but the obvious. ChatGPT-4o on the other hand thinks it'll take three trips. They perform a good approximation of intelligence most of the time but the fact that their error pattern is so distinct from humans in some ways, suggests that we probably shouldn't attribute intelligence to them. At least in a human sense of the word.
  - falcor84 2y ago
    I just tried it with chatgpt and got what seems like the right answer: The farmer can get across the river with the sheep in one trip. Here's how: 1. The farmer and the sheep get into the boat. 2. They both cross the river together. Since the boat can hold one person and one sheep, they can make the journey in a single trip. The fewest number of trips required is just one.
    - eaglefield 2y ago
      That's fair. I get the wrong answer on the gpt3 and gpt4-o models, but there's always some uncertainty involved in these gaps. When I appended "Consider your answer carefully." to my prompt, it answered as though the goal was for only the sheep to get across, and that the farmer had to get back to the original shore.
- knotthebest 2y ago
  Well, they were trained on those tests. What LLMs literally do is map inputs to outputs. That’s why they can’t do much outside of training data.
  - rowanG077 2y ago
    I don't see evidence that humans do better here.
    - krapp 2y ago
      Most humans are aware that adding Elmer's glue to pizza cheese to "get the cheese to stick" is a humorous statement that would not work in reality, despite the fact that "glue" makes things "sticky." This should provide ample evidence that humans do better than LLMs here.
      - r2_pilot 2y ago
        I used Claude 3 Sonnet against the cheese sliding off prompt and it gave sensible responses such as "let it cool, don't put so many toppings on it" and no hint of glue. Then again, I'm finding Anthropic to be a better steward of LLM hype than most other companies, which may be why I use Claude 3 in my company.
      - rowanG077 2y ago
        Humans are trained that glue is not to be eaten. It's a meme even that young children eat glue. The example you give is exactly something Humans are trained for.
        krapp 2y ago
        It's a meme because it's unusual, eating glue is a shorthand for having low intelligence because most humans don't eat glue. Beyond that, most humans understand that given the way glue works and given the way cheese works the premise of using glue to make cheese stick to a pizza doesn't even make sense, thus the statement is immediately understood as a pun, not a credible attempt at a recipe. If humans were no better than LLMs in the way that this input were processed, there would be no meme, nor would there be a thread about how ridiculous Google's LLM is. Humans would simply accept as fact that glue can be added to pizza to make the cheese stickier, because the words make syntactic sense. Yet here we are.
ben_w 2y ago
They are transformers, that are just as good at transforming English into German as they are at turning "halp a bad man noked me dn an ran away, how do i su him" into a coherent legal suggestion (which, for this example, should be ~ "Call the police (and if necessary an ambulance), then get an actual lawyer"). When it comes to complex step-by-step reasoning, sure, they're stupid; when it comes to linguistic comprehension, are better at this than the average human — and GPT-4 beats the average law students taking at least one bar exam.

rowanG077 2y ago

That's a pretty rich statement considering they perform pretty good at tests which we have designed to measure intelligence. I don't see how you can say there is no intelligence.

whythre 2y ago
Sounds like a problem with the tests being administered. There is a lot of woo around LLMs, a lot of people have a vested interest in hyping and selling AI; heck, even referring to LLMs as AI is a form of branding.
- rowanG077 2y ago
  So turn it around then. When is an AI intelligent? Since when it performs at human standard in intelligence tests doesn't seem enough for you.
  - blooalien 2y ago
    https://arstechnica.com/science/2023/07/a-jargon-free-explanation-of-how-ai-large-language-models-work/ https://arstechnica.com/science/2023/07/a-jargon-free-explan... There's no intelligence involved in "artificial intelligence" as it currently stands. It's all marketing hype around a really fancy statistical completion engine. Intelligence would require thought and reasoning, which current "AI" does not do, no matter how convincingly it fakes it.
hunter2_ 2y ago
Every computing task that was previously the domain of humans, even simple arithmetic back when that was astonishingly rare for a machine, could be described as intelligence if you want to anthropomorphize it or as something less if you don't.
- rowanG077 2y ago
  I'm simply applying "We use this test to measure intelligence in humans what does this AI do.". We have a priori here that it measures intelligence, before the AI existed. Now the AI scores high on this measurement. There is nothing else but t conclude the AI is intelligent.
  - genocidicbunny 2y ago
    Yes there is. I could conclude that the test wasn't actually measuring intelligence, but just one component that when summed with other components displays intelligence; That is, if the test was purported to alone measure intelligence, it was a flawed assumption. We used to measure intelligence with IQ tests, those are now known to largely be bunk. What's to say our other intelligence tests aren't similarly flawed?
    - rowanG077 2y ago
      Huh? IQ tests are by far the best measure of intelligence we have. Where did you read it is bunk?
  - knotthebest 2y ago
    You’re not understanding the tests. They were designed for humans. They were also designed to be taken once. LLMs have been trained on these tests numerous times. LLMs are also not humans. We can’t conceivably compare the two
    - rowanG077 2y ago
      Just saying these tests were designed for humans doesn't mean anything. You have to specify why exactly it doesn't work for an AI. Or rather let me pose this question. What is the intellectual test you envisions that proves an AI is intelligent that any non-disabled human can easily pass. I'm willing to bet 500 USD it will pass that hurdle in the coming 10 years if you are willing to put your money where your mouth is.
ben_w 2y ago
They are indeed — every test, even every informal measure, that I saw before about a decade ago —there's some AI which can do that now; but I think this is revealing that what used to be a distinction without a difference in humans is now suddenly very important indeed. I used to see people getting criticised for being "book-smart" and lacking practical experience… but someone who was able to learn from books can quickly learn from real life, too. AI need a lot of examples to learn from, and make up for this by being on hardware that beats biological neurones by the same degree to which marathon runners beat continental drift, so it can go through those examples much faster, leading to fantastic performance. But the shape of that performance graph is very un-human — you never see a human that's approximately 80-90% accurate at every level of mathematics from basic algebra to helping Terence Tao: https://pandaily.com/mathematician-terence-tao-comments-on-chatgpt/ https://pandaily.com/mathematician-terence-tao-comments-on-c...
eaglefield 2y ago
IMO the word intelligence doesn't seem a good description for the thing LLMs possess, they perform pretty well on some tests measuring it. But they're also sometimes wrong in ways intelligent beings (humans) never are. Like this[0] riddle about boats and farmers i stumbled over recently: > A farmer stands on the side of a river with a sheep. There is a boat on the riverbank that has room for exactly one person and one sheep. How can the farmer get across with the sheep in the fewest number of trips? It's obviously riffing on the classic wolf sheep lettuce riddle, but I don't think that's gonna fool any humans into answering anything but the obvious. ChatGPT-4o on the other hand thinks it'll take three trips. They perform a good approximation of intelligence most of the time but the fact that their error pattern is so distinct from humans in some ways, suggests that we probably shouldn't attribute intelligence to them. At least in a human sense of the word.
- falcor84 2y ago
  I just tried it with chatgpt and got what seems like the right answer: The farmer can get across the river with the sheep in one trip. Here's how: 1. The farmer and the sheep get into the boat. 2. They both cross the river together. Since the boat can hold one person and one sheep, they can make the journey in a single trip. The fewest number of trips required is just one.
  - eaglefield 2y ago
    That's fair. I get the wrong answer on the gpt3 and gpt4-o models, but there's always some uncertainty involved in these gaps. When I appended "Consider your answer carefully." to my prompt, it answered as though the goal was for only the sheep to get across, and that the farmer had to get back to the original shore.
knotthebest 2y ago
Well, they were trained on those tests. What LLMs literally do is map inputs to outputs. That’s why they can’t do much outside of training data.
- rowanG077 2y ago
  I don't see evidence that humans do better here.
  - krapp 2y ago
    Most humans are aware that adding Elmer's glue to pizza cheese to "get the cheese to stick" is a humorous statement that would not work in reality, despite the fact that "glue" makes things "sticky." This should provide ample evidence that humans do better than LLMs here.
    - r2_pilot 2y ago
      I used Claude 3 Sonnet against the cheese sliding off prompt and it gave sensible responses such as "let it cool, don't put so many toppings on it" and no hint of glue. Then again, I'm finding Anthropic to be a better steward of LLM hype than most other companies, which may be why I use Claude 3 in my company.
    - rowanG077 2y ago
      Humans are trained that glue is not to be eaten. It's a meme even that young children eat glue. The example you give is exactly something Humans are trained for.
      - krapp 2y ago
        It's a meme because it's unusual, eating glue is a shorthand for having low intelligence because most humans don't eat glue. Beyond that, most humans understand that given the way glue works and given the way cheese works the premise of using glue to make cheese stick to a pizza doesn't even make sense, thus the statement is immediately understood as a pun, not a credible attempt at a recipe. If humans were no better than LLMs in the way that this input were processed, there would be no meme, nor would there be a thread about how ridiculous Google's LLM is. Humans would simply accept as fact that glue can be added to pizza to make the cheese stickier, because the words make syntactic sense. Yet here we are.

ben_w 2y ago

They are transformers, that are just as good at transforming English into German as they are at turning "halp a bad man noked me dn an ran away, how do i su him" into a coherent legal suggestion (which, for this example, should be ~ "Call the police (and if necessary an ambulance), then get an actual lawyer"). When it comes to complex step-by-step reasoning, sure, they're stupid; when it comes to linguistic comprehension, are better at this than the average human — and GPT-4 beats the average law students taking at least one bar exam.