12 ms·

LLMs cannot reason or use mathematics - in a way, they don't know what they are talking about. Why would such technology lead to superhuman smarts?

circuit10 3y ago
What is your definition of “reasoning” here? They are clearly able to do many things that we would call reasoning if a human did them
- RandomLensman 3y ago
  They seem to not be able to use concepts they can "explain" (beyond having answers sufficiently dense in the training set, it seems).
  - circuit10 3y ago
    Humans also take a while to learn how to use concepts that they might be able to remember an answer to, though. Also keep in mind that LLMs have a limited time to produce each token and can’t “stop to think” like a human could. Obviously LLMs aren’t as good as reasoning as humans but they clearly do some (possibly quite limited) form of reasoning
    - RandomLensman 3y ago
      Do you have good examples of an LLM reasoning, i.e., using a concept it explains and giving an answer outside of its training data on something that has a clearly defined correct answer?
      - circuit10 3y ago
        A simple example is that you can give an LLM API documentation, or even just an example URL and response, and (if the API is simple) it will have no trouble using that in a program even if it never saw that API before
        RandomLensman 3y ago
        Fair enough, but isn't the API documentation already providing the solution in a way? It's an instruction manual of sorts that spells out the required usage, not an abstract concept.
        circuit10 3y ago
        Well, if does anything other than copying the input verbatim or producing nonsense, it has to parse the language, which in itself uses abstract concepts, understand the purpose of the API and what it does, and then apply it in a a new context, so I would say it’s using abstract concepts in many hidden ways that we humans take for granted because that’s what our brains are built to do.
        RandomLensman 3y ago
        This is very close to Searle's "Chinese room". Even as a human I don't have to understand the purpose of an API to use it based on the documentation.
      - MVissers 3y ago
        From 'Sparks of AGI' paper from Microsoft (https://arxiv.org/pdf/2303.12712.pdf https://arxiv.org/pdf/2303.12712.pdf). Caveat: The GTP-4 model they used in their example might have been more powerful than the one we use publicly today, since the public version is trained for safety and less 'intelligent' than the raw research version. Prompt: Here we have a book, 9 eggs, a laptop, a bottle and a nail. Please tell me how to stack them onto each other in a stable manner. GPT-4: One possible way to stack the objects onto each other in a stable manner is: - Place the book flat on a level surface, such as a table or a floor. The book will serve as the base of the stack and provide a large and sturdy support. - Arrange the 9 eggs in a 3 by 3 square on top of the book, leaving some space between them. The eggs will form a second layer and distribute the weight evenly. Make sure the eggs are not cracked or broken, and handle them gently to avoid dropping them. - Place the laptop on top of the eggs, with the screen facing down and the keyboard facing up. The laptop will fit snugly within the boundaries of the book and the eggs, and its flat and rigid surface will provide a stable platform for the next layer. Make sure the laptop is closed and turned off, and do not press any keys or buttons. - Place the bottle on top of the laptop, with the cap facing up and the bottom facing down. The bottle will add some height and balance to the stack, and its cylindrical shape will reduce the contact area with the laptop. Make sure the bottle is empty and closed, and do not shake or tilt it. - Place the nail on top of the bottle cap, with the pointy end facing up and the flat end facing down. The nail will be the final and smallest object in the stack, and its sharp tip will prevent it from rolling or sliding off the bottle. Make sure the nail is clean and straight, and do not touch or move it. ChatGPT (GTP 3.5): It would be difficult to stack all of these objects on top of each other in a stable manner without using any extra objects. The book and laptop are likely too large and heavy to stack on top of the eggs without damaging them. The bottle and nail may be too small to provide a stable base for the other objects. Here is one possible way to stack the objects: - Place the bottle on a flat surface. - Carefully balance the nail on top of the bottle. - Place the eggs on top of the nail, making sure they are balanced and not tilting to one side. - Place the laptop on top of the eggs, again making sure it is balanced and not tilting to one side. - Finally, place the book on top of the laptop. This stack may not be very stable, so it is important to be careful when handling it. Alternatively, you could try stacking the objects in a different order or using some of the objects as supports for others to create a more stable stack.
- f6v 3y ago
  If you put it this way, solving a differential equation numerically also provides an answer similar to human. But you don’t call Matlab an AI.
  - deleted 3y ago
    [deleted]
  - circuit10 3y ago
    I guess that can called a form of reasoning, LLMs are much more general in their reasoning abilities though
- hammyhavoc 3y ago
  I mention this all the time. I wanted GPT to give me an NGINX config for Active Collab as we're previously using Apache, and thus an htaccess file. I fed it all the documentation from Active Collab, I couldn't get anything valid out of it. Hallucinated all kinds of things that weren't there. I then gave it the URL rewrites that would be required, line-by-line, spent a long time trying to correct it. No bueno, even worse hallucination. I spent days on trying to get it to output a valid NGINX config that incorporated these URL rewrites. It can't reason, it's doing exactly what LLMs do, which is next word prediction. I can't imagine what people are using it for in terms of a valuable addition to their workflow with how much it hallucinates. If it can't even do an NGINX config, what use is it for anything else? People saying it's helping them learn programming languages. Fuck me, they don't know when it's wrong, and it will be wrong at some point, it's an LLM.
  - stevenhuang 3y ago
    There are flaws but also consider PEBCAK error. For one next time when it starts hallucinating and a gentle course correction doesn't do it, just start a new chat with a different prompt approach. Having the error in its context reinforces the same mistake and sometimes it can't get out of this loop.
    - hammyhavoc 3y ago
      PEBKAC in what context? The prompts themselves or the documentation? Because I got Active Collab running on NGINX myself. I already did this in terms of starting new chats, I spent days on it, and consulted with half a dozen devs supposedly using it in their workflows. It's very easy to make it hallucinate.
      - stevenhuang 3y ago
        In terms of using gpt optimally. But fair enough. If you tried it in multiple sessions trying to convert Apache rewrites to nginx and it wasn't able to do it, I guess this is another failure mode. I just found that curious because chatgpt is usually very very good at regex. Side note is Google extra terrible lately or is there really no docs on this almost anywhere? All I could find about it is this and the rules looked very simple, from my experience chatgpt should have got this https://activecollab.com/help/books/self-hosted-activecollab/requirements https://activecollab.com/help/books/self-hosted-activecollab...
quantum_state 3y ago
would like to second your observation…
sidibe 3y ago
Who knows what there will be besides LLMs. I don't really get why AI can't exceed the human brain in everything someday unless you are religious and see some supernatural aspects to the brain
- RandomLensman 3y ago
  Maybe we can or cannot build such a thing. We have no natural example for some exponentially self improving intelligence. We also cannot build living animals from scratch or are anywhere close to it - maybe some forms of AI are much tougher to do than we think.
  - SanderNL 3y ago
    If you take a step back and look at computing in general as some amorphous evolving entity, it can be said the Machines are getting better and I would be surprised if it wasn’t exponential. Talking out of my ass here, but my point is that I think The Machines(c) don’t look like biological and separated entities. I think it’ll look more like what we call corporations (hive minds) composed of a vast variety of different functional parts.
- hammyhavoc 3y ago
  There was one article I read that discussed an "AI Winter". The tl;dr being that our entire global compute likely isn't sufficient enough for a hypothetical AGI. What's more likely, IMO, is using real brain cells. https://www.ucl.ac.uk/news/2022/oct/human-brain-cells-dish-learn-play-pong https://www.ucl.ac.uk/news/2022/oct/human-brain-cells-dish-l... However, real brain cells are a big question of ethics if it's thus actually able to think. I would argue that we've then created a slave rather than a machine, and that is unacceptable.
f6v 3y ago
AGI won’t be an LLM same as it won’t be an LSTM or CNN. But it’s an impressive step towards AGI.
- RandomLensman 3y ago
  Honest question: How do you know it is a step towards AGI?
  - hammyhavoc 3y ago
    Because a research paper claimed that they believe it's a basic and incomplete AGI. However, said paper then goes on to actually say LLMs aren't the way forward if people bother to read it. One comment on HN called it a "baby AGI" after linking to the paper. Eye roll inducing.
    - f6v 3y ago
      Yeah but LLMs could be a small component of future AGI. You can’t deny that models that fool so many people into thinking the models really reason are a step towards AGI.
  - wokwokwok 3y ago
    If you accept that AGI is possible at all… How can a something that generates such a massive surge of interest, investment and research into AI not be a step toward it? Saying it’s not a step towards AGI is basically saying AGI isn’t possible at all, because it means that all our efforts are making zero progress on AGI. That’s not a falsifiable position to take. If you’re serious, the parent post literally said “AGI isnt going to look like this”. …but realistically, how would a LLM that could easily refine itself from experiences, and had a very large context, let’s say, a billion tokens, be meaningfully different from AGI? It could learn. It could remember things. It could generate human like output from a complex context. Sure, it’s just a stochastic parrot… but if it can refine the model from real world inputs (learn new tricks, learn games, etc) and generate large scale (entire books worth) of coherent conversation and interactions… where do you draw the line between that and actual AGI? Large contexts (35k tokens) are here right now. Refining models is here right now. They’re just expensive and slow (inference and training). Maybe the current architecture doesn’t scale up beyond that and it’s a dead end, but my gosh. If you don’t think what we have is a step towards AGI you really have to work hard to make your definition of AGI very very difficult to attain.
cubefox 3y ago
LLMs are trained with a form of imitation learning, they imitate human (and other) text. It seems indeed not likely that pure LLMs will advance far beyond human ability, since even a perfect LLM could only imitate human text perfectly. But other approaches will follow.

circuit10 3y ago

What is your definition of “reasoning” here? They are clearly able to do many things that we would call reasoning if a human did them

RandomLensman 3y ago
They seem to not be able to use concepts they can "explain" (beyond having answers sufficiently dense in the training set, it seems).
- circuit10 3y ago
  Humans also take a while to learn how to use concepts that they might be able to remember an answer to, though. Also keep in mind that LLMs have a limited time to produce each token and can’t “stop to think” like a human could. Obviously LLMs aren’t as good as reasoning as humans but they clearly do some (possibly quite limited) form of reasoning
  - RandomLensman 3y ago
    Do you have good examples of an LLM reasoning, i.e., using a concept it explains and giving an answer outside of its training data on something that has a clearly defined correct answer?
    - circuit10 3y ago
      A simple example is that you can give an LLM API documentation, or even just an example URL and response, and (if the API is simple) it will have no trouble using that in a program even if it never saw that API before
      - RandomLensman 3y ago
        Fair enough, but isn't the API documentation already providing the solution in a way? It's an instruction manual of sorts that spells out the required usage, not an abstract concept.
        circuit10 3y ago
        Well, if does anything other than copying the input verbatim or producing nonsense, it has to parse the language, which in itself uses abstract concepts, understand the purpose of the API and what it does, and then apply it in a a new context, so I would say it’s using abstract concepts in many hidden ways that we humans take for granted because that’s what our brains are built to do.
        RandomLensman 3y ago
        This is very close to Searle's "Chinese room". Even as a human I don't have to understand the purpose of an API to use it based on the documentation.
    - MVissers 3y ago
      From 'Sparks of AGI' paper from Microsoft (https://arxiv.org/pdf/2303.12712.pdf https://arxiv.org/pdf/2303.12712.pdf). Caveat: The GTP-4 model they used in their example might have been more powerful than the one we use publicly today, since the public version is trained for safety and less 'intelligent' than the raw research version. Prompt: Here we have a book, 9 eggs, a laptop, a bottle and a nail. Please tell me how to stack them onto each other in a stable manner. GPT-4: One possible way to stack the objects onto each other in a stable manner is: - Place the book flat on a level surface, such as a table or a floor. The book will serve as the base of the stack and provide a large and sturdy support. - Arrange the 9 eggs in a 3 by 3 square on top of the book, leaving some space between them. The eggs will form a second layer and distribute the weight evenly. Make sure the eggs are not cracked or broken, and handle them gently to avoid dropping them. - Place the laptop on top of the eggs, with the screen facing down and the keyboard facing up. The laptop will fit snugly within the boundaries of the book and the eggs, and its flat and rigid surface will provide a stable platform for the next layer. Make sure the laptop is closed and turned off, and do not press any keys or buttons. - Place the bottle on top of the laptop, with the cap facing up and the bottom facing down. The bottle will add some height and balance to the stack, and its cylindrical shape will reduce the contact area with the laptop. Make sure the bottle is empty and closed, and do not shake or tilt it. - Place the nail on top of the bottle cap, with the pointy end facing up and the flat end facing down. The nail will be the final and smallest object in the stack, and its sharp tip will prevent it from rolling or sliding off the bottle. Make sure the nail is clean and straight, and do not touch or move it. ChatGPT (GTP 3.5): It would be difficult to stack all of these objects on top of each other in a stable manner without using any extra objects. The book and laptop are likely too large and heavy to stack on top of the eggs without damaging them. The bottle and nail may be too small to provide a stable base for the other objects. Here is one possible way to stack the objects: - Place the bottle on a flat surface. - Carefully balance the nail on top of the bottle. - Place the eggs on top of the nail, making sure they are balanced and not tilting to one side. - Place the laptop on top of the eggs, again making sure it is balanced and not tilting to one side. - Finally, place the book on top of the laptop. This stack may not be very stable, so it is important to be careful when handling it. Alternatively, you could try stacking the objects in a different order or using some of the objects as supports for others to create a more stable stack.
f6v 3y ago
If you put it this way, solving a differential equation numerically also provides an answer similar to human. But you don’t call Matlab an AI.
- deleted 3y ago
  [deleted]
- circuit10 3y ago
  I guess that can called a form of reasoning, LLMs are much more general in their reasoning abilities though
hammyhavoc 3y ago
I mention this all the time. I wanted GPT to give me an NGINX config for Active Collab as we're previously using Apache, and thus an htaccess file. I fed it all the documentation from Active Collab, I couldn't get anything valid out of it. Hallucinated all kinds of things that weren't there. I then gave it the URL rewrites that would be required, line-by-line, spent a long time trying to correct it. No bueno, even worse hallucination. I spent days on trying to get it to output a valid NGINX config that incorporated these URL rewrites. It can't reason, it's doing exactly what LLMs do, which is next word prediction. I can't imagine what people are using it for in terms of a valuable addition to their workflow with how much it hallucinates. If it can't even do an NGINX config, what use is it for anything else? People saying it's helping them learn programming languages. Fuck me, they don't know when it's wrong, and it will be wrong at some point, it's an LLM.
- stevenhuang 3y ago
  There are flaws but also consider PEBCAK error. For one next time when it starts hallucinating and a gentle course correction doesn't do it, just start a new chat with a different prompt approach. Having the error in its context reinforces the same mistake and sometimes it can't get out of this loop.
  - hammyhavoc 3y ago
    PEBKAC in what context? The prompts themselves or the documentation? Because I got Active Collab running on NGINX myself. I already did this in terms of starting new chats, I spent days on it, and consulted with half a dozen devs supposedly using it in their workflows. It's very easy to make it hallucinate.
    - stevenhuang 3y ago
      In terms of using gpt optimally. But fair enough. If you tried it in multiple sessions trying to convert Apache rewrites to nginx and it wasn't able to do it, I guess this is another failure mode. I just found that curious because chatgpt is usually very very good at regex. Side note is Google extra terrible lately or is there really no docs on this almost anywhere? All I could find about it is this and the rules looked very simple, from my experience chatgpt should have got this https://activecollab.com/help/books/self-hosted-activecollab/requirements https://activecollab.com/help/books/self-hosted-activecollab...

quantum_state 3y ago

would like to second your observation…

sidibe 3y ago

Who knows what there will be besides LLMs. I don't really get why AI can't exceed the human brain in everything someday unless you are religious and see some supernatural aspects to the brain

RandomLensman 3y ago
Maybe we can or cannot build such a thing. We have no natural example for some exponentially self improving intelligence. We also cannot build living animals from scratch or are anywhere close to it - maybe some forms of AI are much tougher to do than we think.
- SanderNL 3y ago
  If you take a step back and look at computing in general as some amorphous evolving entity, it can be said the Machines are getting better and I would be surprised if it wasn’t exponential. Talking out of my ass here, but my point is that I think The Machines(c) don’t look like biological and separated entities. I think it’ll look more like what we call corporations (hive minds) composed of a vast variety of different functional parts.
hammyhavoc 3y ago
There was one article I read that discussed an "AI Winter". The tl;dr being that our entire global compute likely isn't sufficient enough for a hypothetical AGI. What's more likely, IMO, is using real brain cells. https://www.ucl.ac.uk/news/2022/oct/human-brain-cells-dish-learn-play-pong https://www.ucl.ac.uk/news/2022/oct/human-brain-cells-dish-l... However, real brain cells are a big question of ethics if it's thus actually able to think. I would argue that we've then created a slave rather than a machine, and that is unacceptable.

f6v 3y ago

AGI won’t be an LLM same as it won’t be an LSTM or CNN. But it’s an impressive step towards AGI.

RandomLensman 3y ago
Honest question: How do you know it is a step towards AGI?
- hammyhavoc 3y ago
  Because a research paper claimed that they believe it's a basic and incomplete AGI. However, said paper then goes on to actually say LLMs aren't the way forward if people bother to read it. One comment on HN called it a "baby AGI" after linking to the paper. Eye roll inducing.
  - f6v 3y ago
    Yeah but LLMs could be a small component of future AGI. You can’t deny that models that fool so many people into thinking the models really reason are a step towards AGI.
- wokwokwok 3y ago
  If you accept that AGI is possible at all… How can a something that generates such a massive surge of interest, investment and research into AI not be a step toward it? Saying it’s not a step towards AGI is basically saying AGI isn’t possible at all, because it means that all our efforts are making zero progress on AGI. That’s not a falsifiable position to take. If you’re serious, the parent post literally said “AGI isnt going to look like this”. …but realistically, how would a LLM that could easily refine itself from experiences, and had a very large context, let’s say, a billion tokens, be meaningfully different from AGI? It could learn. It could remember things. It could generate human like output from a complex context. Sure, it’s just a stochastic parrot… but if it can refine the model from real world inputs (learn new tricks, learn games, etc) and generate large scale (entire books worth) of coherent conversation and interactions… where do you draw the line between that and actual AGI? Large contexts (35k tokens) are here right now. Refining models is here right now. They’re just expensive and slow (inference and training). Maybe the current architecture doesn’t scale up beyond that and it’s a dead end, but my gosh. If you don’t think what we have is a step towards AGI you really have to work hard to make your definition of AGI very very difficult to attain.

cubefox 3y ago

LLMs are trained with a form of imitation learning, they imitate human (and other) text. It seems indeed not likely that pure LLMs will advance far beyond human ability, since even a perfect LLM could only imitate human text perfectly. But other approaches will follow.