4 ms·

As a non-coder I can get away with asking this: Why is it so hard to simulate reason? Logic and reason are based on rules. Then you add values to steer the con

As a non-coder I can get away with asking this: Why is it so hard to simulate reason? Logic and reason are based on rules. Then you add values to steer the conclusions based on the available data. Why not have separate systems for values and logic and memory working together as an AI brain to generate truly reasoned responses? You could even have adversarial parts that duke it out (left-wing vs right-wing, Jefferson versus Adams) to fine tune its conclusions based on the values bias you've selected.

vaylian 2y ago
LLMs operate on word co-occurances. Given an initial prompt (consisting of words), it tries to predict the most likely next word. And then it goes on to predict the next most likely word based on the previous output. There is also a bit of randomization involved, so that it doesn't always produce the same results. Some people call these LLMs "statistical parrots", because the LLMs simply repeat the word combinations that they were trained with. But despite this very simple approach, we get very convincing outputs for things that do not require reasoning. LLMs might have encountered examples of logical reasoning in their training data, but they can't generalize it to arbitrary logical reasoning, because that's not something they are designed to do. Disclaimer: I'm not an expert in AI and I do not follow the developments on a deep technical level.
- mrweasel 2y ago
  The level of correctness from LLMs is actually pretty impressive, given that they are most just attempting to predict the next word, based on the prompt. It shouldn't come as much of a surprise that we can easily formulate questions that it will get wrong, by wording questions in a particular way, or asking about subjects for which is has little to no training data. For me the absolutely terrifying thing isn't that LLMs get answers wrong, it's the confidence with which it express those answers and how much some people / companies do not care. We know that the LLMs will get some answers wrong, they will lie, they will make up facts to justify their answers, but if will only do those last two because we make them and insist that they answer all questions (expect those where the developers put in restriction as to not offend). In some way I feel like the model should be able to rely a confidence score to the user, mostly that might be an interface issue, because we insist on limiting ourselves to the chat bot interface. The confidence score should perhaps exist outside the answer box. So you'd get an answer, and next to it a score from 0 - 100 perhaps, 0 meaning that the model doesn't actually had the training data that would allow it to answer the question.
program_whiz 2y ago
you are describing GOFAI (good old fashioned AI) which was using things like logical production rules. The issue is that the ability to model the complexity of things like human languages and continuous/real-valued systems makes it infeasible to use at scale. For something like the alice brothers/sisters example, that is a canonical example where GOFAI and/or logical reasoning with languages like prolog excel. A common example in prolog is: philosophers => human human => mortal plato => philosopher plato mortal? Yes But encoding enough semantic information to create compelling AI with this type of system is difficult. Some have tried to enter thousands/millions of rules and still the system isn't convincing. The main breakthrough that has enabled LLMs is an encoding of words that relies on their frequency in being near other words in the english language (using all the text available on the internet). Therefore words like "philosopher" and "plato" become associated in a high-dimensional space (so instead of "plato" you have a "token" with thousands of numbers associated with it). You can then perform numeric operations on these numbers to come to conclusions. For example, we would expect something like a "human name" to emerge in this embedding space where we could determine if something "is used like a name" in various contexts by applying some non-linear transformations of the word vector / token. LLMs have simply make a sequence of these transforms, while using prior words it generates as additional input (which allows it to construct sentences). So it is quite different from traditional reasoning. It is better at "fuzzy reasoning" but also worse in situations that require precise results (in fact, at each step it chooses one of a few best possible words based on its stats at random, the variation in this is called 'temperature').
YeGoblynQueenne 2y ago
>> As a non-coder I can get away with asking this: Why is it so hard to simulate reason? It isn't. We know how to do reasoning with computers. The discussion about reasoning in LLMs is carried out in an echo chamber that ignores the prior work on reasoning (for a bit of a summary see my bio). Which of course makes it very hard for the people involved to understand why their systems fail at it; or, often, that they fail at it.

vaylian 2y ago

LLMs operate on word co-occurances. Given an initial prompt (consisting of words), it tries to predict the most likely next word. And then it goes on to predict the next most likely word based on the previous output. There is also a bit of randomization involved, so that it doesn't always produce the same results. Some people call these LLMs "statistical parrots", because the LLMs simply repeat the word combinations that they were trained with. But despite this very simple approach, we get very convincing outputs for things that do not require reasoning. LLMs might have encountered examples of logical reasoning in their training data, but they can't generalize it to arbitrary logical reasoning, because that's not something they are designed to do. Disclaimer: I'm not an expert in AI and I do not follow the developments on a deep technical level.

mrweasel 2y ago
The level of correctness from LLMs is actually pretty impressive, given that they are most just attempting to predict the next word, based on the prompt. It shouldn't come as much of a surprise that we can easily formulate questions that it will get wrong, by wording questions in a particular way, or asking about subjects for which is has little to no training data. For me the absolutely terrifying thing isn't that LLMs get answers wrong, it's the confidence with which it express those answers and how much some people / companies do not care. We know that the LLMs will get some answers wrong, they will lie, they will make up facts to justify their answers, but if will only do those last two because we make them and insist that they answer all questions (expect those where the developers put in restriction as to not offend). In some way I feel like the model should be able to rely a confidence score to the user, mostly that might be an interface issue, because we insist on limiting ourselves to the chat bot interface. The confidence score should perhaps exist outside the answer box. So you'd get an answer, and next to it a score from 0 - 100 perhaps, 0 meaning that the model doesn't actually had the training data that would allow it to answer the question.

program_whiz 2y ago

you are describing GOFAI (good old fashioned AI) which was using things like logical production rules. The issue is that the ability to model the complexity of things like human languages and continuous/real-valued systems makes it infeasible to use at scale. For something like the alice brothers/sisters example, that is a canonical example where GOFAI and/or logical reasoning with languages like prolog excel. A common example in prolog is: philosophers => human human => mortal plato => philosopher plato mortal? Yes But encoding enough semantic information to create compelling AI with this type of system is difficult. Some have tried to enter thousands/millions of rules and still the system isn't convincing. The main breakthrough that has enabled LLMs is an encoding of words that relies on their frequency in being near other words in the english language (using all the text available on the internet). Therefore words like "philosopher" and "plato" become associated in a high-dimensional space (so instead of "plato" you have a "token" with thousands of numbers associated with it). You can then perform numeric operations on these numbers to come to conclusions. For example, we would expect something like a "human name" to emerge in this embedding space where we could determine if something "is used like a name" in various contexts by applying some non-linear transformations of the word vector / token. LLMs have simply make a sequence of these transforms, while using prior words it generates as additional input (which allows it to construct sentences). So it is quite different from traditional reasoning. It is better at "fuzzy reasoning" but also worse in situations that require precise results (in fact, at each step it chooses one of a few best possible words based on its stats at random, the variation in this is called 'temperature').

YeGoblynQueenne 2y ago

>> As a non-coder I can get away with asking this: Why is it so hard to simulate reason? It isn't. We know how to do reasoning with computers. The discussion about reasoning in LLMs is carried out in an echo chamber that ignores the prior work on reasoning (for a bit of a summary see my bio). Which of course makes it very hard for the people involved to understand why their systems fail at it; or, often, that they fail at it.