6 ms·

I don't think it's overly philosophical to point out that these are large language models, not truth engines or AGI or knowledge directories. They're not using

I don't think it's overly philosophical to point out that these are large language models, not truth engines or AGI or knowledge directories. They're not using logic to reason their way to an answer. They're just predicting the next word that would sound like part of a human answer.

farleykr 2y ago
Fair enough. I think a lot of people are going to end up blindly trusting AI because its right often enough. But for those who are interested in what it really means to know something, I wonder if this will push people back towards embracing the idea that there is fundamental, objective, knowable truth at the core of the universe even if we can't ever know that truth perfectly.
kennysoona 2y ago
> They're not using logic to reason their way to an answer. They're just predicting the next word that would sound like part of a human answer. OpenAI claims recent models are actually reasoning to some extent.
- thomastjeffery 2y ago
  Only by conveniently redefining the word. Instead, they predict the next tokens of a "think out loud" example, and wrap it up with a "conclusion and summary" example. It doesn't know why this writing pattern is the semantic space it is exploring: it has simply been set up to do so in the first place.
  - yetihehe 2y ago
    Does it matter if it doesn't know why this particular pattern is suitable? Also, do you always ask yourself why you use that particular pattern all the time, or do you just use them?
    - thomastjeffery 2y ago
      It seems like you are implying that I don't think before I speak. Maybe that is sometimes the case, but I would venture to say, "not usually, and certainly not always." The point I'm making here is that all of these observations are made after-the-fact. We humans see five different categories of output: 1. "I do know X" where X is indeed correct information 2. "I do know X" where X is false information or nonsense 3. "I don't know" when it really doesn't 4. "I don't know" when a slightly different prompt would lead to option #1 5. Output that is not phrased as a direct answer to a question. The article introduced #2 as "hallucinations". I introduced #4 in my previous comment (and just now #5), and propose that all five are hallucinations. As far as the LLM is concerned, there is only one category of output: the most likely next token. Which of the five that will be is determined by the examples present in the training corpus, which are later weighed during training. Logic is not present in the process. It is only present in the result.
      - yetihehe 2y ago
        > It seems like you are implying that I don't think before I speak. I'm implying that most times you don't think before you think or after you think (you or me typically don't meta-think). I'm saying that very often I (and looks like a lot of people around me) don't think much before I speak. I have internal monologue when I'm "thinking something out", but I typically don't think things through when I'm speaking with people in day-to-day conversations, only when I encounter a problem I didn't see yet and I'm not "trained" in solving it. Maybe some people can make fully reasoned sentences in split seconds before they start talking, but not me. IIRC those two modes of thinking are called slow and fast thinking. > Logic is not present in the process. It is only present in the result. I'm talking about that process. Have you seen "thinking" part of current reasoning LLM's? It does indeed look like a process of using logic. After "thinking" part, there is "output" part that makes conclusions form the process of thinking. Recently I asked local version of deepseek about a gas exchange problem and it thought a lot about this, making some small mistakes in logic, correcting them, ultimately returning approximately valid result. It even made some small errors in calculations and corrected itself by multiplying parts of numbers and adding them for correct result. I've put that example online[1] if you'd like to read it, it's pretty interesting. [1] https://pastebin.com/mXyLGCGQ https://pastebin.com/mXyLGCGQ
- techjamie 2y ago
  They're just outputting tokens that resemble a reasoning process. The underlying tech is still the same LLM it always has been. I can't deny that doing it that way improves results, but any model could do the same thing if you add extra prompts to encourage the reasoning process, then use that as context for the final solution. People discovered that trick before "reasoning" models became the hot thing. It's the "Work it out step by step" trick but in a dedicated fine-tune.
  - yetihehe 2y ago
    > They're just outputting tokens that resemble a reasoning process. Looking at one such process of emulating reasoning (got deepseek-70B locally), I'm starting to wonder how does that differ from actual reasoning? We "think" about something, may make errors in that thinking, look for things that don't make sense and correct ourselves. That "think" step is still a blackbox. I asked that llm a typical question of gas exchange between containers, it made some errors and noticed some calculations that didn't make sense: > Moles left A: ~0.0021 mol > Moles entered B: ~0.008 mol > But 0.0021 +0.008=0.0101 mol, which doesn't make sense because that would imply a net increase of moles in the system. Well, that's totally invalid calculation, it should be "-" in there. It also noticed that those quantities should be same in other place. Eventually, after 102 minutes and 10141 tokens, involving checking answers from different angles multiple times, it outputted approximately correct response.

farleykr 2y ago

Fair enough. I think a lot of people are going to end up blindly trusting AI because its right often enough. But for those who are interested in what it really means to know something, I wonder if this will push people back towards embracing the idea that there is fundamental, objective, knowable truth at the core of the universe even if we can't ever know that truth perfectly.

kennysoona 2y ago

> They're not using logic to reason their way to an answer. They're just predicting the next word that would sound like part of a human answer. OpenAI claims recent models are actually reasoning to some extent.

thomastjeffery 2y ago
Only by conveniently redefining the word. Instead, they predict the next tokens of a "think out loud" example, and wrap it up with a "conclusion and summary" example. It doesn't know why this writing pattern is the semantic space it is exploring: it has simply been set up to do so in the first place.
- yetihehe 2y ago
  Does it matter if it doesn't know why this particular pattern is suitable? Also, do you always ask yourself why you use that particular pattern all the time, or do you just use them?
  - thomastjeffery 2y ago
    It seems like you are implying that I don't think before I speak. Maybe that is sometimes the case, but I would venture to say, "not usually, and certainly not always." The point I'm making here is that all of these observations are made after-the-fact. We humans see five different categories of output: 1. "I do know X" where X is indeed correct information 2. "I do know X" where X is false information or nonsense 3. "I don't know" when it really doesn't 4. "I don't know" when a slightly different prompt would lead to option #1 5. Output that is not phrased as a direct answer to a question. The article introduced #2 as "hallucinations". I introduced #4 in my previous comment (and just now #5), and propose that all five are hallucinations. As far as the LLM is concerned, there is only one category of output: the most likely next token. Which of the five that will be is determined by the examples present in the training corpus, which are later weighed during training. Logic is not present in the process. It is only present in the result.
    - yetihehe 2y ago
      > It seems like you are implying that I don't think before I speak. I'm implying that most times you don't think before you think or after you think (you or me typically don't meta-think). I'm saying that very often I (and looks like a lot of people around me) don't think much before I speak. I have internal monologue when I'm "thinking something out", but I typically don't think things through when I'm speaking with people in day-to-day conversations, only when I encounter a problem I didn't see yet and I'm not "trained" in solving it. Maybe some people can make fully reasoned sentences in split seconds before they start talking, but not me. IIRC those two modes of thinking are called slow and fast thinking. > Logic is not present in the process. It is only present in the result. I'm talking about that process. Have you seen "thinking" part of current reasoning LLM's? It does indeed look like a process of using logic. After "thinking" part, there is "output" part that makes conclusions form the process of thinking. Recently I asked local version of deepseek about a gas exchange problem and it thought a lot about this, making some small mistakes in logic, correcting them, ultimately returning approximately valid result. It even made some small errors in calculations and corrected itself by multiplying parts of numbers and adding them for correct result. I've put that example online[1] if you'd like to read it, it's pretty interesting. [1] https://pastebin.com/mXyLGCGQ https://pastebin.com/mXyLGCGQ
techjamie 2y ago
They're just outputting tokens that resemble a reasoning process. The underlying tech is still the same LLM it always has been. I can't deny that doing it that way improves results, but any model could do the same thing if you add extra prompts to encourage the reasoning process, then use that as context for the final solution. People discovered that trick before "reasoning" models became the hot thing. It's the "Work it out step by step" trick but in a dedicated fine-tune.
- yetihehe 2y ago
  > They're just outputting tokens that resemble a reasoning process. Looking at one such process of emulating reasoning (got deepseek-70B locally), I'm starting to wonder how does that differ from actual reasoning? We "think" about something, may make errors in that thinking, look for things that don't make sense and correct ourselves. That "think" step is still a blackbox. I asked that llm a typical question of gas exchange between containers, it made some errors and noticed some calculations that didn't make sense: > Moles left A: ~0.0021 mol > Moles entered B: ~0.008 mol > But 0.0021 +0.008=0.0101 mol, which doesn't make sense because that would imply a net increase of moles in the system. Well, that's totally invalid calculation, it should be "-" in there. It also noticed that those quantities should be same in other place. Eventually, after 102 minutes and 10141 tokens, involving checking answers from different angles multiple times, it outputted approximately correct response.