5 ms·

My mental model for chain-of-thought is not “reasoning”. It’s more of an iterative search through the latent space of the model.

• valine 2y ago
  And human reasoning is somehow more magical? Really struggling to understand the distinction between searching through a latent space and “reasoning”.
  • Exoristos 2y ago
    As you're demonstrating, it depends on the human.
    • error_logic 2y ago
      Indeed, some of us are more willing to believe ourselves somehow special and block out our own irrationalities and missteps. There is a deep need in some people to ignore their own flaws so they can present themselves as more competent. More reasonably, people try to have a thought ready before starting to talk, or will stop and think again if they notice a contradiction in what they're saying. Other people will keep right on going and double down if they've made a mistake, refusing to acknowledge the opportunity for learning and blocking out inconvenient contradiction. Interestingly this is itself quite relevant to the AI alignment problem. Competence requires being able to accurately reflect [on] the patterns, whereas goal-seeking and morality require ignoring some options as costly/dangerous distractions--but not to the point of incompetence. Balancing the two is tricky, and neither people nor governments have solved this universally, let alone an AI.
  • exe34 2y ago
    no, humans do the same thing - even the "intuition" of things to try are probably the results of searches, but we don't have conscious access to them. certainly it's the simplest explanation that fits the observations.
    • sgt101 2y ago
      Given the observations of the workings of brains though (slow, low energy) we're back to some magic to make this happen?
      • exe34 2y ago
        does a machine with ten trillion components need magic to function? maybe. but probably not.
  • sgt101 2y ago
    It would be startling if humans could reason this way with two kilos of meat consuming 40w. Even more surprising given that we get answers in ~1s with a cross brain comms time of about 0.25s To me it's clear that human reasoning is different from a massive search of a latent space. I can say that when I am thinking I maybe try half a dozen ideas or scenarios, but very rarely more. I can't say where those ideas come from or how I make them up though. Maybe we can't frame what it is and how it works with human languages though, which might make it seem magical in some way. Or maybe there's a good framing that I don't know - would love to learn!
    • naasking 2y ago
      > I can say that when I am thinking I maybe try half a dozen ideas or scenarios, but very rarely more You mean you consciously try half a dozen ideas. That's not really reflective of what's happening subconsciously though, which could be a broader kind of search that filters out other possibilities and only bubbles up some options you would consider promising.
    • daveguy 2y ago
      It could be a matter of massively parallel computation (trillions of synapses functioning at once) plus efficiency of chemical-analog vs transistor. But, personally I believe, there are a lot more algorithmic issues between what we are doing now and what the brain is doing. Even if it's just a matter of scale, we are at least a million fold away from the bandwidth and processing power of a single human brain vs exaflop supercomputers, especially in efficiency. Closing that gap, will require 20 years of Moore's law + bandwidth + efficiency + equivalent of 18 years grounded training (probably faster than 18y). We are not close to having our most powerful computers as flexible and general purpose intelligence as a single human. The 20 year limitation is based on processing, efficiency, and bandwidth. Hopefully the algorithms will catch up by then.
      • valine 2y ago
        Even if everything you said is true it doesn’t mean general AI is 20 years away. Computers are many orders of magnitude faster at non-parallel tasks. Who’s to say raw clock speeds won’t easily compensate for any deficiencies vs the brain. You can run a trillion parameter model on a single machine today. We’re maybe 4 or 5 years away from that same 1T model running on an iPhone.
  • numba888 2y ago
    Yes, it has at least two distinct parts: consciousness and sub. First is visible to 'us', it's inner monologue or vision, or other senses. But we don't 'know' what happens in subconsciousness. The answer just pops up from nowhere. 'Reasoning' LLMs for now have the first part. The 'sub' part is questionable, depends how you look at latent space.
• qnleigh 2y ago
  Can you elaborate? I think this is a really interesting question. It comes up over and over again, but it often feels like the two sides of the debate talk past each other. What does the mental model of 'iterative search through latent space' convey that 'reasoning' doesn't? Human reasoning also often searches through a space of potential solution methods and similar problems, and keeps applying them until making progress. I appreciate that there might be danger in using words like 'thinking' and 'reasoning' in that they cause us to anthropomorphize LLMs, but if we are careful not to do so then this is a separate issue.
  • naasking 2y ago
    "Search" a class of fairly well defined algorithms, "reasoning" is vague / ambiguous. If reasoning can be reduced to some kind of search, that makes its meaning more precise and understandable.
    • qnleigh 2y ago
      Sure one word is more precise than the other, but in the context of this question that's beside the point. If one makes the claim 'it's not reasoning, it's just X,' the burden of proof is that X is district from reasoning. That 'reasoning' is an ambiguous term, long makes it harder to argue that X is not compatible with. I was pointing out that, at least superficially, they seem potentially compatible. Again I'm not at all trying to argue that LLMs do reason. I'm trying to understand how one definitively argues that they don't.
      • naasking 2y ago
        I agree I wouldn't a priori exclude reasoning as being a form of search, I was just addressing the question of reduction providing clarity. I do think people are far too dismissive of classifying something as reasoning simply because the latter is vague where the reduction is generally something understood.
• dyauspitr 2y ago
  That sounds like Sheldon from BBT describing human reasoning.