4 ms·
> The first is that reasoning probably requires language use. Even if you don’t think AI models can “really” reason - more on that later - even simulated reason
by LudwigNagasena 1y ago
> The first is that reasoning probably requires language use. Even if you don’t think AI models can “really” reason - more on that later - even simulated reasoning has to be reasoning in human language.
That is an unreasonable assumption. In case of LLMs it seems wasteful to transform a point from latent space into a random token and lose information. In fact, I think in near future it will be the norm for MLLMs to "think" and "reason" without outputting a single "word".
> Whether AI reasoning is “real” reasoning or just a mirage can be an interesting question, but it is primarily a philosophical question. It depends on having a clear definition of what “real” reasoning is, exactly.
It is not a "philosophical" (by which the author probably meant "practically inconsequential") question. If the whole reasoning business is just rationalization of pre-computed answers or simply a means to do some computations because every token provides only a fixed amount of computation to update the model's state, then it doesn't make much sense to focus on improving the quality of chain-of-thought output from human POV.
- kazinator 1y agoNot all reasoning requires language. Symbolic reasoning uses language. Real-time spatial reasoning like driving a car and not hitting things does not seem linguistic. Figuring out how to rotate a cabinet so that it will clear through a stairwell also doesn't seem like it requires language, only to communicate the solution to someone else (where language can turn into a hindrance, compared to a diagram or model).
- vmg12 1y agoSolutions to some of the hardest problems I've had have only come after a night of sleep or when I'm out on a walk and I'm not even thinking about the problem. Maybe what my brain was doing was something different from reasoning?
- andybak 1y agoThis is a very important point and mostly absent from the conversation. We have many words that almost mean the same thing or can mean ment different things - and conversations about intelligence and consciousness are riddled with them.
- tempodox 1y ago> This is a very important point and mostly absent from the conversation. That's because when humans are mentioned at all in the context of coding with “AI”, it's mostly as bad and buggy simulations of those perfect machines.
- jojobas 1y agoAt the very least intermediate points of one's reasoning are grounded in reality.
- safety1st 1y agoI'm pretty much a layperson in this field, but I don't understand why we're trying to teach a stochastic text transformer to reason. Why would anyone expect that approach to work? I would have thought the more obvious approach would be to couple it to some kind of symbolic logic engine. It might transform plain language statements into fragments conforming to a syntax which that engine could then parse deterministically. This is the Platonic ideal of reasoning that the author of the post pooh-poohs, I guess, but it seems to me to be the whole point of reasoning; reasoning is the application of logic in evaluating a proposition. The LLM might be trained to generate elements of the proposition, but it's too random to apply logic.
- shannifin 1y agoProblem is, even with symbolic logic, reasoning is not completely deterministic. Whether one can get to a set of given axioms from a given proposition is sometimes undecidable.
- bubblyworld 1y agoI don't think this is really a problem. The general problem of finding a proof from some axioms to some formula is undecidable (in e.g. first order logic). But that doesn't tell you anything about specific cases, in the same way that we can easily tell whether some specific program halts, like this one: "return 1"
- shannifin 1y agoTrue, I was rather pointing out that being able to parse symbolic language deterministically doesn't imply that we could then "reason" deterministically in general; the reasoning would still need to involve some level of stochasticism. Whether or not that's a problem in practice depends on specifics.
- deleted 1y ago[deleted]
- 1y ago
- emorning4 1y ago>That is an unreasonable assumption.< It definitely is. Neither language nor writing is required to perform logic, see (Existential graph)[https://en.wikipedia.org/wiki/Existential_graph https://en.wikipedia.org/wiki/Existential_graph]. You can perform automated logic by putting/removing beads on a colored tile floor, see (Flower Calculus)[https://arxiv.org/abs/2402.15174 https://arxiv.org/abs/2402.15174].
- limaoscarjuliet 1y ago> In fact, I think in near future it will be the norm for MLLMs to "think" and "reason" without outputting a single "word". It will be outputting something, as this is the only way it can get more compute - output a token, then all context + the next token is fed through the LLM again. It might not be presented to the user, but that's a different story.
- LudwigNagasena 1y agoThat’s the only effective way to get more compute in current production LLMs, but the field is evolving.
- potsandpans 1y ago> It is not a "philosophical" (by which the author probably meant "practically inconsequential") question. I didn't take it that way. I suppose it depends on whether or not you believe philosophy is legitimate
- Terr_ 1y ago> I suppose it depends on whether or not you believe philosophy is legitimate The only way to declare philosophy illegitimate is to be using legitimate philosophy, so... :p
- LudwigNagasena 1y agoThe author called it “the least interesting question possible” and contraposed it with such questions as “how accurately it reflects the actual process going on.” I don’t see any other way to take it.
- pornel 1y agoYou're looking at this from the perspective of what would make sense for the model to produce. Unfortunately, what really dictates the design of the models is what we can train the models with (efficiently, at scale). The output is then roughly just the reverse of the training. We don't even want AI to be an "autocomplete", but we've got tons of text, and a relatively efficient method of training on all prefixes of a sentence at the same time. There have been experiments with preserving embedding vectors of the tokens exactly without loss caused by round-tripping through text, but the results were "meh", presumably because it wasn't the input format the model was trained on. It's conceivable that models trained on some vector "neuralese" that is completely separate from text would work better, but it's a catch 22 for training: the internal representations don't exist in a useful sense until the model is trained, so we don't have anything to feed into the models to make them use them. The internal representations also don't stay stable when the model is trained further.
- LudwigNagasena 1y agoIt’s indeed a very tricky problem with no clear solution yet. But if someone finds a way to bootstrap it, it may be a new qualitative jump that may reverse the current trend of innovating ways to cut inference costs rather than improve models.
- AbrahamParangi 1y agoYou're not allowed to say that it's not reasoning without distinguishing what is reasoning. Absent a strict definition that the models fail and that some other reasoner passes, it is entirely philosophical.
- sdenton4 1y ago"entirely philosophical" I don't think this means what you think it means... Philosophers (at least up to Wittgenstein) love constructing and arguing about definitions.
- LudwigNagasena 1y agoI think it’s perfectly fine to discuss whether it’s reasoning without fully committing to any foundational theory of reasoning. There are practical things we expect from reasoning that we can operationalise. If it’s truly reasoning, then it wouldn’t be able to deceive or to rationalize a leaked answer in a backwards fashion. Asking and answering those questions can help us understand how the research agendas for improving reasoning and improving alignment should be modified.
- esafak 1y agoIt is not obvious that a continuous space is better for thinking than a discrete one.
- kromem 1y agoLatent space reasoners are a thing, and honestly we're probably already seeing emergent latent space reasoners starting to end up embedded into the weights as new models train on extensive reasoning synthetics. If Othello-GPT can build a board in latent space given just the moves, can an exponentially larger transformer build a reasoner in their latent space given a significant number of traces?
- HarHarVeryFunny 1y agoI don't think that the concept of "real reasoning" vs simulated or fake reasoning makes any sense... LLM reasoning can be regarded as a subset of human reasoning, and a more useful comparison would be not real vs fake but rather what is missing from LLM reasoning that would need to be added (likely in a completely new architecture - not an LLM/transformer) to make it more human-like and capable. Human reasoning, and cortical function in general, would also appear to be prediction based, but there are many differences to LLMs, starting with the fact that we learn continuously and incrementally from our own experience and prediction failures and successes. Human reasoning is basically chained what-if prediction, based on predictive outcomes of individual steps that we have learnt, either in terms of general knowledge or domain-specific problem solving steps that we have learnt. Perhaps there is not so much difference between what a human does and an LLM does in, say, tackling a math problem when the RL-trained reasoning-LLM chains together a sequence of reasoning steps that worked before... Where the difference come in, is in how the LLM learned those steps in the first place, and what happens when its reasoning fails. In humans these are essentially the same thing - we learn by predicting and giving it a go, and learn from prediction failure (sensory/etc feedback) to update our context-specific predictions for next time. If we reach a reasoning/predictive impasse - we've tried everything that comes to mind and everything fails, then our innate traits of curiosity and boredom (maybe more?) come to play and we will explore the problem and learn and try again. Curiosity and exploration can of course lead to gain of knowledge from things like imitation and active pursuit (or receipt) of knowledge from sources other then personal experimentation. The LLM of course has no ability to learn (outside of in-context learning - a poor substitute), so is essentially limited in capability to what it has been pre-trained on, and pre-training is never going to be the solution to a world full of infinite ever-changing variety. So, rather than say that an LLM isn't doing "real" reasoning, it seems more productive to acknowledge that prediction is the basis of reasoning, but that the LLM (or rather a future cognitive architecture - not a pass-thru stack of transformer layers!) needs many additional capabilities such as continual/incremental learning, innate traits such as curiosity to expose itself to learning situations, and other necessary cognitive apparatus such as working memory, cognitive iteration/looping (cf thalamo-cortical loop), etc.
- xg15 1y ago> The first is that reasoning probably requires language use. Even if you don’t think AI models can “really” reason - more on that later - even simulated reasoning has to be reasoning in human language. I'd claim that this assumption doesn't even hold true for humans. Reasoning in language is the most "flashy" kind of reasoning and the one that can be most readily shared with other people - because we can articulate it, write it down, publish, etc. But I know for sure that I'm not constantly narrating my life in my head, like the reasoning traces of LLMs. A lot of reasoning happens visually, I.e. by imagining some scene and thinking how it would play out. In other situations, it's spontaneous ideas that "just pop up" - I.e., there are unconscious processes and probably some kind of association involved. None of that uses language.