19 ms·

Reasoning in Large Language Models: A Geometric Perspective

214 points by belter 2y ago

dr_dshiv 2y ago
What does reasoning have to do with geometry? Is this like the idea that different concepts have inherent geometrical forms? A Platonic or noetic take on the geometries of reason? (I struggled to understand much of this paper…)
- qorrect 2y ago
  I think they are talking about the word embeddings, where context is embedded into high geometric dimensions (one dimension might capture how 'feminine' a word is, or how 'blue' it is).
  - cubefox 2y ago
    Which word embeddings get their own dimension though, and which don't? ("feminine" and "blue" are words like any other)
    - exe34 2y ago
      maybe it's like a PCA/Huffman deal, where the more regularly useful ones get to be the eigenvectors.
    - willy_k 2y ago
      My extremely naive understanding is that the more useful ones, which also tend to be structures of language like gender or color, get their own dimensions, and other embedding are represented with combinations. A weak illustration of this is this site[1], from an HN post a few months ago[2]. [1] https://neal.fun/infinite-craft/ https://neal.fun/infinite-craft/ [2] https://news.ycombinator.com/item?id=39205020 https://news.ycombinator.com/item?id=39205020
  - exe34 2y ago
    ooooh what if qualia is just embedding? some philosophers would get their toga in a twist!
    - darby_nine 2y ago
      Not Wittgenstein :cool:
- cornholio 2y ago
  I think the connection is that the authors could convincingly write a paper on this connection, thus inflating the AI publication bubble, furthering their academic acumen and improving their chances of getting research grants or selective jobs in the field. Some other interests of the authors seem to be detecting exoplanets using AI and detecting birds through audio analysis. Since nobody can really say what a good AI department does, companies seem to be driven by credentiallism, load up on machine learning PhDs and masters so they can show their board and investors that they are ready for the AI revolution. This creates economic pressure to write such papers, the vast majority of which will amount to nothing.
  - techbro92 2y ago
    I think a lot of the time you would be correct. But this is published to arxiv so it’s not peer reviewed and doesn’t boost the authors credentials. It could be designed to attract attention to the company they work at. Or it could just be a cool idea the author wanted to share.
- deleted 2y ago
  [deleted]
- magicalhippo 2y ago
  Modern neural networks make heavy use of linear algebra, in particular the transformer[1] architecture that powers modern LLMs. Since linear algebra is closely related to geometry[2], it seems quite reasonable that there are some geometric aspects that define their capabilities and performance. Specifically, in this paper they're considering the intrinsic dimension[3] of the attention layers, and seeing how it correlates with the performance of LLMs. [1]: https://en.wikipedia.org/wiki/Transformer_(deep_learning_architecture)#Scaled_dot-product_attention https://en.wikipedia.org/wiki/Transformer_(deep_learning_arc... [2]: https://en.wikipedia.org/wiki/Linear_algebra#Relationship_with_geometry https://en.wikipedia.org/wiki/Linear_algebra#Relationship_wi... [3]: https://en.wikipedia.org/wiki/Intrinsic_dimension https://en.wikipedia.org/wiki/Intrinsic_dimension
  - darby_nine 2y ago
    > it seems quite reasonable that there are some geometric aspects that define their capabilities and performance. Sure but this doesn't mean terribly much when you can relate either concept to virtually any other concept. "Reasonable" would imply one specific term implies another specific term and you haven't filled in those blanks yet.
    - deleted 2y ago
      [deleted]
  - dr_dshiv 2y ago
    Btw, thanks so much for your thoughtful and insightful replies on this.
- sbierwagen 2y ago
  The paper doesn't make this point at all, but one thing you could do here is an AlphaGeometry-style[1] synthetic benchmark, where you have a geometry engine crank out a hundred million word problems, and have an LLM try to solve them. Geometry problems have the nice property that they're easy to generate and solve mechanically, but there's no particular reason why a vanilla Transformer LLM would be any good at them, and you can have absolutely huge scale. (Unlike, say, the HumanEval benchmark, which only has 164 problems, which resulted in lots of accusations that LLMs can simply memorize the answers) 1: https://deepmind.google/discover/blog/alphageometry-an-olympiad-level-ai-system-for-geometry/ https://deepmind.google/discover/blog/alphageometry-an-olymp...
  - darby_nine 2y ago
    You'd have the second problem of trying to figure out how to relay geometry as a sequence of tokens when surely how you would encode this would affect what things you might reasonably expect an LLM to draw from it.
    - jiggawatts 2y ago
      Image input.
    - energy123 2y ago
      Only if your purpose is to create the best geometry solver. If you're trying to improve the general intelligence of a frontier LLM, you're probably better off feeding in the synthetic data as some combination of raw images and text (as part of its existing tokenisation).
- benreesman 2y ago
  If the curvature metric wasn’t steep to begin with AdamW wouldn’t work. If the regions of interest weren’t roughly Euclidean control vectors wouldn’t work.
- magicalhippo 2y ago
  A follow-up comment after having studied the paper a bit more, since you asked about where the geometry comes into play. One of the references the paper provide is to this[1] paper, which shows how the non-linear layers in modern deep neural networks partitions the input into regions and applies region-dependent affine mappings[2] to generate the output. It also mentions how that connects to vector quantization and k-means clustering. So, the geometric perspective isn't referring to your typical high-school geometry, but more abstract concepts like vector spaces[3] and combinatiorial computational geometry[4]. The submitted paper shows that this partitioning is directly linked to the approximation power of the neural network. They then show how increasing the approximation power results in better answers to math word problems, and hence that the approximation power correlated to the reasoning ability of LLMs. [1]: https://arxiv.org/abs/1805.06576v2 https://arxiv.org/abs/1805.06576v2 [2]: https://en.wikipedia.org/wiki/Affine_transformation https://en.wikipedia.org/wiki/Affine_transformation [3]: https://en.wikipedia.org/wiki/Vector_space https://en.wikipedia.org/wiki/Vector_space [4]: https://en.wikipedia.org/wiki/Computational_geometry#Combinatorial_computational_geometry https://en.wikipedia.org/wiki/Computational_geometry#Combina...
- rlupi 2y ago
  "different concepts have inherent geometrical forms" Absolutely, in fact you can build the foundation of mathematics on this concept. You can build proofs and reasoning (for some value of "reasoning"). That's how dependent type systems work, search for HoTT and modal homotophy theory. That's how lean4, coq, and theorem proofs work. If you remember at the foundation of lambda calculus or boolean algebra, they proceed through a series of transformation of mathematical objects that are organized lattices or semi-lattices, partially ordered sets (e.g. in boolean algebra, where the partial order is provided by the implication). It would be interesting to understand if the density of attention mechanisms follow a similar progression as dependent type systems, and we can find a link between the dependent types involved in a proof and the corresponding spaces in a LLM via some continuous relaxation analogous to a proximal operator + some transformation (from high-level concepts into output tokens). We have found in embeddings that geometry has a meaning. Specific simple concepts correspond to vector directions. I wouldn't be surprised at all that we find that reasoning on dependent concepts correspond to complex subspaces in the paths that a LLM takes, and that with enough training this connections becomes closer and closer to the logical structure of corresponding proofs (for self-consistent corpus of input and, like math proofs, and given enough training data).
lifeisstillgood 2y ago
But I understand there are two sides to the discussion - that by ingesting huge amounts of text these models have somehow built reasoning capabilities (language then reasoning) or that the reasoning was done by humans and then written down so as long as you ask something like “should romeo find another love after Juliet” there is a set of reasoning reflected in a billion English literature essays and the model just reflects those answers Am I missing something?
- magicalhippo 2y ago
  Glossing through the paper, it seems they're noting this issue but kinda skipping over it: In fact, it is clear that approximation capabilities and generalization are not equivalent notions. However, it is not yet determined that the reasoning capabilities of LLMs are tied to their generalization. While these notions are still hard to pinpoint, we will focus in this experimental section on the relationship between intrinsic dimension, thus expressive power, and reasoning capabilities.
  - cowsaymoo 2y ago
    Right, they never claimed to have found a roadmap to AGI, they just found a cool geometric tool to describe how LLMs reason through approximation. Sounds like a handy tool if you want to discover things about approximation or generalization.
- Fripplebubby 2y ago
  > the model just reflects those answers I think there is a lot happening in the word "reflects"! Is it so simple? Does this mean that the model takes on the opinion of a specific lit crit essay it has "read"? Does that mean it takes on some kind of "average" opinion from everything? How would you define the "average" opinion on a topic, anyway? Anyway, although I think this is really interesting stuff and cuts to the core of what an LLM is, this paper isn't where you're going to get the answer to that, because it is much more focused and narrow.
- nshm 2y ago
  It is actually pretty straightforward why those model "reason" or, to be more exact, can operate on a complex concepts. By processing huge amount of texts they build an internal representation where those concepts are represented as a simple nodes (neurons or groups). So they really distill knowledge. Alternatively you can think about it as a very good principal component analysis that can extract many important aspects. Or like a semantic graph built automatically. Once knowledge is distilled you can build on top of it easily by merging concepts for example. So no secret here.
ChicagoDave 2y ago
LLMs do not have the technology to iteratively solve a complex problem. This is a fact. No graph will change this. You want “reasoning,” then you need to invent a new technology to iterate, validate, experiment, validate, query external expertise, and validate again. When we get that technology, then AI will become resilient in solving complex problems.
- sigmoid10 2y ago
  That's false. It has been shown that LLMs can perform e.g. gradient descent internally [1], which can explain why they are so good at few shot prompting. The universal approximation theorem already tells us that a single layer is sufficient to approximate any function, so it should come as no surprise that modern deep networks with many layers should be able to perform iterative optimisations. [1] https://arxiv.org/abs/2212.10559 https://arxiv.org/abs/2212.10559
  - ChicagoDave 2y ago
    What’s the success rate once you hit 20 prompts? What’s the success rate if you hit 30 prompts 40 prompts? I’m pretty sure that as you increase the complexity of your questioning, the LLM is just gonna flat out fail and no change to the vector database is going to improve that.
    - sigmoid10 2y ago
      What are you talking about? This has nothing to do with vector databases. This is about the basic capabilities of attention networks and dense layers.
  - slashdave 2y ago
    So you want to encapsulate reason into a simple function?
    - sigmoid10 2y ago
      Why not? At the most simple level, the human brain also just takes a set of inputs and produces a set of outputs. There is a huge complicated function behind that, but complexity is no longer an issue thanks to modern compute capabilities.
jens-c 2y ago
[flagged]
- belter 2y ago
  Comes from this company: https://www.tenyx.com/about-us https://www.tenyx.com/about-us
- slashdave 2y ago
  Not exactly gobbledygook, but math, figures, and terminology designed to provide the appearance of some deep, conceptual idea. The illusion doesn't stand up to closer inspection, though.
- deleted 2y ago
  [deleted]
magicalhippo 2y ago
I'm not into AI, but I like to watch from the sidelines. Here's my non-AI summary of the paper after glossing through (corrections appreciated): The multilayered perceptron[1] layers used in modern neural networks, like LLMs, essentially partitions the input into multiple regions. They show that the number of regions a single MLP layer can partition into depends exponentially on the intrinsic dimension[2] of the input. The number of regions/partitions increases the approximation power of the MLP layer. Thus you can significantly increase the approximation power of a MLP layer without increasing the number of neurons, by essentially "distilling" the input to it. In the transformer architecture, the inputs to the MLP layers are the self-attention layers[3]. The authors then show that the graph density of the self-attention layers[3] correlates strongly with the intrinsic dimension of the self-attention layer. Thus a more dense self-attention layer means the MLP can do a better job. One way of increasing the density of the attention layers is to add more context. (edited, see comment) They show that prepending any token as context to a question which increases the intrinsic dimension of the final layer makes the LLM perform better. They also note that the transformer architecture is susceptible to compounding approximation errors, and that the much more precise partitioning provided by the MLP layers when fed with high intrinsic-dimensional input can help with this. However the impact of this on generalization remains to be explored further. If the results hold up it does seem like this paper provides nice insight into how to better optimize LLMs and similar neural networks. [1]: https://en.wikipedia.org/wiki/Multilayer_perceptron https://en.wikipedia.org/wiki/Multilayer_perceptron [2]: https://en.wikipedia.org/wiki/Intrinsic_dimension https://en.wikipedia.org/wiki/Intrinsic_dimension [3]: https://en.wikipedia.org/wiki/Transformer_(deep_learning_architecture)#Multi-head_attention https://en.wikipedia.org/wiki/Transformer_(deep_learning_arc...
- Fripplebubby 2y ago
  Awesome summarization by someone who read and actually understood the paper. > One way of increasing the density of the attention layers is to add more context. They show that simply prepending any token as context to a question makes the LLM perform better. Adding relevant context makes it even better. Right, I think a more intuitive way to think about this is to define density: the number of _edges_ in the self-attention graph connecting tokens. Maybe a simpler explanation: the number of times a token had some connection to another token divided by the number of tokens. So, tokens which actually relate to one another and provide information are good, non sequitur tokens don't help except that you say > They show that simply prepending any token as context to a question makes the LLM perform better. I think this is not quite right. What they found was: > pre-pending the question at hand with any type of token does increase the intrinsic dimension at the first layer > however, this increase is not necessarily correlated with the reasoning capability of the model but it is only > when the pre-pended tokens lead to an increase in the intrinsic dimension at the *final layer* of the model, the reasoning capabilities of the LLM improve significantly. (emphasis mine)
bastien2 2y ago
You can't "enhance" from zero. LLMs by design are not capable of reason. We can observe LLM-like behaviour in humans: all those reactionaries who just parrot whatever catchphrases mass media programmed into them. LLMs are just the computer version of that uncle who thinks Fox News is true and is the reason your nieces have to wear long pants at family gatherings. He doesn't understand the catchphrases he parrots any more than the chatbots do. Actual AI will require a kind of modelling that as yet does not exist.
- belter 2y ago
  > LLMs by design are not capable of reason. It is not as clear cut. The argument being, that the patterns they learn in text encodes several layers of abstraction, one of them being some reasoning, as it is encoded in the discourse.
  - wizzwizz4 2y ago
    They are capable of picking up incredibly crude, noisy versions of first-order symbolic reasoning, and specific, commonly-used arguments, and the context for when those might be applied. Taken together and iterated, you get something vaguely resembling a reasoning algorithm, but your average schoolchild with an NLP library and regular expressions could make a better reasoning algorithm. (While I've been calling these "reasoning algorithms" for analogy's sake, they don't actually behave how we expect reasoning to behave.) The language model predicts what reasoning might look like. But it doesn't actually do the reasoning, so (unless it has something capable of reasoning to guide it), it's not going to correctly derive conclusions from premises.
    - belter 2y ago
      Yes and No. I don't entirely disagree with you, but think about when you ask a model to explain step by step a conclusion. It is not doing the reasoning, but in a way abstracted and learned the pattern of doing the reasoning....So it is doing some type of reasoning....and sometimes producing the outcomes that are derived from actual reasoning...Even if defining "actual reasoning" is a whole new challenge.
benreesman 2y ago
I read too much of this stuff to dive deep unless someone on HN resoundingly endorses it. From a cursory glance it looks like we’re once again on the verge of realizing that we’re dealing with complex valued weights. Even Anthropic will be publishing that before the year is out.
- deleted 2y ago
  [deleted]
slashdave 2y ago
Okay. So more weights = more parameter space for expression. And?
- deleted 2y ago
  [deleted]
- magicalhippo 2y ago
  The way I understood it, it's more like the opposite. That is if you feed the non-linear layers "dense" data, ie with higher intrinsic dimension, they perform better. Thus, you could potentially get by using a smaller non-linear layers by "condensing" the input before passing it through the non-linear layers.
  - slashdave 2y ago
    This doesn't make any sense. Higher dimension means less dense. Far less dense, actually.
    - deleted 2y ago
      [deleted]
    - magicalhippo 2y ago
      But the point is to focus on the intrinsic dimension[1], not dimensions of the vector itself. I meant dense in the sense that the two are close, relative to another vector where they are not so close. Perhaps a poor choice of words on my part. [1]: https://en.wikipedia.org/wiki/Intrinsic_dimension https://en.wikipedia.org/wiki/Intrinsic_dimension
      - slashdave 2y ago
        Ah, I see. Well, the data has an intrinsic dimension of a specific size. You don't get to choose that. And, in any case, you want something quite a bit larger than the intrinsic dimension, because deep-learning needs redundancy in its weights in order to train correctly.
TibbityFlanders 2y ago
[dead]
john-tells-all 2y ago
AI has a "bathtub curve" of value. At the low level, it's a super-autocomplete, able to write 1-3 lines of code that works good enough. At the high level, it's great for explaining high-level concepts that are relevant to a task at hand. In the middle... AI doesn't work very well. If an AI writes a multi-step plan, where the pieces have to fit together, I've found it goes off the rails. Parts 1 and 3 of a 4-part plan are fine. So is part 2. However they don't fit together! AI has no concept of "these four parts have to be closely connected, building a whole". It just builds from A to B in four steps... but taking two different paths and stitching the pieces together poorly.
- kelseyfrog 2y ago
  The low-level high-level spectrum might not be the best scale to gauge AI by. We should kernel-trick our scale so that low and high level is seperable from multi-step planning problems. Or in other words, use a different dimension to separate these three problems.
- ADeerAppeared 2y ago
  It's not a bathtub curve. Your low-level and "high"-level tasks are the same thing: Probabilistic text generation. It's not reasoning about your code, nor about the explanation it gives you. > AI has no concept of "these four parts have to be closely connected, building a whole". AI can't think. It doesn't create an internal model of the problem given, it just guesses. It fails at all these "middle" tasks because they require abstract reasoning to be correct.
  - bubblyworld 2y ago
    Not going to comment on the thinking part, because who knows what that means, but there's evidence that transformers do in fact learn predictive models of their input space. There's a cool blog post on this here: https://www.neelnanda.io/mechanistic-interpretability/othello https://www.neelnanda.io/mechanistic-interpretability/othell...
    - ADeerAppeared 2y ago
DrMiaow 2y ago
"just add more dimensions, bro!"
omerhac 2y ago
Each time research about LLM and reasoning comes out Yan LeCun gets an itch
justkk 2y ago
What are regions in this context?, are more regions better, how one delimiter the regions?, can one region be the same concept as several related regions?
- magicalhippo 2y ago
  As I understand it, the regions are simply the pieces that constitute the partitioning of the input domain, ie vector space formed by the the weights. There's some more details in one of the referenced papers[1], section 3.1 and onward. The argument in that paper is that the layers in a typical deep neural network partitions the input domain into regions, where each region has its own affine mapping of the input. For any arbitrary activation function, one would have to find the partitioning as well as the per-region parameters of the affine mappings. However since all the common activation functions are globally convex, they show that one can use this in a way where the partitioning is entirely determined by the per-region affine mapping parameters. Thus the output of the layer for a given input x is a "partition-region-dependent, piecewise affine transformation of x". The affine mapping parameters is effectively what you end up changing during training, and so the number and shape of the regions change during training as well. The submitted paper shows that more regions increase the approximation power of the neural net layer. This in itself doesn't seem that surprising given the above, but they use it as an important stepping stone. [1]: https://arxiv.org/abs/1805.06576v2 https://arxiv.org/abs/1805.06576v2
lngnmn2 2y ago
[dead]
EncomLab 2y ago
Does anyone remember the "Mad Libs" games - you fill out a form with blanks for "verb", "noun", "adjective", etc - then on the next page you fill in the words from the form to create a silly story. The results are funny because the words you provided initially were without context - they were syntactically correct, but were nonsense in context. LLM's are like Mad Libs with a "contextual predictor" - they produce syntactically correct output, and the "contextual predictor" limits the amount of nonsense because statistical correlations can generate meaningful output most of the time. But there is no "reasoning" occurring here - just syntactic templating and statistical auto-complete.
- FeepingCreature 2y ago
  I believe reasoning is (sufficiently advanced) syntactic templating and statistical autocomplete. Reminder that syntactic transformations are Turing complete: https://wiki.c2.com/?RewriteRules https://wiki.c2.com/?RewriteRules
  - enord 2y ago
    That’s neither here nor there. Everything can be statistically modelled but very few things are reasoning. Same with turing machines.
    - mistermann 2y ago
      Speaking of reasoning, how do you know what is happening inside the mind when it "reasons"? And while you're at it: what is happening inside the mind when it reasons?
      - enord 2y ago
        These are very good questions that deserve unequivocal answers. Alas…
        mistermann 2y ago
voidhorse 2y ago
As with many philosophical discussions, there is no point in claiming LLMs can "reason" because "reason" is not a well-defined term and you will not get everyone to agree on a singular definition. Ask a computer scientist, continental philosopher, and anthropologist what "reason" is and they will give you extremely different answers. If by reason we mean deductive reasoning as practiced in mathematics and inductive reasoning as practiced in the sciences, there is no evidence that LLMs do anything of the sort. There is no reason (ha) to believe that linguistic pattern matching is enough to emulate all that we call thinking in man. To claim so is to adopt an drastically narrow definition of "thinking" and to ignore the fact that we are embodied intellects, capable of knowing ourselves in a transparent possibly prelinguistic way. Unless an AI becomes embodied and can do the same, I have no faith that it will ever "think" or "reason" as humans do. It remains a really good statistical parlor trick.
- bkitano19 2y ago
  https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html https://transformer-circuits.pub/2022/in-context-learning-an... there is a lot of evidence to suggest that they are performing induction
- PhunkyPhil 2y ago
  > Unless an AI becomes embodied and can do the same, I have no faith that it will ever "think" or "reason" as humans do. It remains a really good statistical parlor trick. This may be true, but if it's "good enough" then why does that matter? If I can't determine if a user on Slack/Teams is an LLM that covers their tickets on time with decent code quality, then I really don't care if they know themselves in a transparent, prelinguistic fashion.

214 points by belter 2y ago

What does reasoning have to do with geometry? Is this like the idea that different concepts have inherent geometrical forms? A Platonic or noetic take on the geometries of reason? (I struggled to understand much of this paper…)

qorrect 2y ago
I think they are talking about the word embeddings, where context is embedded into high geometric dimensions (one dimension might capture how 'feminine' a word is, or how 'blue' it is).
- cubefox 2y ago
  Which word embeddings get their own dimension though, and which don't? ("feminine" and "blue" are words like any other)
  - exe34 2y ago
    maybe it's like a PCA/Huffman deal, where the more regularly useful ones get to be the eigenvectors.
  - willy_k 2y ago
    My extremely naive understanding is that the more useful ones, which also tend to be structures of language like gender or color, get their own dimensions, and other embedding are represented with combinations. A weak illustration of this is this site[1], from an HN post a few months ago[2]. [1] https://neal.fun/infinite-craft/ https://neal.fun/infinite-craft/ [2] https://news.ycombinator.com/item?id=39205020 https://news.ycombinator.com/item?id=39205020
- exe34 2y ago
  ooooh what if qualia is just embedding? some philosophers would get their toga in a twist!
  - darby_nine 2y ago
    Not Wittgenstein :cool:
cornholio 2y ago
I think the connection is that the authors could convincingly write a paper on this connection, thus inflating the AI publication bubble, furthering their academic acumen and improving their chances of getting research grants or selective jobs in the field. Some other interests of the authors seem to be detecting exoplanets using AI and detecting birds through audio analysis. Since nobody can really say what a good AI department does, companies seem to be driven by credentiallism, load up on machine learning PhDs and masters so they can show their board and investors that they are ready for the AI revolution. This creates economic pressure to write such papers, the vast majority of which will amount to nothing.
- techbro92 2y ago
  I think a lot of the time you would be correct. But this is published to arxiv so it’s not peer reviewed and doesn’t boost the authors credentials. It could be designed to attract attention to the company they work at. Or it could just be a cool idea the author wanted to share.
deleted 2y ago
[deleted]
magicalhippo 2y ago
Modern neural networks make heavy use of linear algebra, in particular the transformer[1] architecture that powers modern LLMs. Since linear algebra is closely related to geometry[2], it seems quite reasonable that there are some geometric aspects that define their capabilities and performance. Specifically, in this paper they're considering the intrinsic dimension[3] of the attention layers, and seeing how it correlates with the performance of LLMs. [1]: https://en.wikipedia.org/wiki/Transformer_(deep_learning_architecture)#Scaled_dot-product_attention https://en.wikipedia.org/wiki/Transformer_(deep_learning_arc... [2]: https://en.wikipedia.org/wiki/Linear_algebra#Relationship_with_geometry https://en.wikipedia.org/wiki/Linear_algebra#Relationship_wi... [3]: https://en.wikipedia.org/wiki/Intrinsic_dimension https://en.wikipedia.org/wiki/Intrinsic_dimension
- darby_nine 2y ago
  > it seems quite reasonable that there are some geometric aspects that define their capabilities and performance. Sure but this doesn't mean terribly much when you can relate either concept to virtually any other concept. "Reasonable" would imply one specific term implies another specific term and you haven't filled in those blanks yet.
  - deleted 2y ago
    [deleted]
- dr_dshiv 2y ago
  Btw, thanks so much for your thoughtful and insightful replies on this.
sbierwagen 2y ago
The paper doesn't make this point at all, but one thing you could do here is an AlphaGeometry-style[1] synthetic benchmark, where you have a geometry engine crank out a hundred million word problems, and have an LLM try to solve them. Geometry problems have the nice property that they're easy to generate and solve mechanically, but there's no particular reason why a vanilla Transformer LLM would be any good at them, and you can have absolutely huge scale. (Unlike, say, the HumanEval benchmark, which only has 164 problems, which resulted in lots of accusations that LLMs can simply memorize the answers) 1: https://deepmind.google/discover/blog/alphageometry-an-olympiad-level-ai-system-for-geometry/ https://deepmind.google/discover/blog/alphageometry-an-olymp...
- darby_nine 2y ago
  You'd have the second problem of trying to figure out how to relay geometry as a sequence of tokens when surely how you would encode this would affect what things you might reasonably expect an LLM to draw from it.
  - jiggawatts 2y ago
    Image input.
  - energy123 2y ago
    Only if your purpose is to create the best geometry solver. If you're trying to improve the general intelligence of a frontier LLM, you're probably better off feeding in the synthetic data as some combination of raw images and text (as part of its existing tokenisation).
benreesman 2y ago
If the curvature metric wasn’t steep to begin with AdamW wouldn’t work. If the regions of interest weren’t roughly Euclidean control vectors wouldn’t work.
magicalhippo 2y ago
A follow-up comment after having studied the paper a bit more, since you asked about where the geometry comes into play. One of the references the paper provide is to this[1] paper, which shows how the non-linear layers in modern deep neural networks partitions the input into regions and applies region-dependent affine mappings[2] to generate the output. It also mentions how that connects to vector quantization and k-means clustering. So, the geometric perspective isn't referring to your typical high-school geometry, but more abstract concepts like vector spaces[3] and combinatiorial computational geometry[4]. The submitted paper shows that this partitioning is directly linked to the approximation power of the neural network. They then show how increasing the approximation power results in better answers to math word problems, and hence that the approximation power correlated to the reasoning ability of LLMs. [1]: https://arxiv.org/abs/1805.06576v2 https://arxiv.org/abs/1805.06576v2 [2]: https://en.wikipedia.org/wiki/Affine_transformation https://en.wikipedia.org/wiki/Affine_transformation [3]: https://en.wikipedia.org/wiki/Vector_space https://en.wikipedia.org/wiki/Vector_space [4]: https://en.wikipedia.org/wiki/Computational_geometry#Combinatorial_computational_geometry https://en.wikipedia.org/wiki/Computational_geometry#Combina...
rlupi 2y ago
"different concepts have inherent geometrical forms" Absolutely, in fact you can build the foundation of mathematics on this concept. You can build proofs and reasoning (for some value of "reasoning"). That's how dependent type systems work, search for HoTT and modal homotophy theory. That's how lean4, coq, and theorem proofs work. If you remember at the foundation of lambda calculus or boolean algebra, they proceed through a series of transformation of mathematical objects that are organized lattices or semi-lattices, partially ordered sets (e.g. in boolean algebra, where the partial order is provided by the implication). It would be interesting to understand if the density of attention mechanisms follow a similar progression as dependent type systems, and we can find a link between the dependent types involved in a proof and the corresponding spaces in a LLM via some continuous relaxation analogous to a proximal operator + some transformation (from high-level concepts into output tokens). We have found in embeddings that geometry has a meaning. Specific simple concepts correspond to vector directions. I wouldn't be surprised at all that we find that reasoning on dependent concepts correspond to complex subspaces in the paths that a LLM takes, and that with enough training this connections becomes closer and closer to the logical structure of corresponding proofs (for self-consistent corpus of input and, like math proofs, and given enough training data).

lifeisstillgood 2y ago

But I understand there are two sides to the discussion - that by ingesting huge amounts of text these models have somehow built reasoning capabilities (language then reasoning) or that the reasoning was done by humans and then written down so as long as you ask something like “should romeo find another love after Juliet” there is a set of reasoning reflected in a billion English literature essays and the model just reflects those answers Am I missing something?

magicalhippo 2y ago
Glossing through the paper, it seems they're noting this issue but kinda skipping over it: In fact, it is clear that approximation capabilities and generalization are not equivalent notions. However, it is not yet determined that the reasoning capabilities of LLMs are tied to their generalization. While these notions are still hard to pinpoint, we will focus in this experimental section on the relationship between intrinsic dimension, thus expressive power, and reasoning capabilities.
- cowsaymoo 2y ago
  Right, they never claimed to have found a roadmap to AGI, they just found a cool geometric tool to describe how LLMs reason through approximation. Sounds like a handy tool if you want to discover things about approximation or generalization.
Fripplebubby 2y ago
> the model just reflects those answers I think there is a lot happening in the word "reflects"! Is it so simple? Does this mean that the model takes on the opinion of a specific lit crit essay it has "read"? Does that mean it takes on some kind of "average" opinion from everything? How would you define the "average" opinion on a topic, anyway? Anyway, although I think this is really interesting stuff and cuts to the core of what an LLM is, this paper isn't where you're going to get the answer to that, because it is much more focused and narrow.
nshm 2y ago
It is actually pretty straightforward why those model "reason" or, to be more exact, can operate on a complex concepts. By processing huge amount of texts they build an internal representation where those concepts are represented as a simple nodes (neurons or groups). So they really distill knowledge. Alternatively you can think about it as a very good principal component analysis that can extract many important aspects. Or like a semantic graph built automatically. Once knowledge is distilled you can build on top of it easily by merging concepts for example. So no secret here.

ChicagoDave 2y ago

LLMs do not have the technology to iteratively solve a complex problem. This is a fact. No graph will change this. You want “reasoning,” then you need to invent a new technology to iterate, validate, experiment, validate, query external expertise, and validate again. When we get that technology, then AI will become resilient in solving complex problems.

sigmoid10 2y ago
That's false. It has been shown that LLMs can perform e.g. gradient descent internally [1], which can explain why they are so good at few shot prompting. The universal approximation theorem already tells us that a single layer is sufficient to approximate any function, so it should come as no surprise that modern deep networks with many layers should be able to perform iterative optimisations. [1] https://arxiv.org/abs/2212.10559 https://arxiv.org/abs/2212.10559
- ChicagoDave 2y ago
  What’s the success rate once you hit 20 prompts? What’s the success rate if you hit 30 prompts 40 prompts? I’m pretty sure that as you increase the complexity of your questioning, the LLM is just gonna flat out fail and no change to the vector database is going to improve that.
  - sigmoid10 2y ago
    What are you talking about? This has nothing to do with vector databases. This is about the basic capabilities of attention networks and dense layers.
- slashdave 2y ago
  So you want to encapsulate reason into a simple function?
  - sigmoid10 2y ago
    Why not? At the most simple level, the human brain also just takes a set of inputs and produces a set of outputs. There is a huge complicated function behind that, but complexity is no longer an issue thanks to modern compute capabilities.

jens-c 2y ago

[flagged]

belter 2y ago
Comes from this company: https://www.tenyx.com/about-us https://www.tenyx.com/about-us
slashdave 2y ago
Not exactly gobbledygook, but math, figures, and terminology designed to provide the appearance of some deep, conceptual idea. The illusion doesn't stand up to closer inspection, though.
deleted 2y ago
[deleted]

magicalhippo 2y ago

I'm not into AI, but I like to watch from the sidelines. Here's my non-AI summary of the paper after glossing through (corrections appreciated): The multilayered perceptron[1] layers used in modern neural networks, like LLMs, essentially partitions the input into multiple regions. They show that the number of regions a single MLP layer can partition into depends exponentially on the intrinsic dimension[2] of the input. The number of regions/partitions increases the approximation power of the MLP layer. Thus you can significantly increase the approximation power of a MLP layer without increasing the number of neurons, by essentially "distilling" the input to it. In the transformer architecture, the inputs to the MLP layers are the self-attention layers[3]. The authors then show that the graph density of the self-attention layers[3] correlates strongly with the intrinsic dimension of the self-attention layer. Thus a more dense self-attention layer means the MLP can do a better job. One way of increasing the density of the attention layers is to add more context. (edited, see comment) They show that prepending any token as context to a question which increases the intrinsic dimension of the final layer makes the LLM perform better. They also note that the transformer architecture is susceptible to compounding approximation errors, and that the much more precise partitioning provided by the MLP layers when fed with high intrinsic-dimensional input can help with this. However the impact of this on generalization remains to be explored further. If the results hold up it does seem like this paper provides nice insight into how to better optimize LLMs and similar neural networks. [1]: https://en.wikipedia.org/wiki/Multilayer_perceptron https://en.wikipedia.org/wiki/Multilayer_perceptron [2]: https://en.wikipedia.org/wiki/Intrinsic_dimension https://en.wikipedia.org/wiki/Intrinsic_dimension [3]: https://en.wikipedia.org/wiki/Transformer_(deep_learning_architecture)#Multi-head_attention https://en.wikipedia.org/wiki/Transformer_(deep_learning_arc...

Fripplebubby 2y ago
Awesome summarization by someone who read and actually understood the paper. > One way of increasing the density of the attention layers is to add more context. They show that simply prepending any token as context to a question makes the LLM perform better. Adding relevant context makes it even better. Right, I think a more intuitive way to think about this is to define density: the number of _edges_ in the self-attention graph connecting tokens. Maybe a simpler explanation: the number of times a token had some connection to another token divided by the number of tokens. So, tokens which actually relate to one another and provide information are good, non sequitur tokens don't help except that you say > They show that simply prepending any token as context to a question makes the LLM perform better. I think this is not quite right. What they found was: > pre-pending the question at hand with any type of token does increase the intrinsic dimension at the first layer > however, this increase is not necessarily correlated with the reasoning capability of the model but it is only > when the pre-pended tokens lead to an increase in the intrinsic dimension at the *final layer* of the model, the reasoning capabilities of the LLM improve significantly. (emphasis mine)

bastien2 2y ago

You can't "enhance" from zero. LLMs by design are not capable of reason. We can observe LLM-like behaviour in humans: all those reactionaries who just parrot whatever catchphrases mass media programmed into them. LLMs are just the computer version of that uncle who thinks Fox News is true and is the reason your nieces have to wear long pants at family gatherings. He doesn't understand the catchphrases he parrots any more than the chatbots do. Actual AI will require a kind of modelling that as yet does not exist.

belter 2y ago
> LLMs by design are not capable of reason. It is not as clear cut. The argument being, that the patterns they learn in text encodes several layers of abstraction, one of them being some reasoning, as it is encoded in the discourse.
- wizzwizz4 2y ago
  They are capable of picking up incredibly crude, noisy versions of first-order symbolic reasoning, and specific, commonly-used arguments, and the context for when those might be applied. Taken together and iterated, you get something vaguely resembling a reasoning algorithm, but your average schoolchild with an NLP library and regular expressions could make a better reasoning algorithm. (While I've been calling these "reasoning algorithms" for analogy's sake, they don't actually behave how we expect reasoning to behave.) The language model predicts what reasoning might look like. But it doesn't actually do the reasoning, so (unless it has something capable of reasoning to guide it), it's not going to correctly derive conclusions from premises.
  - belter 2y ago
    Yes and No. I don't entirely disagree with you, but think about when you ask a model to explain step by step a conclusion. It is not doing the reasoning, but in a way abstracted and learned the pattern of doing the reasoning....So it is doing some type of reasoning....and sometimes producing the outcomes that are derived from actual reasoning...Even if defining "actual reasoning" is a whole new challenge.

benreesman 2y ago

I read too much of this stuff to dive deep unless someone on HN resoundingly endorses it. From a cursory glance it looks like we’re once again on the verge of realizing that we’re dealing with complex valued weights. Even Anthropic will be publishing that before the year is out.

deleted 2y ago
[deleted]

slashdave 2y ago

Okay. So more weights = more parameter space for expression. And?

deleted 2y ago
[deleted]
magicalhippo 2y ago
The way I understood it, it's more like the opposite. That is if you feed the non-linear layers "dense" data, ie with higher intrinsic dimension, they perform better. Thus, you could potentially get by using a smaller non-linear layers by "condensing" the input before passing it through the non-linear layers.
- slashdave 2y ago
  This doesn't make any sense. Higher dimension means less dense. Far less dense, actually.
  - deleted 2y ago
    [deleted]
  - magicalhippo 2y ago
    But the point is to focus on the intrinsic dimension[1], not dimensions of the vector itself. I meant dense in the sense that the two are close, relative to another vector where they are not so close. Perhaps a poor choice of words on my part. [1]: https://en.wikipedia.org/wiki/Intrinsic_dimension https://en.wikipedia.org/wiki/Intrinsic_dimension
    - slashdave 2y ago
      Ah, I see. Well, the data has an intrinsic dimension of a specific size. You don't get to choose that. And, in any case, you want something quite a bit larger than the intrinsic dimension, because deep-learning needs redundancy in its weights in order to train correctly.

TibbityFlanders 2y ago

[dead]

john-tells-all 2y ago

AI has a "bathtub curve" of value. At the low level, it's a super-autocomplete, able to write 1-3 lines of code that works good enough. At the high level, it's great for explaining high-level concepts that are relevant to a task at hand. In the middle... AI doesn't work very well. If an AI writes a multi-step plan, where the pieces have to fit together, I've found it goes off the rails. Parts 1 and 3 of a 4-part plan are fine. So is part 2. However they don't fit together! AI has no concept of "these four parts have to be closely connected, building a whole". It just builds from A to B in four steps... but taking two different paths and stitching the pieces together poorly.

kelseyfrog 2y ago
The low-level high-level spectrum might not be the best scale to gauge AI by. We should kernel-trick our scale so that low and high level is seperable from multi-step planning problems. Or in other words, use a different dimension to separate these three problems.
ADeerAppeared 2y ago
It's not a bathtub curve. Your low-level and "high"-level tasks are the same thing: Probabilistic text generation. It's not reasoning about your code, nor about the explanation it gives you. > AI has no concept of "these four parts have to be closely connected, building a whole". AI can't think. It doesn't create an internal model of the problem given, it just guesses. It fails at all these "middle" tasks because they require abstract reasoning to be correct.
- bubblyworld 2y ago
  Not going to comment on the thinking part, because who knows what that means, but there's evidence that transformers do in fact learn predictive models of their input space. There's a cool blog post on this here: https://www.neelnanda.io/mechanistic-interpretability/othello https://www.neelnanda.io/mechanistic-interpretability/othell...
  - ADeerAppeared 2y ago

DrMiaow 2y ago

"just add more dimensions, bro!"

omerhac 2y ago

Each time research about LLM and reasoning comes out Yan LeCun gets an itch

justkk 2y ago

What are regions in this context?, are more regions better, how one delimiter the regions?, can one region be the same concept as several related regions?

magicalhippo 2y ago
As I understand it, the regions are simply the pieces that constitute the partitioning of the input domain, ie vector space formed by the the weights. There's some more details in one of the referenced papers[1], section 3.1 and onward. The argument in that paper is that the layers in a typical deep neural network partitions the input domain into regions, where each region has its own affine mapping of the input. For any arbitrary activation function, one would have to find the partitioning as well as the per-region parameters of the affine mappings. However since all the common activation functions are globally convex, they show that one can use this in a way where the partitioning is entirely determined by the per-region affine mapping parameters. Thus the output of the layer for a given input x is a "partition-region-dependent, piecewise affine transformation of x". The affine mapping parameters is effectively what you end up changing during training, and so the number and shape of the regions change during training as well. The submitted paper shows that more regions increase the approximation power of the neural net layer. This in itself doesn't seem that surprising given the above, but they use it as an important stepping stone. [1]: https://arxiv.org/abs/1805.06576v2 https://arxiv.org/abs/1805.06576v2

lngnmn2 2y ago

[dead]

EncomLab 2y ago

Does anyone remember the "Mad Libs" games - you fill out a form with blanks for "verb", "noun", "adjective", etc - then on the next page you fill in the words from the form to create a silly story. The results are funny because the words you provided initially were without context - they were syntactically correct, but were nonsense in context. LLM's are like Mad Libs with a "contextual predictor" - they produce syntactically correct output, and the "contextual predictor" limits the amount of nonsense because statistical correlations can generate meaningful output most of the time. But there is no "reasoning" occurring here - just syntactic templating and statistical auto-complete.

FeepingCreature 2y ago
I believe reasoning is (sufficiently advanced) syntactic templating and statistical autocomplete. Reminder that syntactic transformations are Turing complete: https://wiki.c2.com/?RewriteRules https://wiki.c2.com/?RewriteRules
- enord 2y ago
  That’s neither here nor there. Everything can be statistically modelled but very few things are reasoning. Same with turing machines.
  - mistermann 2y ago
    Speaking of reasoning, how do you know what is happening inside the mind when it "reasons"? And while you're at it: what is happening inside the mind when it reasons?
    - enord 2y ago
      These are very good questions that deserve unequivocal answers. Alas…
      - mistermann 2y ago

voidhorse 2y ago

As with many philosophical discussions, there is no point in claiming LLMs can "reason" because "reason" is not a well-defined term and you will not get everyone to agree on a singular definition. Ask a computer scientist, continental philosopher, and anthropologist what "reason" is and they will give you extremely different answers. If by reason we mean deductive reasoning as practiced in mathematics and inductive reasoning as practiced in the sciences, there is no evidence that LLMs do anything of the sort. There is no reason (ha) to believe that linguistic pattern matching is enough to emulate all that we call thinking in man. To claim so is to adopt an drastically narrow definition of "thinking" and to ignore the fact that we are embodied intellects, capable of knowing ourselves in a transparent possibly prelinguistic way. Unless an AI becomes embodied and can do the same, I have no faith that it will ever "think" or "reason" as humans do. It remains a really good statistical parlor trick.

bkitano19 2y ago
https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html https://transformer-circuits.pub/2022/in-context-learning-an... there is a lot of evidence to suggest that they are performing induction
PhunkyPhil 2y ago
> Unless an AI becomes embodied and can do the same, I have no faith that it will ever "think" or "reason" as humans do. It remains a really good statistical parlor trick. This may be true, but if it's "good enough" then why does that matter? If I can't determine if a user on Slack/Teams is an LLM that covers their tickets on time with decent code quality, then I really don't care if they know themselves in a transparent, prelinguistic fashion.