38 ms·

Transformers Can Do Arithmetic with the Right Embeddings

207 points by byt3h3ad 2y ago

infogulch 2y ago
The other day I was wondering if LLMs are bad at at maths because they don't have readily apparent access to the concept of "columns". Apparently the answer is yes. Vertical alignment across lines is pretty important for humans to learn operations on digits, but the way we encode lines with a \n separator doesn't really help. In a recent codebullet video gpt really struggled with any kind of vertical alignment task. I wonder if it would do better on a fixed 80 column width...
- Terr_ 2y ago
  Isn't it more that they don't have ready access to the much-more-fundamental concept of decimal numbers? My understanding was that they tokenized them into chunks and tried to learn associations between the chunks, the same as if one was breaking apart English words. So "2+2=4" isn't being treated that differently from "all's well that ends well." This might lead to a kind of Benny's Rules [0] situation, where sufficient brute-force can make a collection of overfitted non-arithmetic rules appear to work. [0] https://blog.mathed.net/2011/07/rysk-erlwangers-bennys-conception-of.html?m=1 https://blog.mathed.net/2011/07/rysk-erlwangers-bennys-conce...
  - andrepd 2y ago
    Fascinating article!
    - Terr_ 2y ago
      It looks like the math-notation formatting didn't survive, for that you might want to see a PDF, ex: https://people.wou.edu/~girodm/library/benny.pdf https://people.wou.edu/~girodm/library/benny.pdf
  - isaacfung 2y ago
    The current gen llms tokenize numbers digit by digit unlike earlier llms.
    - Last5Digits 2y ago
      They don't. Which you can easily check with any of the dozen web apps currently implementing the GPT-4o tokenizer.
    - mike_hearn 2y ago
      No, it doesn't help. Bloomberg tried this and it didn't seem to make much difference.
      - singularity2001 2y ago
        If someone else is interested in the Bloomberg tokenizer: https://medium.com/generative-ai-insights-for-business-leaders-and/turning-text-into-data-how-bloomberggpt-tokenizes-financial-language-c17679fb72d0 https://medium.com/generative-ai-insights-for-business-leade...
- matrix2596 2y ago
  wouldnt presenting numbers in reverse order, with the least significant digit on the left and most significant on the right help with the reasoning?
  - spencerchubb 2y ago
    They do that in the paper
michaelnny 2y ago
I think the main problem is the way we turn the raw mathematics symbols or equations into tokens, and these suboptimal tokenization may decreases the performance
- threatofrain 2y ago
  That doesn't stop decent code output for many computer languages.
- ynik 2y ago
  I thinks that's far from the only problem. To me the most obvious problem is that we use right-to-left numbers (think about the order you're writing digits when doing long addition) in a left-to-right language. Without a special number-flipping step; the transformer is forced to produce the output token-by-token, i.e. from left-to-right. Without the ability to store additional internal state, this turns addition into an O(N²) problem purely due to the suboptimal output ordering!
  - Shrezzing 2y ago
    The paper discusses this, and the approach taken in the paper implements a number-flip stage, so numbers are formatted with their least significant figure first.
msoad 2y ago
It seems like a hack to be honest. Problem at hand is not to make transformers do addition of 100 digit numbers. Problem is the current systems can’t reason about things, math included. Optimizing for a certain use case is not gonna take us where we wanna be. We want to have a system that can learn to reason.
- josehackernews 2y ago
  how do you argue that these models are not able to reason? deductive reasoning is just drawing specific conclusion from general patterns. something I would argue this models can do (of course not always and are still pretty bad in most cases) the point i’m trying to make is that sometimes reasoning is overrated and put on the top of the cognitive ladder, sometimes I have seen it compared to self-awareness or stuff like that. I know that you are not probably saying it in this way, just wanted to let it out. I believe there is fundamental work still to be done, maybe models that are able to draw patterns comparing experience, but this kind of work can be useful as make us reflect in every step of what these models do, and how much the internal representation learned can be optimized
  - YeGoblynQueenne 2y ago
    >> deductive reasoning is just drawing specific conclusion from general patterns. This is according to whom, please?
    - nicklecompte 2y ago
      The fundamental argument of "Artificial Intelligence, Natural Stupidity" is that AI researchers constantly abuse terms like "reasoning," "deduction," "understanding," and so on, deluding others and themselves that their machine is almost as intelligent as a human when it's clearly dumber than a dog. My cats don't need "general patterns" to form deductions, they deduce many sophisticated things (on their terms) with n=1 data points. In the 80s the computers were indisputably dumber than ants. That's probably not true these days. But the decades-long refusal of most AI researchers to accept humility about the limitations of their knowledge (now they describe multiple-choice science trivia as "graduate level reasoning") suggests to me that none of us will live to see an AI that's smarter than a mouse. There's just too much money and ideology, and too little falsifiability.
YeGoblynQueenne 2y ago
What is the point of this work? 99% on 100-digit arithmetic means there's a 0% chance anyone will ever use a Transformer as an ALU or anything of the kind. We already know how to hard-code a (literally) infinitely more accurate addition machine. And not only addition: all four arithmetic operations. The technique proposed in the article -imposing a strong inductive bias for addition- kiind of works for multiplication, but not for subtraction or division (clearly; I can't even find the words in the paper). As a practical way to build a machine to do arithmetic this is out of the question. We've known how to mechanise arithmetic since the 1850's with Blaize Pascal and his Pascaline. What is the point in demonstrating it's possible to reinvent a broken, partial, buggy version of an arithmetic machine if one tries really hard and shoehorns the necessary patterns in a neural net? We've known that for a long time, too (every proof that a neural net can simulate this or that Turing machine if you design the network diagram and set the weights by hand, ever). So what is the point of this? Transformers are supposed to be the "sparks of AGI" and they can almost do arithmetic if we try very hard to shove it down their heads? Who cares?
- IanCal 2y ago
  There are two sides to this that jump out One is that research into what the limits of the architecture are is useful. Maths has a nice property of being very easy to verify and you can construct logical processes with it. It's a useful testbed. Second is there are a lot more places that understanding how to do arithmetic help, outside of just doing sums on their own.
- zarzavat 2y ago
  It’s not about arithmetic but about embeddings. The positional embeddings used in transformers are rather simplistic. If they can add this one new capability to transformers by using different embeddings then maybe there are other capabilities that are within reach.
  - YeGoblynQueenne 2y ago
    No, because those embeddings only work for addition (very weakly for multiplication and sorting). Imagine needing a specially-crafted bias for every single task. The Deep Learning revolution brought on by Convolutional Neural Nets was supposed to do away with the need to do exactly that.
jiggawatts 2y ago
Something I've been thinking about is how the Minds -- the super-human AI hyper-computers that fly the ships in the Culture series of novels are described. The image built up in my head[1] is that they're hybrids blending neural networks and regular compute substrates. They can calculate, simulate, and reason in combination. There have been crude attempts at this already, hooking in Mathematica and Python into ChatGPT. I say crude, because these add-ons are controlled via output tokens. What I would like to see is a GPT-style AI that also has compute blocks, not just transformer blocks. I don't mean compute in the sense of "matrix multiply for weights and biases", but literally an ALU-style block of basic maths operations available for use by the neurons. One thought that I had was that this could be via activations that have both a floating-point activation value and "baggage" such as a numerical value from the input. Like a token in a traditional parser, that can represent a constant string or an integer with its decoded value. The newer, truly multi-modal models gave me a related idea: Just like how they can have "image" tokens and "audio" tokens, I wonder if they could be given "numeric data" tokens or "math symbol" tokens. Not in the same way that they're given mixed-language text tokens, but dedicated tokens that are fed into both the transformer blocks and also into ALU blocks. Just an idle thought... [1] Every reader reads into a story something unique, which may or may not align with what the author intended. This is my understanding, coloured by my own knowledge, etc, etc...
- vessenes 2y ago
  Fellow huge Banks fan here. I like this idea a lot. Right now we are going the long/hard way round, and post training asking an LLM to know it needs compute, then write a compute request, then feed back the compute answer into a tokenization loop. It probably does make sense to add a mini CPU as a layer / tool / math primitive. I wonder how you'd train it to use such a thing? In my mind it's not really a layer per-se, but it's a set of function calls a layer could route to when it wants, and weight the response appropriately.
- int_19h 2y ago
  The problem, if you embed an ALU like that, is how to train it to use them properly. And then it's not clear if they actually need to be able to do that in the middle of a pass that, at the end, is going to produce a single token anyway. Controlling that stuff via output tokens actually kinda makes sense by analogy, since that is how we use calculators etc. But I do agree that specialized tokens that are used specifically to activate tools like that might be a better idea than just using plain text to signal in-band. And production of such specialized tokens can be easily trained.
teleforce 2y ago
I think understanding mathematics is what LLM really need at the moment far more important than video generation that is just another form of CGI [1]. After deep learning and transformer, understanding mathematics and its proofs not just arithmetic will be the next game changer for LLM and a turning point for humanity. [1] Why LLMs like ChatGPT and Google Bard are bad at math: https://www.xda-developers.com/why-llms-are-bad-at-math/ https://www.xda-developers.com/why-llms-are-bad-at-math/
- staunton 2y ago
  > understanding mathematics and its proofs not just arithmetic will be the next game changer for LLM Why? I definitely agree that such capabilities would represent a major advance (and very likely go together with game changing increases of capabilities in other areas). I also think using AI to write formal math proofs in e.g. Lean is very cool. However, by itself, it seems like this capability wouldn't be very useful, commercially for example. Do you think this capability is exceptionally informative merely because it has to go together with other capabilities? It's not impossible to have a (maybe somewhat limited) formal math AI that will remain mostly irrelevant to the everyday world (like FormalGeo).
  - simiones 2y ago
    Understanding mathematics basically means understading higher-level reasoning. If an AI were able to actually do this + the ability to generate and interpret language that LLMs already show, it would seem to be 90% or more of the way to AGI.
  - ADeerAppeared 2y ago
    > However, by itself, it seems like this capability wouldn't be very useful, commercially for example. Quite the opposite, it's the holy grail of all AI. Consider various work that isn't (and can't) be done by computers/robots/etc right now. The intelligence constraint is universally, a required amount of problem solving. Even the "low skill" labour requires it. And to perform such problem solving, you need advanced logic and reasoning capabilities, which is the same thing as novel mathematics, just applied to a different end.
kjhcvkek77 2y ago
Very cool that it was able to generalise from small numbers to larger ones with such high accuracy.
Shrezzing 2y ago
Since models are very good at writing very short computer programs, and computer programs are very good at mathematical calculations, would it not be considerably more efficient to train them to recognise a "what is x + y" type problem, and respond with the answer to "write and execute a small javascript program to calculate x + y, then share the result"?
- deleted 2y ago
  [deleted]
- Grimblewald 2y ago
  From a getting answers perspective yes, from an understanding LLMs perspective no. If you read the avstract you can see how this goes beyond arithmetic and helps with longform reasoning
- gmerc 2y ago
  That's in fact what ChatGPT does ... because 99% accurate math is not useful to anyone.
- simiones 2y ago
  But that's not all that relevant to the question "can LLMs do math". People don't really need ChatGPT to replace a calculator. They are interested in whether the LLM has learned higher reasoning skills from it's training on language (especially since we know it has "read" more math books than any human could in a lifetime). Responding with a program that reuses the + primitive in JS proves no such thing. Even responding with a description of the addition algorithm doesn't prove that it has "understood" maths, if it can't actually run that algorithm itself - it's essentially looking up a memorized definition. The only real proof is actually having the LLM itself perform the addition (without any special-case logic). This question is of course relevant only in a research sense, in seeking to understand to what extent and in what ways the LLM is acting as a stochastic parrot vs gaining a type of "understanding", for lack of a better word.
  - Shrezzing 2y ago
    That's a fair summary of why the research is happening. Thanks.
gmerc 2y ago
That's great, 99% math is absolutely good enough for real world problems /s
- traverseda 2y ago
  It's probably on-par or better than humans get unaided. Hell, I'd bet due to transcription errors it's better than what humans get in a lot of settings, even when aided by a calculator.
  - gmerc 2y ago
    I guarantee you professionals using math at work - for example in finance - not have a 1% error quota. They use tools. We have tools. Nobody in any serious role (money, etc) works unaided. Math inference is a palor trick as is the whole “world model” bullshit - physics doesn’t work with 99% accuracy. It’s the same reason agents are bullshit right now - error compounding at 95% reliability per step murders them and currently there is no path to triple 9
    - Jensson 2y ago
      People didn't have 1% error before computers either, they just practiced enough to barely ever make errors.
- mike_hearn 2y ago
  How many "real world" problems require 100 digit numbers?
  - gmerc 2y ago
    Not many. But all require 100% accuracy. If not someone has to take responsibility.
torginus 2y ago
I just wonder if numbers were written right to left, llms would be much better at arithmetic. You can 'predict' the least significant digit by reusing the already written digits in the computation, but to generate most significant ones, you generally need to do the entire computation in one go.
- weinzierl 2y ago
  This is an interesting idea but probably hard to verify. A tangent is that positional systems were originally invented with least digit first, I believe. The Babylonian sexagesimal system was like that as was the Arabic one (where first is on the right). The most significant digit first convention came when right-to left numbers were used in left-to-right systems without reversing them in writing. To this day we read the more common smaller numbers least significant digit first to varying degrees. 16 = six teen, sech zehn 98 = acht und neunzig, achten negentig, ثمانية وتسعون
- spencerchubb 2y ago
  They do that in the paper. Least significant digit on the left
- gwern 2y ago
  Yes. This has already been demonstrated by "Teaching Arithmetic to Small Transformers" https://arxiv.org/abs/2307.03381 https://arxiv.org/abs/2307.03381 , I'm not sure what OP adds except demonstrating that you can do that via the embedding itself rather than the tokenization. > We first demonstrate that conventional training data is not the most effective for arithmetic learning, and simple formatting changes can significantly improve accuracy. This leads to sharp phase transitions as a function of training data scale, which, in some cases, can be explained through connections to low-rank matrix completion. Building on prior work, we then train on chain-of-thought style data that includes intermediate step results. Even in the complete absence of pretraining, this approach significantly and simultaneously improves accuracy, sample complexity, and convergence speed. We also study the interplay between arithmetic and text data during training and examine the effects of few-shot prompting, pretraining, and model scale. Additionally, we discuss length generalization challenges.
- 2y ago
Havoc 2y ago
For things like this where we have computationally cheap, well understood, reliable tools available (aka calculator) it seems better to train the model in tool use. I guess perhaps the techniques could be generalized though?
- mike_hearn 2y ago
  Generalizable techniques is mostly the point of papers like this one yes. What they show here is that apparently fundamental problems with transformer reasoning can be fixed by encoding data in a more sophisticated manner. This is exciting. I've been thinking for a long time that the tokenization schemes are a low hanging fruit for improving coding LLM performance, this isn't exactly the same thing but it's in the same general area. Smartness and reasoning ability with the current set of algorithmic techniques seems to have topped out around GPT-4 level, which implies that further leaps in mental abilities must come from improving other things beyond training set size. For example, whilst replacing the need for a calculator isn't very important, one obvious research direction would be to explore adding extra embeddings to code inputs, perhaps that are being computed by an IDE.
  - HarHarVeryFunny 2y ago
    It seems sub-word tokenization vs using character inputs is just a trade off to gain computational efficiency, and obviously isn't how our brain works. We're not born with a fixed visual tokenization scheme - we learn to create our own groupings and object representations. However, transformers seem to struggle a bit with accurately manipulating sequences, so going to character inputs and hoping for those to be aggregated into words/numbers/etc might cause more problems than it solves? I have to wonder if these models would not be better off learning whole-word embeddings rather than tokens. You'd have thought they would learn embeddings that encode any useful relatedness (e.g. corresponding to common prefixes) between words. Perhaps numbers would be better off input as a sequence of individual digit embeddings.
    - mike_hearn 2y ago
      Yeah a tiny vocab of characters doesn't work that well, it was tried very early on and creating large vocabs of tokens was a big improvement. Which makes sense. A lot of tokens are full words and so the token->embedding phase can quickly look up an embedding in vector space that contains a lot of meaning, whereas an embedding of 'z' or whatever is going to be meaningless.
vessenes 2y ago
Wow, a lot of grumpiness in here. If it's true that adding like 20 or so tokens to encode column location / decimal spot triples math performance in out of band tasks, that's a big deal. It's a simple fix, it improves performance A LOT, and they even indicate it's not just a party trick, in that the LLM can use the information to do better on related tasks like sorting and list making. This is basically free to add, and there's no reason it shouldn't be made part of standard tokenization. I'm more interested in the question of how we can find other useful concepts for data -> embedding space like this; can we incept our tokenization inception so it has more inception?
- nprateem 2y ago
  But I don't want tricks. I want to know that it knows so I don't have to continually guess whether it's right or not.
  - naasking 2y ago
    That's simply not possible. Human understanding is still unreliable, even for geniuses.
    - lapitopi 2y ago
      That’s why I am asking a computer.
      - catapart 2y ago
        I'm with you. I get that this is akin to asking a human, because we're trying to reason, so we will bring along (assumedly) unavoidable deficiencies of human reasoning. But if I were to ask a human genius this question, ne would grab a calculator and employ it as ne did the rest of ner reasoning. So it seems like we should probably teach LLMs to "use a calculator", rather than try to get them to be more right when doing math 'in their head'.
        2y ago
pmayrgundter 2y ago
I'm curious about the framing of research like this.. "The poor performance of transformers on arithmetic tasks" (relative to what?) and how that informs the adjacent conversation on progress towards AGI. Some say AGI has already been achieved, others that it's years or decades away. When I dig into the disagreement, it often partially depends on the perspective of how competent humans are on the tasks in question, with the optimists being, I think, more realistic about variance in human intelligence and the pessimists seeming to reserve the term "general intelligence" for possessing a nearly perfect suite of capabilities that many otherwise intelligent people practically don't have. For example with arithmetic, this study cites another [Dziri et al. 2023], that says: "For instance, humans can solve 3-digit by 3-digit multiplication arithmetic after learning basic calculation rules. Yet, off-the-shelf ChatGPT and GPT4 achieve only 55% and 59% accuracies on this task, respectively." But this isn't the case.. 5-6% of the population have https://en.wikipedia.org/wiki/Dyscalculia https://en.wikipedia.org/wiki/Dyscalculia, but can be otherwise normal. I still see value in normative statements about human capability in AI & AGI research, but I think we'll need to move towards explicit statistical framing. DeepMind's Position paper "Levels of AGI for Operationalizing Progress on the Path to AGI" has a schema like this, where AGI capabilities are defined across 2 axes of Performance level X Generality (narrow vs general), and the Performance levels are measured by comparison with "Percentile of skilled adults" able to perform the task.. https://arxiv.org/pdf/2311.02462#page=3.40 https://arxiv.org/pdf/2311.02462#page=3.40 Within that framing, this paper's title or result might be "Achieving AGI Competency in Arithmetic", or "Expertise", or "Virtuosity", i.e. on par respectively with 50th, 90th or 99th percentile of skilled adults.
- Last5Digits 2y ago
  Exactly, we need a much more granular approach to evaluating intelligence and generality. Our current conception of intelligence largely works because humans share evolutionary history and partake in the same 10+ years of standardized training. As such, many dimensions of our intelligence correlate quite a bit, and you can likely infer a person's "general" proficiency or education by checking only a subset of those dimensions. If someone can't do arithmetic then it's very unlikely that they'll be able to compute integrals. LLMs don't share that property, though. Their distribution of proficiency over various dimensions and subfields is highly variable and only slightly correlated. Therefore, it makes no sense to infer the ability or inability to perform some magically global type of reasoning or generalization from just a subset of tasks, the way we do for humans.
r2_pilot 2y ago
Meanwhile I'm over here using Claude 3 Opus to do trig and calculus problems as well as generate the LaTex representation of the equations. It's not necessary to be 100% in my case (purely for fun) but I follow its reasoning and it's pretty consistent at least enough for "orders of magnitude" and first order effects. I was gonna post some of the chats about physics but probably nobody cares.
- r2_pilot 2y ago
  I did do some followup research. The math in its complex reasoning "tracks" but when I asked it to do 4 digit x 4 digit multiplication, it got most of it right except for a weird random digit error in the middle (?!) of the correct answer, lol. Now I want to run CLUTTR against Claude since it seems nobody has published that yet.
zacksiri 2y ago
I think the problem here is that 'understanding' is not the same as curve fitting. If all one is doing is giving a model lots of data and fitting curves it's not really 'understanding' but brute forcing it's way (with gradient descent) and then storing the weights and finally approximate the solution when a query is passed in. This is not the same as understanding. Human intelligence can operate deterministically as well as non-deterministically. We can listen to language, which is by it's nature non-deterministic and convert that into deterministic operations and vice a versa. IE we can operate on some logic and explain it in multiple ways to other people. Understanding requires much less data than brute forcing your way into pattern recognition. When you see a simple number like this 2 * 4 you are able to understand that it's equivalent to 2 + 2 + 2 + 2 and that in turn means 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 <- Count that and you've got your answer. Because you 'understand' this basic concept and all the operations in between you are able to compute more examples. But you only need to understand it once. Once you understand multiplications and additions and all the tricks in between you are able to compute 23 * 10 without being fed 23 * 10 as prior data. Understanding is very different from fitting a curve. You can reach conclusions and understanding through pattern recognition, but it's important to differentiate 'approximation' from 'calculation'. If you understand something in it's entirety you should be able to calculate an outcome deterministically. Right now LLMs lack 'understanding', and seems to only 'approximate' which may seem like 'understanding' but is actually not.
- zyklu5 2y ago
  I think you are mixing layers of abstraction. To make a crude but I think not unhelpful analogy: 'Understanding' is a natural language concept that is our way to describe whats happening in our heads, and like most other such concepts is resistant to any clear definition and will exhibit sorites type paradoxes when one is attempted. It belongs to the presentation layer of the stack. While the process of curve fitting, however it is implemented, with whatever NN structure (like transformers) or maybe something else entirely belongs to the physical layer of the stack -- akin to frequency modulation. While I am unsure whether LLMs are really understanding, whatever that means, I think it is not difficult to believe that any form of understanding we implement will involve 'curve fitting' as a central part.
skyde 2y ago
Why not apply same concept every time a word is split into more than one token? Basically if a word contain a Prefix, suffix or root word. We could have a token position relative to the start of the word in the embedding.
- skyde 2y ago
  It seems it has been done before: "Syntax-Aware Transformer Models for Neural Machine Translation" by Yang et al. (2019). This model enhances the transformer architecture with syntax-aware attention mechanisms that consider dependency parse trees. Context-Aware Neural Machine Translation Learns Anaphora Resolution" by Bawden et al. (2018). This paper explores integrating context and syntax into neural machine translation models.
topherjaynes 2y ago
I went through the paper and thought immediately about how did they implement it; I missed they published their code as well. Here is the link for everyone who skimmed past it: https://github.com/mcleish7/arithmetic/tree/main https://github.com/mcleish7/arithmetic/tree/main
- byt3h3ad 2y ago
  my bad, should have posted it with the link itself
  - topherjaynes 2y ago
    Good to start with the concept, I just had so many implementation questions. Working through the code know... which is way harder to digest.
nerdponx 2y ago
I like to see more focus on the input embeddings. It's basically the same as feature engineering in pre-deep machine learning: constructing features with high information content can significantly reduce the amount of data and computation needed to fit a useful model. And sometimes it's impossible to fit a useful model without careful feature engineering, either because the model itself is constrained in some way or because there isn't enough data or both. It's analogous to making a choice of inductive bias within the model itself. We literally could not do LLMs without the carefully-constructed transformer architecture. Why should we expect to make further progress without paying more attention to the embeddings?
wantsanagent 2y ago
I like these kinds of fixes. It's like realizing your child has vision problems and getting them glasses.
winddude 2y ago
But a calculator wouldn't be very good if it's only correct 99% of the time for arithmetic...
CyberDildonics 2y ago
I'm pretty sure getting computers to do arithmetic is not a giant hurdle.

207 points by byt3h3ad 2y ago

The other day I was wondering if LLMs are bad at at maths because they don't have readily apparent access to the concept of "columns". Apparently the answer is yes. Vertical alignment across lines is pretty important for humans to learn operations on digits, but the way we encode lines with a \n separator doesn't really help. In a recent codebullet video gpt really struggled with any kind of vertical alignment task. I wonder if it would do better on a fixed 80 column width...

Terr_ 2y ago
Isn't it more that they don't have ready access to the much-more-fundamental concept of decimal numbers? My understanding was that they tokenized them into chunks and tried to learn associations between the chunks, the same as if one was breaking apart English words. So "2+2=4" isn't being treated that differently from "all's well that ends well." This might lead to a kind of Benny's Rules [0] situation, where sufficient brute-force can make a collection of overfitted non-arithmetic rules appear to work. [0] https://blog.mathed.net/2011/07/rysk-erlwangers-bennys-conception-of.html?m=1 https://blog.mathed.net/2011/07/rysk-erlwangers-bennys-conce...
- andrepd 2y ago
  Fascinating article!
  - Terr_ 2y ago
    It looks like the math-notation formatting didn't survive, for that you might want to see a PDF, ex: https://people.wou.edu/~girodm/library/benny.pdf https://people.wou.edu/~girodm/library/benny.pdf
- isaacfung 2y ago
  The current gen llms tokenize numbers digit by digit unlike earlier llms.
  - Last5Digits 2y ago
    They don't. Which you can easily check with any of the dozen web apps currently implementing the GPT-4o tokenizer.
  - mike_hearn 2y ago
    No, it doesn't help. Bloomberg tried this and it didn't seem to make much difference.
    - singularity2001 2y ago
      If someone else is interested in the Bloomberg tokenizer: https://medium.com/generative-ai-insights-for-business-leaders-and/turning-text-into-data-how-bloomberggpt-tokenizes-financial-language-c17679fb72d0 https://medium.com/generative-ai-insights-for-business-leade...
matrix2596 2y ago
wouldnt presenting numbers in reverse order, with the least significant digit on the left and most significant on the right help with the reasoning?
- spencerchubb 2y ago
  They do that in the paper

michaelnny 2y ago

I think the main problem is the way we turn the raw mathematics symbols or equations into tokens, and these suboptimal tokenization may decreases the performance

threatofrain 2y ago
That doesn't stop decent code output for many computer languages.
ynik 2y ago
I thinks that's far from the only problem. To me the most obvious problem is that we use right-to-left numbers (think about the order you're writing digits when doing long addition) in a left-to-right language. Without a special number-flipping step; the transformer is forced to produce the output token-by-token, i.e. from left-to-right. Without the ability to store additional internal state, this turns addition into an O(N²) problem purely due to the suboptimal output ordering!
- Shrezzing 2y ago
  The paper discusses this, and the approach taken in the paper implements a number-flip stage, so numbers are formatted with their least significant figure first.

msoad 2y ago

It seems like a hack to be honest. Problem at hand is not to make transformers do addition of 100 digit numbers. Problem is the current systems can’t reason about things, math included. Optimizing for a certain use case is not gonna take us where we wanna be. We want to have a system that can learn to reason.

josehackernews 2y ago
how do you argue that these models are not able to reason? deductive reasoning is just drawing specific conclusion from general patterns. something I would argue this models can do (of course not always and are still pretty bad in most cases) the point i’m trying to make is that sometimes reasoning is overrated and put on the top of the cognitive ladder, sometimes I have seen it compared to self-awareness or stuff like that. I know that you are not probably saying it in this way, just wanted to let it out. I believe there is fundamental work still to be done, maybe models that are able to draw patterns comparing experience, but this kind of work can be useful as make us reflect in every step of what these models do, and how much the internal representation learned can be optimized
- YeGoblynQueenne 2y ago
  >> deductive reasoning is just drawing specific conclusion from general patterns. This is according to whom, please?
  - nicklecompte 2y ago
    The fundamental argument of "Artificial Intelligence, Natural Stupidity" is that AI researchers constantly abuse terms like "reasoning," "deduction," "understanding," and so on, deluding others and themselves that their machine is almost as intelligent as a human when it's clearly dumber than a dog. My cats don't need "general patterns" to form deductions, they deduce many sophisticated things (on their terms) with n=1 data points. In the 80s the computers were indisputably dumber than ants. That's probably not true these days. But the decades-long refusal of most AI researchers to accept humility about the limitations of their knowledge (now they describe multiple-choice science trivia as "graduate level reasoning") suggests to me that none of us will live to see an AI that's smarter than a mouse. There's just too much money and ideology, and too little falsifiability.

YeGoblynQueenne 2y ago

What is the point of this work? 99% on 100-digit arithmetic means there's a 0% chance anyone will ever use a Transformer as an ALU or anything of the kind. We already know how to hard-code a (literally) infinitely more accurate addition machine. And not only addition: all four arithmetic operations. The technique proposed in the article -imposing a strong inductive bias for addition- kiind of works for multiplication, but not for subtraction or division (clearly; I can't even find the words in the paper). As a practical way to build a machine to do arithmetic this is out of the question. We've known how to mechanise arithmetic since the 1850's with Blaize Pascal and his Pascaline. What is the point in demonstrating it's possible to reinvent a broken, partial, buggy version of an arithmetic machine if one tries really hard and shoehorns the necessary patterns in a neural net? We've known that for a long time, too (every proof that a neural net can simulate this or that Turing machine if you design the network diagram and set the weights by hand, ever). So what is the point of this? Transformers are supposed to be the "sparks of AGI" and they can almost do arithmetic if we try very hard to shove it down their heads? Who cares?

IanCal 2y ago
There are two sides to this that jump out One is that research into what the limits of the architecture are is useful. Maths has a nice property of being very easy to verify and you can construct logical processes with it. It's a useful testbed. Second is there are a lot more places that understanding how to do arithmetic help, outside of just doing sums on their own.
zarzavat 2y ago
It’s not about arithmetic but about embeddings. The positional embeddings used in transformers are rather simplistic. If they can add this one new capability to transformers by using different embeddings then maybe there are other capabilities that are within reach.
- YeGoblynQueenne 2y ago
  No, because those embeddings only work for addition (very weakly for multiplication and sorting). Imagine needing a specially-crafted bias for every single task. The Deep Learning revolution brought on by Convolutional Neural Nets was supposed to do away with the need to do exactly that.

jiggawatts 2y ago

Something I've been thinking about is how the Minds -- the super-human AI hyper-computers that fly the ships in the Culture series of novels are described. The image built up in my head[1] is that they're hybrids blending neural networks and regular compute substrates. They can calculate, simulate, and reason in combination. There have been crude attempts at this already, hooking in Mathematica and Python into ChatGPT. I say crude, because these add-ons are controlled via output tokens. What I would like to see is a GPT-style AI that also has compute blocks, not just transformer blocks. I don't mean compute in the sense of "matrix multiply for weights and biases", but literally an ALU-style block of basic maths operations available for use by the neurons. One thought that I had was that this could be via activations that have both a floating-point activation value and "baggage" such as a numerical value from the input. Like a token in a traditional parser, that can represent a constant string or an integer with its decoded value. The newer, truly multi-modal models gave me a related idea: Just like how they can have "image" tokens and "audio" tokens, I wonder if they could be given "numeric data" tokens or "math symbol" tokens. Not in the same way that they're given mixed-language text tokens, but dedicated tokens that are fed into both the transformer blocks and also into ALU blocks. Just an idle thought... [1] Every reader reads into a story something unique, which may or may not align with what the author intended. This is my understanding, coloured by my own knowledge, etc, etc...

vessenes 2y ago
Fellow huge Banks fan here. I like this idea a lot. Right now we are going the long/hard way round, and post training asking an LLM to know it needs compute, then write a compute request, then feed back the compute answer into a tokenization loop. It probably does make sense to add a mini CPU as a layer / tool / math primitive. I wonder how you'd train it to use such a thing? In my mind it's not really a layer per-se, but it's a set of function calls a layer could route to when it wants, and weight the response appropriately.
int_19h 2y ago
The problem, if you embed an ALU like that, is how to train it to use them properly. And then it's not clear if they actually need to be able to do that in the middle of a pass that, at the end, is going to produce a single token anyway. Controlling that stuff via output tokens actually kinda makes sense by analogy, since that is how we use calculators etc. But I do agree that specialized tokens that are used specifically to activate tools like that might be a better idea than just using plain text to signal in-band. And production of such specialized tokens can be easily trained.

teleforce 2y ago

I think understanding mathematics is what LLM really need at the moment far more important than video generation that is just another form of CGI [1]. After deep learning and transformer, understanding mathematics and its proofs not just arithmetic will be the next game changer for LLM and a turning point for humanity. [1] Why LLMs like ChatGPT and Google Bard are bad at math: https://www.xda-developers.com/why-llms-are-bad-at-math/ https://www.xda-developers.com/why-llms-are-bad-at-math/

staunton 2y ago
> understanding mathematics and its proofs not just arithmetic will be the next game changer for LLM Why? I definitely agree that such capabilities would represent a major advance (and very likely go together with game changing increases of capabilities in other areas). I also think using AI to write formal math proofs in e.g. Lean is very cool. However, by itself, it seems like this capability wouldn't be very useful, commercially for example. Do you think this capability is exceptionally informative merely because it has to go together with other capabilities? It's not impossible to have a (maybe somewhat limited) formal math AI that will remain mostly irrelevant to the everyday world (like FormalGeo).
- simiones 2y ago
  Understanding mathematics basically means understading higher-level reasoning. If an AI were able to actually do this + the ability to generate and interpret language that LLMs already show, it would seem to be 90% or more of the way to AGI.
- ADeerAppeared 2y ago
  > However, by itself, it seems like this capability wouldn't be very useful, commercially for example. Quite the opposite, it's the holy grail of all AI. Consider various work that isn't (and can't) be done by computers/robots/etc right now. The intelligence constraint is universally, a required amount of problem solving. Even the "low skill" labour requires it. And to perform such problem solving, you need advanced logic and reasoning capabilities, which is the same thing as novel mathematics, just applied to a different end.

kjhcvkek77 2y ago

Very cool that it was able to generalise from small numbers to larger ones with such high accuracy.

Shrezzing 2y ago

Since models are very good at writing very short computer programs, and computer programs are very good at mathematical calculations, would it not be considerably more efficient to train them to recognise a "what is x + y" type problem, and respond with the answer to "write and execute a small javascript program to calculate x + y, then share the result"?

deleted 2y ago
[deleted]
Grimblewald 2y ago
From a getting answers perspective yes, from an understanding LLMs perspective no. If you read the avstract you can see how this goes beyond arithmetic and helps with longform reasoning
gmerc 2y ago
That's in fact what ChatGPT does ... because 99% accurate math is not useful to anyone.
simiones 2y ago
But that's not all that relevant to the question "can LLMs do math". People don't really need ChatGPT to replace a calculator. They are interested in whether the LLM has learned higher reasoning skills from it's training on language (especially since we know it has "read" more math books than any human could in a lifetime). Responding with a program that reuses the + primitive in JS proves no such thing. Even responding with a description of the addition algorithm doesn't prove that it has "understood" maths, if it can't actually run that algorithm itself - it's essentially looking up a memorized definition. The only real proof is actually having the LLM itself perform the addition (without any special-case logic). This question is of course relevant only in a research sense, in seeking to understand to what extent and in what ways the LLM is acting as a stochastic parrot vs gaining a type of "understanding", for lack of a better word.
- Shrezzing 2y ago
  That's a fair summary of why the research is happening. Thanks.

gmerc 2y ago

That's great, 99% math is absolutely good enough for real world problems /s

traverseda 2y ago
It's probably on-par or better than humans get unaided. Hell, I'd bet due to transcription errors it's better than what humans get in a lot of settings, even when aided by a calculator.
- gmerc 2y ago
  I guarantee you professionals using math at work - for example in finance - not have a 1% error quota. They use tools. We have tools. Nobody in any serious role (money, etc) works unaided. Math inference is a palor trick as is the whole “world model” bullshit - physics doesn’t work with 99% accuracy. It’s the same reason agents are bullshit right now - error compounding at 95% reliability per step murders them and currently there is no path to triple 9
  - Jensson 2y ago
    People didn't have 1% error before computers either, they just practiced enough to barely ever make errors.
mike_hearn 2y ago
How many "real world" problems require 100 digit numbers?
- gmerc 2y ago
  Not many. But all require 100% accuracy. If not someone has to take responsibility.

torginus 2y ago

I just wonder if numbers were written right to left, llms would be much better at arithmetic. You can 'predict' the least significant digit by reusing the already written digits in the computation, but to generate most significant ones, you generally need to do the entire computation in one go.

weinzierl 2y ago
This is an interesting idea but probably hard to verify. A tangent is that positional systems were originally invented with least digit first, I believe. The Babylonian sexagesimal system was like that as was the Arabic one (where first is on the right). The most significant digit first convention came when right-to left numbers were used in left-to-right systems without reversing them in writing. To this day we read the more common smaller numbers least significant digit first to varying degrees. 16 = six teen, sech zehn 98 = acht und neunzig, achten negentig, ثمانية وتسعون
spencerchubb 2y ago
They do that in the paper. Least significant digit on the left
gwern 2y ago
Yes. This has already been demonstrated by "Teaching Arithmetic to Small Transformers" https://arxiv.org/abs/2307.03381 https://arxiv.org/abs/2307.03381 , I'm not sure what OP adds except demonstrating that you can do that via the embedding itself rather than the tokenization. > We first demonstrate that conventional training data is not the most effective for arithmetic learning, and simple formatting changes can significantly improve accuracy. This leads to sharp phase transitions as a function of training data scale, which, in some cases, can be explained through connections to low-rank matrix completion. Building on prior work, we then train on chain-of-thought style data that includes intermediate step results. Even in the complete absence of pretraining, this approach significantly and simultaneously improves accuracy, sample complexity, and convergence speed. We also study the interplay between arithmetic and text data during training and examine the effects of few-shot prompting, pretraining, and model scale. Additionally, we discuss length generalization challenges.
2y ago

Havoc 2y ago

For things like this where we have computationally cheap, well understood, reliable tools available (aka calculator) it seems better to train the model in tool use. I guess perhaps the techniques could be generalized though?

mike_hearn 2y ago
Generalizable techniques is mostly the point of papers like this one yes. What they show here is that apparently fundamental problems with transformer reasoning can be fixed by encoding data in a more sophisticated manner. This is exciting. I've been thinking for a long time that the tokenization schemes are a low hanging fruit for improving coding LLM performance, this isn't exactly the same thing but it's in the same general area. Smartness and reasoning ability with the current set of algorithmic techniques seems to have topped out around GPT-4 level, which implies that further leaps in mental abilities must come from improving other things beyond training set size. For example, whilst replacing the need for a calculator isn't very important, one obvious research direction would be to explore adding extra embeddings to code inputs, perhaps that are being computed by an IDE.
- HarHarVeryFunny 2y ago
  It seems sub-word tokenization vs using character inputs is just a trade off to gain computational efficiency, and obviously isn't how our brain works. We're not born with a fixed visual tokenization scheme - we learn to create our own groupings and object representations. However, transformers seem to struggle a bit with accurately manipulating sequences, so going to character inputs and hoping for those to be aggregated into words/numbers/etc might cause more problems than it solves? I have to wonder if these models would not be better off learning whole-word embeddings rather than tokens. You'd have thought they would learn embeddings that encode any useful relatedness (e.g. corresponding to common prefixes) between words. Perhaps numbers would be better off input as a sequence of individual digit embeddings.
  - mike_hearn 2y ago
    Yeah a tiny vocab of characters doesn't work that well, it was tried very early on and creating large vocabs of tokens was a big improvement. Which makes sense. A lot of tokens are full words and so the token->embedding phase can quickly look up an embedding in vector space that contains a lot of meaning, whereas an embedding of 'z' or whatever is going to be meaningless.

vessenes 2y ago

Wow, a lot of grumpiness in here. If it's true that adding like 20 or so tokens to encode column location / decimal spot triples math performance in out of band tasks, that's a big deal. It's a simple fix, it improves performance A LOT, and they even indicate it's not just a party trick, in that the LLM can use the information to do better on related tasks like sorting and list making. This is basically free to add, and there's no reason it shouldn't be made part of standard tokenization. I'm more interested in the question of how we can find other useful concepts for data -> embedding space like this; can we incept our tokenization inception so it has more inception?

nprateem 2y ago
But I don't want tricks. I want to know that it knows so I don't have to continually guess whether it's right or not.
- naasking 2y ago
  That's simply not possible. Human understanding is still unreliable, even for geniuses.
  - lapitopi 2y ago
    That’s why I am asking a computer.
    - catapart 2y ago
      I'm with you. I get that this is akin to asking a human, because we're trying to reason, so we will bring along (assumedly) unavoidable deficiencies of human reasoning. But if I were to ask a human genius this question, ne would grab a calculator and employ it as ne did the rest of ner reasoning. So it seems like we should probably teach LLMs to "use a calculator", rather than try to get them to be more right when doing math 'in their head'.
      - 2y ago

pmayrgundter 2y ago

I'm curious about the framing of research like this.. "The poor performance of transformers on arithmetic tasks" (relative to what?) and how that informs the adjacent conversation on progress towards AGI. Some say AGI has already been achieved, others that it's years or decades away. When I dig into the disagreement, it often partially depends on the perspective of how competent humans are on the tasks in question, with the optimists being, I think, more realistic about variance in human intelligence and the pessimists seeming to reserve the term "general intelligence" for possessing a nearly perfect suite of capabilities that many otherwise intelligent people practically don't have. For example with arithmetic, this study cites another [Dziri et al. 2023], that says: "For instance, humans can solve 3-digit by 3-digit multiplication arithmetic after learning basic calculation rules. Yet, off-the-shelf ChatGPT and GPT4 achieve only 55% and 59% accuracies on this task, respectively." But this isn't the case.. 5-6% of the population have https://en.wikipedia.org/wiki/Dyscalculia https://en.wikipedia.org/wiki/Dyscalculia, but can be otherwise normal. I still see value in normative statements about human capability in AI & AGI research, but I think we'll need to move towards explicit statistical framing. DeepMind's Position paper "Levels of AGI for Operationalizing Progress on the Path to AGI" has a schema like this, where AGI capabilities are defined across 2 axes of Performance level X Generality (narrow vs general), and the Performance levels are measured by comparison with "Percentile of skilled adults" able to perform the task.. https://arxiv.org/pdf/2311.02462#page=3.40 https://arxiv.org/pdf/2311.02462#page=3.40 Within that framing, this paper's title or result might be "Achieving AGI Competency in Arithmetic", or "Expertise", or "Virtuosity", i.e. on par respectively with 50th, 90th or 99th percentile of skilled adults.

Last5Digits 2y ago
Exactly, we need a much more granular approach to evaluating intelligence and generality. Our current conception of intelligence largely works because humans share evolutionary history and partake in the same 10+ years of standardized training. As such, many dimensions of our intelligence correlate quite a bit, and you can likely infer a person's "general" proficiency or education by checking only a subset of those dimensions. If someone can't do arithmetic then it's very unlikely that they'll be able to compute integrals. LLMs don't share that property, though. Their distribution of proficiency over various dimensions and subfields is highly variable and only slightly correlated. Therefore, it makes no sense to infer the ability or inability to perform some magically global type of reasoning or generalization from just a subset of tasks, the way we do for humans.

r2_pilot 2y ago

Meanwhile I'm over here using Claude 3 Opus to do trig and calculus problems as well as generate the LaTex representation of the equations. It's not necessary to be 100% in my case (purely for fun) but I follow its reasoning and it's pretty consistent at least enough for "orders of magnitude" and first order effects. I was gonna post some of the chats about physics but probably nobody cares.

r2_pilot 2y ago
I did do some followup research. The math in its complex reasoning "tracks" but when I asked it to do 4 digit x 4 digit multiplication, it got most of it right except for a weird random digit error in the middle (?!) of the correct answer, lol. Now I want to run CLUTTR against Claude since it seems nobody has published that yet.

zacksiri 2y ago

I think the problem here is that 'understanding' is not the same as curve fitting. If all one is doing is giving a model lots of data and fitting curves it's not really 'understanding' but brute forcing it's way (with gradient descent) and then storing the weights and finally approximate the solution when a query is passed in. This is not the same as understanding. Human intelligence can operate deterministically as well as non-deterministically. We can listen to language, which is by it's nature non-deterministic and convert that into deterministic operations and vice a versa. IE we can operate on some logic and explain it in multiple ways to other people. Understanding requires much less data than brute forcing your way into pattern recognition. When you see a simple number like this 2 * 4 you are able to understand that it's equivalent to 2 + 2 + 2 + 2 and that in turn means 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 <- Count that and you've got your answer. Because you 'understand' this basic concept and all the operations in between you are able to compute more examples. But you only need to understand it once. Once you understand multiplications and additions and all the tricks in between you are able to compute 23 * 10 without being fed 23 * 10 as prior data. Understanding is very different from fitting a curve. You can reach conclusions and understanding through pattern recognition, but it's important to differentiate 'approximation' from 'calculation'. If you understand something in it's entirety you should be able to calculate an outcome deterministically. Right now LLMs lack 'understanding', and seems to only 'approximate' which may seem like 'understanding' but is actually not.

zyklu5 2y ago
I think you are mixing layers of abstraction. To make a crude but I think not unhelpful analogy: 'Understanding' is a natural language concept that is our way to describe whats happening in our heads, and like most other such concepts is resistant to any clear definition and will exhibit sorites type paradoxes when one is attempted. It belongs to the presentation layer of the stack. While the process of curve fitting, however it is implemented, with whatever NN structure (like transformers) or maybe something else entirely belongs to the physical layer of the stack -- akin to frequency modulation. While I am unsure whether LLMs are really understanding, whatever that means, I think it is not difficult to believe that any form of understanding we implement will involve 'curve fitting' as a central part.

skyde 2y ago

Why not apply same concept every time a word is split into more than one token? Basically if a word contain a Prefix, suffix or root word. We could have a token position relative to the start of the word in the embedding.

skyde 2y ago
It seems it has been done before: "Syntax-Aware Transformer Models for Neural Machine Translation" by Yang et al. (2019). This model enhances the transformer architecture with syntax-aware attention mechanisms that consider dependency parse trees. Context-Aware Neural Machine Translation Learns Anaphora Resolution" by Bawden et al. (2018). This paper explores integrating context and syntax into neural machine translation models.

topherjaynes 2y ago

I went through the paper and thought immediately about how did they implement it; I missed they published their code as well. Here is the link for everyone who skimmed past it: https://github.com/mcleish7/arithmetic/tree/main https://github.com/mcleish7/arithmetic/tree/main

byt3h3ad 2y ago
my bad, should have posted it with the link itself
- topherjaynes 2y ago
  Good to start with the concept, I just had so many implementation questions. Working through the code know... which is way harder to digest.

nerdponx 2y ago

I like to see more focus on the input embeddings. It's basically the same as feature engineering in pre-deep machine learning: constructing features with high information content can significantly reduce the amount of data and computation needed to fit a useful model. And sometimes it's impossible to fit a useful model without careful feature engineering, either because the model itself is constrained in some way or because there isn't enough data or both. It's analogous to making a choice of inductive bias within the model itself. We literally could not do LLMs without the carefully-constructed transformer architecture. Why should we expect to make further progress without paying more attention to the embeddings?

wantsanagent 2y ago

I like these kinds of fixes. It's like realizing your child has vision problems and getting them glasses.

winddude 2y ago

But a calculator wouldn't be very good if it's only correct 99% of the time for arithmetic...

CyberDildonics 2y ago

I'm pretty sure getting computers to do arithmetic is not a giant hurdle.