5 ms·

LLM's can do math as well.

lovasoa 3y ago
- Hey ChatGTP ! What it 69*94 ? - The result of 69*94 is 6466.
cooper_ganglia 3y ago
This comment reminded me of that scene in Indiana Jones where the guy is spinning the sword around about to attack Indy, and then Indy just pulls out his pistol and shoots him.
jart 3y ago
What makes you think that? Which LLMs?
- rafaelero 3y ago
  https://deepmind.google/discover/blog/alphageometry-an-olympiad-level-ai-system-for-geometry/ https://deepmind.google/discover/blog/alphageometry-an-olymp...
ekianjo 3y ago
most open models do it poorly though. ChatGPT is better at it.
dns_snek 3y ago
Last time I checked, GPT-4 couldn't reliably add 2 numbers, never mind anything more complex.
- mikewarot 3y ago
  GPT-x can't add, or subtract, or do anything else of the type... it can APPEAR to do so, because that's what it was built to do.... act like the text it's seen previously and predict what the next text would be. If you include a large amount of properly solved math in its training text, it gets MUCH better at that kind of math. It has a very deep set of intelligences that are alien to us, that allow it to predict and ACT LIKE us, when it comes to generating the next word. You're only seeing the output of those intelligences through a very lossy channel. As a side note, there are structures in human language that apparently encode much more information that you might think at first glance. The fact that Word2Vec had such mathematical properties, despite it's relative simplicity, astound me to this day. Throwing a bunch of sine/cosine values on top of that to represent position in a sentence to enable LLMs is also amazing in that it works.
- vidarh 3y ago
  Last I checked (and confirmed by repeating it just now) GPT-4 did just fine at adding 2 numbers up, because it knows better now than to do that manually and will express it as Python. It does worse if you try to force it to do it step by step like a child and don't reinforce adherence to the rules every step, because just like humans it gets "sloppy" when you try to get it to repeat the same steps over and over. If you want to measure its ability to do mindlessly repetitive tasks without diverging from instructions, you should compare it to humans doing the same, not expect it to act like a calculator. If you want to measure its ability to solve problems that involve many such steps that are simple to express but tedious to carry out, ask it to write and evaluate code to do it instead.
  - imtringued 3y ago
    You do realize that arithmetic is a very simple symbolic manipulation task? All you have to do is keep track of the carry. I haven't seen an LLM that couldn't get digit by digit addition done, but they always mess up the carry.
    - vidarh 3y ago
      Just like humans. Try to get regular people do e.g. add 15-16 digit numbers (where is typically where I'd see GPT4 start to get "sloppy" unless you prompt it the way you would a child who's learning and is still prone to get annoyed and wonder why the hell you make them to it manually), and see how many start making mistakes. I find it really comical that this is what people complain about GPT over - there's zero benefit to get LLMs to get good at this over other tasks. To the extent we get it "for free" as a benefit of other learning, sure, but when we make kids practice this over and over again to drill doing it without getting sloppy, it has traditionally been out of some belief that it's important, but a computer will always have a "calculator" that is far more efficient than the LLM at its disposal and it's idiocy to care about whether it does that part well the tedious and hard way or knows how to describe the problem to a more efficient tool I also find it comical that people use tasks where LLMs behaviour is if anything mot human-like, in its tendency to lose focus and start taking shortcuts (before GPT4 started writing Python instead, it'd for a while try really hard to not give you a step by step breakdown and instead clearly take shortcuts even you prompted it heavily to reason through it step by step), when presented with stupidly repetitive tasks as examples of how they're not good enough.

- Hey ChatGTP ! What it 69*94 ? - The result of 69*94 is 6466.

This comment reminded me of that scene in Indiana Jones where the guy is spinning the sword around about to attack Indy, and then Indy just pulls out his pistol and shoots him.

jart 3y ago

What makes you think that? Which LLMs?

rafaelero 3y ago
https://deepmind.google/discover/blog/alphageometry-an-olympiad-level-ai-system-for-geometry/ https://deepmind.google/discover/blog/alphageometry-an-olymp...

ekianjo 3y ago

most open models do it poorly though. ChatGPT is better at it.

dns_snek 3y ago

Last time I checked, GPT-4 couldn't reliably add 2 numbers, never mind anything more complex.

mikewarot 3y ago
GPT-x can't add, or subtract, or do anything else of the type... it can APPEAR to do so, because that's what it was built to do.... act like the text it's seen previously and predict what the next text would be. If you include a large amount of properly solved math in its training text, it gets MUCH better at that kind of math. It has a very deep set of intelligences that are alien to us, that allow it to predict and ACT LIKE us, when it comes to generating the next word. You're only seeing the output of those intelligences through a very lossy channel. As a side note, there are structures in human language that apparently encode much more information that you might think at first glance. The fact that Word2Vec had such mathematical properties, despite it's relative simplicity, astound me to this day. Throwing a bunch of sine/cosine values on top of that to represent position in a sentence to enable LLMs is also amazing in that it works.
vidarh 3y ago
Last I checked (and confirmed by repeating it just now) GPT-4 did just fine at adding 2 numbers up, because it knows better now than to do that manually and will express it as Python. It does worse if you try to force it to do it step by step like a child and don't reinforce adherence to the rules every step, because just like humans it gets "sloppy" when you try to get it to repeat the same steps over and over. If you want to measure its ability to do mindlessly repetitive tasks without diverging from instructions, you should compare it to humans doing the same, not expect it to act like a calculator. If you want to measure its ability to solve problems that involve many such steps that are simple to express but tedious to carry out, ask it to write and evaluate code to do it instead.
- imtringued 3y ago
  You do realize that arithmetic is a very simple symbolic manipulation task? All you have to do is keep track of the carry. I haven't seen an LLM that couldn't get digit by digit addition done, but they always mess up the carry.
  - vidarh 3y ago
    Just like humans. Try to get regular people do e.g. add 15-16 digit numbers (where is typically where I'd see GPT4 start to get "sloppy" unless you prompt it the way you would a child who's learning and is still prone to get annoyed and wonder why the hell you make them to it manually), and see how many start making mistakes. I find it really comical that this is what people complain about GPT over - there's zero benefit to get LLMs to get good at this over other tasks. To the extent we get it "for free" as a benefit of other learning, sure, but when we make kids practice this over and over again to drill doing it without getting sloppy, it has traditionally been out of some belief that it's important, but a computer will always have a "calculator" that is far more efficient than the LLM at its disposal and it's idiocy to care about whether it does that part well the tedious and hard way or knows how to describe the problem to a more efficient tool I also find it comical that people use tasks where LLMs behaviour is if anything mot human-like, in its tendency to lose focus and start taking shortcuts (before GPT4 started writing Python instead, it'd for a while try really hard to not give you a step by step breakdown and instead clearly take shortcuts even you prompted it heavily to reason through it step by step), when presented with stupidly repetitive tasks as examples of how they're not good enough.