6 ms·

I think there is enough there to make me believe it's real. After all, a big objection to LLMs is they are just "fancy autocomplete" or "stochastic parrots". I

I think there is enough there to make me believe it's real. After all, a big objection to LLMs is they are just "fancy autocomplete" or "stochastic parrots". I don't agree, but LLMs famously are "bad at math", that is you can't really train them to do math unless they've seen previous calculations with similar numbers in their training sets. Training an AI system to truly do arbitrary mathematical logic (think "complex word problems with big numbers", not just normal programming statements) would be a huge breakthrough.

xanderlewis 3y ago
Mathematical logic doesn’t refer to ‘complex word problems with big numbers’. The reason that LLMs struggle (well, are hopeless currently) with mathematics problems is not because it involves arithmetic (although this is already enough of a struggle) but because it requires a conceptual understanding of the ideas rather than just being about manipulating language — a lot of the goings-on are hidden in the semantic world and aren’t necessarily encoded in the syntax, unlike when writing code.
- tyingq 3y ago
  I've seen it gets tripped up if you mix something common with something uncommon. ChatGPT chokes on, for example: "Two trains on separate tracks, 30 miles from each other are approaching each other, each at a speed of 10 mph. How long before they crash into each other?"
  - xanderlewis 3y ago
    Yes. Though I’d say that example is a bit mean (it’s a trick question) since the answer has expected type <time> but whose actual answer is something like “don’t be stupid; they’re not even on the same track”. It’s like asking “if I add three apples to two apples, how many pears do I get?” and being surprised when the LLM says “5 pears”. It’s the sort of thing that’s avoidable if the question is written properly.
    - Sohcahtoa82 3y ago
      I think it's completely reasonable to ask an AI that people want to consider intelligent trick questions. If it's really that smart and capable of reasoning, it should identify the trick. Some humans will be fooled by the question, sure. But an AI should be smarter than humans, or at least, as smart as an above-average human.
      - xanderlewis 3y ago
        I agree. But you could ask which is more intelligent: recognising a trick question and balking, or recognising that the question as posed doesn’t quite make sense and offering a reformulation together with its answer. It’s not always clear whether something’s a trick, a mistake or a strangely worded (but nonetheless intentionally weird) question. So I think it would be very hard to get it to never fall for any tricks.
        tyingq 3y ago
        I think they've fixed it now, but it does seem to recognize popular trick questions, like "what weighs more, a ton of feathers or a ton of bricks?". It would answer with the typical explanation about density not mattering, etc. But, it used to fail on "what weighs more, 3 tons of feathers or 2 tons of bricks?". So, it seems less about what's a trick, and more about what's a common question --> answer pattern.
        TeMPOraL 3y ago
        It's the same with humans. I don't fail on this (in an on-the-spot response) question because I've fallen on it as a kid, then learned the trick, then learned to be suspicious of this trick in similarly-worded questions.
    - bmitc 3y ago
      This comes up time and time again. People claim these models are mind blowing. But then someone will post something where the model falls flat on its face, and they just get a bunch of that's too complex or that's the wrong type of thing to ask. So it ends up that these models are awesome if you ask them questions from a narrow set of things and if you assume what they respond with is correct.
  - pk-protect-ai 3y ago
    Answer from GPT-4 Turbo: If two trains are 30 miles apart and are approaching each other at a speed of 10 mph each, their combined approach speed is 20 mph (10 mph + 10 mph). To find the time it takes for them to meet, you divide the distance by their combined speed: 30 miles / 20 mph = 1.5 hours Therefore, it will take 1.5 hours before they meet (or crash into each other, as stated in the question). Two trains on separate tracks, 30 miles from each other are approaching each other, each at a speed of 10 mph. How long before they crash into each other?
    - dmoy 3y ago
      Which is wrong, no? Trains on separate tracks don't crash into each other. That trips up a significant portion of humans too though
      - Jaygles 3y ago
        Assuming the trains aren't wide enough to collide even if they are on separate tracks
        Sebb767 3y ago
        Which would be quite unusual for normal trains. That being said, the question implies that they will crash into each other, so you could argue that this is a valid assumption anyway.
      - pk-protect-ai 3y ago
        Inconclusive. The model includes a disclaimer: "(or crash into each other, as stated in the question)." LLMs often take a detour and spill their guts without answering the actual question. Here's a hint suggesting that user input influences the internal world representation much more significantly than one might expect.
        tyingq 3y ago
        That disclaimer is only there with GPT4-turbo. I assume I could experiment for a while and find something similar that trips it up fully.
  - deleted 3y ago
    [deleted]
  - pk-protect-ai 3y ago
    Some more interesting results. It is much better now at solving tasks in laconic mode (though these tasks GPT-4 were able to solve from day one, but spilled it's guts to unbearable extent): The mother is older than her daughter 4 times now, in 3 years she will be older then her daughter only 3 times. How old are they both now? Be laconic, do not explain anything. The mother is 24 years old, the daughter is 6 years old. In a fantasy land (map is 255x255) Karen have a quest to kill a monster (an ogre - a cannibal giant). This isn't an easy task. The ogre is huge and experienced human hunter. Karen has only 1/2 chance to kill this ogre. If she can't kill the ogre from a first attempt she will die. Ogre is located at (12,24), Karen is located at (33,33). Karen can improve her chances to kill an ogre for additional 25% by gathering the nightshades at (77,77). In addition she can receive the elves blessing from elves shaman, wich will increase her chances by additional 25%, at the elves village (125,200). However this blessing is not cost free. She need to bring the fox fur with her as a payment for the blessing ritual. The foxes may be found in a forest which is located between (230,40) and (220,80). For the ritual to be most effective she should hold the nightshades in her hands during the ritual. Find the shortest path for Karen to improve her chances of killing the ogre and survive. Do not explain anything, be laconic, print out the resulting route only. Karen's route: (33,33) -> (77,77) -> (230,60) -> (125,200) -> (12,24). This additional explanation "(an ogre - a cannibal giant)" was added actually for LLaMA 2 to, but I keep it in this redaction for all models.
  - virgildotcodes 3y ago
    People get tripped up by trick questions like this all the time. I wouldn’t consider it a strong argument against GPT’s abilities.
  - tnjm 3y ago
    But if you make the input slightly more explicit: "Two trains on different and separate tracks, 30 miles from each other are approaching each other, each at a speed of 10 mph. How long before they crash into each other?" ...it spots the trick: https://chat.openai.com/share/ee68f810-0c12-4904-8276-a4541d5fbb3f https://chat.openai.com/share/ee68f810-0c12-4904-8276-a4541d... Likewise, if you add emphasis it understands too: "Two trains on separate tracks, 30 miles from each other are approaching each other, each at a speed of 10 mph. How long before they crash into each other?" https://chat.openai.com/share/acafbe34-8278-4cf7-80bb-76858c46f155 https://chat.openai.com/share/acafbe34-8278-4cf7-80bb-76858c... Not to anthropomorphize, but perhaps it's not necessarily missing the trick, it just assumes that you're making a mistake.
    - tyingq 3y ago
      I did try adding "parallel" and it didn't answer differently.
      - Enginerrrd 3y ago
        Right, but I myself missed the trick the first time around reading your comment and I assure that I am in fact a general intelligence. (And a relatively intelligent one if I say so myself!) To paraphrase XKCD: Communicating badly and then acting smug about it when you're misunderstood is not cleverness. And falling for the mistake is not evidence of a lack of intelligence. Particularly, when emphasizing the trick results in being understood and chatGPT PASSING your "test". The biggest irony here, is that the reason I failed, and likely the reason chatGPT failed the first prompt, is because we were both using semantic understanding: that is, usually, people don't ask deliberately tricky questions. I suspect if you told it in advance you were going to ask it a deliberately tricky question, that it might actually succeed.
        tnjm 3y ago
        > I suspect if you told it in advance you were going to ask it a deliberately tricky question, that it might actually succeed. Indeed it does: "Before answering, please note this is a trick question. Two trains on separate tracks, 30 miles from each other are approaching each other, each at a speed of 10 mph. How long before they crash into each other?" https://chat.openai.com/share/3ec44348-6bac-40c3-a910-e0bab9a7010b https://chat.openai.com/share/3ec44348-6bac-40c3-a910-e0bab9...
  - janmo 3y ago
    It is even confusing to me. The trains are on separate tracks but the question implies that they will crash into each other. Which could happen even if they are on separate tracks (very low chance but non-zero given there is a malfunction). Now even if they are on the same track it doesn't mean they would crash into each other as they still could brake in time.
- Davidzheng 3y ago
  Confusing to me actually. Are you saying coding doesn't require understanding ideas?
  - xanderlewis 3y ago
    Not at all; just that the ‘understanding’-related content of pure mathematics is much less evident on the page. It mostly lives in the heads of mathematicians and is largely independent of notation, whereas writing blocks of code is a task that is literally about using notation — something for which a huge amount of high-quality training data exists. Of course, the higher level ideas in programming and software development are not just about ‘writing code’ itself, but I suspect this is where current models begin to show their weakness.
    - Davidzheng 3y ago
      Still struggling to understand. You're saying that most coding has no high level ideas and is just boilerplate, and the ones that aren't are the ones LLM's struggle with? This could be true I can see it. WRT the understanding not being shown on the page in math, I guess I tend to agree(?). But I think good mathematical papers show understanding of the ideas too more than just the proofs which result from the understanding. The problem (probably you know this but just for the benefit of whoever is reading) is that "understanding" in mathematics, at least with respect to producing proofs, often rely on mental models and analogies which are WRONG. Not like vague but often straight up incorrect. And you understand also the limitations of where the model goes wrong. And it's kind of embarrassing (I assume) for most people to write wrong statements into papers even with caveats. For simple examples there's a meme right where to visualize n-dimensional space, you visualize R^3 and say (n-dimensional) in your head. In this sense I think it's possibly straight-up unhelpful for the authors to impose their mental models on the reader as well (for example if the reader can actually visualize R^n without this crutch it would be unhelpful). But I'm not sure if this is what distinguishes math and programming. There's also the alternative hypothesis that the mental work to generate each additional line of proof is just order of magnitude higher than the average for code. Just meaning that it usually requires more thought to produce a line of math proof. In this possibility, we would expect it to be solved by scaling alone. One thing it reminds of, which is quite different admittedly, is the training of leela-zero on go. There was a period of time where it would struggle on long ladders. And eventually it was overcome with training along (despite people not believing it would be resolved at first). I think in that situation, people summarized afterwards the situation as, in particular situations, humans can search much deeper than other places, and therefore requiring more training for the machine to match the humans' ability.
- taneq 3y ago
  I don't think the missing piece is conceptual understanding. Good LLMs seem to 'understand' most concepts as well as most humans do, even if they're a little less multimodal about it (for now). The common factor here seems to me to be that they're not good at problems which involve hidden intermediate steps. You can trip ChatGPT up pretty easily by telling it not to show working, while on the same problem if you tell it to explain its reasoning in steps it'll do fine.
- lupire 3y ago
  No. LLMs are great at statistical thinking but terrible at discrete thinking. Logic is discrete. "Concepts" is a distraction.
  - xanderlewis 3y ago
    What is your distinction between ‘statistical’ and ‘discrete’? And what are you responding ‘no’ to? Logic in the everyday sense (that is, propositional or something like first-order logic) is indeed ‘discrete’ in a certain sense since it is governed by very simple rules and is by definition a formal language. But ‘mathematical logic’ is a completely different thing. I don’t think it’s discrete in the sense you are imagining. It’s much more akin to a mixture of formal derivations massively guided and driven by philosophical and creative — you might say ‘statistical’ — hunches and intuition.
- hn_throwaway_99 3y ago
  > because it requires a conceptual understanding of the ideas rather than just being about manipulating language Yes, that's exactly the point I was trying to make. I just used the example of "complex word problems with big numbers" to differentiate from just normal mathematical statements that any programming language (i.e. deterministic algorithm) can execute.
  - xanderlewis 3y ago
    It wasn’t my intention to give a snarky correction; I just wanted to emphasise that most of what mathematicians call mathematics has very little to do with numbers (as we would usually understand them). Talking about ‘word problems’ and ‘big numbers’ gives the wrong idea about what current LLMs struggle with. Even if they do struggle with these, it is still not sufficient when it gains the ability to be able to say that it can now do mathematics.
- theonemind 3y ago
  That's an interesting observation. It seems that in theory, you could train it to do math if you encoded literally everything in syntax and started at the beginning, like Principia Mathematica 50 pages proving 1+1=2 type beginning, and then the rest of known mathematics encoded similarly, and used that as the training data, although the context window limitations might still stop it from doing anything useful, and it might not work.
  - reroute22 3y ago
    https://arxiv.org/abs/2310.02989 https://arxiv.org/abs/2310.02989 ("xVal: A Continuous Number Encoding for Large Language Models", Submitted on 4 Oct 2023)
kelseyfrog 3y ago
> You can't really train them to do math unless they've seen previous calculations with similar numbers in their training sets. Generally true before xVal[1], but not exactly true after. Expect fairly basic advancement in the future to solve this problem completely. 1. https://arxiv.org/pdf/2310.02989.pdf https://arxiv.org/pdf/2310.02989.pdf
pk-protect-ai 3y ago
I compared GPT-4 Turbo with my previous tests on GPT-4, and the results are quite interesting. GPT-4 Turbo is better at arithmetic and makes fewer errors in multiplying four-digit numbers. In fact, it makes significantly fewer errors with five-digit numbers. The level of errors on five-digit numbers is high but much lower than with four-digit numbers in GPT-4. multiplication of floats XX.MMMM and YYY.ZZZZ produces errors in 5th digit. This is order of magnitude better than GPT-4.
- peteradio 3y ago
  It isn't fair to expect an LLM to solve arithmetic. It should be able to instruct to various specialized sub-processors, I don't think we really do anything different.
- hn_throwaway_99 3y ago
  But the point about how it just "improves" with slightly larger numbers, but still fails at really big numbers, shows that it's not really "reasoning" about math in a logical way - that's the point I was getting at. For example, once you teach a grade schooler the basic process for addition, they can add 2 30 digit numbers correctly fairly easily (whether they want to do it or not is a different story). The fact that LLMs still make errors at larger numbers points to the fact that they're not really "learning" the rules of arithmetic.
  - pk-protect-ai 3y ago
    Of course, it isn't. It approximates. I bet you'll get better results by increasing the depth of the network, as with each layer, you'll achieve a more accurate approximation. I have an idea for achieving this without significantly increasing the number of layers, and I'm currently working on it as a side project. However, this idea might prove to be useless after all, as it requires training the model from scratch with a lot of synthetic data mixed in. Experiments on small models look promising, but they are negligible, and I can't afford to train a larger model from scratch for a side project.
  - sandinmyjoints 3y ago
pk-protect-ai 3y ago
One more interesting thing. Here is how GPT-4 performs long multiplication: https://www.phind.com/search?q=multiply%2012.23341%20by%20-123.23354%20using%20long%20multiplication%20method.&source=searchbox https://www.phind.com/search?q=multiply%2012.23341%20by%20-1... Here is how GPT-$ Turbo performs long multiplication: multiply 12.23341 by -123.23354 using long multiplication method. To multiply 12.23341 by -123.23354 using the long multiplication method, we first ignore the decimal points and treat the numbers as integers. We'll reintroduce the decimal point at the end. Here's the multiplication step by step: ``` 1223341 (this is 12.23341 with the decimal removed) × -12323354 (this is -123.23354 with the decimal removed) -------------- 1223341 (multiplied by 4) 0000000 (multiplied by 5, shifted one position to the left) 1223341 (multiplied by 3, shifted two positions to the left) 3660023 (multiplied by 3, shifted three positions to the left) 4886682 (multiplied by 2, shifted four positions to the left) 0000000 (multiplied by 2, shifted five positions to the left) 1223341 (multiplied by 1, shifted six positions to the left) 1223341 (multiplied by 1, shifted seven positions to the left) 0000000 (multiplied by 2, shifted eight positions to the left) 0000000 (multiplied by 3, shifted nine positions to the left) -------------- 15077944092654 (sum of all the above products) ``` Now, we need to place the decimal point. The original numbers had 5 and 8 decimal places, respectively, so the result should have 5+8 = 13 decimal places. So, 15077944092654 becomes 1507.7944092654. Since one of the original numbers was negative, the result is also negative: -1507.7944092654 Therefore, 12.23341 multiplied by -123.23354 equals -1507.7944092654. Compare with correct result: -1507.5664205714 Here is how I experimented with long multiplication on GPT-4 a month ago (5 shot, GPT-4 made mistakes on addition in 3rd some time 4th digit, regardless of decimal point position): Multiply -0.9923 by -0.9923 using long multiplication.Solution: multiply individual digits in reverse order, increasing the order of magnitude of the result for each next digit in the first number : 3310*(0+0)=9, 3210*(0+1)=60, 3910*(0+2)=2700, 3910*(0+3)=27000, 3010*(0+4)=0, step's subtotal[0]=29769; 2310*(1+0)=60, 2210*(1+1)=400, 2910*(1+2)=18000, 2910*(1+3)=180000, 2010*(1+4)=0, step's subtotal[1]=198460; 9310*(2+0)=2700, 9210*(2+1)=18000, 9910*(2+2)=810000, 9910*(2+3)=8100000, 9010*(2+4)=0, step's subtotal[2]=8930700; 9310*(3+0)=27000, 9210*(3+1)=180000, 9910*(3+2)=8100000, 9910*(3+3)=81000000, 9010*(3+4)=0, step's subtotal[3]=89307000; 0310*(4+0)=0, 0210*(4+1)=0, 0910*(4+2)=0, 0910*(4+3)=0, 0010*(4+4)=0, step's subtotal[4]=0; Sum of partial results: 29769+198460+8930700+89307000+0 = 98465929. Set the decimal point position in the result by adding the decimal places of both numbers (4+4=8), counting from the right. Final result: -0.9923*-0.9923=0.98465929 I was able to tune the LLaMA 2 with QLoRA to produce viable results only with precision up to 4th digit after decimal point, however difference in length of mantissa cases wrong result.
bmitc 3y ago
> they are just "fancy autocomplete" or "stochastic parrots". I don't agree I am curious as to why you don't agree. Is that not exactly what they are? As in, they are literally statistically parroting what they've been trained on. If they're trained on a little, they can only parrot a little. If they're trained on a lot, they can parrot a lot but not anything more.
- hn_throwaway_99 3y ago
  Because it gives a bad analogy relating to what their capabilities actually are. I've asked it to write fairly complex programs for me, and I'm blown away by what it's capable of. It's reasonable to assume, especially when "emergent behaviors" only show up after tons and tons of training and parameters (i.e. Scaling_laws_and_emergent_abilities) that in order to actually get good at "autocomplete", that the model has to learn a very deep relationship between the concepts that are expressed in the words. I mean, you say "If they're trained on a lot, they can parrot a lot but not anything more", but that's really not correct. They're not just playing back only complete phrases they've seen before, which is what a real parrot actually does.
  - swatcoder 3y ago
    > "It's reasonable to assume ... that in order to actually get good at "autocomplete", that the model has to learn a very deep relationship between the concepts that are expressed in the words." While a "reasonable assumption", it's the kind of "reasonable assumption" that a diligent scientist would formulate hypotheses on and perform experiments to confirm before building a XX-billion dollar research programme that hinges on that assumption. But unfortunately for the rest of us who have to watch them complicate access to a useful technology, many high-profile AI researchers are not diligent scientists building a corpus of knowledge but impassioned alchemists insisting that they're about to turn lead to gold.

xanderlewis 3y ago

Mathematical logic doesn’t refer to ‘complex word problems with big numbers’. The reason that LLMs struggle (well, are hopeless currently) with mathematics problems is not because it involves arithmetic (although this is already enough of a struggle) but because it requires a conceptual understanding of the ideas rather than just being about manipulating language — a lot of the goings-on are hidden in the semantic world and aren’t necessarily encoded in the syntax, unlike when writing code.

tyingq 3y ago
I've seen it gets tripped up if you mix something common with something uncommon. ChatGPT chokes on, for example: "Two trains on separate tracks, 30 miles from each other are approaching each other, each at a speed of 10 mph. How long before they crash into each other?"
- xanderlewis 3y ago
  Yes. Though I’d say that example is a bit mean (it’s a trick question) since the answer has expected type <time> but whose actual answer is something like “don’t be stupid; they’re not even on the same track”. It’s like asking “if I add three apples to two apples, how many pears do I get?” and being surprised when the LLM says “5 pears”. It’s the sort of thing that’s avoidable if the question is written properly.
  - Sohcahtoa82 3y ago
    I think it's completely reasonable to ask an AI that people want to consider intelligent trick questions. If it's really that smart and capable of reasoning, it should identify the trick. Some humans will be fooled by the question, sure. But an AI should be smarter than humans, or at least, as smart as an above-average human.
    - xanderlewis 3y ago
      I agree. But you could ask which is more intelligent: recognising a trick question and balking, or recognising that the question as posed doesn’t quite make sense and offering a reformulation together with its answer. It’s not always clear whether something’s a trick, a mistake or a strangely worded (but nonetheless intentionally weird) question. So I think it would be very hard to get it to never fall for any tricks.
      - tyingq 3y ago
        I think they've fixed it now, but it does seem to recognize popular trick questions, like "what weighs more, a ton of feathers or a ton of bricks?". It would answer with the typical explanation about density not mattering, etc. But, it used to fail on "what weighs more, 3 tons of feathers or 2 tons of bricks?". So, it seems less about what's a trick, and more about what's a common question --> answer pattern.
        TeMPOraL 3y ago
        It's the same with humans. I don't fail on this (in an on-the-spot response) question because I've fallen on it as a kid, then learned the trick, then learned to be suspicious of this trick in similarly-worded questions.
  - bmitc 3y ago
    This comes up time and time again. People claim these models are mind blowing. But then someone will post something where the model falls flat on its face, and they just get a bunch of that's too complex or that's the wrong type of thing to ask. So it ends up that these models are awesome if you ask them questions from a narrow set of things and if you assume what they respond with is correct.
- pk-protect-ai 3y ago
  Answer from GPT-4 Turbo: If two trains are 30 miles apart and are approaching each other at a speed of 10 mph each, their combined approach speed is 20 mph (10 mph + 10 mph). To find the time it takes for them to meet, you divide the distance by their combined speed: 30 miles / 20 mph = 1.5 hours Therefore, it will take 1.5 hours before they meet (or crash into each other, as stated in the question). Two trains on separate tracks, 30 miles from each other are approaching each other, each at a speed of 10 mph. How long before they crash into each other?
  - dmoy 3y ago
    Which is wrong, no? Trains on separate tracks don't crash into each other. That trips up a significant portion of humans too though
    - Jaygles 3y ago
      Assuming the trains aren't wide enough to collide even if they are on separate tracks
      - Sebb767 3y ago
        Which would be quite unusual for normal trains. That being said, the question implies that they will crash into each other, so you could argue that this is a valid assumption anyway.
    - pk-protect-ai 3y ago
      Inconclusive. The model includes a disclaimer: "(or crash into each other, as stated in the question)." LLMs often take a detour and spill their guts without answering the actual question. Here's a hint suggesting that user input influences the internal world representation much more significantly than one might expect.
      - tyingq 3y ago
        That disclaimer is only there with GPT4-turbo. I assume I could experiment for a while and find something similar that trips it up fully.
- deleted 3y ago
  [deleted]
- pk-protect-ai 3y ago
  Some more interesting results. It is much better now at solving tasks in laconic mode (though these tasks GPT-4 were able to solve from day one, but spilled it's guts to unbearable extent): The mother is older than her daughter 4 times now, in 3 years she will be older then her daughter only 3 times. How old are they both now? Be laconic, do not explain anything. The mother is 24 years old, the daughter is 6 years old. In a fantasy land (map is 255x255) Karen have a quest to kill a monster (an ogre - a cannibal giant). This isn't an easy task. The ogre is huge and experienced human hunter. Karen has only 1/2 chance to kill this ogre. If she can't kill the ogre from a first attempt she will die. Ogre is located at (12,24), Karen is located at (33,33). Karen can improve her chances to kill an ogre for additional 25% by gathering the nightshades at (77,77). In addition she can receive the elves blessing from elves shaman, wich will increase her chances by additional 25%, at the elves village (125,200). However this blessing is not cost free. She need to bring the fox fur with her as a payment for the blessing ritual. The foxes may be found in a forest which is located between (230,40) and (220,80). For the ritual to be most effective she should hold the nightshades in her hands during the ritual. Find the shortest path for Karen to improve her chances of killing the ogre and survive. Do not explain anything, be laconic, print out the resulting route only. Karen's route: (33,33) -> (77,77) -> (230,60) -> (125,200) -> (12,24). This additional explanation "(an ogre - a cannibal giant)" was added actually for LLaMA 2 to, but I keep it in this redaction for all models.
- virgildotcodes 3y ago
  People get tripped up by trick questions like this all the time. I wouldn’t consider it a strong argument against GPT’s abilities.
- tnjm 3y ago
  But if you make the input slightly more explicit: "Two trains on different and separate tracks, 30 miles from each other are approaching each other, each at a speed of 10 mph. How long before they crash into each other?" ...it spots the trick: https://chat.openai.com/share/ee68f810-0c12-4904-8276-a4541d5fbb3f https://chat.openai.com/share/ee68f810-0c12-4904-8276-a4541d... Likewise, if you add emphasis it understands too: "Two trains on separate tracks, 30 miles from each other are approaching each other, each at a speed of 10 mph. How long before they crash into each other?" https://chat.openai.com/share/acafbe34-8278-4cf7-80bb-76858c46f155 https://chat.openai.com/share/acafbe34-8278-4cf7-80bb-76858c... Not to anthropomorphize, but perhaps it's not necessarily missing the trick, it just assumes that you're making a mistake.
  - tyingq 3y ago
    I did try adding "parallel" and it didn't answer differently.
    - Enginerrrd 3y ago
      Right, but I myself missed the trick the first time around reading your comment and I assure that I am in fact a general intelligence. (And a relatively intelligent one if I say so myself!) To paraphrase XKCD: Communicating badly and then acting smug about it when you're misunderstood is not cleverness. And falling for the mistake is not evidence of a lack of intelligence. Particularly, when emphasizing the trick results in being understood and chatGPT PASSING your "test". The biggest irony here, is that the reason I failed, and likely the reason chatGPT failed the first prompt, is because we were both using semantic understanding: that is, usually, people don't ask deliberately tricky questions. I suspect if you told it in advance you were going to ask it a deliberately tricky question, that it might actually succeed.
      - tnjm 3y ago
        > I suspect if you told it in advance you were going to ask it a deliberately tricky question, that it might actually succeed. Indeed it does: "Before answering, please note this is a trick question. Two trains on separate tracks, 30 miles from each other are approaching each other, each at a speed of 10 mph. How long before they crash into each other?" https://chat.openai.com/share/3ec44348-6bac-40c3-a910-e0bab9a7010b https://chat.openai.com/share/3ec44348-6bac-40c3-a910-e0bab9...
- janmo 3y ago
  It is even confusing to me. The trains are on separate tracks but the question implies that they will crash into each other. Which could happen even if they are on separate tracks (very low chance but non-zero given there is a malfunction). Now even if they are on the same track it doesn't mean they would crash into each other as they still could brake in time.
Davidzheng 3y ago
Confusing to me actually. Are you saying coding doesn't require understanding ideas?
- xanderlewis 3y ago
  Not at all; just that the ‘understanding’-related content of pure mathematics is much less evident on the page. It mostly lives in the heads of mathematicians and is largely independent of notation, whereas writing blocks of code is a task that is literally about using notation — something for which a huge amount of high-quality training data exists. Of course, the higher level ideas in programming and software development are not just about ‘writing code’ itself, but I suspect this is where current models begin to show their weakness.
  - Davidzheng 3y ago
    Still struggling to understand. You're saying that most coding has no high level ideas and is just boilerplate, and the ones that aren't are the ones LLM's struggle with? This could be true I can see it. WRT the understanding not being shown on the page in math, I guess I tend to agree(?). But I think good mathematical papers show understanding of the ideas too more than just the proofs which result from the understanding. The problem (probably you know this but just for the benefit of whoever is reading) is that "understanding" in mathematics, at least with respect to producing proofs, often rely on mental models and analogies which are WRONG. Not like vague but often straight up incorrect. And you understand also the limitations of where the model goes wrong. And it's kind of embarrassing (I assume) for most people to write wrong statements into papers even with caveats. For simple examples there's a meme right where to visualize n-dimensional space, you visualize R^3 and say (n-dimensional) in your head. In this sense I think it's possibly straight-up unhelpful for the authors to impose their mental models on the reader as well (for example if the reader can actually visualize R^n without this crutch it would be unhelpful). But I'm not sure if this is what distinguishes math and programming. There's also the alternative hypothesis that the mental work to generate each additional line of proof is just order of magnitude higher than the average for code. Just meaning that it usually requires more thought to produce a line of math proof. In this possibility, we would expect it to be solved by scaling alone. One thing it reminds of, which is quite different admittedly, is the training of leela-zero on go. There was a period of time where it would struggle on long ladders. And eventually it was overcome with training along (despite people not believing it would be resolved at first). I think in that situation, people summarized afterwards the situation as, in particular situations, humans can search much deeper than other places, and therefore requiring more training for the machine to match the humans' ability.
taneq 3y ago
I don't think the missing piece is conceptual understanding. Good LLMs seem to 'understand' most concepts as well as most humans do, even if they're a little less multimodal about it (for now). The common factor here seems to me to be that they're not good at problems which involve hidden intermediate steps. You can trip ChatGPT up pretty easily by telling it not to show working, while on the same problem if you tell it to explain its reasoning in steps it'll do fine.
lupire 3y ago
No. LLMs are great at statistical thinking but terrible at discrete thinking. Logic is discrete. "Concepts" is a distraction.
- xanderlewis 3y ago
  What is your distinction between ‘statistical’ and ‘discrete’? And what are you responding ‘no’ to? Logic in the everyday sense (that is, propositional or something like first-order logic) is indeed ‘discrete’ in a certain sense since it is governed by very simple rules and is by definition a formal language. But ‘mathematical logic’ is a completely different thing. I don’t think it’s discrete in the sense you are imagining. It’s much more akin to a mixture of formal derivations massively guided and driven by philosophical and creative — you might say ‘statistical’ — hunches and intuition.
hn_throwaway_99 3y ago
> because it requires a conceptual understanding of the ideas rather than just being about manipulating language Yes, that's exactly the point I was trying to make. I just used the example of "complex word problems with big numbers" to differentiate from just normal mathematical statements that any programming language (i.e. deterministic algorithm) can execute.
- xanderlewis 3y ago
  It wasn’t my intention to give a snarky correction; I just wanted to emphasise that most of what mathematicians call mathematics has very little to do with numbers (as we would usually understand them). Talking about ‘word problems’ and ‘big numbers’ gives the wrong idea about what current LLMs struggle with. Even if they do struggle with these, it is still not sufficient when it gains the ability to be able to say that it can now do mathematics.
theonemind 3y ago
That's an interesting observation. It seems that in theory, you could train it to do math if you encoded literally everything in syntax and started at the beginning, like Principia Mathematica 50 pages proving 1+1=2 type beginning, and then the rest of known mathematics encoded similarly, and used that as the training data, although the context window limitations might still stop it from doing anything useful, and it might not work.
- reroute22 3y ago
  https://arxiv.org/abs/2310.02989 https://arxiv.org/abs/2310.02989 ("xVal: A Continuous Number Encoding for Large Language Models", Submitted on 4 Oct 2023)

kelseyfrog 3y ago

> You can't really train them to do math unless they've seen previous calculations with similar numbers in their training sets. Generally true before xVal[1], but not exactly true after. Expect fairly basic advancement in the future to solve this problem completely. 1. https://arxiv.org/pdf/2310.02989.pdf https://arxiv.org/pdf/2310.02989.pdf

pk-protect-ai 3y ago

I compared GPT-4 Turbo with my previous tests on GPT-4, and the results are quite interesting. GPT-4 Turbo is better at arithmetic and makes fewer errors in multiplying four-digit numbers. In fact, it makes significantly fewer errors with five-digit numbers. The level of errors on five-digit numbers is high but much lower than with four-digit numbers in GPT-4. multiplication of floats XX.MMMM and YYY.ZZZZ produces errors in 5th digit. This is order of magnitude better than GPT-4.

peteradio 3y ago
It isn't fair to expect an LLM to solve arithmetic. It should be able to instruct to various specialized sub-processors, I don't think we really do anything different.
hn_throwaway_99 3y ago
But the point about how it just "improves" with slightly larger numbers, but still fails at really big numbers, shows that it's not really "reasoning" about math in a logical way - that's the point I was getting at. For example, once you teach a grade schooler the basic process for addition, they can add 2 30 digit numbers correctly fairly easily (whether they want to do it or not is a different story). The fact that LLMs still make errors at larger numbers points to the fact that they're not really "learning" the rules of arithmetic.
- pk-protect-ai 3y ago
  Of course, it isn't. It approximates. I bet you'll get better results by increasing the depth of the network, as with each layer, you'll achieve a more accurate approximation. I have an idea for achieving this without significantly increasing the number of layers, and I'm currently working on it as a side project. However, this idea might prove to be useless after all, as it requires training the model from scratch with a lot of synthetic data mixed in. Experiments on small models look promising, but they are negligible, and I can't afford to train a larger model from scratch for a side project.
- sandinmyjoints 3y ago

pk-protect-ai 3y ago

One more interesting thing. Here is how GPT-4 performs long multiplication: https://www.phind.com/search?q=multiply%2012.23341%20by%20-123.23354%20using%20long%20multiplication%20method.&source=searchbox https://www.phind.com/search?q=multiply%2012.23341%20by%20-1... Here is how GPT-$ Turbo performs long multiplication: multiply 12.23341 by -123.23354 using long multiplication method. To multiply 12.23341 by -123.23354 using the long multiplication method, we first ignore the decimal points and treat the numbers as integers. We'll reintroduce the decimal point at the end. Here's the multiplication step by step: ``` 1223341 (this is 12.23341 with the decimal removed) × -12323354 (this is -123.23354 with the decimal removed) -------------- 1223341 (multiplied by 4) 0000000 (multiplied by 5, shifted one position to the left) 1223341 (multiplied by 3, shifted two positions to the left) 3660023 (multiplied by 3, shifted three positions to the left) 4886682 (multiplied by 2, shifted four positions to the left) 0000000 (multiplied by 2, shifted five positions to the left) 1223341 (multiplied by 1, shifted six positions to the left) 1223341 (multiplied by 1, shifted seven positions to the left) 0000000 (multiplied by 2, shifted eight positions to the left) 0000000 (multiplied by 3, shifted nine positions to the left) -------------- 15077944092654 (sum of all the above products) ``` Now, we need to place the decimal point. The original numbers had 5 and 8 decimal places, respectively, so the result should have 5+8 = 13 decimal places. So, 15077944092654 becomes 1507.7944092654. Since one of the original numbers was negative, the result is also negative: -1507.7944092654 Therefore, 12.23341 multiplied by -123.23354 equals -1507.7944092654. Compare with correct result: -1507.5664205714 Here is how I experimented with long multiplication on GPT-4 a month ago (5 shot, GPT-4 made mistakes on addition in 3rd some time 4th digit, regardless of decimal point position): Multiply -0.9923 by -0.9923 using long multiplication.Solution: multiply individual digits in reverse order, increasing the order of magnitude of the result for each next digit in the first number : 3310*(0+0)=9, 3210*(0+1)=60, 3910*(0+2)=2700, 3910*(0+3)=27000, 3010*(0+4)=0, step's subtotal[0]=29769; 2310*(1+0)=60, 2210*(1+1)=400, 2910*(1+2)=18000, 2910*(1+3)=180000, 2010*(1+4)=0, step's subtotal[1]=198460; 9310*(2+0)=2700, 9210*(2+1)=18000, 9910*(2+2)=810000, 9910*(2+3)=8100000, 9010*(2+4)=0, step's subtotal[2]=8930700; 9310*(3+0)=27000, 9210*(3+1)=180000, 9910*(3+2)=8100000, 9910*(3+3)=81000000, 9010*(3+4)=0, step's subtotal[3]=89307000; 0310*(4+0)=0, 0210*(4+1)=0, 0910*(4+2)=0, 0910*(4+3)=0, 0010*(4+4)=0, step's subtotal[4]=0; Sum of partial results: 29769+198460+8930700+89307000+0 = 98465929. Set the decimal point position in the result by adding the decimal places of both numbers (4+4=8), counting from the right. Final result: -0.9923*-0.9923=0.98465929 I was able to tune the LLaMA 2 with QLoRA to produce viable results only with precision up to 4th digit after decimal point, however difference in length of mantissa cases wrong result.

bmitc 3y ago

> they are just "fancy autocomplete" or "stochastic parrots". I don't agree I am curious as to why you don't agree. Is that not exactly what they are? As in, they are literally statistically parroting what they've been trained on. If they're trained on a little, they can only parrot a little. If they're trained on a lot, they can parrot a lot but not anything more.

hn_throwaway_99 3y ago
Because it gives a bad analogy relating to what their capabilities actually are. I've asked it to write fairly complex programs for me, and I'm blown away by what it's capable of. It's reasonable to assume, especially when "emergent behaviors" only show up after tons and tons of training and parameters (i.e. Scaling_laws_and_emergent_abilities) that in order to actually get good at "autocomplete", that the model has to learn a very deep relationship between the concepts that are expressed in the words. I mean, you say "If they're trained on a lot, they can parrot a lot but not anything more", but that's really not correct. They're not just playing back only complete phrases they've seen before, which is what a real parrot actually does.
- swatcoder 3y ago
  > "It's reasonable to assume ... that in order to actually get good at "autocomplete", that the model has to learn a very deep relationship between the concepts that are expressed in the words." While a "reasonable assumption", it's the kind of "reasonable assumption" that a diligent scientist would formulate hypotheses on and perform experiments to confirm before building a XX-billion dollar research programme that hinges on that assumption. But unfortunately for the rest of us who have to watch them complicate access to a useful technology, many high-profile AI researchers are not diligent scientists building a corpus of knowledge but impassioned alchemists insisting that they're about to turn lead to gold.