4 ms·

What is 10 + 10? Answer: 0 + 10 = 10 + 10 = 10 + 10 = 10 + 10 = 10 + 10 = This seems like a waste of compute and time.

DarmokJalad1701 3y ago
The model may not be fine-tuned for instruct/chat.
coder543 3y ago
The linked page claims that LiteLlama scored a zero on the GSM8K benchmark, so let's just say math probably isn't its forte.
badgersnake 3y ago
Pretty sure your computer can calculate 10+10 without an LLM.
- minimaxir 3y ago
  It's a test case.
  - viraptor 3y ago
    It's a bad test case. You should know why you're choosing a given test. I hope people will understand why "solve this equation" doesn't work with LLMs and why "transform this into python code" does much better.
    - minimaxir 3y ago
      Why not both? They're distinct enough, and there's no such thing as too many distinct test cases.
      - deleted 3y ago
        [deleted]
    - Solvency 3y ago
      Why can't it transform 10 + 10 into 20 (equivalency) if it can do so with code?
      - minimaxir 3y ago
        Addition is a different problem for LLMs than code solving, since the latter is easier to memorize.
diggan 3y ago
Throwing math problems at a LLM just shows your level of understanding about the basics of LLMs. They're not trained to solve straight math calculations. I'm guess you could train one to be, I suppose, but the ones being released today are not. You could instead ask it how to calculate something, and it could give you accurate instructions for how to achieve that. Then you either perform the calculation yourself, or use something like ChatGPT that has a built in python evaluator, so it can perform the calculation. Quick example: https://chat.openai.com/share/9f76f5e5-d933-48fb-99e8-4a65306b20ad https://chat.openai.com/share/9f76f5e5-d933-48fb-99e8-4a6530...
- jackblemming 3y ago
  You've actually shown your poor understanding of LLMs. I just asked Llama-2 7b the same question and it answered perfectly fine. It did not need to use an external python interpreter or a function call, or need to be prompted with chain of thought reasoning. You're correct, LLMs are not (usually) explicitly trained to solve math calculations, this does not mean they cannot solve basic math equations (they can!).
  - viraptor 3y ago
    LLMs don't solve basic math equations. They can pattern match on some aspects, but it's not calculation. Try random numbers for the sum and you'll find examples where it fails. Especially with longer numbers with repeating digits.
    - jackblemming 3y ago
      I chose a simple math equation that should almost assuredly be in a dataset of 1 trillion tokens exactly to check its basic pattern matching skills. The token "10" and "20" are even in its hardcoded vocabulary set. I think GPT-4 might be exceeding HN discourse on ML at this point.
      - viraptor 3y ago
        I guess we have a disagreement about what "solve basic math equations" actually means.
jxy 3y ago
Just need some prompt. Give it the following Q: 1+1 A: 2 Q: 1+2 A: 3 Q: 3+2 A: 5 Q: 9+8 A: 17 Q: 10+2 A: 12 Q: 10+10 and it spits out A: 20 Q: 20+10 A: 30 Not too bad.

The model may not be fine-tuned for instruct/chat.

The linked page claims that LiteLlama scored a zero on the GSM8K benchmark, so let's just say math probably isn't its forte.

badgersnake 3y ago

Pretty sure your computer can calculate 10+10 without an LLM.

minimaxir 3y ago
It's a test case.
- viraptor 3y ago
  It's a bad test case. You should know why you're choosing a given test. I hope people will understand why "solve this equation" doesn't work with LLMs and why "transform this into python code" does much better.
  - minimaxir 3y ago
    Why not both? They're distinct enough, and there's no such thing as too many distinct test cases.
    - deleted 3y ago
      [deleted]
  - Solvency 3y ago
    Why can't it transform 10 + 10 into 20 (equivalency) if it can do so with code?
    - minimaxir 3y ago
      Addition is a different problem for LLMs than code solving, since the latter is easier to memorize.

diggan 3y ago

Throwing math problems at a LLM just shows your level of understanding about the basics of LLMs. They're not trained to solve straight math calculations. I'm guess you could train one to be, I suppose, but the ones being released today are not. You could instead ask it how to calculate something, and it could give you accurate instructions for how to achieve that. Then you either perform the calculation yourself, or use something like ChatGPT that has a built in python evaluator, so it can perform the calculation. Quick example: https://chat.openai.com/share/9f76f5e5-d933-48fb-99e8-4a65306b20ad https://chat.openai.com/share/9f76f5e5-d933-48fb-99e8-4a6530...

jackblemming 3y ago
You've actually shown your poor understanding of LLMs. I just asked Llama-2 7b the same question and it answered perfectly fine. It did not need to use an external python interpreter or a function call, or need to be prompted with chain of thought reasoning. You're correct, LLMs are not (usually) explicitly trained to solve math calculations, this does not mean they cannot solve basic math equations (they can!).
- viraptor 3y ago
  LLMs don't solve basic math equations. They can pattern match on some aspects, but it's not calculation. Try random numbers for the sum and you'll find examples where it fails. Especially with longer numbers with repeating digits.
  - jackblemming 3y ago
    I chose a simple math equation that should almost assuredly be in a dataset of 1 trillion tokens exactly to check its basic pattern matching skills. The token "10" and "20" are even in its hardcoded vocabulary set. I think GPT-4 might be exceeding HN discourse on ML at this point.
    - viraptor 3y ago
      I guess we have a disagreement about what "solve basic math equations" actually means.

jxy 3y ago

Just need some prompt. Give it the following Q: 1+1 A: 2 Q: 1+2 A: 3 Q: 3+2 A: 5 Q: 9+8 A: 17 Q: 10+2 A: 12 Q: 10+10 and it spits out A: 20 Q: 20+10 A: 30 Not too bad.