4 ms·

LLMs are text completion engines and can't really do math. Even if it happened to know the correct answer to 9.11 - 9.9, it could still fall flat on 9.12 - 9.9

LLMs are text completion engines and can't really do math. Even if it happened to know the correct answer to 9.11 - 9.9, it could still fall flat on 9.12 - 9.9.

cableshaft 2y ago
If it doesn't already (doesn't seem like it does last time I checked), it could potentially recognize that as a mathematical equation and feed that portion of the query into a calculation engine, and then include that in its answer.
- pseudosavant 2y ago
  It is interesting that one of the best ways to deal with math problems in LLMs is to have it write Python code to solve the problem. It is good at writing that kind of fairly straight-forward Python, and Python is good at accurately being able to do math. It does mean that you need to safely implement a sandboxed Python interpreter to do the calculations though.
  - visarga 2y ago
    As a human you can hardly catch me doing a math calculation by hand. Almost always use calculators.
- jononor 2y ago
  Companies like Cohere are focusing a lot on this. It is a bit surprising if OpenAI does not already do it?
- Terretta 2y ago
  This is from current ChatGPT 4o: Q: Write parameterized Python that gives the answer to 9.11 - 9.9 and 9.12 - 9.9. Run the Python to show the answer to each pair of parameters. A: The results for the parameter pairs are as follows: - 9.11 - 9.9 = -0.79 - 9.12 - 9.9 = -0.78 [>_] The [>_] is a link. Clicking it opens a pop-up window, showing: def calculate_difference(a, b): return a - b # Define the pairs of parameters pairs = [(9.11, 9.9), (9.12, 9.9)] # Calculate and print the results for each pair results = {f"{a} - {b}": calculate_difference(a, b) for a, b in pairs} results Result: {'9.11 - 9.9': -0.7900000000000009, '9.12 - 9.9': -0.7800000000000011} // Note, if I ask it simply "compute the answer to 9.11 - 9.9", I get nonsense such as a negative number, and no Python. It doesn't "automagically" write code, it has to be nudged. If you see "Analyzing", it's fired up a sandboxed Python container.
  - cableshaft 2y ago
    Cool! I haven't checked in a while, so good to know it's at least possible somehow.
  - Slyfox33 2y ago
    I tried asking it to compare 9.11 to 9.9 using a python sandbox and it wrote the python code and then told me the result of the code would be 9.11 > 9.9 without actually running it. lol.
Sohcahtoa82 2y ago
> LLMs are text completion engines and can't really do math It blows my mind that there are people on HN that don't know this. LLMs are basically Markov Chains on a massive dose of steroids. It looks at the last $context_window tokens and decides what the next token should be. They just work differently in that a neural network creates more "fuzzy" matching than a Markov Chain, which is purely statistical.
- visarga 2y ago
  That is RNNs. LLMs (usually transformers) are non-markovian.
  - Jensson 2y ago
    > That is RNNs. LLMs (usually transformers) are non-markovian. How? They have a hard cut off context, and randomly generates next state from that. That is the definition of a markov chain. It is a markov chain with a pretty large state at every point, but still a markov chain.
fsloth 2y ago
Yes, but chatgpt will happily go browsing the web if needed or appropriate. I guess there is a reason why they haven't hooked up a symbolic math package to do the math when encountered, but feels weird it's not a feature yet.

LLMs are text completion engines and can't really do math. Even if it happened to know the correct answer to 9.11 - 9.9, it could still fall flat on 9.12 - 9.9.

cableshaft 2y ago

If it doesn't already (doesn't seem like it does last time I checked), it could potentially recognize that as a mathematical equation and feed that portion of the query into a calculation engine, and then include that in its answer.

pseudosavant 2y ago
It is interesting that one of the best ways to deal with math problems in LLMs is to have it write Python code to solve the problem. It is good at writing that kind of fairly straight-forward Python, and Python is good at accurately being able to do math. It does mean that you need to safely implement a sandboxed Python interpreter to do the calculations though.
- visarga 2y ago
  As a human you can hardly catch me doing a math calculation by hand. Almost always use calculators.
jononor 2y ago
Companies like Cohere are focusing a lot on this. It is a bit surprising if OpenAI does not already do it?
Terretta 2y ago
This is from current ChatGPT 4o: Q: Write parameterized Python that gives the answer to 9.11 - 9.9 and 9.12 - 9.9. Run the Python to show the answer to each pair of parameters. A: The results for the parameter pairs are as follows: - 9.11 - 9.9 = -0.79 - 9.12 - 9.9 = -0.78 [>_] The [>_] is a link. Clicking it opens a pop-up window, showing: def calculate_difference(a, b): return a - b # Define the pairs of parameters pairs = [(9.11, 9.9), (9.12, 9.9)] # Calculate and print the results for each pair results = {f"{a} - {b}": calculate_difference(a, b) for a, b in pairs} results Result: {'9.11 - 9.9': -0.7900000000000009, '9.12 - 9.9': -0.7800000000000011} // Note, if I ask it simply "compute the answer to 9.11 - 9.9", I get nonsense such as a negative number, and no Python. It doesn't "automagically" write code, it has to be nudged. If you see "Analyzing", it's fired up a sandboxed Python container.
- cableshaft 2y ago
  Cool! I haven't checked in a while, so good to know it's at least possible somehow.
- Slyfox33 2y ago
  I tried asking it to compare 9.11 to 9.9 using a python sandbox and it wrote the python code and then told me the result of the code would be 9.11 > 9.9 without actually running it. lol.

Sohcahtoa82 2y ago

> LLMs are text completion engines and can't really do math It blows my mind that there are people on HN that don't know this. LLMs are basically Markov Chains on a massive dose of steroids. It looks at the last $context_window tokens and decides what the next token should be. They just work differently in that a neural network creates more "fuzzy" matching than a Markov Chain, which is purely statistical.

visarga 2y ago
That is RNNs. LLMs (usually transformers) are non-markovian.
- Jensson 2y ago
  > That is RNNs. LLMs (usually transformers) are non-markovian. How? They have a hard cut off context, and randomly generates next state from that. That is the definition of a markov chain. It is a markov chain with a pretty large state at every point, but still a markov chain.

fsloth 2y ago

Yes, but chatgpt will happily go browsing the web if needed or appropriate. I guess there is a reason why they haven't hooked up a symbolic math package to do the math when encountered, but feels weird it's not a feature yet.