4 ms·
They can do exactly that, it's called Tool Use and nearly all modern models can handle it. For example, I have a consumer GPU that can run a R1 Qwen distill, w
by luma 2y ago
They can do exactly that, it's called Tool Use and nearly all modern models can handle it. For example, I have a consumer GPU that can run a R1 Qwen distill, which, when prompted for a large multiplication, will elect to write a python script to find the answer.
This is a table stakes feature for even the open/free models today.
- deleted 2y ago[deleted]
- 13years 2y agoYou still can't get 100% reliability that would be necessary for certain problem domains. There are going to be some level of hallucination errors in the translation to the agent or code. If it is a complex problem, those will compound.
- zeroxfe 2y agoYou can't get 100% reliability from a human either.
- 13years 2y agoWhich is why humans use calculators. That is the key point being made secondary to the reliability. The LLM "knows" it is bad at Math. It knows the purpose of calculators. However, doesn't use this information to inform the user. It could also propose to the user it could write the answer using code. It doesn't do that either.
- lern_too_spel 2y agoIt has been prompted to give an answer to the user. If you prompted a human to give an answer, they would do the same thing. This example is awful.
- amelius 2y agoYes, but the takeaway message is that even laypeople can invent how LLMs should work.
- xg15 2y agoThe author adressss this and argues that it misses the point: > Now, some might interject here and say we could, of course, train the LLM to ask for a calculator. However, that would not make them intelligent. Humans require no training at all for calculators, as they are such intuitive instruments. We use them simply because we have understanding for the capability they provide. So the real question behind the headline is why LLMs don't learn to ask for a calculator by themselves, if both the the definition of a calculator and the fact that LLMs are bad at math are part of the training data.
- luma 2y agoI’d contest the statement that humans don’t need to be trained to use a calculator. It certainly isn’t instinctive behavior.
- neom 2y agoI have dyscalculia and I still have no clue about calculators except I was taught how to make it give me the answer to math problems, I'm a bit embarrassed to say: even still I sometimes take a few seconds to boot into being able to use one. We often discuss LLMs like there is no divergence in humans, I don't know how many people math is intuitive for, but I know plenty of people like me.
- 13years 2y agoI first used a calculator as a kid. Took about 30 seconds. Never had instruction or training. We aren't talking about scientific calculators.
- doubled112 2y agoYeah, the buttons do what the symbol says. Then about 30 seconds after that somebody showed me I could spell “boobs” if I flipped it upside down.
- recursive 2y ago> Humans require no training at all for calculators, as they are such intuitive instruments I don't think that's even true though. If you think this, I would suggest you've just internalized your training on the subject.
- roger_ 2y agoBasic math problems should be treated as a special vs general tool use. Why use Python when the computer evaluating the LLM can do math itself?
- mhh__ 2y agoBecause AI programmers only know python (is the path dependency / real reason)
- luma 2y agoIt does do the math itself, it creates the code to do so and then executes that code to get the answer.
- mystified5016 2y agoThe same reason you don't use your desk calculator to run Python scripts on huge datasets. It's the wrong tool. Running an LLM and parsing and computing mathematical expressions are entirely disjoint operations. You need highly specialized code for each, it makes just as much sense to put a calculator in your LLM as it does to stuff a Python interpreter in a calculator. Could you? Of course, software is infinitely flexible. Does it make sense to do it? No, it makes more sense to connect two different specialized applications than to try shoehorning one into the other.
- eternityforest 2y agoBut they only know about specific tools with few or zero shot methods, whereas modern humans are highly trained to specifically always route all math through a calculator, it's "in the fine tune" not "in the prompt". It doesn't use up any "context" to remember that calculators exist. With the partial exception of Python code, models know how to solve problems with code and even 1.5B models do amazing things if you prompt them with something like "Use the Z3 solver"
- IanCal 2y agoBut if you give them a search they'll use that to look for tools and if you give them the ability they can write new tools for themselves.
- eternityforest 2y agoBut for tiny models on CPUs, that kind of thing doesn't seem practical, it seems like you get at most one LLM call to a 360M model with maybe a few paragraphs of context, before a user will probably be tired of waiting.