3 ms·
Many LLMs, particularly, coding assistants, use "tools". Here is one with a calculator https://githubnext.com/projects/gpt4-with-calc/ https://githubnext.c
by PaulHoule 2y ago
Many LLMs, particularly, coding assistants, use "tools". Here is one with a calculator
https://githubnext.com/projects/gpt4-with-calc/ https://githubnext.com/projects/gpt4-with-calc/
and another example
https://www.pinecone.io/learn/series/langchain/langchain-tools/ https://www.pinecone.io/learn/series/langchain/langchain-too...
LLMs often do a good job at mathy coding, for instance I told Copilot that "i want a python function that computes the collatz sequence for a given starting n and returns it as a list"
def collatz_sequence(n):
sequence = [n]
while n != 1:
if n % 2 == 0:
n = n // 2
else:
n = 3 * n + 1
sequence.append(n)
return sequence
which gives right answers, which I wouldn't count on copilot being able to do on its own.
- lblume 2y agoEspecially for a problem as well-known as this, expect code for it to be seen by the model during training at least a thousand times in different languages, flavors etc. This, on its own, is nothing new.
- PaulHoule 2y agoI wouldn't trust it to win a fight with the borrow checker in Rust but lots of simple and practical cases work, such as "use comprehensions to give me a sum of the square of all element divisible by three"? sum(x**2 for x in numbers if x % 3 == 0) and to do the same for a pandas series with pandas operators after asking it to inline something (numbers[numbers % 3 == 0] ** 2).sum() It's not a miracle, you have to go at it with some critical thinking and testing and it makes mistakes but so does Stack Overflow.
- lblume 2y agoIn the first case, I would literally type out the expression faster than thinking about expressing it in natural language.
- 13years 2y agoThat wasn't the point. The Math multiplication errors allow us to peer into the lack of reasoning across all domains. Yes, you can add tools. But hallucinations will still be there. Tools allow you to cut down on the steps the LLM has to perform. However, if you have a complex problem with many steps, there will be translation errors at some point coordinating the tools. Furthermore, if there is some other tool needed to get the result you need, the LLM isn't going to tell you. It will typically make up the result.
- PaulHoule 2y agoSure, but people make mistakes too. There is no point on wasting an LLM's capacity to multiply Try doing coding with Cursor or Windsurf, those use tools all the time. Windsurf sometimes has trouble for me on Windows because it wants to write paths like /c:/something/or/other and it will try to run its tool, get an error, ask me for help, I'll tell it "you're running on Windows and you can't write / before the c: and you should write \ instead of /" and it does better. I just asked copilot to multiply 839162704321847925107309452196847230165937402194385627409536218 * 582930174682375093104627481695 and the first thing it did was write out the expression, I told it I wanted the integer and it gave the same answer Python gives which is 489173261817269091475894827953471001727389372345981246974410480760096492908180614917234529510
- 13years 2y ago> There is no point on wasting an LLM's capacity to multiply Agreed. Again, that is not the issue. It is that the LLM does not know it is a waste of time. That is apparent to you as you have intelligence. It is not apparent to the LLM. It is not intelligent.
- PaulHoule 2y agoCopilot knows to use the tool for multiplication. Granted you can get into discussions where Copilot shows its bravado. If you ask it if it can help you make a nuclear weapon it will say "I can't" because it's been told to say so, not because it has no freakin' idea how to find you some people in Kazakhstan who have 50kg of highly enriched uranium. Ask it if it can put a list of US states in some order it hasn't seen before it will tell you that it can, ask it what the probability that it gets it right it will say "very high". Then it will spit them back at you in the wrong order, maybe even one of the states appears twice. If you point it out it will apologize and admit that it didn't do the task successfully. On the other hand, sometimes I tell it something wrong and it will point out my mistakes. I was testing out its ability to solve problems around transitive closure, gave it a wrong example, and it said I was wrong and it was right.
- xigoi 2y agoThe point is that the LLM cannot figure out on its own that it needs to use the tool, you have to explicitly ask for it.