3 ms·
It looks like the key insight here is to have the LLM generate its own tools (as in GPT/Claude tool calling) via Python code generation and apply cosine similar
by throwup238 2y ago
It looks like the key insight here is to have the LLM generate its own tools (as in GPT/Claude tool calling) via Python code generation and apply cosine similarity RAG to select which tools are available at each step using the tool description and the problem/step while using recent history to error correct.
The agent starts with some human created tooling like a tool to read the file system or create another tool using python code, then starts accumulating custom Python functions it wrote itself with tool calling metadata like descriptions and input/output types. Every time it reaches a step, if it doesn't find a relevant tool, it creates a new one. Apparently this improves performance on complex tasks (via GAIA benchmark) with diminishing returns on simpler tasks.
- IanCal 2y agoI played around making these things before, it's a fun exercise. Interesting to see that's where things may be heading. My example was asking for a poem about the headlines (good example of info they don't have, and something that's very hard to do mechanically). https://news.ycombinator.com/item?id=37015591 https://news.ycombinator.com/item?id=37015591
- deleted 2y ago[deleted]
- llm_trw 2y agoI ended up training a bert on nothing but python for the embedding search. The results were crap. Then I used an llm to write a new docstring for each class/function definition in the training data and the results were better than state of the art. There's so much wide open space to explore. It's a shame that everyone is wasting their time with the biggest possible models they can afford.
- digdugdirk 2y agoDo you have any more detailed info on this process? I've played around with using LLMs, but nothing in the training realm. I'd love to see a writeup or guide to the process you used there.
- llm_trw 2y agoNo and it won't do you much good even if I did. The tools have broken again since then - thanks tensorflow data loaders - and my code only works against a version of python that's no longer supported in LTS Ubuntu/Debian10+. I have been mulling about running a subscription service where you get up to date code that works on topics like the above. If you're interested drop me a line at my profile email and I'll add you to a mailing list when/if I ever get around to doing it.
- mountainriver 2y agoThis is what Voyager did awhile back, it’s interesting but I think only part of the answer
- thom 2y agoSeems like you could go further than this with something like DSPy and start evaluating which tools contribute to successful outcomes. Funny how much things start to look like Eurisko the more time goes on.