3 ms·
> TL;DR: Giving LLMs the full output of tool calls is costly and slow. Is this true for all tool calls? Even if the tool returns little data?
by koakuma-chan 1y ago
> TL;DR: Giving LLMs the full output of tool calls is costly and slow.
Is this true for all tool calls? Even if the tool returns little data?
- fullstackchris 1y agofrom my experience its about the speed of a very competant human - one of my favorite custom tools ive written is just access to a series of bash commands - havent tested with others but claude very quickly browses through files, reads them, and so on to do whatever it was you prompted. But even then it is all contextual - for example, I had to remove 'find' because as one would expect, running 'find' against a huge directory set is very slow!
- koakuma-chan 1y agoWell, the bottleneck there would usually be the LLM, because, e.g., a tool to inspect a filesystem directory would be very fast, and it wouldn't necessarily return a lot of data, so I am confused what this article is really trying to say.