3 ms·
The Gemini API runs the Python code for you as part of your single API call, without you having to handle the tool call request yourself.
by simonw 2y ago
The Gemini API runs the Python code for you as part of your single API call, without you having to handle the tool call request yourself.
- tempaccount420 2y agoThis is so much cheaper than re-prompting each tool use. I wish this was extended to things like: you could give the model an API endpoint that it can call to execute JS code, and the only requirement is that your API has to respond within 5 seconds (maybe less actually). I wonder if this is what OpenAI is planning to do in the upcoming API update to support tools in o3.
- danpalmer 2y agoI imagine there wouldn’t bd much of a cost to the provider on the API call there so much longer times may be possible. It’s not like this would hold up the LLM in any way, execution would get suspended while the call is made and the TPU/GPU will serve another request.
- suchar 2y agoThey need to keep KV cache to avoid prompt reprocessing, so they would need to move it to ram/nvme during longer api calls to use gpu for another request