3 ms·

Thanks a lot for llm, Simon. I've started to integrate it more and more in what I do as an abstraction layer over llm apis. By the way, do you know of any mode

by fzzzy 2y ago

Thanks a lot for llm, Simon. I've started to integrate it more and more in what I do as an abstraction layer over llm apis. By the way, do you know of any models that support streaming input tokens to reduce latency to output tokens?

simonw 2y ago
I don't think I've seen one that does that. Groq is pretty amazing to play with to see what fast models are like (llm-groq plugin).
- fzzzy 2y ago
  Thanks. The problem with groq is time to first token (being in a queue, presumably).