2 ms·
This resource looks very bad to me as they don't check batched inference at all. This might make sense now when most people a just running single query at once,
by lostmsu 2y ago
This resource looks very bad to me as they don't check batched inference at all. This might make sense now when most people a just running single query at once, but pretty soon almost everything will be running queries in parallel to take advantage of the compute.
- menaerus 2y agoHow do you run multiple queries from multiple clients simultaneously on the same HW without affecting each other context?
- lostmsu 2y agoIt depends on the framework. Here's a LlamaSharp example: https://github.com/SciSharp/LLamaSharp/blob/master/LLama.Examples/Examples/BatchedExecutorBoolQ.cs https://github.com/SciSharp/LLamaSharp/blob/master/LLama.Exa...
- menaerus 2y agoMy question wasn't about how to run multiple queries against the LLM but rather how is it even possible from transformer architecture PoV to have a single LLM hosting multiple and different end clients. I'm probably missing something but can't figure that out yet.
- lostmsu 2y agoIf you have a branchless program, you can execute the same step of the program on multiple different inputs. https://en.wikipedia.org/wiki/SIMD https://en.wikipedia.org/wiki/SIMD