3 ms·
Sounds like something that could be implemented with llm-d, though I've not experimented with it. https://llm-d.ai/blog/intelligent-inference-scheduling-with-l
by jochalek 1y ago
Sounds like something that could be implemented with llm-d, though I've not experimented with it.
https://llm-d.ai/blog/intelligent-inference-scheduling-with-llm-d https://llm-d.ai/blog/intelligent-inference-scheduling-with-...