3 ms·
Is this tied to a specific framework like pytorch or an inference server like vLLM? Our inference stack is built using candle in Rust, how hard would it be to
by zackangelo 2y ago
Is this tied to a specific framework like pytorch or an inference server like vLLM?
Our inference stack is built using candle in Rust, how hard would it be to integrate?
- tovacinni 2y agoWe’d just need to write a Rust client for the daemon and load the weights in a way that is compatible with candle- we can definitely look into this since parts of what we are building is already in Rust!