3 ms·
Kyanite: NN inference library, in/for Rust, using CPU or Nvidia GPUs
- galangalalgol 3y agoAll these inference libraries are interesting. * Kyanite * Tract * Wonnx * onnxruntime bindings * candle So far, I've only compared the bindings with tract for cpu inference on x86_64 (not tract's main focus), and the bindings won by 30x. Converting the same onnx models to ggml to load into candle would be interesting. For gpu inference, with nvidia at least, I'd guess everything but wonnx would come out the same, as they are all using cuda under there. Wonnx is using wglsl so I imagine there is a pretty big price for that portability. Tract seems like the right answer if you are using embedded arm. For general cpu inference, my current guess would be the bindings for onnxruntime, but maybe this or candle could win?