Shameless plug, this is why I built Mighty Inference Server [0]. It wraps ONNX Runtime in a production ready rust binary, and can be installed quickly and pain

Shameless plug, this is why I built Mighty Inference Server [0]. It wraps ONNX Runtime in a production ready rust binary, and can be installed quickly and painlessly on most machines. The sentence transformer models in the example notebook from the OP will infer queries in <10ms, which is fast enough for a production customer facing service…and lets you run on commodity instances without expensive GPUs [0] https://max.io https://max.io