3 ms·
Is anyone using Onnx-compiled models with Triton Inference Server? Is it worth it? How does it compare to other options like torchscript or tensorrt?
by claytonjy 3y ago
Is anyone using Onnx-compiled models with Triton Inference Server? Is it worth it? How does it compare to other options like torchscript or tensorrt?
- maininformer 3y agoSuper worth it.
- machinekob 3y agoTensorRT with torchscript is a king as its a lot easier to modify, but ONNX is fine as you can also import some ONNX models to TensorRT -> https://docs.nvidia.com/deeplearning/tensorrt/api/python_api/parsers/Onnx/pyOnnx.html https://docs.nvidia.com/deeplearning/tensorrt/api/python_api... But ofc. as everything it depends on OPs version of ONNX.