3 ms·
How does this compare to Nvidia Triton? It's also a dockerized inference server capable of running ONNX, TRT, and many other compiled models. Automatic request
by claytonjy 3y ago
How does this compare to Nvidia Triton? It's also a dockerized inference server capable of running ONNX, TRT, and many other compiled models. Automatic request queuing, HTTP & GRPC interfaces, configurable delays for dynamic batching, multiple models per GPU, integrations for metrics and tracing, etc. etc.
Is Inference looking to compete directly with Triton, or in a different niche (CV)? Does it make a different set of tradeoffs?
- yeldarb 3y agoInference operates at a higher layer of abstraction (tasks & architectures vs weights), targets a different audience (software engineers vs deep machine learning experts), and is focused on computer vision specifically (which informs features like its data integrations & interfaces). I could even imagine a future version of Inference using Triton as an option for its inference engine behind the scenes. Edit: I'd also be remiss to mention that NVIDIA have been great partners to Roboflow and we also have integrations with Tao Toolkit[1] & Omniverse[2]. [1] https://blog.roboflow.com/nvidia-tao-toolkit-roboflow/ https://blog.roboflow.com/nvidia-tao-toolkit-roboflow/ [2] https://developer.nvidia.com/blog/how-to-train-a-defect-detection-model-using-synthetic-data-with-nvidia-omniverse-replicator/ https://developer.nvidia.com/blog/how-to-train-a-defect-dete...