4 ms·
No one uses ONNXRuntime (nor the new engine in OpenCV 5) in production. For anything performance-sensitive, one would run models under TensorRT, as an example.
by nnevatie 4mo ago
No one uses ONNXRuntime (nor the new engine in OpenCV 5) in production. For anything performance-sensitive, one would run models under TensorRT, as an example.
- gunalx 4mo agoProduction dosent have to be performance sensitive, so devex may still outcompete the performance differences in some scenarios.
- snovv_crash 4mo agoStrong statement to make when I have at least 2 datapoints contradicting it, in SaaS and embedded/robotics.
- amorroxic 4mo agoCurious on what backs this assertion. As a counterpoint we’ve been running 200+ models in production for more than 5 years - language models, embedding, classifiers, low tens to hundred M params. Traffic in the order of 1-2M requests/day and everything is enabled by onnx with some cgo (or Rust) plumbing on top. What’s your SLA?
- nnevatie 4mo agoAhh, I should have probably added some context around my hyperbole. I was referring to real-time computer vision - think of e.g. segmenting FHD/UHD video.
- pzo 4mo agohow are supposed to use TensorRT on iOS, iPadOS, Android or even Web? Production is not only cloud.
- OvervCW 4mo agoYou can use ONNXRuntime with a TensorRT backend, so one does not exclude the other.
- antonvs 4mo agoWe use this in production: https://docs.rs/onnxruntime/latest/onnxruntime/ https://docs.rs/onnxruntime/latest/onnxruntime/ It’s a Rust wrapper around ONNX Runtime. We currently serve 5+ million inference requests per day for a highly performance-sensitive application, for a long list of major enterprise clients. We don’t use GPUs for inference, because it would be cost-prohibitive. We launch tens of thousands of VMs per day to run these workloads.
- monster_truck 4mo agoI've never understood how anyone comes into contact with it and thinks its anything more than an incredible inconvenience masked as the easy way of doing things. Given it a few good shakes for various uses and regretted the time spent each time
- cik 4mo agoUmmm embedded robotics is all about this. For years.
- dTal 4mo agoOpenTrack uses it for its AI headtracking, which works extremely well.