5 ms·

Show HN: Pip install inference, open source computer vision deployment

71 points by zerojames 3y ago

Deploying vision models is time consuming and tedious. Setting up dependencies. Fixing conflicts. Configuring TRT acceleration. Flashing (and re-flashing) NVIDIA Jetsons. A streamlined, developer-friendly solution for inference is needed. We, the Roboflow team, have been hard at work open sourcing Inference, an open source vision deployment solution. Our solution is designed with developers in mind, offering a HTTP-based interface. Run models on your hardware without having to write architecture-specific inference code. Here's a demo showing how to go from a model to GPU inference on a video of a football game in ~10 minutes: https://www.youtube.com/watch?v=at-yuwIMiN4 https://www.youtube.com/watch?v=at-yuwIMiN4 Inference powers millions of daily API calls for global sports broadcasts, one of the world’s largest railways, a leading electric car manufacturer, and multiple other Fortune 500 companies, along with countless hackers’ hobby and research projects. Inference works in Docker and supports CPU (ARM and x86), NVIDIA GPU, and TRT. Inference manages dependencies and the environment. All you need to do is make HTTP requests to the server. YOLOv5, YOLOv8, YOLACT, CLIP, SAM, and other popular vision models are supported (some models need to be hosted on Roboflow first, see the docs; we're working on bring your own model weights!). Try it out and tell us what you think!

thangngoc89 3y ago
Is it possible to run this without supplying Roboflow API key? Why the requirement?
- yeldarb 3y ago
  Yes, but also not easily right now. We’ll be adding more supported ways to load models soon, but right now things largely assume using our model registry (and API Key is how it validates who is authorized to use which models). The “why” is because this has been an internal project for years that we just open sourced, and while it was internal this wasn’t needed. But we’ll add it soon! (PRs also welcome; we’re happy to talk through how that might be architected if you start an issue/discussion on the repo.)
nl 3y ago
Looks interesting. Missing a trick by not having some images in the README though. Especially since this post here is where I found out it can run CLIP instead of the readme.
- yeldarb 3y ago
  Thanks for the suggestion! Definitely agree, we’ve seen that work extremely well for Supervision[1] and Autodistill[2], some of our other open source projects. There’s still a lot of polish like this we need to do; we’ve spent most of our effort cleaning up the code and documentation to prep for open sourcing the repo. Next step is improving the usability of the pip pathway (the native Python interface was just added; the http server was all we had for internal use). Then we’re going to focus on improving the content and expanding the models it supports. [1] https://github.com/roboflow/supervision https://github.com/roboflow/supervision [2] https://github.com/autodistill/autodistill https://github.com/autodistill/autodistill
- zerojames 3y ago
  I have added in a video showing inference into the README -- thank you for the feedback!
  - noman-land 3y ago
    Thanks. The video blew my mind.
Aspos 3y ago
Interested, but after spending 2 min reading it I am still not sure what I am looking at exactly.
- zerojames 3y ago
  Thank you for your comment! Inference is all about reducing the friction associated with deployment. Set up a Docker container (which works on a range of architectures, with TRT acceleration), load your model (or use SAM or CLIP, which come out of the box), then you can start sending HTTP requests to get predictions. I'd love to know more about how we can improve the docs (I'm a big documentarian; feedback on docs is always sincerely appreciated).
  - Aspos 3y ago
    Genuine feedback: I guess your readme.md describes the HOW well, but does not touch the WHY. Why should I use this? Why not?
    - yeldarb 3y ago
      We'll work on adding this to the repo, but I'll try to summarize here. Why should I use this? If you're looking to deploy computer vision models to production (on edge devices or self-hosting in the cloud), Roboflow Inference is the easiest way to get up and running quickly & to iteratively improve your model's performance over time. It's also in very active use with many real-world customers in production. That means we've ironed out (and are continuing to find and fix) many of the bugs and edge-cases that you'd invariably encounter building something on your own or using something less mature. When should you _not_ use this? For everything else. * If you're prototyping & hacking on core model primitives there are likely too many layers of abstraction & you should wait to productionize the model until you're happy with the underlying architecture. * If you're not doing computer vision you're probably better off with something focused on the specific needs of LLMs or other types of models. * If you need the absolute highest speed possible you may want to use lower-level tools like DeepStream and Triton (for now). Out of the box we sacrifice a bit of speed to get a better interface & end-user experience. We do have a TensorRT provider that sacrifices some of that convenience for speed & are working on additional optimizations (including potentially integrating with DeepStream/Triton behind the scenes). Edit: @zerojames updated the README.
jvanillaaaa 3y ago
This is sick
pkiv 3y ago
Looks great! Congrats on the launch.
claytonjy 3y ago
How does this compare to Nvidia Triton? It's also a dockerized inference server capable of running ONNX, TRT, and many other compiled models. Automatic request queuing, HTTP & GRPC interfaces, configurable delays for dynamic batching, multiple models per GPU, integrations for metrics and tracing, etc. etc. Is Inference looking to compete directly with Triton, or in a different niche (CV)? Does it make a different set of tradeoffs?
- yeldarb 3y ago
  Inference operates at a higher layer of abstraction (tasks & architectures vs weights), targets a different audience (software engineers vs deep machine learning experts), and is focused on computer vision specifically (which informs features like its data integrations & interfaces). I could even imagine a future version of Inference using Triton as an option for its inference engine behind the scenes. Edit: I'd also be remiss to mention that NVIDIA have been great partners to Roboflow and we also have integrations with Tao Toolkit[1] & Omniverse[2]. [1] https://blog.roboflow.com/nvidia-tao-toolkit-roboflow/ https://blog.roboflow.com/nvidia-tao-toolkit-roboflow/ [2] https://developer.nvidia.com/blog/how-to-train-a-defect-detection-model-using-synthetic-data-with-nvidia-omniverse-replicator/ https://developer.nvidia.com/blog/how-to-train-a-defect-dete...
tlack 3y ago
Love & recommend Roboflow. Thanks for releasing this yall!

71 points by zerojames 3y ago

thangngoc89 3y ago

Is it possible to run this without supplying Roboflow API key? Why the requirement?

yeldarb 3y ago
Yes, but also not easily right now. We’ll be adding more supported ways to load models soon, but right now things largely assume using our model registry (and API Key is how it validates who is authorized to use which models). The “why” is because this has been an internal project for years that we just open sourced, and while it was internal this wasn’t needed. But we’ll add it soon! (PRs also welcome; we’re happy to talk through how that might be architected if you start an issue/discussion on the repo.)

nl 3y ago

Looks interesting. Missing a trick by not having some images in the README though. Especially since this post here is where I found out it can run CLIP instead of the readme.

yeldarb 3y ago
Thanks for the suggestion! Definitely agree, we’ve seen that work extremely well for Supervision[1] and Autodistill[2], some of our other open source projects. There’s still a lot of polish like this we need to do; we’ve spent most of our effort cleaning up the code and documentation to prep for open sourcing the repo. Next step is improving the usability of the pip pathway (the native Python interface was just added; the http server was all we had for internal use). Then we’re going to focus on improving the content and expanding the models it supports. [1] https://github.com/roboflow/supervision https://github.com/roboflow/supervision [2] https://github.com/autodistill/autodistill https://github.com/autodistill/autodistill
zerojames 3y ago
I have added in a video showing inference into the README -- thank you for the feedback!
- noman-land 3y ago
  Thanks. The video blew my mind.

Aspos 3y ago

Interested, but after spending 2 min reading it I am still not sure what I am looking at exactly.

zerojames 3y ago
Thank you for your comment! Inference is all about reducing the friction associated with deployment. Set up a Docker container (which works on a range of architectures, with TRT acceleration), load your model (or use SAM or CLIP, which come out of the box), then you can start sending HTTP requests to get predictions. I'd love to know more about how we can improve the docs (I'm a big documentarian; feedback on docs is always sincerely appreciated).
- Aspos 3y ago
  Genuine feedback: I guess your readme.md describes the HOW well, but does not touch the WHY. Why should I use this? Why not?
  - yeldarb 3y ago
    We'll work on adding this to the repo, but I'll try to summarize here. Why should I use this? If you're looking to deploy computer vision models to production (on edge devices or self-hosting in the cloud), Roboflow Inference is the easiest way to get up and running quickly & to iteratively improve your model's performance over time. It's also in very active use with many real-world customers in production. That means we've ironed out (and are continuing to find and fix) many of the bugs and edge-cases that you'd invariably encounter building something on your own or using something less mature. When should you _not_ use this? For everything else. * If you're prototyping & hacking on core model primitives there are likely too many layers of abstraction & you should wait to productionize the model until you're happy with the underlying architecture. * If you're not doing computer vision you're probably better off with something focused on the specific needs of LLMs or other types of models. * If you need the absolute highest speed possible you may want to use lower-level tools like DeepStream and Triton (for now). Out of the box we sacrifice a bit of speed to get a better interface & end-user experience. We do have a TensorRT provider that sacrifices some of that convenience for speed & are working on additional optimizations (including potentially integrating with DeepStream/Triton behind the scenes). Edit: @zerojames updated the README.

jvanillaaaa 3y ago

This is sick

pkiv 3y ago

Looks great! Congrats on the launch.

claytonjy 3y ago

How does this compare to Nvidia Triton? It's also a dockerized inference server capable of running ONNX, TRT, and many other compiled models. Automatic request queuing, HTTP & GRPC interfaces, configurable delays for dynamic batching, multiple models per GPU, integrations for metrics and tracing, etc. etc. Is Inference looking to compete directly with Triton, or in a different niche (CV)? Does it make a different set of tradeoffs?

yeldarb 3y ago
Inference operates at a higher layer of abstraction (tasks & architectures vs weights), targets a different audience (software engineers vs deep machine learning experts), and is focused on computer vision specifically (which informs features like its data integrations & interfaces). I could even imagine a future version of Inference using Triton as an option for its inference engine behind the scenes. Edit: I'd also be remiss to mention that NVIDIA have been great partners to Roboflow and we also have integrations with Tao Toolkit[1] & Omniverse[2]. [1] https://blog.roboflow.com/nvidia-tao-toolkit-roboflow/ https://blog.roboflow.com/nvidia-tao-toolkit-roboflow/ [2] https://developer.nvidia.com/blog/how-to-train-a-defect-detection-model-using-synthetic-data-with-nvidia-omniverse-replicator/ https://developer.nvidia.com/blog/how-to-train-a-defect-dete...

tlack 3y ago

Love & recommend Roboflow. Thanks for releasing this yall!