4 ms·

Hey Everyone! I lead the development of Neuropod. Happy to answer any questions There's also a blog post that has more detail: https://eng.uber.com/introducing

Hey Everyone! I lead the development of Neuropod. Happy to answer any questions There's also a blog post that has more detail: https://eng.uber.com/introducing-neuropod/ https://eng.uber.com/introducing-neuropod/ Super excited to open-source it!

jaredtn 6y ago
What will the continued support for this project be, given that Uber has shuttered their AI Labs?
- tourist_on_road 6y ago
  From the looks of it. It is from uber ATG not from AI labs. I believe those are two different orgs. Someone from uber can clarify
- mysterEFrank 6y ago
  I'm a former Uber AI Labs member - they're different orgs. Begs the question though, who at Uber will use this now?
  - mysterEFrank 6y ago
    (EDIT there are many projects at Uber using neuropods)
    - mysterEFrank 6y ago
      Also it's great.
- vpanyam 6y ago
  Neuropod was created at Uber ATG (not AI Labs) and powers hundreds of models across the company (ATG and the core business). It's been used in production for over a year and we're continuing to actively work on it. The blog post I linked above goes into more detail, but here's a relevant quote about usage within Uber: > Neuropod has been instrumental in quickly deploying new models at Uber since its internal release in early 2019. Over the last year, we have deployed hundreds of Neuropod models across Uber ATG, Uber AI, and the core Uber business. These include models for demand forecasting, estimated time of arrival (ETA) prediction for rides, menu transcription for Uber Eats, and object detection models for self-driving vehicles.
m0zg 6y ago
TensorRT offers significant advantages wrt inference and it takes ONNX files. Best I can tell this does not have a TensorRT backend (https://github.com/uber/neuropod/search?q=nvinfer.h&unscoped_q=nvinfer.h https://github.com/uber/neuropod/search?q=nvinfer.h&unscoped...). Why not?
- emef 6y ago
  I wrote our internal lightweight version of neuropod at another SDC startup where we did use TensorRT. Our ML researchers worked in pytorch and more often than not, the pytorch -> onnx -> tensorrt conversion did not work. We ended up needing to replicate the network architecture using the tensorrt library and manually convert the weights from pytorch. Then we'd use the tensorrt serialization to compile the models so they could be run in c++. I imagine that they may have tried this in neuropod and saw the same conversion problems. TensorRT was a big investment to get running smoothly but it did shave off 20% or so off our inference latency
  - m0zg 6y ago
    It's gotten better in TensorRT7. I'm using it quite successfully. It does have a lot of corner cases though, that much is true, and the documentation is really poor, which, coupled with it being mostly closed source, severely limits adoption. That said, I'm getting ridiculously good performance with it, even without using the TensorCores.
- vpanyam 6y ago
  Adding backends for TensorRT, ONNX, JAX, etc are on our TODO list (and we'd love to see PRs to add support for these and others)! We actually do use TensorRT with several of our models, but our approach is generally to do all TRT related processing before the Neuropod export step. For example, we might do something like TF model -> TF-TRT optimization -> Neuropod export or PyTorch model -> (convert subset of model to a torchscript engine) -> PyTorch model + custom op to run TRT engine -> TorchScript model + custom op to run TRT engine -> Neuropod export Since Neuropod wraps the underlying model (including custom ops), this approach works well for us.
yshvrdhn 6y ago
any possible support for tensorrt ?

jaredtn 6y ago

What will the continued support for this project be, given that Uber has shuttered their AI Labs?

tourist_on_road 6y ago
From the looks of it. It is from uber ATG not from AI labs. I believe those are two different orgs. Someone from uber can clarify
mysterEFrank 6y ago
I'm a former Uber AI Labs member - they're different orgs. Begs the question though, who at Uber will use this now?
- mysterEFrank 6y ago
  (EDIT there are many projects at Uber using neuropods)
  - mysterEFrank 6y ago
    Also it's great.
vpanyam 6y ago
Neuropod was created at Uber ATG (not AI Labs) and powers hundreds of models across the company (ATG and the core business). It's been used in production for over a year and we're continuing to actively work on it. The blog post I linked above goes into more detail, but here's a relevant quote about usage within Uber: > Neuropod has been instrumental in quickly deploying new models at Uber since its internal release in early 2019. Over the last year, we have deployed hundreds of Neuropod models across Uber ATG, Uber AI, and the core Uber business. These include models for demand forecasting, estimated time of arrival (ETA) prediction for rides, menu transcription for Uber Eats, and object detection models for self-driving vehicles.

m0zg 6y ago

TensorRT offers significant advantages wrt inference and it takes ONNX files. Best I can tell this does not have a TensorRT backend (https://github.com/uber/neuropod/search?q=nvinfer.h&unscoped_q=nvinfer.h https://github.com/uber/neuropod/search?q=nvinfer.h&unscoped...). Why not?

emef 6y ago
I wrote our internal lightweight version of neuropod at another SDC startup where we did use TensorRT. Our ML researchers worked in pytorch and more often than not, the pytorch -> onnx -> tensorrt conversion did not work. We ended up needing to replicate the network architecture using the tensorrt library and manually convert the weights from pytorch. Then we'd use the tensorrt serialization to compile the models so they could be run in c++. I imagine that they may have tried this in neuropod and saw the same conversion problems. TensorRT was a big investment to get running smoothly but it did shave off 20% or so off our inference latency
- m0zg 6y ago
  It's gotten better in TensorRT7. I'm using it quite successfully. It does have a lot of corner cases though, that much is true, and the documentation is really poor, which, coupled with it being mostly closed source, severely limits adoption. That said, I'm getting ridiculously good performance with it, even without using the TensorCores.
vpanyam 6y ago
Adding backends for TensorRT, ONNX, JAX, etc are on our TODO list (and we'd love to see PRs to add support for these and others)! We actually do use TensorRT with several of our models, but our approach is generally to do all TRT related processing before the Neuropod export step. For example, we might do something like TF model -> TF-TRT optimization -> Neuropod export or PyTorch model -> (convert subset of model to a torchscript engine) -> PyTorch model + custom op to run TRT engine -> TorchScript model + custom op to run TRT engine -> Neuropod export Since Neuropod wraps the underlying model (including custom ops), this approach works well for us.

yshvrdhn 6y ago

any possible support for tensorrt ?