4 ms·

Good to have a gVisor developer on here. At Dropbox one way we use secure containers is to run machine learning models. Do you know if TensorFlow works in a gVi

Good to have a gVisor developer on here. At Dropbox one way we use secure containers is to run machine learning models. Do you know if TensorFlow works in a gVisor container? How is GPU support in the container? If running on a CPU, are BLAS libraries supported to speed up matrix math in the container? Finally, do you know if OpenCV currently runs in gVisor containers?

flx42_ 8y ago
nvidia-docker[1] maintainer here. Curious to know, are you using docker today? If yes, is there anything missing to satisfy your security requirements? [1] https://github.com/NVIDIA/nvidia-docker https://github.com/NVIDIA/nvidia-docker
- bradneuberg 8y ago
  We aren't using Docker today; we have jailing infrastructure using linux cgroups, namespaces, seclist, etc.
- deleted 8y ago
  [deleted]
dgacmu 8y ago
Tensorflow security geek here: TF works in gVisor. I strongly recommend a VM solution when dealing with GPUs. Haven't tried GPU in gVisor because the size of the attack surface against the GPU device driver is so large. Depends on your level of paranoia, though. For CPU, gVisor is great. You can absolutely use BLAS libraries. Eigen (the TF default) works for sure. I don't see any reason MKL wouldn't, but I haven't personally tested it.
- bradneuberg 8y ago
  Yeah, its still unclear to me how to create a security boundary for using GPUs at inference time in containers, or even in VMs. Any thoughts?
  - dgacmu 8y ago
    Sadly, at least without violating NVidia's license agreement, you can't without a Tesla-series GPU. The consumer drivers won't let you put them in a VM using an IOMMU, which is the "right" way to do it. Your main options are a Tesla P40 if you've got six grand to drop, which you can happily stuff in a VM and use the IOMMU to guarantee isolation for, or trying to massively optimize for your host CPU. Fortunately, with a lot of inference tasks, CPU isn't too bad. If you can get your batch sizes up, using MKL or MKL-DNN is a quite reasonable option on a decent Intel CPU. Installing TF with MKL-DNN is pretty easy these days, too (https://www.tensorflow.org/performance/performance_guide#tensorflow_with_intel_mkl_dnn https://www.tensorflow.org/performance/performance_guide#ten... ). That would honestly be my first try if you need serious isolation on a budget. You can map a GPU into a general container, it's just... not very good security. Any flaw in the (very large, complex) nvidia binary blob will leave you exposed to potential container escapes (or outright root). This really depends on your threat model. Putting it in a container is better than handing someone the root ssh keys to your server, it's just not very satisfying if you're serious about security. If you're just accepting untrusted, e.g., CSV or image input from a customer and then running your own trusted model on it, you could half-a## it and do the format decode and some validation and initial processing in a gVisor sandbox, and then pass that to your own trusted process that has GPU access. It doesn't protect you against all exploits, and if you were doing it at Google, I'd tell you not to do that, but if you have less to lose it may be an acceptable middle ground. It's kinda complicated, though, and would incur decent data copy overhead This is one area where using a cloud hosted service is appealing, 'cause they've bought the datacenter GPUs or TPUs or whateverPUs for you and handled the isolation story. (Disclaimer - as you probably gathered, I also work part time at Google, frequently on tensorflow and cloudml security, but this is all my opinion.)
    - throwaway9d0291 8y ago
      > The consumer drivers won't let you put them in a VM using an IOMMU, which is the "right" way to do it. For Windows guests at least, the workaround is pretty easy. You just prevent KVM from identifying itself. I have a gaming machine with a GTX1080Ti running in KVM at home. http://vfio.blogspot.ch/2014/08/vfiovga-faq.html http://vfio.blogspot.ch/2014/08/vfiovga-faq.html question 10 has the answers.

flx42_ 8y ago

nvidia-docker[1] maintainer here. Curious to know, are you using docker today? If yes, is there anything missing to satisfy your security requirements? [1] https://github.com/NVIDIA/nvidia-docker https://github.com/NVIDIA/nvidia-docker

bradneuberg 8y ago
We aren't using Docker today; we have jailing infrastructure using linux cgroups, namespaces, seclist, etc.
deleted 8y ago
[deleted]

dgacmu 8y ago

Tensorflow security geek here: TF works in gVisor. I strongly recommend a VM solution when dealing with GPUs. Haven't tried GPU in gVisor because the size of the attack surface against the GPU device driver is so large. Depends on your level of paranoia, though. For CPU, gVisor is great. You can absolutely use BLAS libraries. Eigen (the TF default) works for sure. I don't see any reason MKL wouldn't, but I haven't personally tested it.

bradneuberg 8y ago
Yeah, its still unclear to me how to create a security boundary for using GPUs at inference time in containers, or even in VMs. Any thoughts?
- dgacmu 8y ago
  Sadly, at least without violating NVidia's license agreement, you can't without a Tesla-series GPU. The consumer drivers won't let you put them in a VM using an IOMMU, which is the "right" way to do it. Your main options are a Tesla P40 if you've got six grand to drop, which you can happily stuff in a VM and use the IOMMU to guarantee isolation for, or trying to massively optimize for your host CPU. Fortunately, with a lot of inference tasks, CPU isn't too bad. If you can get your batch sizes up, using MKL or MKL-DNN is a quite reasonable option on a decent Intel CPU. Installing TF with MKL-DNN is pretty easy these days, too (https://www.tensorflow.org/performance/performance_guide#tensorflow_with_intel_mkl_dnn https://www.tensorflow.org/performance/performance_guide#ten... ). That would honestly be my first try if you need serious isolation on a budget. You can map a GPU into a general container, it's just... not very good security. Any flaw in the (very large, complex) nvidia binary blob will leave you exposed to potential container escapes (or outright root). This really depends on your threat model. Putting it in a container is better than handing someone the root ssh keys to your server, it's just not very satisfying if you're serious about security. If you're just accepting untrusted, e.g., CSV or image input from a customer and then running your own trusted model on it, you could half-a## it and do the format decode and some validation and initial processing in a gVisor sandbox, and then pass that to your own trusted process that has GPU access. It doesn't protect you against all exploits, and if you were doing it at Google, I'd tell you not to do that, but if you have less to lose it may be an acceptable middle ground. It's kinda complicated, though, and would incur decent data copy overhead This is one area where using a cloud hosted service is appealing, 'cause they've bought the datacenter GPUs or TPUs or whateverPUs for you and handled the isolation story. (Disclaimer - as you probably gathered, I also work part time at Google, frequently on tensorflow and cloudml security, but this is all my opinion.)
  - throwaway9d0291 8y ago
    > The consumer drivers won't let you put them in a VM using an IOMMU, which is the "right" way to do it. For Windows guests at least, the workaround is pretty easy. You just prevent KVM from identifying itself. I have a gaming machine with a GTX1080Ti running in KVM at home. http://vfio.blogspot.ch/2014/08/vfiovga-faq.html http://vfio.blogspot.ch/2014/08/vfiovga-faq.html question 10 has the answers.