4 ms·

Since I see that some of the developers are in this thread, I'll post my question to them here. What are your plans to deal with the overhead and nastiness of

by catern 8y ago

Since I see that some of the developers are in this thread, I'll post my question to them here. What are your plans to deal with the overhead and nastiness of ptrace? Beyond the performance losses, there's also the annoyance that you can't ptrace a single task twice, so no debuggers. Are you familiar with the FlexSC paper? Have you considered using a FlexSC-like RPC interface (a secure one, of course) to achieve your syscall interception, instead of ptrace? That would allow you to not just match the performance of native system calls, but even theoretically exceed their performance, while still having the same level of control. (I have been working on such an approach, so I was excited to see this gvisor project posted - I hoped you might have already done this and saved me some work :)) Not sure how far this project can go if it sticks with ptrace...

amscanne 8y ago
(As requested, I work on this project.) To correct one misconception, the project is not bound to ptrace. There is a generic platform interface, and the repository also includes a KVM-based platform (wherein the Sentry acts as guest kernel and host VMM simultaneously) described in the README. The default platform is ptrace so that it works out of the box everywhere. > What are your plans to deal with the overhead and nastiness of ptrace? Beyond the performance losses, there's also the annoyance that you can't ptrace a single task twice, so no debuggers. It's true that you can't trace the sandbox itself, and that's annoying, but you can still use ptrace inside the sandbox (ptrace is implemented by the Sentry). Just wanted to make sure that was clear. > Are you familiar with the FlexSC paper? Have you considered using a FlexSC-like RPC interface (a secure one, of course) to achieve your syscall interception, instead of ptrace? That would allow you to not just match the performance of native system calls, but even theoretically exceed their performance, while still having the same level of control. (I have been working on such an approach, so I was excited to see this gvisor project posted - I hoped you might have already done this and saved me some work :)) I am familiar with FlexSC. There are certainly opportunities for improvement, including kernel-hooks, shared regions for async system calls, etc. Given the pace that this space is evolving, our priority was to share these pieces so that we can discuss things in the open. While I don't think we'll be able to save you work (sorry!), we're aiming for collaboration and cross-fertilization.
- bradneuberg 8y ago
  Good to have a gVisor developer on here. At Dropbox one way we use secure containers is to run machine learning models. Do you know if TensorFlow works in a gVisor container? How is GPU support in the container? If running on a CPU, are BLAS libraries supported to speed up matrix math in the container? Finally, do you know if OpenCV currently runs in gVisor containers?
  - flx42_ 8y ago
    nvidia-docker[1] maintainer here. Curious to know, are you using docker today? If yes, is there anything missing to satisfy your security requirements? [1] https://github.com/NVIDIA/nvidia-docker https://github.com/NVIDIA/nvidia-docker
    - bradneuberg 8y ago
      We aren't using Docker today; we have jailing infrastructure using linux cgroups, namespaces, seclist, etc.
    - deleted 8y ago
      [deleted]
  - dgacmu 8y ago
    Tensorflow security geek here: TF works in gVisor. I strongly recommend a VM solution when dealing with GPUs. Haven't tried GPU in gVisor because the size of the attack surface against the GPU device driver is so large. Depends on your level of paranoia, though. For CPU, gVisor is great. You can absolutely use BLAS libraries. Eigen (the TF default) works for sure. I don't see any reason MKL wouldn't, but I haven't personally tested it.
    - bradneuberg 8y ago
      Yeah, its still unclear to me how to create a security boundary for using GPUs at inference time in containers, or even in VMs. Any thoughts?
stphantom 8y ago
You are right in that ptrace is slow and nasty. The key problem, I believe, is the tracer-tracee mode that involves two host processes and the switch is asynchronous (ptrace(SYSEMU) and then waitpid). We do have the KVM platform that offers the synchronous switch, which performs better if you have bare-metal virtualization support.
deleted 8y ago
[deleted]
tych0 8y ago
You guys might all be interested in this: https://lkml.org/lkml/2018/2/4/28 https://lkml.org/lkml/2018/2/4/28

by catern 8y ago

amscanne 8y ago

(As requested, I work on this project.) To correct one misconception, the project is not bound to ptrace. There is a generic platform interface, and the repository also includes a KVM-based platform (wherein the Sentry acts as guest kernel and host VMM simultaneously) described in the README. The default platform is ptrace so that it works out of the box everywhere. > What are your plans to deal with the overhead and nastiness of ptrace? Beyond the performance losses, there's also the annoyance that you can't ptrace a single task twice, so no debuggers. It's true that you can't trace the sandbox itself, and that's annoying, but you can still use ptrace inside the sandbox (ptrace is implemented by the Sentry). Just wanted to make sure that was clear. > Are you familiar with the FlexSC paper? Have you considered using a FlexSC-like RPC interface (a secure one, of course) to achieve your syscall interception, instead of ptrace? That would allow you to not just match the performance of native system calls, but even theoretically exceed their performance, while still having the same level of control. (I have been working on such an approach, so I was excited to see this gvisor project posted - I hoped you might have already done this and saved me some work :)) I am familiar with FlexSC. There are certainly opportunities for improvement, including kernel-hooks, shared regions for async system calls, etc. Given the pace that this space is evolving, our priority was to share these pieces so that we can discuss things in the open. While I don't think we'll be able to save you work (sorry!), we're aiming for collaboration and cross-fertilization.

bradneuberg 8y ago
Good to have a gVisor developer on here. At Dropbox one way we use secure containers is to run machine learning models. Do you know if TensorFlow works in a gVisor container? How is GPU support in the container? If running on a CPU, are BLAS libraries supported to speed up matrix math in the container? Finally, do you know if OpenCV currently runs in gVisor containers?
- flx42_ 8y ago
  nvidia-docker[1] maintainer here. Curious to know, are you using docker today? If yes, is there anything missing to satisfy your security requirements? [1] https://github.com/NVIDIA/nvidia-docker https://github.com/NVIDIA/nvidia-docker
  - bradneuberg 8y ago
    We aren't using Docker today; we have jailing infrastructure using linux cgroups, namespaces, seclist, etc.
  - deleted 8y ago
    [deleted]
- dgacmu 8y ago
  Tensorflow security geek here: TF works in gVisor. I strongly recommend a VM solution when dealing with GPUs. Haven't tried GPU in gVisor because the size of the attack surface against the GPU device driver is so large. Depends on your level of paranoia, though. For CPU, gVisor is great. You can absolutely use BLAS libraries. Eigen (the TF default) works for sure. I don't see any reason MKL wouldn't, but I haven't personally tested it.
  - bradneuberg 8y ago
    Yeah, its still unclear to me how to create a security boundary for using GPUs at inference time in containers, or even in VMs. Any thoughts?

stphantom 8y ago

You are right in that ptrace is slow and nasty. The key problem, I believe, is the tracer-tracee mode that involves two host processes and the switch is asynchronous (ptrace(SYSEMU) and then waitpid). We do have the KVM platform that offers the synchronous switch, which performs better if you have bare-metal virtualization support.

deleted 8y ago

[deleted]

tych0 8y ago

You guys might all be interested in this: https://lkml.org/lkml/2018/2/4/28 https://lkml.org/lkml/2018/2/4/28