4 ms·

The Kubernetes docs don't say anything about having to use privileged containers for GPU support. Privileged containers are given tens of Linux capabilities; wh

by puzzle 10y ago

The Kubernetes docs don't say anything about having to use privileged containers for GPU support. Privileged containers are given tens of Linux capabilities; which of those are actually needed in your setup? Or, conversely, which specific step would fail for an unprivileged container? Just because I want to use a GPU shouldn't require the power to change the clock, switch UIDs, chown files, mess with logs, reboot the machine, etc.

marcoceppi 10y ago
Since the GPU libraries are hosted on the node, privileged flag is typically required to make that possible. I'm sure there will be improvements to not require privileged, but today it's mostly a requirement to get anything useful out of containers tapping into GPU. That said, if you set the allow-privileged flag to false GPU drivers will still be installed but you may not be able to make use of the cuda cores
- puzzle 10y ago
  That's weird, because all the times I tried the experimental support, it didn't need privileged containers. From the YAML files, it looks like it's using hostPath directories, but those don't require special privileges, unless you need to write to them: https://kubernetes.io/docs/concepts/storage/volumes/#hostpath https://kubernetes.io/docs/concepts/storage/volumes/#hostpat... I suspect that there is a bug somewhere.
  - puzzle 10y ago
    Ah, wait: https://github.com/madeden/blogposts/blob/master/k8s-gpu-cloud/src/nvidia-smi.yaml https://github.com/madeden/blogposts/blob/master/k8s-gpu-clo... You don't need to mount the /dev entries into the container at all. The experimental support creates them automatically for you when you are using GPU resources. Perhaps it's device nodes, not the libraries that required privileges?
    - samnco 10y ago
      Aaah that is interesting. Let me dive into this later today and test my charts without that. It would actually make my life way easier for charting. I got that from a very early stage work and never questioned it again (the /dev stuff). Thanks for pointing that out.
    - samnco 10y ago
      Hello, OK I gave it a try and you are absolutely right. For the nvidia-smi, I could run it the /dev/nvidia0, which is cool. I was also able to run it unprivileged. I guess my mistake was to believe the example from the docs and not test without. Thanks for sharing that, I'll update my charts and the post accordingly.
      - puzzle 10y ago
        Awesome! Happy to hear that more containers will run without unneeded privileges.

by puzzle 10y ago

marcoceppi 10y ago

Since the GPU libraries are hosted on the node, privileged flag is typically required to make that possible. I'm sure there will be improvements to not require privileged, but today it's mostly a requirement to get anything useful out of containers tapping into GPU. That said, if you set the allow-privileged flag to false GPU drivers will still be installed but you may not be able to make use of the cuda cores

puzzle 10y ago
That's weird, because all the times I tried the experimental support, it didn't need privileged containers. From the YAML files, it looks like it's using hostPath directories, but those don't require special privileges, unless you need to write to them: https://kubernetes.io/docs/concepts/storage/volumes/#hostpath https://kubernetes.io/docs/concepts/storage/volumes/#hostpat... I suspect that there is a bug somewhere.
- puzzle 10y ago
  Ah, wait: https://github.com/madeden/blogposts/blob/master/k8s-gpu-cloud/src/nvidia-smi.yaml https://github.com/madeden/blogposts/blob/master/k8s-gpu-clo... You don't need to mount the /dev entries into the container at all. The experimental support creates them automatically for you when you are using GPU resources. Perhaps it's device nodes, not the libraries that required privileges?
  - samnco 10y ago
    Aaah that is interesting. Let me dive into this later today and test my charts without that. It would actually make my life way easier for charting. I got that from a very early stage work and never questioned it again (the /dev stuff). Thanks for pointing that out.
  - samnco 10y ago
    Hello, OK I gave it a try and you are absolutely right. For the nvidia-smi, I could run it the /dev/nvidia0, which is cool. I was also able to run it unprivileged. I guess my mistake was to believe the example from the docs and not test without. Thanks for sharing that, I'll update my charts and the post accordingly.
    - puzzle 10y ago
      Awesome! Happy to hear that more containers will run without unneeded privileges.