10 ms·

Putting GPUs to work with Kubernetes

133 points by marcoceppi 9y ago

aub3bhat 9y ago
Great read, on a smaller scale, I have found nvidia-docker with nvidia-docker-compose to be a great solution for deploying docker containers on AWS P2 machines with 8 GPUs.
Tossrock 9y ago
Unprofitable for ETH mining maybe, but it seems like a natural fit to rent time on it to deep learning people with slow training models. Although that could still be unprofitable after the cost of electricity, I guess it's a question of market size/demand. A lot of deep learning is already at big infrastructure players anyway who wouldn't need the service, leaving academics / smaller companies. But maybe some people would find a reliable, scalable GPU cluster valuable.
- samnco 9y ago
  Ahah, good point. Really the ETH stuff was "because I can". But in the same charts repository you will find a Tensorflow chart. My previous series of blogs [0] was about exactly that. A nice addition as well for compute intensive workloads is the use of LXD [1] Another use case is in media for transcoding. It is not a trivial job to orchestrate transcoding at scale, and Kubernetes with or without GPUs is an excellent solution for that as it is trivial to setup a completely automated job queue. Also another interesting field will eventually be HPC but there are some constraints about compute that K8s does not tick scheduling wise at this point in time. There is a pluggable scheduler in the works I think, and this will eventually help. Also the LXD example is a nice optimization but it would not replace the scheduler in any way. [0]: https://medium.com/intuitionmachine/gpus-kubernetes-for-deep-learning-part-3-3-automating-tensorflow-deployment-5dc4d5472e91 https://medium.com/intuitionmachine/gpus-kubernetes-for-deep... [1]: https://hackernoon.com/job-concurrency-in-kubernetes-lxd-and-cpu-pinning-to-the-rescue-b9fb7b44f99d https://hackernoon.com/job-concurrency-in-kubernetes-lxd-and...
Seanny123 9y ago
I keep seeing Kubernetes appear on HackerNews. Is there a quick thing I can read to explain why everyone's so excited about? I know it's container orchestration, but I'm not sure what people are using it for or what pain point it is revealing.
- moondev 9y ago
  https://vishh.github.io/docs/concepts/overview/what-is-kubernetes/ https://vishh.github.io/docs/concepts/overview/what-is-kuber...
  - Seanny123 9y ago
    If you'd like to engage with me further, how does a company know it needs Kubernetes? If I'm Soylent and I'm processing a few orders a minute, I'm probably safe with a few redundant monoliths. Do I have to be Uber? What's the middle-ground between Soylent and Uber that would still need this? Is the answer the same as the question "who needs a microservice architecture"?
    - samnco 9y ago
      There are a few killer features that you would benefit at any size and that I really love * self healing: when you create a deployment/replica set. it will be maintained at all cost, so if the app has a memory leak or anything goes wrong, it will be contained and kept up and running * Rolling update: even when you run 5 frontends, it is a pain to use capistrano or other tools to just update at git repo. it is literally a one liner in Kubernetes. If you use CI/CD the setup is just a few lines in any Jenkins/Gitlab/Travis... * Service discovery: the combination of ENV and predictable DNS endpoints is just awesome * Ecosystem: PaaS, Serverless... Many of the new world infra is built on K8s, so it is a door to the next gen, whether you know you will use it or not. As for Micro Service Architecture, just starting with the web frontend and a couple of lightly dockerize middleware makes it sooooo simple that you instantly want to get more out of it. As the overhead of running K8s vs. set of servers is relatively low especially at small scale, it is definitely worth looking at. Happy to do a run through with you and show you how the deployment of a tiered app works as a demo, ping me on @SaMnCo_23 if interested.
shaklee3 9y ago
Is the author of this working on official support or just testing? I know there's a gpu roadmap for k8s, but I can't tell from this blog if this was part of it.
- samnco 9y ago
  Canonical will officially support GPUs when they lands GA upstream. The flag is beta as of now in the Canonical Distribution of Kubernetes. Paying customers either for the managed or supported solutions get a best effort for GPU, and this feature is enabled by default.
  - puzzle 9y ago
    What is the requirement for privileged containers? The post never explains it.
    - samnco 9y ago
      privileged containers are required for the GPU to be shared with the containers. By default, the bundle come with a "auto" tag, which will activate privileged containers just when GPUs are detected. You can enforce "false" to remove that, but then you won't be able to run GPU workloads. Or you can enforce "yes" and have them activated all the time. Does that answer the question? Not sure if I understood it right.
      - puzzle 9y ago
        The Kubernetes docs don't say anything about having to use privileged containers for GPU support. Privileged containers are given tens of Linux capabilities; which of those are actually needed in your setup? Or, conversely, which specific step would fail for an unprivileged container? Just because I want to use a GPU shouldn't require the power to change the clock, switch UIDs, chown files, mess with logs, reboot the machine, etc.
        marcoceppi 9y ago
nrki 9y ago
"1060GTX at home but on consumer grade Intel NUC" A bit OT, but I'd like to see how this works... Ah, very cool - https://www.youtube.com/watch?v=wyY-lTmgb8c https://www.youtube.com/watch?v=wyY-lTmgb8c
- samnco 9y ago
  Actually, it was a fun DIY project I did a while ago. You can read about it here: https://hackernoon.com/installing-a-diy-bare-metal-gpu-cluster-for-kubernetes-364200254187 https://hackernoon.com/installing-a-diy-bare-metal-gpu-clust... It works, but the GPUs aren't very stable at 4x vs. a normal 16x.
  - jacquesm 9y ago
    That's one problem, another is the size of the powersupply. And maybe that's the only problem, I don't see why a GPU would become unstable when using fewer lanes, all it should do is get slower.
    - samnco 9y ago
      I don't know. Maybe the make of the extenders isn't very good, I saw other people with similar issues. The PSU is the Corsair AX1500i (1500W), with 10x lines for GPUs. It's robust on paper, didn't have any problem with just 4 plugged in. But I must say... The T630 are very noisy compared to these, but so much more powerful #NotGoingBack
      - jacquesm 9y ago
        I just bought a GTX1080ti + a similar corsair as an upgrade for my 3 year old Dell, it works like a charm. If you have a PSU that big then that probably isn't the problem. I thought you might be using the PSU that comes with those extender boxes and they usually are very puny (250 W or so). Do you use it for gaming or for CUDA? Do you run the 4 GPUs in the extender?
        9y ago

133 points by marcoceppi 9y ago

Great read, on a smaller scale, I have found nvidia-docker with nvidia-docker-compose to be a great solution for deploying docker containers on AWS P2 machines with 8 GPUs.

Tossrock 9y ago

Unprofitable for ETH mining maybe, but it seems like a natural fit to rent time on it to deep learning people with slow training models. Although that could still be unprofitable after the cost of electricity, I guess it's a question of market size/demand. A lot of deep learning is already at big infrastructure players anyway who wouldn't need the service, leaving academics / smaller companies. But maybe some people would find a reliable, scalable GPU cluster valuable.

samnco 9y ago
Ahah, good point. Really the ETH stuff was "because I can". But in the same charts repository you will find a Tensorflow chart. My previous series of blogs [0] was about exactly that. A nice addition as well for compute intensive workloads is the use of LXD [1] Another use case is in media for transcoding. It is not a trivial job to orchestrate transcoding at scale, and Kubernetes with or without GPUs is an excellent solution for that as it is trivial to setup a completely automated job queue. Also another interesting field will eventually be HPC but there are some constraints about compute that K8s does not tick scheduling wise at this point in time. There is a pluggable scheduler in the works I think, and this will eventually help. Also the LXD example is a nice optimization but it would not replace the scheduler in any way. [0]: https://medium.com/intuitionmachine/gpus-kubernetes-for-deep-learning-part-3-3-automating-tensorflow-deployment-5dc4d5472e91 https://medium.com/intuitionmachine/gpus-kubernetes-for-deep... [1]: https://hackernoon.com/job-concurrency-in-kubernetes-lxd-and-cpu-pinning-to-the-rescue-b9fb7b44f99d https://hackernoon.com/job-concurrency-in-kubernetes-lxd-and...

Seanny123 9y ago

I keep seeing Kubernetes appear on HackerNews. Is there a quick thing I can read to explain why everyone's so excited about? I know it's container orchestration, but I'm not sure what people are using it for or what pain point it is revealing.

moondev 9y ago
https://vishh.github.io/docs/concepts/overview/what-is-kubernetes/ https://vishh.github.io/docs/concepts/overview/what-is-kuber...
- Seanny123 9y ago
  If you'd like to engage with me further, how does a company know it needs Kubernetes? If I'm Soylent and I'm processing a few orders a minute, I'm probably safe with a few redundant monoliths. Do I have to be Uber? What's the middle-ground between Soylent and Uber that would still need this? Is the answer the same as the question "who needs a microservice architecture"?
  - samnco 9y ago
    There are a few killer features that you would benefit at any size and that I really love * self healing: when you create a deployment/replica set. it will be maintained at all cost, so if the app has a memory leak or anything goes wrong, it will be contained and kept up and running * Rolling update: even when you run 5 frontends, it is a pain to use capistrano or other tools to just update at git repo. it is literally a one liner in Kubernetes. If you use CI/CD the setup is just a few lines in any Jenkins/Gitlab/Travis... * Service discovery: the combination of ENV and predictable DNS endpoints is just awesome * Ecosystem: PaaS, Serverless... Many of the new world infra is built on K8s, so it is a door to the next gen, whether you know you will use it or not. As for Micro Service Architecture, just starting with the web frontend and a couple of lightly dockerize middleware makes it sooooo simple that you instantly want to get more out of it. As the overhead of running K8s vs. set of servers is relatively low especially at small scale, it is definitely worth looking at. Happy to do a run through with you and show you how the deployment of a tiered app works as a demo, ping me on @SaMnCo_23 if interested.

shaklee3 9y ago

Is the author of this working on official support or just testing? I know there's a gpu roadmap for k8s, but I can't tell from this blog if this was part of it.

samnco 9y ago
Canonical will officially support GPUs when they lands GA upstream. The flag is beta as of now in the Canonical Distribution of Kubernetes. Paying customers either for the managed or supported solutions get a best effort for GPU, and this feature is enabled by default.
- puzzle 9y ago
  What is the requirement for privileged containers? The post never explains it.
  - samnco 9y ago
    privileged containers are required for the GPU to be shared with the containers. By default, the bundle come with a "auto" tag, which will activate privileged containers just when GPUs are detected. You can enforce "false" to remove that, but then you won't be able to run GPU workloads. Or you can enforce "yes" and have them activated all the time. Does that answer the question? Not sure if I understood it right.
    - puzzle 9y ago
      The Kubernetes docs don't say anything about having to use privileged containers for GPU support. Privileged containers are given tens of Linux capabilities; which of those are actually needed in your setup? Or, conversely, which specific step would fail for an unprivileged container? Just because I want to use a GPU shouldn't require the power to change the clock, switch UIDs, chown files, mess with logs, reboot the machine, etc.
      - marcoceppi 9y ago

nrki 9y ago

"1060GTX at home but on consumer grade Intel NUC" A bit OT, but I'd like to see how this works... Ah, very cool - https://www.youtube.com/watch?v=wyY-lTmgb8c https://www.youtube.com/watch?v=wyY-lTmgb8c

samnco 9y ago
Actually, it was a fun DIY project I did a while ago. You can read about it here: https://hackernoon.com/installing-a-diy-bare-metal-gpu-cluster-for-kubernetes-364200254187 https://hackernoon.com/installing-a-diy-bare-metal-gpu-clust... It works, but the GPUs aren't very stable at 4x vs. a normal 16x.
- jacquesm 9y ago
  That's one problem, another is the size of the powersupply. And maybe that's the only problem, I don't see why a GPU would become unstable when using fewer lanes, all it should do is get slower.
  - samnco 9y ago
    I don't know. Maybe the make of the extenders isn't very good, I saw other people with similar issues. The PSU is the Corsair AX1500i (1500W), with 10x lines for GPUs. It's robust on paper, didn't have any problem with just 4 plugged in. But I must say... The T630 are very noisy compared to these, but so much more powerful #NotGoingBack
    - jacquesm 9y ago
      I just bought a GTX1080ti + a similar corsair as an upgrade for my 3 year old Dell, it works like a charm. If you have a PSU that big then that probably isn't the problem. I thought you might be using the PSU that comes with those extender boxes and they usually are very puny (250 W or so). Do you use it for gaming or for CUDA? Do you run the 4 GPUs in the extender?
      - 9y ago