4 ms·

Do CUDA AI projects not containerize well? I understand that AI projects often have many, many dependencies, but this seems like something containers ("docker")

Do CUDA AI projects not containerize well? I understand that AI projects often have many, many dependencies, but this seems like something containers ("docker") would be ideally suited for. Something like `docker pull dall-e` and `docker run -ie -n dalle dall-e` probably volume mount in a local folder? I'm curious what the hurdles are here for packaging something like this.

indiv0 4y ago
They do containerize but it’s not enough. You still need the right drivers and CUDA stuff installed on the docker host, which can be finicky to setup. Not to mention figuring out how to get docker to actually pass the GPU to the container.
- analyte123 4y ago
  I see this has not gotten any better in the last 5 years. For future reference, if you as a dev are interacting with a product that is this hard to use and there is no other option but to use it (CUDA), you should buy their stock.
  - danuker 4y ago
    I'd much rather look out for competitors. Resting on laurels won't last. If a company stops improving its offer, a competitor might catch up.
    - ceeplusplus 4y ago
      The competition has even more broken software (AMD). At least Nvidia's works when you need it to.
      - moffkalast 4y ago
        I think OpenCL support has been getting better, and it may be possible to run a lot of models with it, but that just doubles your already frustrating amount of hours trying to set the damn thing up.
- boolemancer 4y ago
  > Not to mention figuring out how to get docker to actually pass the GPU to the container. Should just be passing `--gpus all` these days, shouldn't it?
sailingparrot 4y ago
> I'm curious what the hurdles are here for packaging something like this. No hurdle, it's just that Docker is not part of the standard toolkit of ML people. ML people all use conda (or virtualenv etc.) which already solves most of the dependency problems, making learning Docker not especially appealing. But virtually all training/inference platform (including the ones used by OpenAI) are using docker. It's not a technical limitation.
- visarga 4y ago
  No, ML people use Docker to deploy their models. Nobody got time or patience to replicate environments for inference.

indiv0 4y ago

They do containerize but it’s not enough. You still need the right drivers and CUDA stuff installed on the docker host, which can be finicky to setup. Not to mention figuring out how to get docker to actually pass the GPU to the container.

analyte123 4y ago
I see this has not gotten any better in the last 5 years. For future reference, if you as a dev are interacting with a product that is this hard to use and there is no other option but to use it (CUDA), you should buy their stock.
- danuker 4y ago
  I'd much rather look out for competitors. Resting on laurels won't last. If a company stops improving its offer, a competitor might catch up.
  - ceeplusplus 4y ago
    The competition has even more broken software (AMD). At least Nvidia's works when you need it to.
    - moffkalast 4y ago
      I think OpenCL support has been getting better, and it may be possible to run a lot of models with it, but that just doubles your already frustrating amount of hours trying to set the damn thing up.
boolemancer 4y ago
> Not to mention figuring out how to get docker to actually pass the GPU to the container. Should just be passing `--gpus all` these days, shouldn't it?

sailingparrot 4y ago

> I'm curious what the hurdles are here for packaging something like this. No hurdle, it's just that Docker is not part of the standard toolkit of ML people. ML people all use conda (or virtualenv etc.) which already solves most of the dependency problems, making learning Docker not especially appealing. But virtually all training/inference platform (including the ones used by OpenAI) are using docker. It's not a technical limitation.

visarga 4y ago
No, ML people use Docker to deploy their models. Nobody got time or patience to replicate environments for inference.