3 ms·
People use "Docker-based" all the time but what they mean is that they ship $SOFTWARE in a Docker image. "Docker-based" reads, to me, as if you were doing Infe
by leonheld 2y ago
People use "Docker-based" all the time but what they mean is that they ship $SOFTWARE in a Docker image.
"Docker-based" reads, to me, as if you were doing Inference on AMD cards with Docker somehow, which doesn't make sense.
- a_vanderbilt 2y agoYou can do inference from a Docker container, just as you'd do it with NVidia. OpenAI runs a K8s cluster doing this. I have personally only worked with NVidia, but the docs are present for AMD too. Like anything AI and AMD, you need the right card(s) and rocm version along with sheer dumb luck to get it working. AMD has Docker images with rocm support, so you could merge your app in with that as the base layer. Just pass through the GPU to the container and you should get it working. It might just be the software in a Docker image, but it removes a variable I would otherwise have to worry about during deployment. It literally is inference on AMD with Docker, if that's what you meant.
- mikepurvis 2y agoDocker became part of the standard toolkit for ML because deploying Python that links to underlying system libraries is a gong show unless you ship that layer too.
- tannhaeuser 2y agoEven Docker doesn't guarantee reproducible results due to sensitivity towards host GPU drivers, and ML frontends/integrations bringing their own "helpful" newby-friendly all-in-one dependency checks and updater services.
- jeffhuys 2y agoWhy doesn’t it make sense? You can talk to devices from a Docker container - you just have to attach it.
- fazkan 2y agoyou can mount a specific device to docker. If you read the script, we are mounting GPUs https://github.com/slashml/amd_inference/blob/main/run-docker-amd.sh https://github.com/slashml/amd_inference/blob/main/run-docke...
- steeve 2y agoHi, we (ZML), fix that: https://github.com/zml/zml https://github.com/zml/zml
- latchkey 2y agoWorks out of the box on our MI300x. Fantastic work steeve! https://x.com/HotAisle/status/1842245896085356949 https://x.com/HotAisle/status/1842245896085356949
- fazkan 2y agoThis is pretty cool. Is there a document that shows which AMD drivers are supported out of the box?
- steeve 2y agoWe are in line with ROCm 6.2 support. We actually just opened a PR to bump to 6.2.2: https://github.com/zml/zml/pull/39 https://github.com/zml/zml/pull/39
- bongodongobob 2y agoYeah, they're using docker to wrap up the software packages, which is what Docker is used for. I don't understand why that confuses you or what you think Docker is otherwise used for.
- leonheld 2y agoI'm pretty comfortable with Docker/cgroups/namespaces, I have quite a deep understanding of it. But I read "Docker-based inference" like you literally took Docker code to... do inference? The wording in my opinion doesn't make much sense. It's like saying, I don't know, "Flatpak-based inference" or "SSD-based inference". Semantics.