3 ms·
Currently, Docker does not support Metal GPUs. When running LLMs on Docker with an Apple M3 or M4 chip, they will operate in CPU mode regardless of the chip's
by jjuliano 2y ago
Currently, Docker does not support Metal GPUs.
When running LLMs on Docker with an Apple M3 or M4 chip, they will operate in CPU mode regardless of the chip's class, as Docker only supports Nvidia and Radeon GPUs.
If you're developing LLMs on Docker, consider getting a Framework laptop with an Nvidia or Radeon GPU instead.
Source: I develop an AI agent framework that runs LLMs inside Docker on an M3 Max (https://kdeps.com https://kdeps.com).
- rtwld 2y agoPodman does support GLU acceleration through libkrun with virtio-gpu (venus) on Mac: https://podman-desktop.io/docs/podman/gpu https://podman-desktop.io/docs/podman/gpu
- asadalt 2y agohow is it in practice (if you have tried). I have some vulkan work and I am too lazy to setup a new ec2 for it.