3 ms·

Ah yes, running on GPU isn't supported at the moment. But CUDA (for Nvidia GPUs) and Metal support is on the roadmap!

• samspenc 3y ago
  Ah fascinating, just curious, what's the technical blocker? I thought most of the Llama models were optimized to run on GPUs?
  • mayankchhabra 3y ago
    It's fairly straightforward to add GPU support when running on the host, but LlamaGPT runs inside a Docker container, and that's where it gets a bit challenging.
    • stavros 3y ago
      It shouldn't, nVidia provides a CUDA Docker plugin that lets you expose your GPU to the container, and it works quite well.
      • dicriseg 3y ago
        See above if you're interested in that. It does work quite well, even with nested virtualization (WSL2).
        • stavros 3y ago
          I am, thanks!