3 ms·

This might tempt me to get a graphics card and run local. What do I need minimum to run it?

• ipnon 2y ago
  It will run faster than you can read on a MacBook Pro with 192GB.
  • graeme 2y ago
    You can only run a distilled model. They're quite good but not nearly as good as the full thing. As for as fast as you can read, depends on the distilled size. I have a mac mini 64 GB Ram. The 32 GB models are quite slow. 14B and lower are very very fast.
    • yborg 2y ago
      M4 or M4 Pro?
• buyucu 2y ago
  I'm running the 32b version on a home server (Ryzen 7) using llama.cpp, which uses Vulkan for hardware acceleration on the iGPU. For hobbyist inference, getting a iGPU with lots of system ram is probably better than getting a dedicated Nvidia gpu.