10 ms·

Working great in ollama: https://mastodon.social/@rcarmo/112837520236956526 https://mastodon.social/@rcarmo/112837520236956526

• jxy 2y ago
  https://github.com/ollama/ollama/issues/5881 https://github.com/ollama/ollama/issues/5881 https://github.com/ggerganov/llama.cpp/issues/8650 https://github.com/ggerganov/llama.cpp/issues/8650
  • rcarmo 2y ago
    Still works fine for me. Latest ollama, running on NVIDIA.
• raminf 2y ago
  FWIW, 405B not working with Ollama on a Mac M3-pro Max with 128GB RAM. Times out.
  • pbmonster 2y ago
    Did you get a 2 bit quant? You need to chain several Mac Studios via Exo to get enough memory for a useful quant to work.