3 ms·

You can get Macs with 192GB of UMA, so maybe 180GBusable VRAM. But of course the GPU horsepower is much less tban an A100.

• atemerev 2y ago
  Sort of works for inference, but not for anything else. For now, I bought a 4x4090 workstation… but I am crying how power-inefficient it is.
  • ra120271 2y ago
    For my education, would you be able to expand on what it doesn't work for and why? Thank you!
    • Gracana 2y ago
      Inference (token generation) is memory-bound, KV cache prefill (prompt processing) is compute-bound. The ARM Macintoshes have lots of memory bandwidth but not a lot of compute power, so they're great for outputting text but terrible for tasks like analyzing documents. I've never done fine-tuning but my understanding is that that is a highly-parallelizable compute hog as well. You might like this article, which looks at the arithmetic intensity of LLM processing: https://www.baseten.co/blog/llm-transformer-inference-guide/ https://www.baseten.co/blog/llm-transformer-inference-guide/