4 ms·
To start with, an Epyc server or Mac Studio with 512GB RAM.
by trvz 1y ago
To start with, an Epyc server or Mac Studio with 512GB RAM.
- criddell 1y agoI looked up the price of the Mac Studio: $9500. That's actually a lot less than I was expecting... I'm guessing an Epyc machine is even less.
- graeme 1y agoHow does the mac studio load the trillion parameter model?
- deleted 1y ago[deleted]
- petu 11mo agoBy using ~3 bit quantized model with llama.cpp, Unsloth makes good quants: https://docs.unsloth.ai/models/tutorials-how-to-fine-tune-and-run-llms/kimi-k2-how-to-run-locally#model-uploads https://docs.unsloth.ai/models/tutorials-how-to-fine-tune-an... Note that llama.cpp doesn't try to be production-grade engine, more focused on local usage.