4 ms·
Or separate concerns and shell out on a Studio display with the-mac-you-like on usb-c
by whiplash451 2y ago
Or separate concerns and shell out on a Studio display with the-mac-you-like on usb-c
- talldayo 2y agoOr buy a real computer, plug in an Nvidia GPU, and save your DRAM for compute while using your VRAM for inference.
- nickthegreek 2y agoSo now you just limited to 24gb unless you are running dual 3090s or leave the consumer market for a gpu.
- talldayo 2y agoI mean, model layering has been around for several years now: https://huggingface.co/blog/lyogavin/airllm https://huggingface.co/blog/lyogavin/airllm
- Philpax 2y agoMoving the weights between the CPU and the GPU significantly limits performance. It's not comparable to having the entire model resident within video/unified memory.
- sroussey 2y agoYet it gets done for really big models. I can’t find the description for how Cerebras is doing Llama 405b for reference, but they are splitting by layers for that one (and why it’s not available right now). https://cerebras.ai/blog/introducing-cerebras-inference-ai-at-instant-speed/ https://cerebras.ai/blog/introducing-cerebras-inference-ai-a...
- Philpax 2y agoThat's not quite the same thing. There are three layers in the memory hierarchy (there are more, but for the purpose of this discussion, three is sufficient) - CPU RAM - GPU RAM - GPU SRAM The grandposter was talking about moving layers between CPU and GPU RAM, while Cerebras store models entirely in SRAM. CPU <-> GPU is _much_ slower than GPU RAM <-> GPU SRAM: the former will never be competitive with keeping the weights entirely on accelerator. Comparing the bandwidth numbers given in that article with the bandwidth of PCI-E should demonstrate the problem. Larger models are split between multiple accelerators, but even then they avoid going through the CPU as much as possible and instead use a direct interconnect (NVLink, etc). The weights are uploaded ahead of time to ensure they don't need to be reloaded during active inference. The considerations for training are different, but the general principle of avoiding CPU traffic still holds.
- lynguist 2y ago16 GB or 24 GB for inference doesn’t cut it for large models… Mac hardware offers up to 128 GB shared RAM.
- diffeomorphism 2y agoIt is in a weird middle ground. It is much worse ram and much slower hardware (factor 4 or so IIRC) and only an option if you need more than 24 but less than 200. Also, only if you think that 9000€ is pocket change but 20000€ is cost prohibitive. Also, you need all that power but no server, no ECC,... And you only ever want to do inference but no training or tuning. If you want a Mac anyway, sure. But if you don't care, this seems like a very, very specific Venn diagram.
- int_19h 2y agoIt's not much worse RAM, though. RTX 4090 has memory bandwidth of 1050 Gb/s. M2 Ultra is 800 Gb/s. And you can get a Mac Studio with Ultra and 128Gb of RAM for $3K or less. It's great for 70-150B models. You're correct that it's only good for inference, but most people running local LLMs only do inference.
- keybits 2y agoThat config Mac Studio costs 5,800 Euros (minimum). Where do you get it for 3,000 USD?
- int_19h 2y agoYou don't need an M2 for this, an M1 will do just fine. Nor do you need the one with maxed-out SSD, which jacks up the price considerably. Finding a brand new M1 Ultra for $4K or less right now is pretty easy. When I got mine, about a year ago, $3K was the best deal I could find.