4 ms·
> For instance, serving a single 175 billion LLM requires at least 350GB of GPU memory using specialized infrastructure Apple ships the Mac Studio which suppor
by threeseed 3y ago
> For instance, serving a single 175 billion LLM requires at least 350GB of GPU memory using specialized infrastructure
Apple ships the Mac Studio which support up to 144GB of usable GPU memory.
Would be amusing if they were to release a Mac Pro with 300+ GB and dominate the LLM serving space.
- rubyn00bie 3y agoApple's hardware advantage is something I'm hoping to see really unleashed with the M3 generation. The fact the A17 Pro has ray tracing support is giving me hope they can catch up with the incumbents quickly. It's honestly the one thing that has kept me away from their newer hardware-- I primarily use my computer at a desk and PC hardware (GPU primarily) is leagues ahead of what their very best can do. I honestly can't justify spending nearly 4k when Linux works great for everything I do professionally AND when I'm done with work I can still play a game.
- behnamoh 3y ago3 most important things for Apple are: - HW design - UI - UX That Macs come with awesome VRAM is only because #3. Apple has no explicit incentive to make LLM-ready machines.
- rubyn00bie 3y agoI disagree. ML is eating the world by storm and one thing Apple loves is high margins. Folks like myself would gladly pay that margin to someone besides Nvidia. I paid $2k for my last GPU and was happy to do it. That doesn’t even factor the fact that businesses would pay even more when the alternative are machines that start in the tens of thousands for just the GPU ability. GPUs right now are carrying fat margins and that makes more than enough incentive for Apple IMHO.
- behnamoh 3y agoBut you wouldn't be able to upgrade an Apple Mac when the next generation of LLMs is out.
- nborwankar 3y agoYou do get some money back for a trade-in depending on how old the machine is.
- leetharris 3y agoYou can find many leaked email threads from Apple where they talk about the value of high powered local compute. It is a fundamental part of their strategy. They do not want the cloud approach for high end compute like Google, they want you to have high end and powerful devices in your hands.
- flangola7 3y agoGot a link to those emails?
- deleted 3y ago[deleted]
- stevenhuang 3y agoThat number is not quantized either. Quantize 175B param at 4 bits then it'd fit in ~120GB VRAM (one can fit 34B models at 4 bit quantization in a single RTX3090 24GB VRAM)
- brucethemoose2 3y ago> Would be amusing if they were to release a Mac Pro with 300+ GB and dominate the LLM serving space Is there any framework that can batch LLMs on Metal? I don't think GGML or MLC have it yet. Otherwise that is just another reason they wouldn't be good for LLM hosting at this moment. Anyway, the real disruptor is Intel. They could theoretically barge in with a 2x48GB Arc card and undercut the market that AMD/Nvidia refuse to dive into because of their Pro card clients.
- astrange 3y agoIt's coming to llama.cpp soon. https://github.com/ggerganov/llama.cpp/pull/3228 https://github.com/ggerganov/llama.cpp/pull/3228
- brucethemoose2 3y agoAmazing! Its really a shame that Apple (and AMD/Intel or pretty much any other infrence vendor) are not directly contributing to llama.cpp. The feature set is amazing and growing at a stunning pace.
- nomel 3y agoI imagine the legal concerns around it are part of the reason. Where I am, our legal team has been very direct, that anything llama based cannot touch a work computer.
- stringtoint 3y agoCurious to know why this is the case? I haven't played around with LLAMA yet but I figured it being open-source would make it more trustworthy than models provided by OpenAI.
- nomel 3y ago> it being open-source Absolutely not. The LLAMA license [1] is clear that it's not open source. It's for non-commercial, research only, and only by explicit permission from Meta. The weights were leaked, on 4chan [2], illegally, according to the license. Very very few people are using it legally. This interpretation is clear from its wording, and also matches the interpretation of our flock of lawyers. [1] https://github.com/facebookresearch/llama/issues/266 https://github.com/facebookresearch/llama/issues/266 [2] https://levelup.gitconnected.com/metas-chatgpt-is-now-illegally-free-for-everyone-5b58524caeb5 https://levelup.gitconnected.com/metas-chatgpt-is-now-illega...
- oezi 3y agoI am wondering who will be the first to raise RAM capacity of their hardware offerings dramatically to win the LLM crowd. Seems like the way to win market share.
- marcyb5st 3y agoAgreed. I would love to have something with the FLOPs of a 3070/4060 and 80 gigs of even slower VRAM (not necessarily HBM/GDDR6x) that can run the XXB models.