4 ms·
> - 96GB of weights. You won't be able to run this on your home GPU. You can these days, even in a portable device running on battery. 96GB fits comfortably i
by jlokier 3y ago
> - 96GB of weights. You won't be able to run this on your home GPU.
You can these days, even in a portable device running on battery.
96GB fits comfortably in some laptop GPUs released this year.
- michaelt 3y agoBe a lot cooler if you said what laptop, and how much quantisation you're assuming :)
- tvararu 3y agoThey're probably referring to the new MacBook Pros with up to 128GB of unified memory.
- jlokier 3y agoSibling commenter tvararu is correct. 2023 Apple Macbook with 128GiB RAM, all available to the GPU. No quantisation required :) Other sibling commenter refulgentis is correct too. The Apple M{1-3} Max chips have up to 400GB/s memory bandwidth. I think that's noticably faster than every other consumer CPU out there. But it's slower than a top Nvidia GPU. If the entire 96GB model has to be read by the GPU for each token, that will limit unquantised performance to 4 tokens/s at best. However, as the "Mixtral" model under discussion is a mixture-of-experts, it doesn't have to read the whole model for each token, so it might go faster. Perhaps still single-digit tokens/s though, for unquantised.
- refulgentis 3y agoThis is extremely misleading. source: been working in local LLMs since 10 months ago. Got my Mac laptop too. I'm bullish too. But we shouldn't breezily dismiss those concerns out of hand. In practice, it's single digit tokens a second on a $4500 laptop for a model with weights half this size (Llama 2 70B Q2 GGUF => 29 GB, Q8 => 36 GB)
- coolspot 3y ago> $4500 Which is more than a price of RTX A6000 48gb ($4k used on ebay)
- CamperBob2 3y agoHow fast does it run on that?
- refulgentis 3y agoquantization makes it hard to have exactly one answer -- I'd make a q0 joke, except that's real now -- i.e. reduce the 3.4 * 10^38 range of float 32 to 2, a boolean. it's not very good, at all, but now we can claim some pretty massive speedups. I can't find anything for llama 2 70B on 4090 after 10 minutes of poking around, 13B is about 30 tkn/s. it looks like people generally don't run 70B unless they have multiple 4090s.
- brucethemoose2 3y agoWhich is outrageously priced, in case thats not clear. Its an 2020 RTX 3090 with doubled up memory ICs, which is not much extra BoM.
- baq 3y agoClearly it’s worth what people are willing to pay for it. At least it isn’t being used to compute hashes of virtual gold.
- tucnak 3y agoPeople are also willing to die for all kinds of stupid reasons, and it's not indicative of _anything_ let alone a clever comment on the online forum. Show some decorum, please!
- brucethemoose2 3y agoIts a artificial supply constraint due to artificial market segmentation enabled by Nvidia/AMD. Honestly its crazy that AMD indulges in this, especially now. Their workstation market share is comparatively tiny, and instead they could have a swarm of devs (like me) pecking away at AMD compatibility on AI repos if they sold cheap 32GB/48GB cards.