8 ms·
Next year, devices equipped with AMD's Strix Halo APU will be available, capable of using ~96GB of VRAM across 4 relatively fast channels from a total of 128GB
by pella 2y ago
Next year, devices equipped with AMD's Strix Halo APU will be available, capable of using ~96GB of VRAM across 4 relatively fast channels from a total of 128GB unified memory, along with a 50 TOPS NPU. This could partially serve as an alternative to the MacBook Pro models with M2/M3/M4 chips, featuring 128GB or 196GB unified memory.
- https://videocardz.com/newz/amd-ryzen-ai-max-395-to-feature-16-zen5-cores-and-40-rdna3-5-cus-strix-halo-lineup-leaks-out https://videocardz.com/newz/amd-ryzen-ai-max-395-to-feature-...
- diggan 2y agoAccording to Tom's (https://www.tomshardware.com/pc-components/cpus/amd-pushes-ryzen-to-the-max-ryzen-ai-max-300-strix-halo-reportedly-has-up-to-16-zen-5-cores-and-40-rdna-3-cus https://www.tomshardware.com/pc-components/cpus/amd-pushes-r...), those are supposed to be laptop CPUs, which makes me wonder what AMD has planned for us desktop users.
- adrian_b 2y agoThey are laptop CPUs for bigger laptops, like those that now use both a CPU and a discrete GPU, i.e. gaming laptops or mobile workstations. It seems that the thermal design power for Strix Halo can be configured between 55 W and 120 W, which is similar to the power used now by a combo laptop CPU + discrete GPU.
- MobiusHorizons 2y agoIf I remember right, in the press conference they suggested desktop users would use a gpu because desktop uses are less power sensitive. That doesn’t address the vram limitations of discrete GPUs though.
- wkat4242 2y agoTrue but try to find a 96GB GPU.
- teaearlgraycold 2y agoH100 NVL is easily available. It’s just that it’s close to $20k.
- andersa 2y agoIt is actually much more expensive than 20k. I can find them between 25-30k only.
- teaearlgraycold 2y agoMy bad. Looks like it’s $29k at the local store.
- VikingCoder 2y agoThis drives me insane. PC Part Picker, DDR5-8400 48 GB (2x24GB) is... $340 right now. For $680 you can get 96 GB of very fast RAM. How about someone make an NVidia GPU with 96 GB of RAM at a reasonable price? Please?
- tymscar 2y agoI totally agree that we should get higher vram sizes on GPUs but they are not the same as DDR5. The one you listed does around 50Gbps. A really good gpu does almost 450Gbps. Prices as you know also don’t scale linearly. For something twice as good sometimes you pay 4x the price and so on.
- MobiusHorizons 2y agoThis is a really important point. The architecture and the bus ram is connected to is also very different on on a discrete card than it is in a cpu. Each compute unit needs dedicated bus width connected to dedicated memory. So for most GPUs you can double the ram, but you couldn’t just add the equivalent of a stick of ddr5 because it wouldn’t match the dedicated bus
- 2y ago
- TibbityFlanders 2y ago[dead]
- aurareturn 2y agoIt will have around 250GB/s of bandwidth which makes it nearly unusable for 70b models. So the high amount of RAM doesn’t help with large models.
- Dalewyn 2y agoFast. Large. Cheap. You may only pick two.
- meiraleal 2y agoFor a few years ago standard the current "small" models like mistral and phind are fast, large and cheap.
- throwaway314155 2y ago> nearly unusable for 70b models Can Apple Silicon manage this? Would it be feasible to do with some quantization perhaps?
- aurareturn 2y agoYes at around 8 tokens/s. Also quite slow.
- bearjaws 2y agoAny of the newer M2+ Max chips runs 400GB/s and can run 70b pretty well. It's not fast though, 3-4 token/s. You can get better performance using a good CPU + 4090 + offloading layers to GPU. However one is a laptop and the other is a desktop...
- staticman2 2y agoApparently Mac purchasers like to talk about tokens per second without talking about Mac's atrocious time to first token. They also like to enthusiastically talk about tokens per second asking a 200 token question rather than a longer prompt. I'm not sure what the impact is on a 70b model but it seems there's a lot of exaggeration going on in this space by Mac fans.
- jstummbillig 2y agoAlso, next year, there will be GPT 5. I find it fascinating how much attention small models get, when at the same time the big models just get bigger and prohibitively expensive to train. No leading lab would do that if they thought it a decent chance that small models were able to compete. So who will be interested in a shitty assistant next year when you can have an amazing one, is what I wonder? Is this just the biggest cup of wishful thinking that we have ever seen?
- jabroni_salad 2y agoOne of the reasons I run local is that the models are completely uncensored and unfiltered. If you're doing anything slightly 'risky' the only thing APIs are good for is a slew of very politely written apology letters, and the definition of 'risky' will change randomly without notice or fail to accommodate novel situations. It is also evident in the moderation that your usage is subject to human review and I don't think that should even be possible.
- Tempest1981 2y agoThere is also a long time-window before most laptops are upgraded to screaming-fast 128GB AI monsters. Either way, it will be fun to watch the battle.
- Larrikin 2y agoWhy would anyone buy a Raspberry Pi when they can get a fully decked out Mac Pro? There are different use cases and computers are already pretty powerful. Maybe your local model won't be able to produce tests that check all the corner cases of the class you just wrote for work in your massive code base. But the small model is perfectly capable of summarizing the weather from an API call and maybe tack on a joke that can be read out to you on your speakers in the morning.
- talldayo 2y ago> Why would anyone buy a Raspberry Pi when they can get a fully decked out Mac Pro? They want compliant Linux drivers?