3 ms·
Sometimes I daydream about a world where GPUs just have the equivalent of LPCAMM and you could put in as much RAM as you can afford and as much as the hardware
by KronisLV 2y ago
Sometimes I daydream about a world where GPUs just have the equivalent of LPCAMM and you could put in as much RAM as you can afford and as much as the hardware supports, much like is the case with motherboards, even if something along the way would bottleneck somewhat. It'd really extend the life of some hardware, yet companies don't want that.
That said, it's cool that you can even get an L4 with 24 GB of VRAM that actually performs okay, yet is passively cooled and consumes like 70W, at that point you can throw a bunch of them into a chassis and if you haven't bankrupted yourself by then, they're pretty good.
I did try them out on Scaleway, the pricing isn't even that exorbitant, using consumer GPUs for LLM use cases doesn't quite hit the same since.
- perching_aix 2y agoSaw a video from Bolt Graphics, a startup trying to do that among other things. Supposedly they'll have demos at several conventions later this year, like at Hot Chips. I think it's an incredibly tall order to get into the GPU game for a startup, but it should be good entertainment if nothing else.
- someothherguyy 2y agohttps://www.youtube.com/watch?v=8m-gSSIheno https://www.youtube.com/watch?v=8m-gSSIheno Sounds like hype
- someothherguyy 2y ago> yet companies don't want that I think there are quite a few constraints you are glossing over here.
- KronisLV 2y agoOh, I definitely am! It's always cool when someone with domain specific knowledge drops by and proceeds to shatter that dream with technical reasons that are cool nonetheless, the same way how LPCAMM2 doesn't work with every platform either and how VRAM has pretty stringent requirements. That said, it's understandable that companies are sticking with whatever works for now, but occasionally you get an immensely cool project that attempts to do something differently, like Intel's Larrabee did, for example.
- Cane_P 2y agoSOCAMM will enter production in the end of the year: https://www.tomshardware.com/pc-components/dram/nvidia-reportedly-developing-socamm-memory-standard https://www.tomshardware.com/pc-components/dram/nvidia-repor... The CEO of SK Hynix have confirmed that it is in the works: https://www.mk.co.kr/en/it/11245259 https://www.mk.co.kr/en/it/11245259 And big companies have started to analyze it, to figure out how it will fit into their domain: https://x.com/Jukanlosreve/status/1892916771692421228?t=3ikBUuWWJcMo-GybSgGELg&s=19 https://x.com/Jukanlosreve/status/1892916771692421228?t=3ikB...