12 ms·
JEDEC publishes GDDR7 graphics memory standard
- paddy_m 3y agoHow much relevance does JEDEC still have? I would think NVIDIA in particular and other chip makers/integrators like apple make up their own standards now. It also seems less relevant because memory is rarely interchangeable anymore
- sliken 3y agoDo you have any evidence that Nvidia or Apple are using non-standard memory chips? Link? From what I can tell they are both using standard chips, but very wide memory interfaces. Apple's lowest end is 128 bit, but offer 256, 512, and 1024 bit wide memory interfaces for more bandwidth, which is mostly a benefit for the iGPU in all of Apple's m series CPUs. This is part of why Apple are pretty good at LLMs, especially those needing more ram than is in even the most expensive GPUs. Sad that the vast majority of x86-64 laptops and desktops have the same bus width of decades ago, while the core counts are ever increasing.
- jsheard 3y agoNvidia and Micron came up with GDDR6X, which isn't a JEDEC standard. JEDEC did standardize GDDR5X before that, but only Micron ever made it and only Nvidia ever used it AFAIK.
- zeusk 3y agoJEDEC never "standardized" GDDR6x that Nvidia does use; Micron and Nvidia worked closely on both GDDR6x and GDDR5x
- kllrnohj 3y ago> Apple's lowest end is 128 bit, but offer 256, 512, and 1024 bit wide memory interfaces for more bandwidth Apple doesn't have any 1024-bit offerings. The Ultras are 2x 512-bit which is different from a true 1024-bit since it's a NUMA configuration (two different memory controllers). > Sad that the vast majority of x86-64 laptops and desktops have the same bus width of decades ago, while the core counts are ever increasing. The consistent trend is CPU performance on consumer workloads is still much more sensitive to memory latency than it is bandwidth. This is probably also why those 512-bit M1/2 Max's don't even bother to let the CPU make use of it, they top out at "just" 200GB/s memory bandwidth even though there's 400GB/s available to the SoC as a whole. Being able to pair a lot of DRAM with the GPU of an M* Max/Ultra is definitely a currently unique perk, but the bandwidth numbers available to those GPUs are not actually all that special. Desktop GPUs passed 400GB/s mark way back in 2016, and are currently pushing 1TB/s in modern flagship offerings.
- sliken 3y agoAgreed. I did see a limit of 220GB or so for the M2 max, but haven't seen an updated number for the M3 max. Certainly GPUs have great bandwidth, but get quite expensive if you need more than 24GB ram. I've been impressed how well the M3 max runs 70B or larger models, plenty for single user use anyways.
- hedgehog 3y agoApple and NVIDIA are both members of JEDEC...
- monocasa 3y agoHBM is a JEDEC spec these days. Apple's on package memory is still LPDDR.
- deleted 3y ago[deleted]
- ajross 3y agoJEDEC isn't a memory technology monopoly or anything, they're just a standards organization. You have a situation where lots of companies need to make products that interoperate, but interoperation is complicated in electrical engineering. There are a *lot* of ways to get a memory interconnect wrong. So the solution is to pick (or create) a separate, notionally independent body staffed and supported by representatives of all the relevant stakeholders, and have them write "standards" that everyone agrees to adhere to. The body doesn't invent the technology, that happens at the individual chip companies. They then present their proposals to JEDEC[1] and everyone argues and agrees on what will go into GDDR19 or whatnot. And JEDEC then publishes the standards for all to see. [1] Or whoever, JEDEC does DRAM, but there's a USB Consortium, Bluetooth SIG, WiFi is under IEEE, etc...
- pillusmany 3y agoStandardized memory chips allows economy of scale to work.
- Dalewyn 3y ago>How much relevance does JEDEC still have? Ah, a FOSSy dev insistent that his bits are gibi- and mebi- and kibi-bytes.
- ryukoposting 3y agoThe physical memory modules might not be interchangeable, but JEDEC is still extremely important to hardware and firmware engineers. It gives us a baseline - we know anything that says "JEDEC" will play nice with our SoC's QSPI/DDR peripherals.
- paulmd 3y agoNVIDIA has basically been doing that with GDDR5X and GDDR6X already - they are "official standards" but nobody besides NVIDIA actually uses them.
- sliken 3y agoDoubling the channels from GDDR6 sounds good, the speed of light isn't changing, so at least we can handle more parallelism with the same latency.
- oorza 3y agoThe ratio of light speed to the area of the universe is so stupidly small that I'm convinced our simulation is determining how low the speed of light can get before interstellar travel is outright impossible.
- adtac 3y agoWith sufficiently advanced technology, travelling between stars will be more of a transfer of your consciousness using interstellar WiFi (remember to set TCP_NODELAY!) rather than transporting slow and heavy atoms. You just get transferred from one biological substrate to another. None of the time dilation, all of the space exploration.
- WJW 3y ago> all of the space exploration. How does the receiving technology get built? Surely at least someone will have to go there the first time, and they will have to take the long way. It will still be quite a problem to get to a system 10k light years away.
- kraquepype 3y agoMy inner sci-fi geek tells me that by this time, we discover faster than light travel, only it isn't compatible with life as we know it. So we ship off these receivers to circumvent that limitation. Instead of travelling ourselves, we can send off our consciousness to inhabit a human-life analog to explore. What that does to your psyche, and your body in limbo, are probably good material for a story, if it hasn't already been written.
- 3y ago
- Scene_Cast2 3y agoOne interesting thing to note is that all the high speed interconnect standards (GDDR, PCIE, USB, Ethernet) are moving (or have already moved) to RF-style non-binary communication (as opposed to bumping up clock speeds or increasing pin count). I wonder what the next steps will be for interconnects - full-blown baseband style transceivers with QAM perhaps?
- wmf 3y agoOptical communication is using QAM so that's probably the next step.
- klysm 3y agoSharp signal edges are hard to get at higher frequencies - it seems quite natural
- SmellTheGlove 3y agoThis is probably a stupid question but I don’t claim any knowledge here. Even though interconnect standards are moving to non-binary communication, doesn’t the signal eventually need to be converted back to a bitstream at its destination? Does this just push the bottleneck around, or do I just not understand the problem being solved? It’s almost certainly the latter and I’d love to understand more.
- kevvok 3y agoYou’re right that the signals have to be converted back into bits at the destination. Basically, this solves the problem of pumping those bits at high speed across traces on a circuit board vs within a chip. The longer a signal has to go, the harder it is to maintain its integrity.
- wmf 3y agoA serializer/deserializer (serdes) is used to convert between high-speed serial I/O outside the chip (e.g. 100 Gbps) and lower-clocked parallel signals inside the chip (e.g. 64 bits at 1.5 GHz). Using serial protocols reduces the cost and thickness of cables while parallel wires are cheaper inside the chip.
- deleted 3y ago[deleted]
- imtringued 3y agoUnless Nvidia can somehow massively increase memory capacity, it is looking bleak for them in the consumer AI inference space. From the left Apple has a fully integrated SoC with insane memory bandwidth and capacity, from the right AMD is tackling the FLOPs advantage using their XDNA AI Engines that they got from their Xilinx acquisition and they are going to open source the compiler for their AI Engines. The only competitive advantage that NVidia has left is its high memory bandwidth, but even that is being threatened by Strix Point so they will need to adopt GDDR7 with 32 GB to 64GB VRAM fast or they will become irrelevant except for training. Oh and by the way AMD GPUs will stay completely irrelevant for AI so that explains why they didn't want to waste so much time on ROCm for consumer GPUs. Nobody is going to buy those for AI anymore by the end of the year.
- rubatuga 3y agoAMD was the first to introduce consumer HBM cards.
- account42 3y agoWell, on "consumer" cards that were AMDs desperate attempt to stay relevant in that space by rebadging their enterprise compute-focused design for consumers. They haven't used HBM in any of their actual consumer designs unfortunately.
- oasisaimlessly 3y agoAre you sure? https://www.techpowerup.com/gpu-specs/radeon-r9-fury-x.c2677 https://www.techpowerup.com/gpu-specs/radeon-r9-fury-x.c2677 > AMD has paired 4 GB HBM memory with the Radeon R9 FURY X, which are connected using a 4096-bit memory interface.
- wmf 3y agoIt's not clear that the "consumer (local) AI inference space" is a real market. Ultimately Nvidia has access to all the same technologies as their competitors and better software so anything they can do Nvidia can do better... if they want to.
- Lramseyer 3y agoJust got my hands on the spec and read through it for a bit. A couple interesting things I noticed that were notable changes from GDDR6. Obviously, the big news is PAM3 signaling, and on die ECC. These aren't all that new, as NVIDIA's GDDR6X used PAM4 signaling (at lower frequencies than traditional GDDR6) and an unnamed DRAM vendor had GDDR6 DRAMs with on die ECC, though at the cost of having annoyingly high read and write latencies. Thankfully, only the DQ (data) pins use PAM3 signaling. I'm going to do my best to explain how this works in GDDR7, since I need to understand this for my work: If you don't know what PAM3 is, it stands for "Pulse Amplitude Modulation, 3 levels" Traditional communication could be thought of as PAM2, since there's a level for 0 and 1. But we usually call it NRZ for "Non-return to Zero" There's a bit of nuance, since not all binary data communication is NRZ. But that's a different discussion. Now you might ask, why PAM3 and not PAM4? With 4 levels you can transfer 2 bits, and that seems much easier to work with. Well, it's because we hate ourselves, and we hate you. That's why. For context, a GDDR6 channel uses 16 DQ (data) pins + 2 EDC (Error Detection/Correction) pins and 16 transfers for 256 data bits and 32 CRC bits transaction. GDDR6X (from my understanding) does the same thing, but since it's PAM4, it sends 2 bits per transfer with (I think) half the number of transfers. GDDR7 on the other hand has 11 DQ pins of PAM3 signaling. Since PAM3 is 3 state {-1, 0, 1} these are defined as "symbols" or "trits" (I really hate the word "trit".) These symbols have are 3 separate encoding methodologies (yikes) that are used in this protocol: 11b7S, 3b2S, 2b1S. These essentially determine how many bits of data you can encode in a given number of symbols. 11b7S means 11 bits of data encoded in 7 symbols. 11 bits of data has 2048 unique combinations, 7 symbols have 2187 unique combinations (3^7). 3b2S is 3 bits (8 combinations) encoded in 2 symbols (9 combinations). And 2b1S is a misnomer but it's application specific to the Poison and Severity flag bits and the Severity flag takes precedent, so the unrepresented combination is invalid. Like GDDR6, there are still 16 transfers, but the DQ bus width is reduced to 11 DQ pins, and a parity error pin. With this, we get a total of 176 symbols per transaction. When decoded, this allows us to send 256 bits of data, 2 bits for Poison/Severity flags (this is the 2b1S symbol), and 18 bits of CRC. Encoded though is a bit of a doozy: 163 symbols for the data, 1 symbol (2b1S) for the Poison/Severity flags, and 12 symbols (3b2s) for the CRC. If that's not confusing enough, The 163 data symbols don't even all use the same encoding. The first 161 symbols encode 253 bits in 23 sets of 7 symbol to 11 bit (11b7S) sets, for 23 sets of 11 bits. The remaining 3 bits a are encoded in the remaining 2 symbols as a 3b2S set. How this is mapped to each pin is outlined in the spec. I'll take a look at that part later, as I have had enough unfriendly math for the day. On a different note, there were some other notable changes that I noticed: They also shrunk the CA (Command Address) bus width from 10 pins down to 5 and run it twice as fast relative to the CK and WCK clocks. Instead of 2 cycles of 10 bits, it's 4 cycles of 5 bits. The 5 bits are split into "Row" [0:2] and "Column" [3:4] bits. It was a bit weird at first glance, but it's actually kind of nice. The commands make a lot more sense now. Interestingly enough, it looks like the CABI (Command Address Bus Inversion) and CAPAR (Command Address Parity) are part of the 20 bit CA command now. Though I'm not sure how useful a CABI is if it's once every 4 cycles. Not sure how much power you're really saving. There are 64 mode registers instead of 16. Still 12 bit wide though. Wait WTF? This (kind of) made sense in GDDR6, since the address and the data fit nicely into 16 bits (4 address, 12 data) and you could use the other 4 bits as the command ID for MRS. Now we have 12 bit wide registers, 6 bit wide addresses, and the MRS command is a double length command because it doesn't use the column bits. This is really weird. Registers 0-31 are defined by the spec, 32-47 are reserved for future use, and 48-63 are Vendor Specific. Found this funny - There's a feature for addressing clock drift between CA and WCK clocks due to varying Voltage and Temperature. It's called the "Command Address Oscillator" I wonder why they picked that name... "There is a CAOSC associated with each channel and operates fully independent of any channel’s operating frequency or state" Someone had fun with that one! Other Quick blurbs: - No mention of Quad Data Rate (QDR) - CTLE looks like it's supported from the DRAM side. - They added an RCK (read clock) pair, and is generated on the DRAM side. - They added a static data scrambler! You don't know how happy I am to see this!
- Sparkyte 3y agoI just want memory that can dynamically change on the basis of workload so it can be great for graphics and great for system performance.
- mistyvales 3y agoStill interested to see if a newer gen of HBM will ever be on retail GPUs again