4 ms·

So, when Apple says 128-bit or 512-bit or 1024-bit wide interface to LPDDR4x/LPDDR5, does this really mean just a number that is derived from number_of_channels

So, when Apple says 128-bit or 512-bit or 1024-bit wide interface to LPDDR4x/LPDDR5, does this really mean just a number that is derived from number_of_channels x channel_width? E.g. 128-bit memory wide bus in case of LPDDR type of memory is really just 8x 16-bit channels or, in a different word-size configuration, 4x 32-bit channels. If that is so, my understanding is that Intel Tiger Lake with its 2x 64-bit channel design would equal to the same 128-bit memory bus width as M1, although, coupled with different type of memory chip technology - DDR4 instead of LPDDR.

wtallis 3y ago
DDR4 and earlier DIMMs are 64 bits per channel. DDR5 is still 64 bits per DIMM, but split into two 32-bit sub-channels. LPDDR is typically 16 bits per channel. For processors whose memory controller supports more than one of these standards, the total bit width is what's fixed in silicon and the processor's pin count; which kind of memory the processor is connected to will determine how many (sub)channels the bus is split into. For about two decades, mainstream consumer x86 platforms have almost always had a 128-bit memory bus. On desktop platforms with DDR4 and earlier, this can be accurately described as dual-channel, and that's the terminology marketing departments and end users are most familiar with. But it's never been accurate for mobile systems that use LPDDR, and DDR5 makes it less accurate for desktops, too. High-end platforms like AMD's Threadripper have supported memory configurations like 256-bit ("quad channel DDR4") and 512-bit ("8 channel DDR4") and now 768-bit on the latest EPYC servers, and over the years Intel has had server or workstation platforms with 192-bit (LGA 1366 socket), 256-bit (LGA 2011), 384-bit (LGA 3467), 512-bit (LGA 4677) memory interfaces (per socket). I believe Apple was the first to ship something wider than 128-bit in a laptop, and AMD plans to be the second in a year or so (not too surprising given what they've been shipping for the console market).
- menaerus 3y ago
  > I believe Apple was the first to ship something wider than 128-bit in a laptop Yes, that is now more obvious to me after your clarification and a short history walk-through. Thanks! Given that Apple never discloses much about their design, I was never actually 100% sure if the difference would only be related to the number of memory channels and thus capability of a memory controller or there was something else that was novel and that nobody else did. I am much more familiar with the server grade CPUs and was aware, as you very neatly summarized, of the differences and high memory bandwidths available in that spectrum. I was checking just this morning how many channels there are on Epycs and last generation of Xeons! Since we have different 128-bit wide designs available, it is interesting that Apple M1 chose to have 128B cache line size instead what others have - 64B. I never thought about that but it makes me wonder if it is more challenging to design a memory subsystem with the cache line size larger than what your DIMM actually supports.
  - wtallis 3y ago
    Memory bus widths are usually measured in bits while cache line sizes are bytes. The problem isn't with having a cache line that's too big, but rather too small: DDR3 and DDR4 both used burst sizes of eight transfers of 64 bits each, for a total burst of 64 bytes. DDR5 increased the burst size to 16 transfers, and split the DIMM into two subchannels to compensate—otherwise, it would be impossible to use the full bandwidth when fetching individual 64Byte cache lines. Transferring two bursts of data from RAM (not necessarily from the same channel) to fetch a single cache line worked fine in the DDR2 era.
    - menaerus 3y ago
      I see that the burst length for LPDDR4/LPDDR5 is configurable as either 16 transfers (32B) or 32 transfers (64B) which means that fetching a single 128B cache line requires Apple M memory controller to issue 2 or 4 read bursts depending on the setting used. According to the page 78 from https://www.micron.com/-/media/client/global/documents/products/data-sheet/dram/mobile-dram/low-power-dram/lpddr4/z19m_embedded_lpddr4_lpddr4x.pdf https://www.micron.com/-/media/client/global/documents/produ..., burst read operation will take 16 clock cycles to clock in, and roughly 4 more cycles to get the data out. At 2133Mhz clock cycle is 468 ps or 0.46 ns so to get 64 bytes out it will take 20 cycles x 0.46 ns = 9.2 ns. This means that to populate the 128B cache line it will take at least two such bursts leading to a total latency of 18.4 ns. This latency translates to ~6GB/s of burst reads. I guess this is a number only for a single channel and it's relatively close to what Micron mentions in the PDF: "Up to 8.5 GB/s per die". What I don't understand is how the LPDDR4 data rate transfer of 3200 MT/s (~25GB/s) is achieved then? The biggest package Micron LPDDR4 module comes in is 2 channels, hence, per their numbers we get a maximum bandwidth of 2 x 8.5GB/s = 17GB/s.
      - menaerus 3y ago
        > The biggest package Micron LPDDR4 module comes in is 2 channels Okay, this is wrong, I couldn't find it before but there is a larger Micron LPDDR4 module equipped with 4 channels and thus this theoretical LPDDR4 limit can be hit. E.g. MT53D1024M32D4 from https://www.micron.com/-/media/client/global/documents/products/data-sheet/dram/mobile-dram/low-power-dram/lpddr4/200b_z11m_non_auto_lpddr4_lpddr4x.pdf?rev=6e4d27ad107040949c35ab17b491ad2a https://www.micron.com/-/media/client/global/documents/produ...

wtallis 3y ago

DDR4 and earlier DIMMs are 64 bits per channel. DDR5 is still 64 bits per DIMM, but split into two 32-bit sub-channels. LPDDR is typically 16 bits per channel. For processors whose memory controller supports more than one of these standards, the total bit width is what's fixed in silicon and the processor's pin count; which kind of memory the processor is connected to will determine how many (sub)channels the bus is split into. For about two decades, mainstream consumer x86 platforms have almost always had a 128-bit memory bus. On desktop platforms with DDR4 and earlier, this can be accurately described as dual-channel, and that's the terminology marketing departments and end users are most familiar with. But it's never been accurate for mobile systems that use LPDDR, and DDR5 makes it less accurate for desktops, too. High-end platforms like AMD's Threadripper have supported memory configurations like 256-bit ("quad channel DDR4") and 512-bit ("8 channel DDR4") and now 768-bit on the latest EPYC servers, and over the years Intel has had server or workstation platforms with 192-bit (LGA 1366 socket), 256-bit (LGA 2011), 384-bit (LGA 3467), 512-bit (LGA 4677) memory interfaces (per socket). I believe Apple was the first to ship something wider than 128-bit in a laptop, and AMD plans to be the second in a year or so (not too surprising given what they've been shipping for the console market).

menaerus 3y ago
> I believe Apple was the first to ship something wider than 128-bit in a laptop Yes, that is now more obvious to me after your clarification and a short history walk-through. Thanks! Given that Apple never discloses much about their design, I was never actually 100% sure if the difference would only be related to the number of memory channels and thus capability of a memory controller or there was something else that was novel and that nobody else did. I am much more familiar with the server grade CPUs and was aware, as you very neatly summarized, of the differences and high memory bandwidths available in that spectrum. I was checking just this morning how many channels there are on Epycs and last generation of Xeons! Since we have different 128-bit wide designs available, it is interesting that Apple M1 chose to have 128B cache line size instead what others have - 64B. I never thought about that but it makes me wonder if it is more challenging to design a memory subsystem with the cache line size larger than what your DIMM actually supports.
- wtallis 3y ago
  Memory bus widths are usually measured in bits while cache line sizes are bytes. The problem isn't with having a cache line that's too big, but rather too small: DDR3 and DDR4 both used burst sizes of eight transfers of 64 bits each, for a total burst of 64 bytes. DDR5 increased the burst size to 16 transfers, and split the DIMM into two subchannels to compensate—otherwise, it would be impossible to use the full bandwidth when fetching individual 64Byte cache lines. Transferring two bursts of data from RAM (not necessarily from the same channel) to fetch a single cache line worked fine in the DDR2 era.
  - menaerus 3y ago
    I see that the burst length for LPDDR4/LPDDR5 is configurable as either 16 transfers (32B) or 32 transfers (64B) which means that fetching a single 128B cache line requires Apple M memory controller to issue 2 or 4 read bursts depending on the setting used. According to the page 78 from https://www.micron.com/-/media/client/global/documents/products/data-sheet/dram/mobile-dram/low-power-dram/lpddr4/z19m_embedded_lpddr4_lpddr4x.pdf https://www.micron.com/-/media/client/global/documents/produ..., burst read operation will take 16 clock cycles to clock in, and roughly 4 more cycles to get the data out. At 2133Mhz clock cycle is 468 ps or 0.46 ns so to get 64 bytes out it will take 20 cycles x 0.46 ns = 9.2 ns. This means that to populate the 128B cache line it will take at least two such bursts leading to a total latency of 18.4 ns. This latency translates to ~6GB/s of burst reads. I guess this is a number only for a single channel and it's relatively close to what Micron mentions in the PDF: "Up to 8.5 GB/s per die". What I don't understand is how the LPDDR4 data rate transfer of 3200 MT/s (~25GB/s) is achieved then? The biggest package Micron LPDDR4 module comes in is 2 channels, hence, per their numbers we get a maximum bandwidth of 2 x 8.5GB/s = 17GB/s.
    - menaerus 3y ago
      > The biggest package Micron LPDDR4 module comes in is 2 channels Okay, this is wrong, I couldn't find it before but there is a larger Micron LPDDR4 module equipped with 4 channels and thus this theoretical LPDDR4 limit can be hit. E.g. MT53D1024M32D4 from https://www.micron.com/-/media/client/global/documents/products/data-sheet/dram/mobile-dram/low-power-dram/lpddr4/200b_z11m_non_auto_lpddr4_lpddr4x.pdf?rev=6e4d27ad107040949c35ab17b491ad2a https://www.micron.com/-/media/client/global/documents/produ...