3 ms·

I don't know how any of this works, and there seem to be a number of experts in this thread so please allow me some very newbie questions. > [can't] get anywhe

I don't know how any of this works, and there seem to be a number of experts in this thread so please allow me some very newbie questions. > [can't] get anywhere near their peak throughput if accesses are serialized But first of all, doesn't DRAM have an optimisation for serial accesses, which is (I hope I get this the right way round) to hold the RAS steady then do CAS/CAS/CAS/CAS instead of RAS/CAS + RAS/CAS + RAS/CAS + RAS/CAS? In addition, if you do access RAM serially then there is some interleaving which is automatically inserted so that sequential accesses at some level are sent off to different banks (but I've never been clear if there interleaved at a 64 bytes, to fit a cache line, or at a 4K page size to suit the TLB, or some other size, and I'd really like to know – anyone please?)

pclmulqdq 4y ago
Yeah, DRAM controllers optimize for serial access in the mapping from physical address space to DRAM DIMMs, banks, rows, and columns. There's actually a pretty sophisticated algorithm for mapping "physical" addresses to DRAM addresses.
Tuna-Fish 4y ago
Sorry, two entirely different definitions of serial, and mine is less useful. What I mean is that if you are doing an infinite pointer hop (that is, load a value, then use the that value as a pointer to load another value, use that as a a pointer... etc), you only get a tiny fraction of the IOPS you get than if you launch 100 different loads to different addresses that you already know. And the inverse of your load to use latency is basically how many IOPS you get if you do that infinite pointer chase. > In addition, if you do access RAM serially then there is some interleaving which is automatically inserted so that sequential accesses at some level are sent off to different banks (but I've never been clear if there interleaved at a 64 bytes, to fit a cache line, or at a 4K page size to suit the TLB, or some other size, and I'd really like to know – anyone please?) They are interleaved at the size of the DRAM row, which is usually 2^16 bits, or 8192 bytes. The reasoning here is that if you are launching a ton of linear accesses, that is, doing CAS/CAS/CAS/CAS... to the same row, you can get full throughput from it. This is because when you have opened a row, you are not reading from the DRAM anymore, the entire row is in a SRAM buffer, and can read full interface throughput from it. Then you only have to open a new row once the current one runs out, which is always in a different bank so can be done in parallel with the read operations from the current row so long as your memory prefetchers are smart enough to start doing it early enough.
- _a_a_a_ 4y ago
  Thanks that's really good, and the interleaving question's been nagging me a long time. There's a lot of depth here when you start asking. Much appreciated.

pclmulqdq 4y ago

Yeah, DRAM controllers optimize for serial access in the mapping from physical address space to DRAM DIMMs, banks, rows, and columns. There's actually a pretty sophisticated algorithm for mapping "physical" addresses to DRAM addresses.

Tuna-Fish 4y ago

Sorry, two entirely different definitions of serial, and mine is less useful. What I mean is that if you are doing an infinite pointer hop (that is, load a value, then use the that value as a pointer to load another value, use that as a a pointer... etc), you only get a tiny fraction of the IOPS you get than if you launch 100 different loads to different addresses that you already know. And the inverse of your load to use latency is basically how many IOPS you get if you do that infinite pointer chase. > In addition, if you do access RAM serially then there is some interleaving which is automatically inserted so that sequential accesses at some level are sent off to different banks (but I've never been clear if there interleaved at a 64 bytes, to fit a cache line, or at a 4K page size to suit the TLB, or some other size, and I'd really like to know – anyone please?) They are interleaved at the size of the DRAM row, which is usually 2^16 bits, or 8192 bytes. The reasoning here is that if you are launching a ton of linear accesses, that is, doing CAS/CAS/CAS/CAS... to the same row, you can get full throughput from it. This is because when you have opened a row, you are not reading from the DRAM anymore, the entire row is in a SRAM buffer, and can read full interface throughput from it. Then you only have to open a new row once the current one runs out, which is always in a different bank so can be done in parallel with the read operations from the current row so long as your memory prefetchers are smart enough to start doing it early enough.

_a_a_a_ 4y ago
Thanks that's really good, and the interleaving question's been nagging me a long time. There's a lot of depth here when you start asking. Much appreciated.