7 ms·
I wonder if it will ever see a chance to go mainstream. The memory bottleneck is pretty much the only thing in CPU design that didn't see a dramatic improvemen
by baybal2 7y ago
I wonder if it will ever see a chance to go mainstream.
The memory bottleneck is pretty much the only thing in CPU design that didn't see a dramatic improvement over the years. Its elimination is the only obvious improvement pathway still left with expectation of double digit performance gains.
So we need either very big and very fast caches, or extremely wide and low latency memory. Both options are quite costly.
Adding on die DRAM that can work at least as fast as 500mhz will surely require some specialty process with a lot of compromises like the one in the article.
Gluing something like HBM2 to the die for a second option moves the cost from the specialty process to the specialty packaging. Not much better.
- jiggawatts 7y ago> Gluing something like HBM2 to the die for a second This is something AMD is already doing for EPYC, and they've already used HBM2 in their GPUs. So I'm surprised they haven't released any CPU models with crazy huge L4 caches using a few GB of HBM2. Then again, Intel made a laptop CPU with a huge 128MB cache and their comment was that it didn't make that big of a difference. I believe the performance boost was less than 5% for going from 64MB to 128MB.
- vvanders 7y agoRead access patterns matter more than cache sizes, triple digit improvements are possible of you have linear reads.
- toohotatopic 7y agoHasn't Intel bought the company that was on the brink of producing those CPU-memory combos? Unfortunately, I haven't been able to find the name of the company or an article about it.
- deepnotderp 7y agoYou mean UpMem or Venray Technology?
- toohotatopic 7y agoSorry but I don't remember the name at all.
- hinkley 7y agoMaybe there’s an inflection point where imperative management of the cache is more effective than heuristic management.
- mhh__ 7y agoWhat do you mean by imperative? Static control by the program?
- cryptonector 7y agoI'm guessing u/hinkley meant having an API for the cache, so a compiler could generate instructions to prefetch cache lines (and maybe even keep them hot) rather than depend on automatic (heuristic) cache management.
- Reelin 7y agoJust FYI, prefetch instructions are already a thing (at least on x86). At present they're only hints to the underlying heuristic system though. (https://www.felixcloutier.com/x86/prefetchh https://www.felixcloutier.com/x86/prefetchh) Cache-as-RAM is also a thing, and allows the sort of pinning you described. Pretty specialized use cases though. (https://stackoverflow.com/questions/41775371/what-use-is-the-invd-instruction https://stackoverflow.com/questions/41775371/what-use-is-the...)
- gwking 7y agoI saw Alan Kay speak once, I think at lambda jam 2013. He made a comment that he’s still waiting for mainstream CPUs to allow us to fit an entire interpreter into on-core cache.
- adrianN 7y agoForth interpreters probably fit into L1 on modern processors.
- 7y ago
- rbanffy 7y agoLate Xeon Phis had 16GB of HBM around the die. It could either complement or mirror main memory.
- baybal2 7y agoIts all about how fast the memory is. 960MiB is prodigiously large for such a microscopic chip, but if it "only" gain 3-5 times latency reduction over external DRAM, it's still very far from a proper L3 implementation, and far behind L2. Make DRAM work on 1Ghz+, and then you will see miracles. Imagine a fully synchronous on-die DRAM that can sit just behind L1, or even be connected to load registers directly. The problem is that effective frequencies for memory round-trip haven't got up much since nineties. If you work with 100% cache misses, your mem will still be working at effective frequency of around 100 to 200Mhz
- Dylan16807 7y ago> If you work with 100% cache misses, your mem will still be working at effective frequency of around 100 to 200Mhz I think you mean 20MHz. Current DRAM is abysmal at random access.
- baybal2 7y agoYes, more so like this if you discard how it looks from the electrical side. Even after bytes arrive to DRAM controller analog side, a lot of things have to happen before the data gets to the register. This accounts for the further five to tenfold increase in round trip latency
- RantyDave 7y agoOh no, it's paging to ram.
- deleted 7y ago[deleted]
- thedance 7y agoIntel laptop parts had 128MB of eDRAM starting in 2013. Is that mainstream enough for you?
- baybal2 7y agoOn what frequency did that eDRAM worked? If the effective freq was 100mhz, you being hit by cache misses will indeed be "only" 2-3 times less bad than as if you used memory round trip
- LargoLasskhyfv 7y agoHad. And then floated away down the stream, never to be seen again, at least not with 128MB.
- Dylan16807 7y agoIf they had ever made it available at the mid-high end or released it outside of a couple weird SKUs before they quietly buried it, then it would count.
- dfox 7y agoThe reason why mainstream DRAM interfaces are narrow and “slow” is that you need row-at-a-time access patterns to really saturate the interconnect which is something that does not happen for general purpose workloads and causing such access patterns requires large caches which by themselves solve the issue and then also physical package pins and pad structures are one of the most expensive things in semiconductor design. In end the DRAM array is bunch of analog magic and the interface works by copying the row you want into SRAM buffer on the chip which you then can access however you want. And the slowest operation in all that are the copies between SRAM row buffer and the actual DRAM array. (what I call SRAM buffer is usually called “column sense amplifiers”, but for the highlevel view it in fact is surprisingly wide array of 6T SRAM flipflops and some analog magic)
- hinkley 7y agoSo how many levels of cache do we have now between the ALU and the memory cell of record??
- blattimwind 7y agoPhysical register <-> L1 <-> Fill Buffer <-> L2 <-> L3 [<-> L4 ] <-> [bus buffer <->] memory module I wouldn't count buffer registers inside DRAM chips as a separate cache.
- kragen 7y ago↔ SSD ↔ spinning rust ↔ tarsnap or something
- BeeOnRope 7y agoI wouldn't really count the fill buffer between L1 and L2 as a separate caching level, or at least if you wanted to do that you should do something similar for L2 <-> L3, L3 <-> memory controller, etc: since such buffers or queues will exist at all those places.
- blattimwind 7y ago
- deepnotderp 7y agoCommodity DRAM latency is mostly array line dominated, not due to proximity/distance, see eg https://ieeexplore.ieee.org/document/6522354 https://ieeexplore.ieee.org/document/6522354 Also eDRAM is difficult to scale
- hinkley 7y agoWith all of the speculation security bugs in chip cache management, I can’t help but wonder if we won’t eventually go full NUMA and turn the cache memory (or at least L2+) into a directly addressable space, either by the kernel or directly by application code. At which point your working set is explicitly on the processor, instead of implicitly. I also wonder if chiplets will be the vehicle by which this comes to pass.
- the8472 7y agoCache can already be made directly addressable (cache-as-ram mode). But outside of some niche solutions that's mostly done during boot due to the limitations of that mode and cache being too valuable.
- ChuckMcM 7y agoI beg to differ, the thing that make the original AMD64 chips so freakishly awesome was multiple memory controllers in a multi-socket system. That forced Intel to do something kinda similar but the Opteron line and then the Ryzen and EPYC lines have always been better than Intel in terms of raw memory bandwidth[1]. And when there was a clear need for the bandwidth, as in GPUs, circuit designers stepped up and created some amazing wide and deep memory bus architectures. I would not be surprised to see AMD partner with IBM to utilize some of their EDRAM tech to keep themselves out ahead of Intel in the data center space. The more ways they can distinguish themselves, the more pressure they put on Intel's design teams. [1] Yes, lookaside buffers and page tables in the Opterons took some of that advantage back, but it was still substantial.
- moring 7y agoWhat is the state of adding higher-level logic to the DRAM chips? E.g. clear a whole DRAM row to 0 (blank memory for new processes). With today's processes, we can probably fit whole simple processor cores there easily.
- baybal2 7y agoLatest DDR standard, and supposedly HBM3 in making support range requests
- imtringued 7y agoThis is called PIM (processing in memory) and UPMEM [0] has a working implementation. [0] https://www.anandtech.com/show/14750/hot-chips-31-analysis-inmemory-processing-by-upmem https://www.anandtech.com/show/14750/hot-chips-31-analysis-i...
- kev009 7y agoThe same process tech is used on POWER9, and a lot of the physical design is shared between z and p, so this eDRAM is pretty mainstream in POWER. OpenCAPI seems like it will provide almost as fast (latency) as an on die controller with DDR4 attached RAM at much cheaper prices than contemporary designs.. and opens the door to 3rd party innovation. The real issue is trading latency versus bandwidth.. IBM's scale up designs have had ridiculously large memory bandwidth forever, but latency suffers to some extent with buffers like these SCs in the article. Whether that matters for your workload really depends.. but people need to break out of lock/latches mindset and use safe memory reclamation techniques like RCU and EBR to avoid common critical sections for lifecycle management, and in general minimize synchronization to make full use of current designs even like the now common AMD Rome. I firmly believe we've been in a time where better programmers aware of hardware/software interface have been needed for a while, rather than some kind of hardware physics brick wall as the pundits often claim.
- RantyDave 7y agoOh no, the packaging connections are (guessing) thousands of nanometres. Much easier to build, much cheaper. Also, the process of moving to multi-chip packages means the process yield for the chiplets is much higher. It's a bloody good idea :)
- zong0dD2 7y agoQuoting: 'The memory bottleneck' (Raises the head from her electric soldering iron:) Maybe "in a world without any 'North- or Southbridge'-bottleneck" ? Or do you actually have figured out to attach a keyboard directly to your graphics-card (still powered by a PSU), cos... your graphics-board has (plain and simply) CPU-Power, RAM and for sure 'ports' to attach something... ?! (-;
- baybal2 7y agono