6 ms·

This is made to sound so revolutionary and novel but in fact it is only an intermediary, fast, transparent to the processes, automatically managed memory. Badly

This is made to sound so revolutionary and novel but in fact it is only an intermediary, fast, transparent to the processes, automatically managed memory. Badly managed as it turned out. What hurts the most is that there definitely is some sort of influence on how a process can take advantage of caching, yet not in its control, and even with the best practices on memory optimization it all falls flat when a few processing layers get stacked on this hardware clever cache memory management. The fast memory should have been left in the control of the software. It should have been left small and accessible like in the old computers (16 bit range) and what is now considered "the main memory" to be developed as a new thing, that could be left to grow massively. I think that the current situation is less the result of quality engineering vision and more of the result of marketing departments putting pressure on the development teams to come out with clever tricks to beat the competition. ___________ P.S.: I already thought about the backward compatibility considerations being primordial. That would have meant that the technical advancement transparent to the existing usage was a reason in itself, but that doesn't quite hold water. The 16 bit to 32 bit architecture switch opened the possibility of adopting a secondary (but slower) layer of RAM going over 64 KiB and keeping the 16 bit manageable memory range close to CPU, as (at least one of) the cache layers.

obl 10y ago
Unless you want programmers to explicitely manage this fast memory the hardware is a better place to do those optimizations than the software. Compilers (even the high-end modern ones) are very quickly clueless about both memory and execution profiles. Those problems are hard to reason about statically. Is this pointer the same as this other one ? Will this loop be executed 10 times or 10^6 times ? The hardware being a dynamic optimizer has a lot of runtime information (pointer values, cache entry usage, ...) to take advantage of. Sure, one can argue that we just need better JITs, PGO, and programming languages that are easier to analyze statically. In the current state of affair we are way better off leaving this to the hardware. By the way you do have "some" control over it using NT moves on x86. Turns out almost nobody uses them except by hand in very specific peak-bandwidth code. Compilers don't dare emitting those since the penalty of getting it wrong outweights the benefits of getting it right.
- restalis 10y ago
  "In the current state of affair we are way better off leaving this to the hardware." That is something I'm not so sure about. That might have been true before, when the hardware served well the then simple computational model of a single high frequency processing core. Things got more complicated since the times of first CPU cache offerings and hardware general use-case implementation can only give you so much. The performance boost that once could be relied on without much care now dissipated. This cache thing now become just a leaky abstraction. We can not abstract it away completely if we care for performance, nor we can control it in any meaningful way. It is not a good model anymore. Just recognize it as a failed experiment. "Unless you want programmers to explicitely manage this fast memory..." Yes, that's exactly what I have in mind (and implied before). Most applications do not need maximal performance and can disregard the fast memory completely. That will only mean that those processes either will not consume their share and leave it more for the processes that need it, or that the fast memory will (occasionally) be used indirectly, through the optimized bits in the layers that the given process relies on. This memory, however, won't be unproductively overwritten or invalidated on every cold cache or whatnot.
- dman 10y ago
  I use non temporal stores all the time when writing database related code.
- gaius 10y ago
  Unless you want programmers to explicitely manage this fast memory That's how we did it in the good old days on the 6502 (zp address mode) and we liked it like that...
tveita 10y ago
The idea of scratchpad memory is as old as the hills, and you may still find it on some specialty platforms like gaming consoles, but the lesson learned is mostly that it ends up chronically underutilized. A cache speeds up nearly all code written for any system, not just the 0.1% that someone bothered to hand optimize the memory access patterns for.
- restalis 10y ago
  "the lesson learned is mostly that it [scratchpad memory] ends up chronically underutilized. A cache speeds up nearly all code written for any system" I recall that when that technology came out, there was a heavy marketing stunt involved. For the boast's sake, performance had to be conjured. And as seamlessly as possible. And as noticeable as possible. That really affected the design development, and not in a good way. The problem (in what I quoted) here is the poor measurement base, the wrong comparison base, and what is counted or discounted as value. So that cache really speeds up nearly all the code written for any system. Such a flat model does not, however, make much distinction between a memory hogging application that might not require performance and some server process that has to be up and running with as little lag as possible. What is counted there in the terms of cache performance really? That 0.1% (that someone bothered to hand optimize the memory access patterns for) might be the only one really needing that cache. That, however, is not how you sell technology to the public, isn't it? You have to show them a general speed-up, even when it doesn't matter. You have to show them that the same application that used to do something in a given period of time does it now in less. That's how you sell. But that's not how you develop, hopefully.
  - tveita 10y ago
    > That 0.1% (that someone bothered to hand optimize the memory access patterns for) might be the only one really needing that cache. I am going to claim that it isn't. I think you underestimate the amount of speedup the cache gives you. Few applications require performance so little that they will tolerate a 50-100 times slowdown. Even if they could, who's to say the right 0.1% ends up optimized? You add a new function without profiling it, or a user has a slightly different use case than you do, and suddenly the hot path is waiting for main memory in every loop. This was all decided a long time ago, general purpose caches won and scratch memory lost. > You have to show them that the same application that used to do something in a given period of time does it now in less. That's how you sell. Yes, that sounds like a swell idea. What is wrong with that?
usefulcat 10y ago
Here's the real problem: which new processor is likely to be more commercially successful: the one with transparent caching that can benefit all existing software without so much as a recompile, or the one with the manually-managed "fast memory" that will only benefit that software which has been specifically tailored to use it?
- restalis 10y ago
  I conceded in one of my previous comments¹ that "that's how you sell" (being focused in the short term if not on "now"). If, however, one designs an architecture for the long term, then "one boot fits all" should be out of the question, on principle. Of course we have an inertia that will keep things going as they are for some time, but if one thinks about designing something afresh, then the lesson learned from the cache model experiment is not one that should be glossed over. ¹ https://news.ycombinator.com/item?id=12186357 https://news.ycombinator.com/item?id=12186357
  - oldmanjay 10y ago
    You seem to forget that trying to plan long-term technology trends is folly, because no one gets to be the technology dictator. What you ascribe to inertia is better assigned to evolution.
- corysama 10y ago
  But, that's not what this project is about. It's not an either-or situation. There are chips in the wild today that give you the option to move some or all of your cache to manual control if you want to. This project is about giving you the software interface needed for you to get access to the feature those chips already have. I feel I have to point this out not necessarily to usefulcat, but because I see a lot of comments arguing against a false-dichotomy strawman of this project.

obl 10y ago

Unless you want programmers to explicitely manage this fast memory the hardware is a better place to do those optimizations than the software. Compilers (even the high-end modern ones) are very quickly clueless about both memory and execution profiles. Those problems are hard to reason about statically. Is this pointer the same as this other one ? Will this loop be executed 10 times or 10^6 times ? The hardware being a dynamic optimizer has a lot of runtime information (pointer values, cache entry usage, ...) to take advantage of. Sure, one can argue that we just need better JITs, PGO, and programming languages that are easier to analyze statically. In the current state of affair we are way better off leaving this to the hardware. By the way you do have "some" control over it using NT moves on x86. Turns out almost nobody uses them except by hand in very specific peak-bandwidth code. Compilers don't dare emitting those since the penalty of getting it wrong outweights the benefits of getting it right.

restalis 10y ago
"In the current state of affair we are way better off leaving this to the hardware." That is something I'm not so sure about. That might have been true before, when the hardware served well the then simple computational model of a single high frequency processing core. Things got more complicated since the times of first CPU cache offerings and hardware general use-case implementation can only give you so much. The performance boost that once could be relied on without much care now dissipated. This cache thing now become just a leaky abstraction. We can not abstract it away completely if we care for performance, nor we can control it in any meaningful way. It is not a good model anymore. Just recognize it as a failed experiment. "Unless you want programmers to explicitely manage this fast memory..." Yes, that's exactly what I have in mind (and implied before). Most applications do not need maximal performance and can disregard the fast memory completely. That will only mean that those processes either will not consume their share and leave it more for the processes that need it, or that the fast memory will (occasionally) be used indirectly, through the optimized bits in the layers that the given process relies on. This memory, however, won't be unproductively overwritten or invalidated on every cold cache or whatnot.
dman 10y ago
I use non temporal stores all the time when writing database related code.
gaius 10y ago
Unless you want programmers to explicitely manage this fast memory That's how we did it in the good old days on the 6502 (zp address mode) and we liked it like that...

tveita 10y ago

The idea of scratchpad memory is as old as the hills, and you may still find it on some specialty platforms like gaming consoles, but the lesson learned is mostly that it ends up chronically underutilized. A cache speeds up nearly all code written for any system, not just the 0.1% that someone bothered to hand optimize the memory access patterns for.

restalis 10y ago
"the lesson learned is mostly that it [scratchpad memory] ends up chronically underutilized. A cache speeds up nearly all code written for any system" I recall that when that technology came out, there was a heavy marketing stunt involved. For the boast's sake, performance had to be conjured. And as seamlessly as possible. And as noticeable as possible. That really affected the design development, and not in a good way. The problem (in what I quoted) here is the poor measurement base, the wrong comparison base, and what is counted or discounted as value. So that cache really speeds up nearly all the code written for any system. Such a flat model does not, however, make much distinction between a memory hogging application that might not require performance and some server process that has to be up and running with as little lag as possible. What is counted there in the terms of cache performance really? That 0.1% (that someone bothered to hand optimize the memory access patterns for) might be the only one really needing that cache. That, however, is not how you sell technology to the public, isn't it? You have to show them a general speed-up, even when it doesn't matter. You have to show them that the same application that used to do something in a given period of time does it now in less. That's how you sell. But that's not how you develop, hopefully.
- tveita 10y ago
  > That 0.1% (that someone bothered to hand optimize the memory access patterns for) might be the only one really needing that cache. I am going to claim that it isn't. I think you underestimate the amount of speedup the cache gives you. Few applications require performance so little that they will tolerate a 50-100 times slowdown. Even if they could, who's to say the right 0.1% ends up optimized? You add a new function without profiling it, or a user has a slightly different use case than you do, and suddenly the hot path is waiting for main memory in every loop. This was all decided a long time ago, general purpose caches won and scratch memory lost. > You have to show them that the same application that used to do something in a given period of time does it now in less. That's how you sell. Yes, that sounds like a swell idea. What is wrong with that?

usefulcat 10y ago

Here's the real problem: which new processor is likely to be more commercially successful: the one with transparent caching that can benefit all existing software without so much as a recompile, or the one with the manually-managed "fast memory" that will only benefit that software which has been specifically tailored to use it?

restalis 10y ago
I conceded in one of my previous comments¹ that "that's how you sell" (being focused in the short term if not on "now"). If, however, one designs an architecture for the long term, then "one boot fits all" should be out of the question, on principle. Of course we have an inertia that will keep things going as they are for some time, but if one thinks about designing something afresh, then the lesson learned from the cache model experiment is not one that should be glossed over. ¹ https://news.ycombinator.com/item?id=12186357 https://news.ycombinator.com/item?id=12186357
- oldmanjay 10y ago
  You seem to forget that trying to plan long-term technology trends is folly, because no one gets to be the technology dictator. What you ascribe to inertia is better assigned to evolution.
corysama 10y ago
But, that's not what this project is about. It's not an either-or situation. There are chips in the wild today that give you the option to move some or all of your cache to manual control if you want to. This project is about giving you the software interface needed for you to get access to the feature those chips already have. I feel I have to point this out not necessarily to usefulcat, but because I see a lot of comments arguing against a false-dichotomy strawman of this project.