4 ms·

Awesome work! But I don't understand why they haven't implemented a neural network to take care of this. Jeff Dean even mentioned something about it[0]. [0] -

Awesome work! But I don't understand why they haven't implemented a neural network to take care of this. Jeff Dean even mentioned something about it[0]. [0] - https://learningsys.org/nips17/assets/slides/dean-nips17.pdf https://learningsys.org/nips17/assets/slides/dean-nips17.pdf (pp 29, 50 etc)

magicalhippo 6y ago
I guess your comment has been downvoted since it comes off a bit like "neural network all the things". But it did make me wonder, so I went searching and found this paper: "Applying Deep Learning to the Cache Replacement Problem"[1], which sounded quite interesting. In the paper they train a LSTM network to predict if an item should be cached or not, but then analyze the resulting model and identify key features of its performance. From the paper: We use these insights to design a new hand-crafted feature that represents a program’s control-flow history compactly and that can be used with a much simpler linear learning model known as a support vector machine (SVM). Our SVM is trained online in hardware, and it matches the LSTM’s offline accuracy with significantly less overhead; in fact, we show that with our hand-crafted feature, an online SVM is equivalent to a perceptron, which has been used in commercial branch predictors. We use these insights to produce the Glider4 cache replacement policy, which uses an SVM-based predictor to outperform the best cache replacement policies from the 2nd Cache Replacement Championship. Glider significantly improves upon Hawkeye, the previous state-of-the-art and winner of the 2nd Cache Replacement Championship. Training a neural network and then using it to restate the problem in more optimal way is an approach I haven't seen before (though I'm just a casual observer of the field), and which sounds very interesting. [1]: https://www.cs.utexas.edu/~lin/papers/micro19c.pdf https://www.cs.utexas.edu/~lin/papers/micro19c.pdf
- throwaway29303 6y ago
  I don't think I've read that before, thanks! Yeah, I believe there's also a very primitive NN on AMD's latest chips as well. I believe they used it on their branch predictors. I can't find the URL at such a short notice, though.
- NovaX 6y ago
  Two other papers that may be of interest are RC-Cache [1] (admission policy) and Deep Cache [2] (prefetch policy). [1] https://ieeexplore.ieee.org/document/9109339 https://ieeexplore.ieee.org/document/9109339 [2] https://dl.acm.org/doi/10.1145/3229543.3229555 https://dl.acm.org/doi/10.1145/3229543.3229555
hderms 6y ago
Do you think its possible to run a neural net all the time without destroying cache coherency? Would it need to be running on dedicated silicon? I think the idea could be interesting but it seems like there are practical ramifications to work through
- throwaway29303 6y ago
  From my understanding of NNs and operating systems I don't see any problem with it, but then again I'm not an expert. But Jeff definitely knows what he's talking about.

magicalhippo 6y ago

I guess your comment has been downvoted since it comes off a bit like "neural network all the things". But it did make me wonder, so I went searching and found this paper: "Applying Deep Learning to the Cache Replacement Problem"[1], which sounded quite interesting. In the paper they train a LSTM network to predict if an item should be cached or not, but then analyze the resulting model and identify key features of its performance. From the paper: We use these insights to design a new hand-crafted feature that represents a program’s control-flow history compactly and that can be used with a much simpler linear learning model known as a support vector machine (SVM). Our SVM is trained online in hardware, and it matches the LSTM’s offline accuracy with significantly less overhead; in fact, we show that with our hand-crafted feature, an online SVM is equivalent to a perceptron, which has been used in commercial branch predictors. We use these insights to produce the Glider4 cache replacement policy, which uses an SVM-based predictor to outperform the best cache replacement policies from the 2nd Cache Replacement Championship. Glider significantly improves upon Hawkeye, the previous state-of-the-art and winner of the 2nd Cache Replacement Championship. Training a neural network and then using it to restate the problem in more optimal way is an approach I haven't seen before (though I'm just a casual observer of the field), and which sounds very interesting. [1]: https://www.cs.utexas.edu/~lin/papers/micro19c.pdf https://www.cs.utexas.edu/~lin/papers/micro19c.pdf

throwaway29303 6y ago
I don't think I've read that before, thanks! Yeah, I believe there's also a very primitive NN on AMD's latest chips as well. I believe they used it on their branch predictors. I can't find the URL at such a short notice, though.
NovaX 6y ago
Two other papers that may be of interest are RC-Cache [1] (admission policy) and Deep Cache [2] (prefetch policy). [1] https://ieeexplore.ieee.org/document/9109339 https://ieeexplore.ieee.org/document/9109339 [2] https://dl.acm.org/doi/10.1145/3229543.3229555 https://dl.acm.org/doi/10.1145/3229543.3229555

hderms 6y ago

Do you think its possible to run a neural net all the time without destroying cache coherency? Would it need to be running on dedicated silicon? I think the idea could be interesting but it seems like there are practical ramifications to work through

throwaway29303 6y ago
From my understanding of NNs and operating systems I don't see any problem with it, but then again I'm not an expert. But Jeff definitely knows what he's talking about.