3 ms·
From my understanding in this implementation there is some amount of knowledge about the model itself needed to determine what parts to place in system memory v
by jupp0r 3y ago
From my understanding in this implementation there is some amount of knowledge about the model itself needed to determine what parts to place in system memory vs what parts to place in GPU memory. Can this ideally be computed automatically or will future models have some sort of interface for placement algorithms like this to help automate this? If the algorithm needs to be adopted for each model architecture, it's going to be a lot of work to maintain this project.
- loudmax 3y agoThat sounds about right. They provide a script to combine their "Predictor" weights to the original models, but I don't see anything obvious in the front page of the Github repo about how to create those weights. A 10x speed improvement is really impressive. If this kind of improvement is reproducible across other models, then presumably identifying hot and cold neurons for inference optimization should go on to become a normal part of model development process.
- thelastparadise 3y agoLike JVM "hot spots," or JIT optimization.
- jupp0r 3y agoOr profile guided optimization.