It took me a while to understand that paper, because it builds on the techniques of the Deja Vu paper for leveraging sparsity which are already pretty complex:

It took me a while to understand that paper, because it builds on the techniques of the Deja Vu paper for leveraging sparsity which are already pretty complex: - First, the Deja Vu paper observes that models with low weight sparsity have what they call high "contextual sparsity". Basically, the matrix multiplications will produce vectors with lots of zeros in them, but where the zeros are depends on the input. - The paper notes that you can use that sparsity to skip loading some rows of your matrices. - However, to get good performance benefits, you can to predict in advance which rows you're going to skip. You can do that with a low-rank matrix. The Apple paper then suggests these findings can not just improve your performance loading from RAM, but even allow you to load from flash memory without sacrificing bandwidth: - The paper notes that attention matrices are rather light; the FFNs are the ones you want to load sparsely. - The paper notes that you can get much better sparsity by predicting the output of the ReLU layer rather than the input of the FFN. Basically saying "if you can predict that after matmul this slot of the vector will have a negative value before ReLU, you can skip loading the matrix column and output zero". - The paper suggests that you don't need to load most rows of the FFN at all; you can just keep a cache of recently used FFN rows for each FFN, and update it from flash memory on-demand. There's a bunch more about chunk loading and the correlation between projection layers, but the above is where I think the main insights are. (FFN = Feed Forward Network; in the context of transformers they're the biggest blocks.)