You can use gather operations: https://stackoverflow.com/questions/22330322/dereference-pointers-in-xmm-register-gather https://stackoverflow.com/questions/2233

You can use gather operations: https://stackoverflow.com/questions/22330322/dereference-pointers-in-xmm-register-gather https://stackoverflow.com/questions/22330322/dereference-poi... And they are not "order of magnitude slower": https://stackoverflow.com/questions/24756534/in-what-situation-would-the-avx2-gather-instructions-be-faster-than-individually https://stackoverflow.com/questions/24756534/in-what-situati... Unless you operate in binary, of course. Results from above suggest that you can leave 12.5% of your weights to be non-zero and get nice 4x speed up (1/8 operations done twice as slow). Accuracy start to drop at about 12.5% of weights remaining in both OP and OpenAI papers.