I did research on vector-matrix last year: https://kolinko.github.io/effort/ https://kolinko.github.io/effort/ For semi-random weights you cam get down to 20-

I did research on vector-matrix last year: https://kolinko.github.io/effort/ https://kolinko.github.io/effort/ For semi-random weights you cam get down to 20-30% multiplications/mem reads and maintain ~0.98 cosine similarity output between the approximated and full result. As far as LLM inference goes, the speedup from removing multiplications is at best comparable to the speedup of quantisation (that is - you get at best similar KL divergence score whether you remove calculations or quantise).