You may decompose matrix for each layer into two matrices with a bottleneck layer. E.g., A=BC (approximately) where A is MxN matrix and B and C are MxK and KxN

You may decompose matrix for each layer into two matrices with a bottleneck layer. E.g., A=BC (approximately) where A is MxN matrix and B and C are MxK and KxN matrices, where K is small enough. It is done using SVD and improves speed without sacrificing memory access patterns.