The main idea is to take each weight matrix W, divide it into smaller blocks, each of dimension, say, n×n, and make each block a circulant matrix that can be sp

The main idea is to take each weight matrix W, divide it into smaller blocks, each of dimension, say, n×n, and make each block a circulant matrix that can be specified by a vector of only n elements.[a] This reduces W's memory consumption by a factor of n, and makes other gains in computational efficiency possible. Read the paper for details. However, as far as I can tell, it appears the authors have tested this technique only with one RNN architecture, in one task. It's hard to know whether the technique will hold up well in a broad range of RNN architectures/tasks. [a] https://en.wikipedia.org/wiki/Circulant_matrix https://en.wikipedia.org/wiki/Circulant_matrix