Sparse matrices (i.e. filled mostly with zeroes) are apt to represent discrete structures, since you can interpret them as binary/n-hot vectors as opposed to po

Sparse matrices (i.e. filled mostly with zeroes) are apt to represent discrete structures, since you can interpret them as binary/n-hot vectors as opposed to points in space. I'd imagine there's some potential to do sparse matrix optimization on GPT since natural language has a tree-like structure, but it's a well known technique and I'm sure the big players are already doing it