This maps wonderfully onto SVD, Neural networks and embeddings. Word embeddings frequently encode particular traits in different 'regions' of a 256(ish) dimens

This maps wonderfully onto SVD, Neural networks and embeddings. Word embeddings frequently encode particular traits in different 'regions' of a 256(ish) dimensional space. AFAIK, It is also why we think of element wise addition (merging) in neural networks as an efficient and relatively loss-less computation. The aggregation after attention step used in Transformers (GPT-3) fundamentally relies on this being true. Although from my reading, there is an inherent assumption of sparsity in such situations. So, is it reasonable to assume that human neurons are also relatively sparse in how information is stored ?