3 ms·
Now I’m wondering what is the eigenspace of an LLM? If I take a set of LLM’s with the same number of parameters, then what are the eigenvectors? Do they have di
by harveyrook 3mo ago
Now I’m wondering what is the eigenspace of an LLM? If I take a set of LLM’s with the same number of parameters, then what are the eigenvectors? Do they have different personalities?
- bee_rider 3mo agoNeural networks are non-linear, so I think you wouldn’t be able to compute typical eigenvalues. You could compute the eigenvalues and/or singular of the individual weight matrices (I’m sure this has been studied). SVDs are very conventional for making low-rank approximations, so it must have been studied. The concept of nonlinear eigenvalues exists, but it is a bit more exotic.
- dapperdrake 3mo agoI saw a presentation about this in 2022. Someone found a way to get "something like" a tri-diagonal matrix that was equivalent to the LLM they were studying in 2022. Apologies for being informal and hand-wavey. Been a long time and I probably forgot a few important points.