2 ms·
Perhaps the most important public discussion, at least among technical people, is making sure we really understand the experimental setup. (I admit I could be
by xpe 3y ago
Perhaps the most important public discussion, at least among technical people, is making sure we really understand the experimental setup.
(I admit I could be projecting my lack of understanding; I’m not rock-solid on the experimental setup yet.)
Still to me, it matters if the linearity been described can be observed directly _in_ the model. My understanding is that it is not; rather, it is tested with an additional model. The term probing can be misunderstood; it isn’t -just- an internal probe. There are ‘derivative’ models as well.
This raises questions about what we mean when we we are talking about internal representations! In my view, a derivative of an internal representation is not the same thing.
(I welcome people who can educate, correct me, or even reframe the topics. The above thoughts are subject to change.)
- karelpeeters 3y agoIt's important to note that these probe models are very simple - they're "trained" by just doing a linear regression between the hidden activations and the desired output. This means that the probes can barely do any computation themselves, so if they work at all this is a strong indication that the signal they predict was already in the hidden activations. For even more proof, see "Figure 5: Space and time neurons in Llama-2 models" for single neurons in LLMs that already encode this information, without even having to use a probe model to extract it.
- xpe 3y agoThanks. After some additional research, I tend to agree. One thing I learned of note is that while a _positive_ result (i.e. a probe showing a relationship) is proof that a label is encoded somehow; a _negative_ result does _not_ prove that the network under test does _not_ encode the information _somehow_. (The information could be encoded in more complex ways that the probe does not discover).