3 ms·
I wonder if we'll have something akin to AVs for these models in the future. I can imagine it looking for malicious patterns in the weights, like analyzing if t
by Dig1t 3y ago
I wonder if we'll have something akin to AVs for these models in the future. I can imagine it looking for malicious patterns in the weights, like analyzing if the model's concept of a "secret command" has any special connections/activations, etc.
As a thought experiment: you can imagine that if we had a technology to read all the neurons in the human brain in real time, we could probably build a lie detector that works pretty well by looking for patterns that are commonly observed when people are lying.
- ivan_gammel 3y agoI do not think it is possible to guess model behavior by just looking at the weights. My feeling is that this may even have a mathematical proof. Your thought experiment is not really the same: it would require the model to be self-conscious, to understand when it expresses a behavior contradicting the expectations and to measure those patterns in vivo (what btw is neuron activation in case of AI model?)