3 ms·
It is much worse. Binaries can be analyzed and AVs do it very efficiently. It is hard to even imagine an AV for an AI model.
by ivan_gammel 3y ago
It is much worse. Binaries can be analyzed and AVs do it very efficiently. It is hard to even imagine an AV for an AI model.
- Dig1t 3y agoI wonder if we'll have something akin to AVs for these models in the future. I can imagine it looking for malicious patterns in the weights, like analyzing if the model's concept of a "secret command" has any special connections/activations, etc. As a thought experiment: you can imagine that if we had a technology to read all the neurons in the human brain in real time, we could probably build a lie detector that works pretty well by looking for patterns that are commonly observed when people are lying.
- ivan_gammel 3y agoI do not think it is possible to guess model behavior by just looking at the weights. My feeling is that this may even have a mathematical proof. Your thought experiment is not really the same: it would require the model to be self-conscious, to understand when it expresses a behavior contradicting the expectations and to measure those patterns in vivo (what btw is neuron activation in case of AI model?)