3 ms·
The exact same thing is true of binaries in general. If you can't see the source, you are just trusting whoever compiled the binary that the thing you are runni
by Dig1t 3y ago
The exact same thing is true of binaries in general. If you can't see the source, you are just trusting whoever compiled the binary that the thing you are running is not doing anything malicious. This just seems like framing an old problem in a new scary-sounding AI way.
- ivan_gammel 3y agoIt is much worse. Binaries can be analyzed and AVs do it very efficiently. It is hard to even imagine an AV for an AI model.
- Dig1t 3y agoI wonder if we'll have something akin to AVs for these models in the future. I can imagine it looking for malicious patterns in the weights, like analyzing if the model's concept of a "secret command" has any special connections/activations, etc. As a thought experiment: you can imagine that if we had a technology to read all the neurons in the human brain in real time, we could probably build a lie detector that works pretty well by looking for patterns that are commonly observed when people are lying.
- ivan_gammel 3y agoI do not think it is possible to guess model behavior by just looking at the weights. My feeling is that this may even have a mathematical proof. Your thought experiment is not really the same: it would require the model to be self-conscious, to understand when it expresses a behavior contradicting the expectations and to measure those patterns in vivo (what btw is neuron activation in case of AI model?)