2 ms·
This is a good point, and I expect in the near future we will be able to simply ask a neural network to explain and change its behavior without retraining. I al
by p1esk 4y ago
This is a good point, and I expect in the near future we will be able to simply ask a neural network to explain and change its behavior without retraining. I already see signs of that in prompt engineering used to interact with GPT3 or Dalle2.
- tinco 4y agoIt's just going to be as good as asking a human to explain its behaviour. It might be a semi-accurate interpretation of its actions based on its internal knowledge, but it's never going to be the actual thing. The actual decision making inside a neural network is fundamentally not something you could simplify into language exactly.
- p1esk 4y agoYeah, it's not clear if we will be satisfied with those explanations: "I decided to slow down because I see these (seemingly random and irrelevant) objects around me", because during training on trillions of video frames the model has learned it should slow down when similar object configurations are present to reduce the chance of an accident by 0.0002%. Even if the given reason is simple, like "I decided to slow down because the car in front of me is red", explicit override of the learned rule ("don't slow down when you see red cars") might potentially increase the chance of the accident by a lot more than 0.0002% because we are messing with the model decision making process.