The 'sentiment neuron' two posts over gives an indication of what this could look like. 'Oh, I see you're giving a strong positive update to the porn neuron...'

The 'sentiment neuron' two posts over gives an indication of what this could look like. 'Oh, I see you're giving a strong positive update to the porn neuron...' More generally, there's the notion that a sufficiently complex model encodes the training data. There's been work on extracting training data (or highly deformed versions of it) from a fully trained neural network. We should be under no illusions that networks offer cryptographically​ strong protections to their memories. It's simply not a design goal.