4 ms·

This is speculative, but it seems like the privacy aspect is oversold as it may be possible to reverse engineer the input data from the model updates. The point

This is speculative, but it seems like the privacy aspect is oversold as it may be possible to reverse engineer the input data from the model updates. The point is that the model updates themselves are specific to each user.

ehsankia 10y ago
Well you obviously can't fully reverse engineer, since whatever model update being sent is far far smaller than the overall data. Now, could you theoretically extract "some" data? Maybe, but it still is strictly better than sending all of the data.
- sdenton4 10y ago
  The 'sentiment neuron' two posts over gives an indication of what this could look like. 'Oh, I see you're giving a strong positive update to the porn neuron...' More generally, there's the notion that a sufficiently complex model encodes the training data. There's been work on extracting training data (or highly deformed versions of it) from a fully trained neural network. We should be under no illusions that networks offer cryptographically strong protections to their memories. It's simply not a design goal.
  - justonepost 10y ago
    Yep. Basically you just reverse the network and predict the source - if not done right.
- argonaut 10y ago
  No, generally models are higher dimensional than their inputs (AKA deep learning models generally have more parameters than the number of features).
fiter 10y ago
I would not be surprised if the specific contents of what you wrote was not accessible but I would expect that the general theme would be captured. The weights that are updated would correspond to neurons that would then correspond to specific themes/subjects.
justonepost 10y ago
Yes! You can reverse engineer if you're not careful. You have to make sure you have a reasonably sufficient baseline that you're training locally on. Otherwise your delta update is extremely revealing. Even then, you have to cautious because if not done correctly you can extract the delta.

ehsankia 10y ago

Well you obviously can't fully reverse engineer, since whatever model update being sent is far far smaller than the overall data. Now, could you theoretically extract "some" data? Maybe, but it still is strictly better than sending all of the data.

sdenton4 10y ago
The 'sentiment neuron' two posts over gives an indication of what this could look like. 'Oh, I see you're giving a strong positive update to the porn neuron...' More generally, there's the notion that a sufficiently complex model encodes the training data. There's been work on extracting training data (or highly deformed versions of it) from a fully trained neural network. We should be under no illusions that networks offer cryptographically strong protections to their memories. It's simply not a design goal.
- justonepost 10y ago
  Yep. Basically you just reverse the network and predict the source - if not done right.
argonaut 10y ago
No, generally models are higher dimensional than their inputs (AKA deep learning models generally have more parameters than the number of features).

fiter 10y ago

I would not be surprised if the specific contents of what you wrote was not accessible but I would expect that the general theme would be captured. The weights that are updated would correspond to neurons that would then correspond to specific themes/subjects.

justonepost 10y ago

Yes! You can reverse engineer if you're not careful. You have to make sure you have a reasonably sufficient baseline that you're training locally on. Otherwise your delta update is extremely revealing. Even then, you have to cautious because if not done correctly you can extract the delta.