3 ms·
If you're concerned about data leakage, it's worth noting that model weights can very easily be used to reconstruct the original data that it was trained on: so
by cpmpcpmp 4y ago
If you're concerned about data leakage, it's worth noting that model weights can very easily be used to reconstruct the original data that it was trained on: so it could be misleading to claim that user data isn't being shared over the network. To avoid this, you'd need to look into techniques like Secure Aggregation or local differential privacy. Flower does provide some of this, FWIW.
- onethought 4y agoThis doesn’t sound right, if they don’t know the structure of the NN how can the reconstruct from the weights alone? (Perhaps the structure is communicated within the weights?)
- aix1 4y agoEvery agent training the model on their proprietary data has to have access to the model form in some way (otherwise how would they train it?) For this reason, one must assume that the model form is known to the adversary. With this, the question becomes: is it possible to reconstruct training data from a trained model? We already know that, at least for some image models, the answer to that question is "yes": https://arxiv.org/pdf/2301.13188.pdf https://arxiv.org/pdf/2301.13188.pdf
- onethought 4y agoThat must only be true if there isn’t a one way compression step occurring, or any approximation in the whole model.
- aix1 4y agoI don't think lossy compression is sufficient. The very first example in the paper I linked to is clearly not identical to the original image (=lossily compressed) yet leaks a training image in a way that would be highly problematic in certain domains, e.g. medical imaging.
- onethought 4y agoI see what you are saying. Agree. Seems we need some set patterns in NN models that will reliably remove reversibility without effecting loss too drastically.