3 ms·

That's bonkers. And the beauty of it is that now someone could realistically do a GDPR Erasure request on the Neural Net. I do hope that they're able to reverse

That's bonkers. And the beauty of it is that now someone could realistically do a GDPR Erasure request on the Neural Net. I do hope that they're able to reverse data out.

qayxc 5y ago
Since the information is encoded in model weights, I doubt that erasure is even possible. Only post-retrieval filtering would be an option. It only goes to show that intransparent black-box models have no place in the industry. The networks leak information left and right, because it's way too easy to just crawl the web and throw terabytes of unfiltered data at the training process.
- bencollier49 5y ago
  > Only post-retrieval filtering would be an option. And illegal, if the original information remains. I assume that there must be a process for altering the training data set and rerunning the entire thing.
  - gmueckl 5y ago
    The problem is that the information is in an opaque encoding that nobody can reverse engineer today. So it's impossible to prove that a certain subset of data has been removed from the model. Say, you have a model that repeats certain PII when prompted in a way that I figure out. I show you the prompt, you retrain the model to give a different, non-offensive answer. But now I go and alter the prompt and the same PII reappears. What now?
  - computerex 5y ago
    Yes, but the compute costs required for training are probably in the range of hundreds of thousands of usd to potentially millions of usd. Not to mention potentially months of training time.
    - 6gvONxR4sf7o 5y ago
      Good thing “compliance is really expensive” isn’t a valid legal defense.
- ohazi 5y ago
  I think the fact that there's no way to delete the data in question without throwing away the entire model is a feature... The strategic goal of a GPDR erasure request would be to force GitHub to nuke this thing from orbit.
- jeroenhd 5y ago
  If this system includes personal information that cannot be removed, corrected or controlled, it's probably a gross violation of all European and some American privacy laws. Designing a system that you cannot control does not grant you legal immunity for whatever the system does. As Github operates inside the EU, personal information this system contains MUST be deleteable, correctable and retrievable, or it's simply illegal.

That's bonkers. And the beauty of it is that now someone could realistically do a GDPR Erasure request on the Neural Net. I do hope that they're able to reverse data out.

qayxc 5y ago

Since the information is encoded in model weights, I doubt that erasure is even possible. Only post-retrieval filtering would be an option. It only goes to show that intransparent black-box models have no place in the industry. The networks leak information left and right, because it's way too easy to just crawl the web and throw terabytes of unfiltered data at the training process.

bencollier49 5y ago
> Only post-retrieval filtering would be an option. And illegal, if the original information remains. I assume that there must be a process for altering the training data set and rerunning the entire thing.
- gmueckl 5y ago
  The problem is that the information is in an opaque encoding that nobody can reverse engineer today. So it's impossible to prove that a certain subset of data has been removed from the model. Say, you have a model that repeats certain PII when prompted in a way that I figure out. I show you the prompt, you retrain the model to give a different, non-offensive answer. But now I go and alter the prompt and the same PII reappears. What now?
- computerex 5y ago
  Yes, but the compute costs required for training are probably in the range of hundreds of thousands of usd to potentially millions of usd. Not to mention potentially months of training time.
  - 6gvONxR4sf7o 5y ago
    Good thing “compliance is really expensive” isn’t a valid legal defense.
ohazi 5y ago
I think the fact that there's no way to delete the data in question without throwing away the entire model is a feature... The strategic goal of a GPDR erasure request would be to force GitHub to nuke this thing from orbit.
jeroenhd 5y ago
If this system includes personal information that cannot be removed, corrected or controlled, it's probably a gross violation of all European and some American privacy laws. Designing a system that you cannot control does not grant you legal immunity for whatever the system does. As Github operates inside the EU, personal information this system contains MUST be deleteable, correctable and retrievable, or it's simply illegal.