6 ms·

Differential privacy techniques are provably impossible to de-anonymize, if implemented correctly. It is possible. But fraught with possibility for error or man

by bo1024 3y ago

Differential privacy techniques are provably impossible to de-anonymize, if implemented correctly. It is possible. But fraught with possibility for error or manipulation.

Aachen 3y ago
This is the answer. The person above can speculate and fearmonger what magic "AI" is going to be able to do, but if there is no personal data there to begin with, or if you use math like in differential privacy, there's not going to be a way to identify individuals. That is, if you suspect they'd change their minds and start trying to deanonymise previously collected data anyway — remember that open source distributions (I don't know fedora-the-organisation specifically) are generally made up of volunteers like you and me. Notable exceptions obviously exist, like for-profit Canonical; that's not the org type I mean or trust.
- jacquesm 3y ago
  In practice though none of that matters because they'll do a slipshod job of implementing it.
  - Aachen 3y ago
    With that logic, we shouldn't have police or a judicial system either If we can't trust people ever, what's the point in doing anything?
    - jacquesm 3y ago
      No, obviously that's not even remotely the same. You need police and a judicial system and you fix them whenever they break. But you don't need telemetry, it's entirely optional and shoddy implementations translate into unnecessary risk. Also: https://lwn.net/ml/fedora-devel/H5JEXR.LLU011IQ4I6K@redhat.com/ https://lwn.net/ml/fedora-devel/H5JEXR.LLU011IQ4I6K@redhat.c...
    - JohnFen 3y ago
      You have recourse is the police or judicial system abuses you. You have no recourse if a company does.
      - Aachen 3y ago
        Given it is open source software, I'd say we have not only recourse but self-determination here. It's also transparent what is being collected and whom it is being sent to. If you want to know what they store about you, it should take one email and a GDPR reference to find out. I'm no fan of privacy invasion, always bother clicking through the banner to find the reject cookies option, and sent out plenty of GDPR requests whereas almost nobody else I know ever sent one. I'm not in favor of tracking, but collecting anonymous statistics, especially when they open with "privacy-preserving" and the business is not Facebook or Google or so where we know there's shit about to hit the fan, the cynicism and mistrust in this thread baffles me. Nobody minds when they browse the web and every site keeps access logs invisibly, but oh boy if someone announces keeping a visitor counter for a configuration screen to see if people can find their way to it
        hakre 3y ago
        - It does not ask for consent - It does not allow a reasonable decision as it does not show the data before it sends it
        JohnFen 3y ago
        > it should take one email and a GDPR reference to find out. I am not in Europe. The GDPR doesn't help me. > collecting anonymous statistics, especially when they open with "privacy-preserving" I am far from convinced that such statistics are gathered in a "privacy preserving" way, but that's neither here nor there. > the business is not Facebook or Google or so where we know there's shit about to hit the fan The problem is that you can't just trust the current devs. You also have to trust all future devs and companies that may buy the thing. It's not Facebook or Google now, but it could be in the future. And this is Fedora, which is connected to Red Hat, which is connected to IBM. And it's also not just about privacy. It's also about impact on product development. It's not exactly rare that software has been made much worse as a result of decision-making based on telemetry data. > Nobody minds when they browse the web and every site keeps access logs invisibly No? I think quite a lot of people mind this. But there's nothing that can be done about that. It's still worth trying to keep everything from getting even worse, though.
    - deleted 3y ago
      [deleted]
- JohnFen 3y ago
  > if there is no personal data there to begin with Which is a very large "if". There is a tendency for people to think that the only personal data consists of what legally counts as PII, when in fact there is much more personally identifying information than is covered in those definitions.
TrueDuality 3y ago
While true, the requestor had never heard of differential privacy techniques and that is not what they were planning on implementing. This was brought up in the discussion thread.
hedora 3y ago
That's not true. Differential privacy just says that the output of the ML model will not change much if you add or remove a particular user, so you can't reverse the training process to infer what training data a given user provided. It says nothing about whether or not you can join the output of multiple ML models with other telemetry to build a deanonymization model.
- bo1024 3y ago
  Differential privacy satisfies a post-processing guarantee. It says that if you take the output of a differentially private process and do any amount of processing and combining with outside information, then you don't learn any more than you would have gotten with the outside information alone (up to epsilon).

by bo1024 3y ago

Differential privacy techniques are provably impossible to de-anonymize, if implemented correctly. It is possible. But fraught with possibility for error or manipulation.

Aachen 3y ago

This is the answer. The person above can speculate and fearmonger what magic "AI" is going to be able to do, but if there is no personal data there to begin with, or if you use math like in differential privacy, there's not going to be a way to identify individuals. That is, if you suspect they'd change their minds and start trying to deanonymise previously collected data anyway — remember that open source distributions (I don't know fedora-the-organisation specifically) are generally made up of volunteers like you and me. Notable exceptions obviously exist, like for-profit Canonical; that's not the org type I mean or trust.

jacquesm 3y ago
In practice though none of that matters because they'll do a slipshod job of implementing it.
- Aachen 3y ago
  With that logic, we shouldn't have police or a judicial system either If we can't trust people ever, what's the point in doing anything?
  - jacquesm 3y ago
    No, obviously that's not even remotely the same. You need police and a judicial system and you fix them whenever they break. But you don't need telemetry, it's entirely optional and shoddy implementations translate into unnecessary risk. Also: https://lwn.net/ml/fedora-devel/H5JEXR.LLU011IQ4I6K@redhat.com/ https://lwn.net/ml/fedora-devel/H5JEXR.LLU011IQ4I6K@redhat.c...
  - JohnFen 3y ago
    You have recourse is the police or judicial system abuses you. You have no recourse if a company does.
    - Aachen 3y ago
      Given it is open source software, I'd say we have not only recourse but self-determination here. It's also transparent what is being collected and whom it is being sent to. If you want to know what they store about you, it should take one email and a GDPR reference to find out. I'm no fan of privacy invasion, always bother clicking through the banner to find the reject cookies option, and sent out plenty of GDPR requests whereas almost nobody else I know ever sent one. I'm not in favor of tracking, but collecting anonymous statistics, especially when they open with "privacy-preserving" and the business is not Facebook or Google or so where we know there's shit about to hit the fan, the cynicism and mistrust in this thread baffles me. Nobody minds when they browse the web and every site keeps access logs invisibly, but oh boy if someone announces keeping a visitor counter for a configuration screen to see if people can find their way to it
      - hakre 3y ago
        - It does not ask for consent - It does not allow a reasonable decision as it does not show the data before it sends it
      - JohnFen 3y ago
        > it should take one email and a GDPR reference to find out. I am not in Europe. The GDPR doesn't help me. > collecting anonymous statistics, especially when they open with "privacy-preserving" I am far from convinced that such statistics are gathered in a "privacy preserving" way, but that's neither here nor there. > the business is not Facebook or Google or so where we know there's shit about to hit the fan The problem is that you can't just trust the current devs. You also have to trust all future devs and companies that may buy the thing. It's not Facebook or Google now, but it could be in the future. And this is Fedora, which is connected to Red Hat, which is connected to IBM. And it's also not just about privacy. It's also about impact on product development. It's not exactly rare that software has been made much worse as a result of decision-making based on telemetry data. > Nobody minds when they browse the web and every site keeps access logs invisibly No? I think quite a lot of people mind this. But there's nothing that can be done about that. It's still worth trying to keep everything from getting even worse, though.
  - deleted 3y ago
    [deleted]
JohnFen 3y ago
> if there is no personal data there to begin with Which is a very large "if". There is a tendency for people to think that the only personal data consists of what legally counts as PII, when in fact there is much more personally identifying information than is covered in those definitions.

TrueDuality 3y ago

While true, the requestor had never heard of differential privacy techniques and that is not what they were planning on implementing. This was brought up in the discussion thread.

hedora 3y ago

That's not true. Differential privacy just says that the output of the ML model will not change much if you add or remove a particular user, so you can't reverse the training process to infer what training data a given user provided. It says nothing about whether or not you can join the output of multiple ML models with other telemetry to build a deanonymization model.

bo1024 3y ago
Differential privacy satisfies a post-processing guarantee. It says that if you take the output of a differentially private process and do any amount of processing and combining with outside information, then you don't learn any more than you would have gotten with the outside information alone (up to epsilon).