4 ms·

This is not the way it works at all (see Secure Aggregation). There are a number of techniques out there that permit privacy safe services and learning, like Di

This is not the way it works at all (see Secure Aggregation). There are a number of techniques out there that permit privacy safe services and learning, like Differential Privacy, Federated Learning, and there's even Deep Neural Nets using homomorphic computing (e.g. https://arxiv.org/abs/1711.05189 https://arxiv.org/abs/1711.05189) How about we lay off the conspiracy theories every time any new paper is published.

krick 7y ago
"Secure aggregation" doesn't mean a thing, because I would argue, as I did in another comment already, that "you" are not "your device".
- cromwellian 7y ago
  You can put as many words as you want in quotes, but if you can point out a specific flaw in the secure aggregation paper, either in the algorithm description, or the math, it would be taken a lot more seriously. It's like your responding to the Bitcoin paper by saying "proof of work" doesn't mean a thing because "hashing" isn't "money". There are papers published, critique those specifically, instead of relying on a handwavy respond to a comic. https://ai.google/research/pubs/pub47246 https://ai.google/research/pubs/pub47246
  - krick 7y ago
    It is somehow understandable for a Google employee to take such a defensive position, but you either didn't read or didn't try to understand what I'm saying. I don't believe there is anything wrong with federated learning or that there are any problems with "secure aggregation" you are so fondly referring to. It's just that they don't mean shit, because, as I said several times already, you don't need to know that some data comes from my device, to learn something sensitive about me. It depends entirely on the application, no crypto has to be broken in order for that to happen.
    - cromwellian 7y ago
      Well, I've been working on crypto and privacy since before the Web, 25+ years before I joined Google, released some of the first anonymizing proxy servers on the internet, designed one of the first mix-net PGP remailer networks, etc. I wouldn't be critiquing your response if you had something a little less handwavey to say. When people point out flaws in protocols, I like to see specific exploit examples, like the kind you'd put into a Spectre/Meltdown Advisory. If you said "out of order speculative execution in CPUs might eventually allow exploits", I may even have vaguely agreed, but without a concrete criticism, it's more of an "uneasy feeling" you have. I can make loose arguments too. Everything you do in this word leaks entropy. Your information is entangled with other people, leaving a wake behind you from the moment you're born. There's a gazillion side channels hanging off of you. So learning information about you is pretty much a given. The question is, is it relevant or important information? There's a huge difference between "learning something", "learning something sensitive about me (that I share with a large number of other people)" and "learning something sensitive about me that's individually traceable to me" Most people won't care if a Federated Learning model, using data from your phone, learns that people who stay up late, and search for Coke, also end up with diabetes, anymore than a double blind study learns about the risks of smoking and lung cancer -- the doctors have learned information about you (you're a smoker, and you have/don't have lung cancer), but they haven't learned that you, krick, are a smoker. The whole point of differential privacy, federated learning, and other techniques, is that aggregate statistics, and aggregate models can be learned without any personally traceable information. Now, you could argue that somewhere, deep within the logical depth of the weights of a DNN is some kind of personal information that could be deanonymized, but this is like a claim that you found a weakness in a hash function -- until you show it, it's just a claim, and mathematics and security research is full of wrong claims on both sides.
- CyanTas 7y ago
  Secure aggregation isn’t meaningless, but you’re right that it’s the wrong tool for the problem you’re talking about. The right tool is differential privacy. Differential privacy is exactly meant for this, in fact. Differential privacy adds a certain amount of randomly-generated noise to client inputs. The result is that, statistically speaking, it’s impossible to tell the difference between a model with your data in it and a model without your data in it. Arguably the reason the comic doesn’t mention differential privacy is that it’s neither new nor invented at Google. Or maybe just because it’s not technically part of federated learning. But the “federated learning at scale” paper Google put out mentions it, and says they have implemented DP techniques.
  - krick 7y ago
    This one is interesting, I'll have to read about that. Right now it doesn't seem to make any sense to me. I mean, if statistically model w/ my data is no different from the model w/o my data, then by definition it must be no better or worse. If it would truly be the case, there wouldn't be any reason to even include the result of such training, would it?
    - CyanTas 7y ago
      I definitely recommend reading more about it because I’m not the best at explaining it. But differential privacy (without federated learning) is what Apple has been doing.

krick 7y ago

"Secure aggregation" doesn't mean a thing, because I would argue, as I did in another comment already, that "you" are not "your device".

cromwellian 7y ago
You can put as many words as you want in quotes, but if you can point out a specific flaw in the secure aggregation paper, either in the algorithm description, or the math, it would be taken a lot more seriously. It's like your responding to the Bitcoin paper by saying "proof of work" doesn't mean a thing because "hashing" isn't "money". There are papers published, critique those specifically, instead of relying on a handwavy respond to a comic. https://ai.google/research/pubs/pub47246 https://ai.google/research/pubs/pub47246
- krick 7y ago
  It is somehow understandable for a Google employee to take such a defensive position, but you either didn't read or didn't try to understand what I'm saying. I don't believe there is anything wrong with federated learning or that there are any problems with "secure aggregation" you are so fondly referring to. It's just that they don't mean shit, because, as I said several times already, you don't need to know that some data comes from my device, to learn something sensitive about me. It depends entirely on the application, no crypto has to be broken in order for that to happen.
  - cromwellian 7y ago
    Well, I've been working on crypto and privacy since before the Web, 25+ years before I joined Google, released some of the first anonymizing proxy servers on the internet, designed one of the first mix-net PGP remailer networks, etc. I wouldn't be critiquing your response if you had something a little less handwavey to say. When people point out flaws in protocols, I like to see specific exploit examples, like the kind you'd put into a Spectre/Meltdown Advisory. If you said "out of order speculative execution in CPUs might eventually allow exploits", I may even have vaguely agreed, but without a concrete criticism, it's more of an "uneasy feeling" you have. I can make loose arguments too. Everything you do in this word leaks entropy. Your information is entangled with other people, leaving a wake behind you from the moment you're born. There's a gazillion side channels hanging off of you. So learning information about you is pretty much a given. The question is, is it relevant or important information? There's a huge difference between "learning something", "learning something sensitive about me (that I share with a large number of other people)" and "learning something sensitive about me that's individually traceable to me" Most people won't care if a Federated Learning model, using data from your phone, learns that people who stay up late, and search for Coke, also end up with diabetes, anymore than a double blind study learns about the risks of smoking and lung cancer -- the doctors have learned information about you (you're a smoker, and you have/don't have lung cancer), but they haven't learned that you, krick, are a smoker. The whole point of differential privacy, federated learning, and other techniques, is that aggregate statistics, and aggregate models can be learned without any personally traceable information. Now, you could argue that somewhere, deep within the logical depth of the weights of a DNN is some kind of personal information that could be deanonymized, but this is like a claim that you found a weakness in a hash function -- until you show it, it's just a claim, and mathematics and security research is full of wrong claims on both sides.
CyanTas 7y ago
Secure aggregation isn’t meaningless, but you’re right that it’s the wrong tool for the problem you’re talking about. The right tool is differential privacy. Differential privacy is exactly meant for this, in fact. Differential privacy adds a certain amount of randomly-generated noise to client inputs. The result is that, statistically speaking, it’s impossible to tell the difference between a model with your data in it and a model without your data in it. Arguably the reason the comic doesn’t mention differential privacy is that it’s neither new nor invented at Google. Or maybe just because it’s not technically part of federated learning. But the “federated learning at scale” paper Google put out mentions it, and says they have implemented DP techniques.
- krick 7y ago
  This one is interesting, I'll have to read about that. Right now it doesn't seem to make any sense to me. I mean, if statistically model w/ my data is no different from the model w/o my data, then by definition it must be no better or worse. If it would truly be the case, there wouldn't be any reason to even include the result of such training, would it?
  - CyanTas 7y ago
    I definitely recommend reading more about it because I’m not the best at explaining it. But differential privacy (without federated learning) is what Apple has been doing.