3 ms·

In practice you add large gaussians to the deltas, which gives you reasonably strong guarantees.

• max_ 6y ago
  How is the strength of these gurantees quantified? Any resources?
  • why_only_15 6y ago
    This is what I remember from working closely with the people doing this at Apple. I believe they may have published a paper, but I'm not sure. Vojta Jina was the relevant manager.
• est31 6y ago
  It's a common fallacy. Adding noise does not anonymize data sufficiently. If your traffic monitoring software adds gaussians to car speeds, you can still average over multiple reports about two cars to identify their respective average speeds. Those can be used to determine whether they are trucks or cars, etc. In neural networks, weight changes do carry meaning. If your network has particularly large updates for the horse recognition neuron, you likely watched horse pictures. If it has updates for handbags, you likely watched pictures of them. If the company averages over all your submissions, the noise will be less relevant and the handbags and horse pictures will eventually show up.
  • why_only_15 6y ago
    This isn't about averaging over all your submissions, it's about averaging over the entire population (in this case, of iphone users, ~1,000,000,000). Data about the entire population, e.g. how many people click this button, or how many results do most people get from this page, or what kind of things do people take pictures of, i think are safe.
    • est31 6y ago
      If you average over the entire population, it's privacy preserving indeed. But this averaging happens on the side of the server, not the client, so I can't really verify whether the server stores away the deltas of interesting/all users or it averages really all user deltas. I know apple has some means to anonymize users by using random changing IDs instead of the apple ID, but they can correlate it to an apple ID via the ip address or other means.
      • why_only_15 6y ago
        Ultimately, what can you really verify as a user? Apple is very different from bitcoin, or IPFS or matrix or the like, in that they have an existing reputation. Instead of trying to provide perfect guarantees, they make a real attempt at preserving privacy, and you just have to trust them on that. I personally trust them after having attempted experiments stymied by the privacy team, but i think in general there's reasonable evidence that apple really does try to be privacy-preserving. For your example, sure it's possible for the server to do something you don't know. But it's the same people on the server doing the aggregation as on the client doing the obfuscation. If they really wanted they could easily e.g. set the seed for the noise to be based on a key which they have on the server, which would be very difficult to detect. I think you just have to trust that apple wouldn't let that happen.
    • t-vi 6y ago
      Well, the game in machine learning is averages conditional on the input. So if the input essentially identifies and individual you'll get to average over the inputs for the individual.