3 ms·

How do they assure you that the training algorithm isn't just exfiltrating your data? Edit: By that I mean... What's stopping the model from being as simple as

How do they assure you that the training algorithm isn't just exfiltrating your data? Edit: By that I mean... What's stopping the model from being as simple as "learn my personal information"?

pas 7y ago
Nothing, of course. But they are probably going to use an open source implementation with probably some kind of deterministic build/compile systems, so they can show you what the training algorithm is (that it does not contain a hidden user id or such to magically "overfit" for that user), and similarly the training questions should also be knowable and provably non-user-specific.
AlexCoventry 7y ago
It looks from the comic like they aggregate data with linear structure (probably derivatives on the model parameters.) Each device adds a mask to their part of the data, and somehow the masks are coordinated across devices so that when the data are summed on the central training server, the masks cancel out. It's unclear from the comic how the masks are coordinated, or how they compensate for the risk that a participating device drops out (which will make all the other data from that iteration useless, if you set the masks up in a naive way.)
- wybiral 7y ago
  > It looks from the comic like they aggregate data with linear structure (probably derivatives on the model parameters.) Thanks, that's the part I must have glossed over. It looks like they're using secret sharing to distribute as shares that all need to be together to reassemble [1]. [1] https://storage.googleapis.com/pub-tools-public-publication-data/pdf/ae87385258d90b9e48377ed49d83d467b45d5776.pdf https://storage.googleapis.com/pub-tools-public-publication-...
  - AlexCoventry 7y ago
    Thanks, just came back to share that link. :) From the introduction, it looks like they group participants into smaller clusters, coordinate between those via the centralized server in a star topology, use Diffie-Helman to share secrets between the participants in a cluster, and construct the canceling noise within that cluster. The Shamir secret sharing squicks me a bit. It looks like if the adversary can control cluster membership (and Google is the adversary, here), they can recover the gradients. > To prevent the server from simulating an arbitrary number of clients (in the active-adversary model), we require the support of a public key infrastructure that allows clients to register identities, and sign messages using their identity, such that other clients can verify this signature, but cannot impersonate them. In this model, each party u will register to a public bulletin board during the setup phase. The bulletin board will only allow parties to register keys for themselves, so it will not be possible for the attacking parties to impersonate honest parties.
- deleted 7y ago
  [deleted]

How do they assure you that the training algorithm isn't just exfiltrating your data? Edit: By that I mean... What's stopping the model from being as simple as "learn my personal information"?

pas 7y ago

Nothing, of course. But they are probably going to use an open source implementation with probably some kind of deterministic build/compile systems, so they can show you what the training algorithm is (that it does not contain a hidden user id or such to magically "overfit" for that user), and similarly the training questions should also be knowable and provably non-user-specific.

AlexCoventry 7y ago

It looks from the comic like they aggregate data with linear structure (probably derivatives on the model parameters.) Each device adds a mask to their part of the data, and somehow the masks are coordinated across devices so that when the data are summed on the central training server, the masks cancel out. It's unclear from the comic how the masks are coordinated, or how they compensate for the risk that a participating device drops out (which will make all the other data from that iteration useless, if you set the masks up in a naive way.)

wybiral 7y ago
> It looks from the comic like they aggregate data with linear structure (probably derivatives on the model parameters.) Thanks, that's the part I must have glossed over. It looks like they're using secret sharing to distribute as shares that all need to be together to reassemble [1]. [1] https://storage.googleapis.com/pub-tools-public-publication-data/pdf/ae87385258d90b9e48377ed49d83d467b45d5776.pdf https://storage.googleapis.com/pub-tools-public-publication-...
- AlexCoventry 7y ago
  Thanks, just came back to share that link. :) From the introduction, it looks like they group participants into smaller clusters, coordinate between those via the centralized server in a star topology, use Diffie-Helman to share secrets between the participants in a cluster, and construct the canceling noise within that cluster. The Shamir secret sharing squicks me a bit. It looks like if the adversary can control cluster membership (and Google is the adversary, here), they can recover the gradients. > To prevent the server from simulating an arbitrary number of clients (in the active-adversary model), we require the support of a public key infrastructure that allows clients to register identities, and sign messages using their identity, such that other clients can verify this signature, but cannot impersonate them. In this model, each party u will register to a public bulletin board during the setup phase. The bulletin board will only allow parties to register keys for themselves, so it will not be possible for the attacking parties to impersonate honest parties.
deleted 7y ago
[deleted]