4 ms·

The enclaves will flush the data (much like the hardware you mention in your first post) after some number of wrong PIN guesses. To do this securely made the w

by kijiki 6y ago

The enclaves will flush the data (much like the hardware you mention in your first post) after some number of wrong PIN guesses. To do this securely made the whole thing quite difficult, because it means we can never persist anything outside the enclave, or an attacker could unpersist into 1000 unmodified enclaves (who would never know about each other) and get 1000*N guesses.

JoshTriplett 6y ago
That was precisely the kind of architecture I was hoping to see documented, yes. So you're locking data not just to the code running in the enclave but to the specific hardware the enclave runs on, and further doing something to prevent replay? And at the same time, providing resilience against hardware failure of the machine any given enclave runs on, and providing persistence?
- kijiki 6y ago
  Yeah, there is a 32 byte random serviceid generated inside the enclave when it starts in "new" mode, so it is locked to the serviceid, not the HW, so we can migrate off failing/failed hardware. The only way to get a non-random serviceid is to have keyspace transferred to you from an existing enclave, after mutual attestation. Clients verify the serviceid after attestation verifies the enclave. Each partition of the keyspace lives on a 5 node cluster, running RAFT inside the enclaves to ensure the pin guess count is consistent. If a node fails, or is failing, or we need to do maintenance, we transfer the partition from the old nodes to new ones, which transfers the serviceid as well. If you start a new enclave from scratch, it'll generate a new serviceid, and clients won't talk to it, and existing partitions will refuse to transfer to it since it already has a serviceid. In fact, it won't even be listening for incoming transfers, since it thinks it owns the entire keyspace. We can also do a partial partition transfer that leaves the source cluster live, which we use to split up the keyspace more when we need to scale up. There is no persistence (wouldn't be secure, attackers could do many parallel enclave resumes to get extra guesses), so the 5 nodes in the cluster live in geographically separated data centers, and we're real careful about things that could kill a node or process.

by kijiki 6y ago

JoshTriplett 6y ago

That was precisely the kind of architecture I was hoping to see documented, yes. So you're locking data not just to the code running in the enclave but to the specific hardware the enclave runs on, and further doing something to prevent replay? And at the same time, providing resilience against hardware failure of the machine any given enclave runs on, and providing persistence?

kijiki 6y ago
Yeah, there is a 32 byte random serviceid generated inside the enclave when it starts in "new" mode, so it is locked to the serviceid, not the HW, so we can migrate off failing/failed hardware. The only way to get a non-random serviceid is to have keyspace transferred to you from an existing enclave, after mutual attestation. Clients verify the serviceid after attestation verifies the enclave. Each partition of the keyspace lives on a 5 node cluster, running RAFT inside the enclaves to ensure the pin guess count is consistent. If a node fails, or is failing, or we need to do maintenance, we transfer the partition from the old nodes to new ones, which transfers the serviceid as well. If you start a new enclave from scratch, it'll generate a new serviceid, and clients won't talk to it, and existing partitions will refuse to transfer to it since it already has a serviceid. In fact, it won't even be listening for incoming transfers, since it thinks it owns the entire keyspace. We can also do a partial partition transfer that leaves the source cluster live, which we use to split up the keyspace more when we need to scale up. There is no persistence (wouldn't be secure, attackers could do many parallel enclave resumes to get extra guesses), so the 5 nodes in the cluster live in geographically separated data centers, and we're real careful about things that could kill a node or process.