3 ms·

Numbers 1 and 3 are great. Number 2 is effectively impossible. You correctly point out the entire netsec industry, but it goes much much further. Cisco, Oracle

by quesera 10y ago

Numbers 1 and 3 are great. Number 2 is effectively impossible. You correctly point out the entire netsec industry, but it goes much much further. Cisco, Oracle, Google, and Facebook would all immediately fail that test. I would shed no tears, but it'd be a tough sell with most people.

tptacek 10y ago
You'd be surprised. Sit down with a group of smart people, isolate a single problem, and pick it apart. For instance: you've got a giant database of sensitive information that could be used to build a(nother) national Muslim registry. So: * Aggressively aggregate data into statistical aggregates that address your business questions while discarding, the way we already do for credit card data, sensitive information we don't immediately need. * Seed the database with false information in a way that allows it to answer the business logic questions your application needs, but that would respond to an unexpected compelled production request with data that would be untenably expensive to true back up. * Design the system so that it can efficiently answer the questions the system was designed to answer, but that takes proof-of-work-scale effort to answer other questions. There are already database systems with these designs. There are plenty of other answers; this was like a 5 minute conversation in a bar in San Francisco a few weeks ago. The point is that rule #2 isn't a professional obligation today, so there's little incentive to design comprehensive safeguards --- or really even to conduct the misuse modeling exercises that would even suggest what kinds of countermeasures are needed. What I think is the case is that we won't come up with good solutions to these problems until we make them professionally mandatory to solve, so that their solutions enable new revenue by enabling applications to be built that could not be built ethically otherwise.
- quesera 10y ago
  That's differential privacy, right? It's getting more attention now after the Apple news, and it would be a great start industry-wide. But it isn't cheap, and takes a real commitment from the business. If there was enough support among tech workers to force industry to align with that sort of model, it might obviate the competitive efficiency problem.

by quesera 10y ago

tptacek 10y ago

You'd be surprised. Sit down with a group of smart people, isolate a single problem, and pick it apart. For instance: you've got a giant database of sensitive information that could be used to build a(nother) national Muslim registry. So: * Aggressively aggregate data into statistical aggregates that address your business questions while discarding, the way we already do for credit card data, sensitive information we don't immediately need. * Seed the database with false information in a way that allows it to answer the business logic questions your application needs, but that would respond to an unexpected compelled production request with data that would be untenably expensive to true back up. * Design the system so that it can efficiently answer the questions the system was designed to answer, but that takes proof-of-work-scale effort to answer other questions. There are already database systems with these designs. There are plenty of other answers; this was like a 5 minute conversation in a bar in San Francisco a few weeks ago. The point is that rule #2 isn't a professional obligation today, so there's little incentive to design comprehensive safeguards --- or really even to conduct the misuse modeling exercises that would even suggest what kinds of countermeasures are needed. What I think is the case is that we won't come up with good solutions to these problems until we make them professionally mandatory to solve, so that their solutions enable new revenue by enabling applications to be built that could not be built ethically otherwise.

quesera 10y ago
That's differential privacy, right? It's getting more attention now after the Apple news, and it would be a great start industry-wide. But it isn't cheap, and takes a real commitment from the business. If there was enough support among tech workers to force industry to align with that sort of model, it might obviate the competitive efficiency problem.