3 ms·

> Distributed systems are hard. It takes time to determine where all possible information could live. And you have to make sure you're providing the correct inf

by jsnell 5y ago

> Distributed systems are hard. It takes time to determine where all possible information could live. And you have to make sure you're providing the correct information. And do this flawlessly, every single time lest you open yourself up to bad press and potential fines. This all takes time in systems as large and distributed as Amazon. This implies that Amazon is serving GDPR data requests manually, rather than the whole process being automated. Surely that can't be true? I agree that identifying where data can be stored, and extracting it correctly, is a difficult problem. But that problem is identical for every user, and it should only need to be solved once. You aren't determining from scratch which databases contain user data on every request, right? Nor are you re-defining your export schema for each user, or re-implementing the identity authentication, or deciding which pieces of data don't need to be in the data export for some legal reason, or any of these other systematic difficulties. And if this automated, why suggest that the difficulty applies to serving each individual data access request rather than just to defining and implemeting a repeatable process?

whoknew1122 5y ago
GDPR requests are handled, at least in part, manually. I have direct knowledge of how GDPR is handled within the product/service I support. And yes, it's manual.
- jsnell 5y ago
  Thanks! That's my mind blown, then :) I can believe that the volume is low enough that manual work is acceptable, but even then I'd have thought that you'd want things entirely automated to eliminate the chance of human error.
  - whoknew1122 5y ago
    I'm not sure how low in volume the requests are. It's really hard to automate because you have to gather data from many internal teams and products, and the data is intentionally siloed to enhance customer privacy and data security. Manual work isn't the best way to handle it, but the costs of automating (in terms of security, intricacy regarding different storage systems, etc.) is too high to really automate it on a grand scale. Where I work, which is low traffic generally, we process around 10 requests or so a week (from what I've seen).
whoknew1122 5y ago
To create a script/bot/application/whatever that can access all potential data, you have to give something read privileges to possibly hundreds of backend systems and products. This is horrendously bad idea security wise. If that service account gets compromised (either from an external or internal threat), you have a single account that has access to everything Amazon stores. This is bad for the company and bad for its customers. There necessarily have to be multiple workflows to maintain the data segregation necessary to protect data at the scale we're talking with Amazon. And assuming you could securely create this automated workflow, you'd still need a person manually verifying the end result to ensure that all the data scraped is in fact owned by the person who made the request. Within the past couple of years, there was a news story where someone got a different person's Alexa data after asking Amazon for their own data. That can't happen again.
- alex_young 5y ago
  The automation would be a bigger risk than granting humans carte blanche access to customer data? That seems like an odd security conclusion.
- jsnell 5y ago
  Sorry, I don't buy any of this. Automating the process doesn't need to imply that there's a single service with direct access to all of the data. Just from a basic software engineering perspective, it makes a ton of sense each product's data export to be a separate service owned by the product team, so no disagreements there. But by talking about how hard it is to figure out what data you have stored and export it correctly, you were implying that you had no such per-product service either, and each export is an artisanal custom job. The question of safeguards is interesting. I don't really see how having a human in the loop is adding any real security: a computer is going to be far better at deciding whether the request is valid or not. As an operator, being assigned a ticket to do an export of account 123456, what are you going to do other than do that export? A computer, on the other hand, can actually verify whether the request is actually authorized. That can be done in a way where a compromise of your central data export service account can't be used to fake the authorization. (A quick design sketch for one option: each account has a public key encryption keypair, managed by the identity system. When the central data export service requests an email verification, that is done via asking the identity system to sign a ticket. The identity system triggers a flow that asks the user to validate the request, and as part of the flow informs them of just what operation they are validating. User approval of the request signs the ticket with their private key. This ticket is sent to each data export service, which checks that the user id they're exporting has signed the ticket, and that the ticket contents match the request: i.e. same userid, operating is a data export, the data export covers this service. You will need to trust your identity system to not be compromised, but if it is, you're completely screwed anyway.) > And assuming you could securely create this automated workflow, you'd still need a person manually verifying the end result to ensure that all the data scraped is in fact owned by the person who made the request. Within the past couple of years, there was a news story where someone got a different person's Alexa data after asking Amazon for their own data. That can't happen again. The odds of a human doing a good job of this kind of validation are basically zero. Either they are following a checklist that a computer could execute more reliably, or they are just randomly poking at some 1 GB data dump trying to find the needle in the haystack.

by jsnell 5y ago

whoknew1122 5y ago

GDPR requests are handled, at least in part, manually. I have direct knowledge of how GDPR is handled within the product/service I support. And yes, it's manual.

jsnell 5y ago
Thanks! That's my mind blown, then :) I can believe that the volume is low enough that manual work is acceptable, but even then I'd have thought that you'd want things entirely automated to eliminate the chance of human error.
- whoknew1122 5y ago
  I'm not sure how low in volume the requests are. It's really hard to automate because you have to gather data from many internal teams and products, and the data is intentionally siloed to enhance customer privacy and data security. Manual work isn't the best way to handle it, but the costs of automating (in terms of security, intricacy regarding different storage systems, etc.) is too high to really automate it on a grand scale. Where I work, which is low traffic generally, we process around 10 requests or so a week (from what I've seen).

whoknew1122 5y ago

To create a script/bot/application/whatever that can access all potential data, you have to give something read privileges to possibly hundreds of backend systems and products. This is horrendously bad idea security wise. If that service account gets compromised (either from an external or internal threat), you have a single account that has access to everything Amazon stores. This is bad for the company and bad for its customers. There necessarily have to be multiple workflows to maintain the data segregation necessary to protect data at the scale we're talking with Amazon. And assuming you could securely create this automated workflow, you'd still need a person manually verifying the end result to ensure that all the data scraped is in fact owned by the person who made the request. Within the past couple of years, there was a news story where someone got a different person's Alexa data after asking Amazon for their own data. That can't happen again.

alex_young 5y ago
The automation would be a bigger risk than granting humans carte blanche access to customer data? That seems like an odd security conclusion.
jsnell 5y ago
Sorry, I don't buy any of this. Automating the process doesn't need to imply that there's a single service with direct access to all of the data. Just from a basic software engineering perspective, it makes a ton of sense each product's data export to be a separate service owned by the product team, so no disagreements there. But by talking about how hard it is to figure out what data you have stored and export it correctly, you were implying that you had no such per-product service either, and each export is an artisanal custom job. The question of safeguards is interesting. I don't really see how having a human in the loop is adding any real security: a computer is going to be far better at deciding whether the request is valid or not. As an operator, being assigned a ticket to do an export of account 123456, what are you going to do other than do that export? A computer, on the other hand, can actually verify whether the request is actually authorized. That can be done in a way where a compromise of your central data export service account can't be used to fake the authorization. (A quick design sketch for one option: each account has a public key encryption keypair, managed by the identity system. When the central data export service requests an email verification, that is done via asking the identity system to sign a ticket. The identity system triggers a flow that asks the user to validate the request, and as part of the flow informs them of just what operation they are validating. User approval of the request signs the ticket with their private key. This ticket is sent to each data export service, which checks that the user id they're exporting has signed the ticket, and that the ticket contents match the request: i.e. same userid, operating is a data export, the data export covers this service. You will need to trust your identity system to not be compromised, but if it is, you're completely screwed anyway.) > And assuming you could securely create this automated workflow, you'd still need a person manually verifying the end result to ensure that all the data scraped is in fact owned by the person who made the request. Within the past couple of years, there was a news story where someone got a different person's Alexa data after asking Amazon for their own data. That can't happen again. The odds of a human doing a good job of this kind of validation are basically zero. Either they are following a checklist that a computer could execute more reliably, or they are just randomly poking at some 1 GB data dump trying to find the needle in the haystack.