5 ms·

I have a genuine question about this. How are we supposed to isolate user data stored in, say Postgres, by region, and still do multi-region replication, and re

I have a genuine question about this. How are we supposed to isolate user data stored in, say Postgres, by region, and still do multi-region replication, and restoring correctly? And how is the application supposed to find which region a user's record is stored? For example, if the user's data is only supposed to be kept in the EU, and if the user tries to login connected to a US node, should the app forward the request to the EU node (and how would it know?), which then accesses the EU database? If all nodes are connected to all regional databases, does it even make a difference? Basically, I am very confused about regional separation of data while still running a single application.

jbigelow76 4y ago
I have a genuine question about this. How are we supposed to isolate user data stored in, say Postgres, by region, and still do multi-region replication, and restoring correctly? And how is the application supposed to find which region a user's record is stored? What part of your question is the concern of a bureaucrat? We can drop a couple 9s of uptime in the name of local data primacy right? (I specifically didn't say "data privacy") :)
- timenova 4y ago
  I believe it'll lead to much higher costs, CO2 emissions, and other issues, simply because this architecture will require more message passing, wasted bandwidth, will be less reliable, and have higher operational complexity. I get that those in decision-making power aren't usually technologically adept at figuring out solutions for these issues, but they should invest in solutions first.
  - wmf 4y ago
    I would predict that data localization leads to data traveling shorter distances because most people access their local region rather than everything going to us-east-1.
lstamour 4y ago
In a very small number of cases, regional storage of data becomes regional restrictions. You see that already in how it's unlikely you will be able to access your consumer account from both Russia and the US at the same time unless you have a travel history to one or both countries. Similarly, an app storing data for EU citizens might indeed say, no, we can't share this data outside the EU, we're not allowed. How it figures out if you're outside a region is, of course, problematic. But that's rare. Generally the data lives on a regional server and gets accessed by a CDN in a region near where the data will be delivered. If you're allowed to sign in from that region, then you're likely authorizing transfer of data (for the duration of that session) to that same region. As for multi-region replication, it's often the case there are multiple regions in the same country - for example, us-east-1 and us-central-1. No jurisdictional problems to host the data in two regions in the same country, after all. As to how you look up what region the user's data is stored in, well, you can keep a list of users and regions. Worst case, force the user to remember their region, and use separate copies of the website (and DNS) to pick a region to sign in on. For example, the same app/website, but at different country domains.
- timenova 4y ago
  I understand that one way is to block users from logging in from other regions, or giving them a region selector at login (which is terrible UX IMO). However, if any of these approaches are done, the operational complexity will be immense, and latency for requests much higher than they could have been. A lot of message passing would be required to build such an architecture, and a lot of bandwidth would get wasted, not to mention the unnecessary costs. A simpler approach would to store core business data regionally, but peripheral data everywhere. For example, a todo app would store all user records, project lists, and other data everywhere, but the actual todos regionally. And yes this becomes infinitely complex will collaborative cross-region applications. I don't think there are any good solutions yet to this, and governments should invest in creating better solutions before forcing to go down this route.
josefx 4y ago
The basic Problem is that if some CCP goon turns up at your Chinese branch and requests all data on your US/EU users and your local branch is able to do that you failed. Same if some goon from the US government wants to access all data on your EU users. Or some German guy looking for US data in your German data center. You have to take steps to prevent this.
- coffeeblack 4y ago
  Difference is that your US and German branches have access to a justice system that can stop the government. In China, the justice system is part of the CCP and if you don’t comply, your company is gone.
  - tirpen 4y ago
    > Difference is that your US and German branches have access to a justice system that can stop the government. Germany sure, but The US? Do you not remember the Snowden leaks at all? Almost every company I've worked with the past several years has had a very strict No-US-Servers-Ever-policy, because people don't trusts the US government to not snoop around.
    - fmajid 4y ago
      It should be “No US company, period” because the Cloud Act empowers the US government to compel, say Amazon Web Services to disclose information they hold on a server hosted in Germany even if this violates EU law.
  - deleted 4y ago
    [deleted]
  - techdragon 4y ago
    The whole US / EU safe harbour provisions issue is because the EU is not satisfied that in the United States you don’t in fact have a justice system that can stop the government… that there are insufficient powers to stop certain parts of the government like the the NSA, CIA, DEA, FBI counter terrorism groups etc… who are granted extraordinary powers to do questionable shit like sucking up as much internet data as possible without a warrant in the name off “efficiency” using such flimsy reasoning as “it’s only a search for legal purposes when we run a database query on a US citizens data later” and “we’re legally allowed to do anything we like with the data of anyone who isn’t a US citizen… so these EU guys can fuck off” (or the appropriate US lawyer speak equivalent)
philips 4y ago
It seems a product opportunity might exist to ensure only encrypted copies are replicated out of region with the keys remaining in region. The side effect is a region might be down if something catastrophic happens but data could be restored
- timenova 4y ago
  Encrypted copies for some isolated data is a good solution, but for a user record when you want to login a user, will not work. Then you would need to fetch the data from another datacenter anyway, increasing costs, latency, and operational complexity.
- antonvs 4y ago
  You're assuming the laws would allow for that. I wouldn't count on it.
PeterStuer 4y ago
If you need geographic redundancy beyond the European borders, what is your threat model trying to mitigate? E.g. Let's say you have servers in Norway, Germany and Spain, what would be solved by e.g. having an extra one in the US? Roaming access can be handled through a VPN
spiffytech 4y ago
Bonus question: what do you do if data needs to stay in the creating user's region by default, but users can share data across regions, including creating new data associated with a record from another region? For example, American and EU users sharing documents and requesting comments be added. How do you manage both isolation and sharing? Do you copy such documents to both regions? Do comments live in the same region as the document, or in the region of the creating user? Do you just give up on foreign key integrity for documents, comments, and users because they could be in separate databases? If your American servers can request data out of the European region on demand (subject to business logic), are you really protecting data to the standard the law demands?
- 908B64B197 4y ago
  > are you really protecting data to the standard the law demands? This assumes the law's purpose is to protect citizen's data and that it's not just a way to bully tech companies into submission by writing impossible to implement laws and selectively enforcing them.
wmf 4y ago
The simple answer is that it's no longer a single application. It's a separate copy of the app per country/region. should the app forward the request to the EU node (and how would it know?) You have to add a home jurisdiction field to the account.
- InfoSecErik 4y ago
  The word that jumps out (to me) with what you describe is "Partition".

jbigelow76 4y ago

timenova 4y ago
I believe it'll lead to much higher costs, CO2 emissions, and other issues, simply because this architecture will require more message passing, wasted bandwidth, will be less reliable, and have higher operational complexity. I get that those in decision-making power aren't usually technologically adept at figuring out solutions for these issues, but they should invest in solutions first.
- wmf 4y ago
  I would predict that data localization leads to data traveling shorter distances because most people access their local region rather than everything going to us-east-1.

lstamour 4y ago

In a very small number of cases, regional storage of data becomes regional restrictions. You see that already in how it's unlikely you will be able to access your consumer account from both Russia and the US at the same time unless you have a travel history to one or both countries. Similarly, an app storing data for EU citizens might indeed say, no, we can't share this data outside the EU, we're not allowed. How it figures out if you're outside a region is, of course, problematic. But that's rare. Generally the data lives on a regional server and gets accessed by a CDN in a region near where the data will be delivered. If you're allowed to sign in from that region, then you're likely authorizing transfer of data (for the duration of that session) to that same region. As for multi-region replication, it's often the case there are multiple regions in the same country - for example, us-east-1 and us-central-1. No jurisdictional problems to host the data in two regions in the same country, after all. As to how you look up what region the user's data is stored in, well, you can keep a list of users and regions. Worst case, force the user to remember their region, and use separate copies of the website (and DNS) to pick a region to sign in on. For example, the same app/website, but at different country domains.

timenova 4y ago
I understand that one way is to block users from logging in from other regions, or giving them a region selector at login (which is terrible UX IMO). However, if any of these approaches are done, the operational complexity will be immense, and latency for requests much higher than they could have been. A lot of message passing would be required to build such an architecture, and a lot of bandwidth would get wasted, not to mention the unnecessary costs. A simpler approach would to store core business data regionally, but peripheral data everywhere. For example, a todo app would store all user records, project lists, and other data everywhere, but the actual todos regionally. And yes this becomes infinitely complex will collaborative cross-region applications. I don't think there are any good solutions yet to this, and governments should invest in creating better solutions before forcing to go down this route.

josefx 4y ago

The basic Problem is that if some CCP goon turns up at your Chinese branch and requests all data on your US/EU users and your local branch is able to do that you failed. Same if some goon from the US government wants to access all data on your EU users. Or some German guy looking for US data in your German data center. You have to take steps to prevent this.

coffeeblack 4y ago
Difference is that your US and German branches have access to a justice system that can stop the government. In China, the justice system is part of the CCP and if you don’t comply, your company is gone.
- tirpen 4y ago
  > Difference is that your US and German branches have access to a justice system that can stop the government. Germany sure, but The US? Do you not remember the Snowden leaks at all? Almost every company I've worked with the past several years has had a very strict No-US-Servers-Ever-policy, because people don't trusts the US government to not snoop around.
  - fmajid 4y ago
    It should be “No US company, period” because the Cloud Act empowers the US government to compel, say Amazon Web Services to disclose information they hold on a server hosted in Germany even if this violates EU law.
- deleted 4y ago
  [deleted]
- techdragon 4y ago
  The whole US / EU safe harbour provisions issue is because the EU is not satisfied that in the United States you don’t in fact have a justice system that can stop the government… that there are insufficient powers to stop certain parts of the government like the the NSA, CIA, DEA, FBI counter terrorism groups etc… who are granted extraordinary powers to do questionable shit like sucking up as much internet data as possible without a warrant in the name off “efficiency” using such flimsy reasoning as “it’s only a search for legal purposes when we run a database query on a US citizens data later” and “we’re legally allowed to do anything we like with the data of anyone who isn’t a US citizen… so these EU guys can fuck off” (or the appropriate US lawyer speak equivalent)

philips 4y ago

It seems a product opportunity might exist to ensure only encrypted copies are replicated out of region with the keys remaining in region. The side effect is a region might be down if something catastrophic happens but data could be restored

timenova 4y ago
Encrypted copies for some isolated data is a good solution, but for a user record when you want to login a user, will not work. Then you would need to fetch the data from another datacenter anyway, increasing costs, latency, and operational complexity.
antonvs 4y ago
You're assuming the laws would allow for that. I wouldn't count on it.

PeterStuer 4y ago

If you need geographic redundancy beyond the European borders, what is your threat model trying to mitigate? E.g. Let's say you have servers in Norway, Germany and Spain, what would be solved by e.g. having an extra one in the US? Roaming access can be handled through a VPN

spiffytech 4y ago

Bonus question: what do you do if data needs to stay in the creating user's region by default, but users can share data across regions, including creating new data associated with a record from another region? For example, American and EU users sharing documents and requesting comments be added. How do you manage both isolation and sharing? Do you copy such documents to both regions? Do comments live in the same region as the document, or in the region of the creating user? Do you just give up on foreign key integrity for documents, comments, and users because they could be in separate databases? If your American servers can request data out of the European region on demand (subject to business logic), are you really protecting data to the standard the law demands?

908B64B197 4y ago
> are you really protecting data to the standard the law demands? This assumes the law's purpose is to protect citizen's data and that it's not just a way to bully tech companies into submission by writing impossible to implement laws and selectively enforcing them.

wmf 4y ago

The simple answer is that it's no longer a single application. It's a separate copy of the app per country/region. should the app forward the request to the EU node (and how would it know?) You have to add a home jurisdiction field to the account.

InfoSecErik 4y ago
The word that jumps out (to me) with what you describe is "Partition".