7 ms·

Google Infrastructure Security Design Overview

317 points by emilburzo 10y ago

mnm1 10y ago
I wonder what their data deletion policies really are for something like Photos. I deleted all my old photos weeks ago but when I pull down the archive of my Google data, they're still there. With such a policy, I could see that data sitting around for years while Google claims that it's in the process of deletion, something that is not actual deletion. Then again, I doubt they actually ever delete anything.
- londons_explore 10y ago
  In which case, you likley didn't delete them correctly. If you delete them from your device, it doesn't delete the cloud copy. If you delete from an album, it removes the image from the album, but not from your account. Google's privacy policy says has limits to delete user data, and I can assure you they are very strict about that. (Lots of data is deleted within hours, but the multiple days is to ensure all backups of it are gone too) See http://blog.tech-and-law.com/2010/11/google-data-retention-periods-for.html?m=1 http://blog.tech-and-law.com/2010/11/google-data-retention-p...
  - petters 10y ago
    This is correct. A deletion should be effective/visible immediately, but it can take some time before all backups are guaranteed to be gone.
    - drieddust 10y ago
      No it isn't google takes your data to tapes as well as offline long term storage. [1]https://www.youtube.com/watch?v=eNliOm9NtCM https://www.youtube.com/watch?v=eNliOm9NtCM
      - mkj 10y ago
        They could be deleting encryption keys to the tapes? All speculation.
        puzzle 10y ago
        There was a talk about the backup infrastructure. The speaker talked about the issue of keys, but didn't provide specific details: http://highscalability.com/blog/2014/2/3/how-google-backs-up-the-internet-along-with-exabytes-of-othe.html http://highscalability.com/blog/2014/2/3/how-google-backs-up...
- pawadu 10y ago
  do you see the actual photos or just the folder thumbnails?
- puzzle 10y ago
  There are whole teams and pipelines dedicated to making sure data is deleted on all media, tapes included. The long tail can be affected by things such as a machine holding a bunch of GFS chunks from your files that went to the hardware repair queue in the meantime. Those chunks might not even be that useful without the others stored on other machines, but in the general case you can't make guarantees that e.g. they don't hold information that a skilled person could use to identify you.
- Klathmon 10y ago
  Also, I'm not sure if photos has this, but you also need to "empty the trash" in your drive account after you delete something before it will be actually deleted. IIRC it gets auto-deleted after 30 days or something.
fowl2 10y ago
so they've reinvented kerberos, presumably in a way that works. interesting. (and there are many other things)
- puzzle 10y ago
  Yeah, it's called LOAS: https://mobile.twitter.com/jbeda/status/715373975182807040 https://mobile.twitter.com/jbeda/status/715373975182807040
- Godel_unicode 10y ago
  Why do you think that Kerberos doesn't work?
petters 10y ago
> ... and laser-based intrusion detection systems Huh? I thought that was exclusive to movies like Entrapment and Mission Impossible.
- scrollaway 10y ago
  It's a fancy term for motion detectors. If my neighbour can afford one for his yard, it's not that crazy to put some in a datacenter :) Edit: I obviously wasn't implying they're using the same ones. Come on, now >.>
  - foxylad 10y ago
    Motion detectors are usually passive infra-red (PIR) sensors - no lasers involved. Unless you can cite a consumer-grade laser-based motion detector, I think this means Google's data centers are protected by slightly higher level gear.
    - dalore 10y ago
      https://www.amazon.com/Homesafe-Safety-Motion-Detector-Sensor/dp/B00YWDDCOW https://www.amazon.com/Homesafe-Safety-Motion-Detector-Senso... Homesafe Safety Beam Laser Motion Detector Sensor & Alert Only $39.99!
      - runeks 10y ago
        > This high tech device creates an invisible infrared beam up to 60 feet long and sounds a loud alarm, pleasant chime, or mutiple chimes when the beam is crossed.
    - SEJeff 10y ago
      From my understanding (after doing a few weeks of research on this for my own home security) the current "top of the line" tech is the Tomographic motion detectors which build up a mesh. It has been commercialized by a security company named Xandem, some info on it: https://en.wikipedia.org/wiki/Tomography https://en.wikipedia.org/wiki/Tomography https://www.youtube.com/watch?v=Y8updJWoSxE https://www.youtube.com/watch?v=Y8updJWoSxE I'll be purchasing a Xandem system soon
contingencies 10y ago
Many of these solutions are unavailable below a certain scale, and there is currently little commercial utility or pressure in offering these features in a wholly-owned-and-operated fashion to small businesses or individuals. The new deal (eg. DDOS resistance) is to rent an implementation, or go without. Basically, the gap between everyone else and the Googles of the world is large and growing. On the other hand, I wonder how useful some of them are. Boot-level security sounds fantastic but the cost of engineering and at the rate they probably cycle hardware, with decent service-level signatures this probably largely wasted money (eg. unexpected behavior like comms from service X to service Y is default-denied at multiple levels, logged, triggers hard shutdown/reset of system). While performance is cited as a concern, you'd save a lot of money removing the design/deployment/maintenance of all that complexity and could afford a little extra (more standard) hardware.
- lima 10y ago
  > Boot-level security You can get pretty far with commodity hardware. Even Secure Boot with custom keys prevents most threats.
  - contingencies 10y ago
    IMHO the biggest problem with commodity hardware is IPMI BMCs, a problem so insidious and widespread as to limit the utility of implementing trusted boot. (I designed datacenters for a major bitcoin exchange.) I would hazard a guess that Google's custom hardware has a more intelligent/limited/secure (and crypto-validated firmware based) IPMI implementation, and this contributes far more to security versus commodity hardware than cryptographically secured main processor / system boot.
    - lima 10y ago
      I agree. Is there any serious effort at making an open source BMC firmware? At least Intel AMT improves the situation a bit.
      - 10y ago
AJRF 10y ago
In the CIO summary they mention every service uses KeyCZAR. First line on KeyCZAR repo: "Important note: KeyCzar has some known security issues which may influence your decision to use it." https://github.com/google/keyczar#known-security-issues https://github.com/google/keyczar#known-security-issues
- jcims 10y ago
  They could be design decisions that are tailored for Google's use or issues for which Google has other compensating controls.
  - apeace 10y ago
    The first listed issue is "Use of SHA 1 and 1024 bit DSA", which they admit are "considered weak by current security standards". Not sure why the OP has been downvoted. Definitely something interesting to note.
- noja 10y ago
  Like "Use of SHA 1 and 1024 bit DSA". Ouch.
- nealmueller 10y ago
  I work at Google. The final bullet in the CIO Summary on Keyczar was a typographical error, taken from our paper on encryption at rest (https://goo.gl/hSordh https://goo.gl/hSordh). It's since been removed from this Security Design Overview. The encryption at rest paper goes into additional detail and includes the important clarification that while a very old version of Keyczar was open-sourced, the open-sourced version has not been updated to reflect internal developments.
  - AJRF 10y ago
    Thanks for the reply and follow on information. Wondering why those internal changes didnt get rolled into the public release, especially if they were security focused updates? Lack of adoption of the library maybe?
bogomipz 10y ago
I have a question about Step 5 in the post, it states: Is "Step 5: Add '1' to the end" Is this a delimiter for beginning of the padding or does it server some other purpose?
- timdierks 10y ago
  Did you mean for this to be somewhere else?
  - bogomipz 10y ago
    Yeah I did, thanks. That's what I get for multitasking. Unfortunately its too late to delete :(
deleted 10y ago
[deleted]
zbjornson 10y ago
> We have started rolling out automatic encryption for the WAN traversal hop of customer VM to VM traffic. ... all control plane WAN traffic within the infrastructure is already encrypted. In the future we plan to ... also encrypt inter-VM LAN traffic within the data center. It would be nice if this was more explicit. For example, is traffic that is TLS-terminated at their LB reencrypted all the way to the back end VM? At what point is it decrypted again? Are those keys unique to us or are they used for whatever traffic happens to traverse the same network paths? (I assume shared but with software-defined networking maybe it's practical for them to be unique.) What does the "control plane" encompass? In any case, I'm curious what people think about trusting the service provider for inter-service and inter-VM encryption. Do you use the LB's TLS termination? Do you still enable encryption for your DB connections even if it is (or will soon be) redundant with their network encryption?
- mentat 10y ago
  Anyone with access to the hypervisor at the service provider will have access to plaintext. TLS protects you from service provider network compromise within whatever scopes that covers. If you're in the cloud, you do have to have some basic trust in your service provider as compute is always in plaintext (barring homomorphic encryption).
  - blazespin 10y ago
    It's not really an issue of trust but rather defense in depth. You want to protect against rogue employees who can tap into the network, for example.
  - xyzzyz 10y ago
    Anyone with access to the hypervisor at the service provider will have access to plaintext. This is mostly true with today's state of the industry, but with upcoming technologies like Intel SGX[1], the hypervisor will not be able to access the plaintext anymore. [1] - https://software.intel.com/en-us/blogs/2013/09/26/protecting-application-secrets-with-intel-sgx https://software.intel.com/en-us/blogs/2013/09/26/protecting...
dwheeler 10y ago
This is great to see. For those who don't know, this is an "assurance case" (definition: "a body of evidence organized into an argument demonstrating that some claim about a system holds, i.e., is assured") - https://www.us-cert.gov/bsi/articles/knowledge/assurance-cases/arguing-security-creating-security-assurance-cases https://www.us-cert.gov/bsi/articles/knowledge/assurance-cas... I'm glad to see more assurance cases. You can't just do one thing and have a secure system. And if you want people to trust you, you need to give them a reason to trust. The CII best practices badge ( https://bestpractices.coreinfrastructure.org https://bestpractices.coreinfrastructure.org ) also has an assurance case; details at https://github.com/linuxfoundation/cii-best-practices-badge/blob/master/doc/security.md https://github.com/linuxfoundation/cii-best-practices-badge/... . If you want to help us make that better, let us know!

317 points by emilburzo 10y ago

mnm1 10y ago

I wonder what their data deletion policies really are for something like Photos. I deleted all my old photos weeks ago but when I pull down the archive of my Google data, they're still there. With such a policy, I could see that data sitting around for years while Google claims that it's in the process of deletion, something that is not actual deletion. Then again, I doubt they actually ever delete anything.

londons_explore 10y ago
In which case, you likley didn't delete them correctly. If you delete them from your device, it doesn't delete the cloud copy. If you delete from an album, it removes the image from the album, but not from your account. Google's privacy policy says has limits to delete user data, and I can assure you they are very strict about that. (Lots of data is deleted within hours, but the multiple days is to ensure all backups of it are gone too) See http://blog.tech-and-law.com/2010/11/google-data-retention-periods-for.html?m=1 http://blog.tech-and-law.com/2010/11/google-data-retention-p...
- petters 10y ago
  This is correct. A deletion should be effective/visible immediately, but it can take some time before all backups are guaranteed to be gone.
  - drieddust 10y ago
    No it isn't google takes your data to tapes as well as offline long term storage. [1]https://www.youtube.com/watch?v=eNliOm9NtCM https://www.youtube.com/watch?v=eNliOm9NtCM
    - mkj 10y ago
      They could be deleting encryption keys to the tapes? All speculation.
      - puzzle 10y ago
        There was a talk about the backup infrastructure. The speaker talked about the issue of keys, but didn't provide specific details: http://highscalability.com/blog/2014/2/3/how-google-backs-up-the-internet-along-with-exabytes-of-othe.html http://highscalability.com/blog/2014/2/3/how-google-backs-up...
pawadu 10y ago
do you see the actual photos or just the folder thumbnails?
puzzle 10y ago
There are whole teams and pipelines dedicated to making sure data is deleted on all media, tapes included. The long tail can be affected by things such as a machine holding a bunch of GFS chunks from your files that went to the hardware repair queue in the meantime. Those chunks might not even be that useful without the others stored on other machines, but in the general case you can't make guarantees that e.g. they don't hold information that a skilled person could use to identify you.
Klathmon 10y ago
Also, I'm not sure if photos has this, but you also need to "empty the trash" in your drive account after you delete something before it will be actually deleted. IIRC it gets auto-deleted after 30 days or something.

fowl2 10y ago

so they've reinvented kerberos, presumably in a way that works. interesting. (and there are many other things)

puzzle 10y ago
Yeah, it's called LOAS: https://mobile.twitter.com/jbeda/status/715373975182807040 https://mobile.twitter.com/jbeda/status/715373975182807040
Godel_unicode 10y ago
Why do you think that Kerberos doesn't work?

petters 10y ago

> ... and laser-based intrusion detection systems Huh? I thought that was exclusive to movies like Entrapment and Mission Impossible.

scrollaway 10y ago
It's a fancy term for motion detectors. If my neighbour can afford one for his yard, it's not that crazy to put some in a datacenter :) Edit: I obviously wasn't implying they're using the same ones. Come on, now >.>
- foxylad 10y ago
  Motion detectors are usually passive infra-red (PIR) sensors - no lasers involved. Unless you can cite a consumer-grade laser-based motion detector, I think this means Google's data centers are protected by slightly higher level gear.
  - dalore 10y ago
    https://www.amazon.com/Homesafe-Safety-Motion-Detector-Sensor/dp/B00YWDDCOW https://www.amazon.com/Homesafe-Safety-Motion-Detector-Senso... Homesafe Safety Beam Laser Motion Detector Sensor & Alert Only $39.99!
    - runeks 10y ago
      > This high tech device creates an invisible infrared beam up to 60 feet long and sounds a loud alarm, pleasant chime, or mutiple chimes when the beam is crossed.
  - SEJeff 10y ago
    From my understanding (after doing a few weeks of research on this for my own home security) the current "top of the line" tech is the Tomographic motion detectors which build up a mesh. It has been commercialized by a security company named Xandem, some info on it: https://en.wikipedia.org/wiki/Tomography https://en.wikipedia.org/wiki/Tomography https://www.youtube.com/watch?v=Y8updJWoSxE https://www.youtube.com/watch?v=Y8updJWoSxE I'll be purchasing a Xandem system soon

contingencies 10y ago

Many of these solutions are unavailable below a certain scale, and there is currently little commercial utility or pressure in offering these features in a wholly-owned-and-operated fashion to small businesses or individuals. The new deal (eg. DDOS resistance) is to rent an implementation, or go without. Basically, the gap between everyone else and the Googles of the world is large and growing. On the other hand, I wonder how useful some of them are. Boot-level security sounds fantastic but the cost of engineering and at the rate they probably cycle hardware, with decent service-level signatures this probably largely wasted money (eg. unexpected behavior like comms from service X to service Y is default-denied at multiple levels, logged, triggers hard shutdown/reset of system). While performance is cited as a concern, you'd save a lot of money removing the design/deployment/maintenance of all that complexity and could afford a little extra (more standard) hardware.

lima 10y ago
> Boot-level security You can get pretty far with commodity hardware. Even Secure Boot with custom keys prevents most threats.
- contingencies 10y ago
  IMHO the biggest problem with commodity hardware is IPMI BMCs, a problem so insidious and widespread as to limit the utility of implementing trusted boot. (I designed datacenters for a major bitcoin exchange.) I would hazard a guess that Google's custom hardware has a more intelligent/limited/secure (and crypto-validated firmware based) IPMI implementation, and this contributes far more to security versus commodity hardware than cryptographically secured main processor / system boot.
  - lima 10y ago
    I agree. Is there any serious effort at making an open source BMC firmware? At least Intel AMT improves the situation a bit.
    - 10y ago

AJRF 10y ago

In the CIO summary they mention every service uses KeyCZAR. First line on KeyCZAR repo: "Important note: KeyCzar has some known security issues which may influence your decision to use it." https://github.com/google/keyczar#known-security-issues https://github.com/google/keyczar#known-security-issues

jcims 10y ago
They could be design decisions that are tailored for Google's use or issues for which Google has other compensating controls.
- apeace 10y ago
  The first listed issue is "Use of SHA 1 and 1024 bit DSA", which they admit are "considered weak by current security standards". Not sure why the OP has been downvoted. Definitely something interesting to note.
noja 10y ago
Like "Use of SHA 1 and 1024 bit DSA". Ouch.
nealmueller 10y ago
I work at Google. The final bullet in the CIO Summary on Keyczar was a typographical error, taken from our paper on encryption at rest (https://goo.gl/hSordh https://goo.gl/hSordh). It's since been removed from this Security Design Overview. The encryption at rest paper goes into additional detail and includes the important clarification that while a very old version of Keyczar was open-sourced, the open-sourced version has not been updated to reflect internal developments.
- AJRF 10y ago
  Thanks for the reply and follow on information. Wondering why those internal changes didnt get rolled into the public release, especially if they were security focused updates? Lack of adoption of the library maybe?

bogomipz 10y ago

I have a question about Step 5 in the post, it states: Is "Step 5: Add '1' to the end" Is this a delimiter for beginning of the padding or does it server some other purpose?

timdierks 10y ago
Did you mean for this to be somewhere else?
- bogomipz 10y ago
  Yeah I did, thanks. That's what I get for multitasking. Unfortunately its too late to delete :(

deleted 10y ago

[deleted]

zbjornson 10y ago

> We have started rolling out automatic encryption for the WAN traversal hop of customer VM to VM traffic. ... all control plane WAN traffic within the infrastructure is already encrypted. In the future we plan to ... also encrypt inter-VM LAN traffic within the data center. It would be nice if this was more explicit. For example, is traffic that is TLS-terminated at their LB reencrypted all the way to the back end VM? At what point is it decrypted again? Are those keys unique to us or are they used for whatever traffic happens to traverse the same network paths? (I assume shared but with software-defined networking maybe it's practical for them to be unique.) What does the "control plane" encompass? In any case, I'm curious what people think about trusting the service provider for inter-service and inter-VM encryption. Do you use the LB's TLS termination? Do you still enable encryption for your DB connections even if it is (or will soon be) redundant with their network encryption?

mentat 10y ago
Anyone with access to the hypervisor at the service provider will have access to plaintext. TLS protects you from service provider network compromise within whatever scopes that covers. If you're in the cloud, you do have to have some basic trust in your service provider as compute is always in plaintext (barring homomorphic encryption).
- blazespin 10y ago
  It's not really an issue of trust but rather defense in depth. You want to protect against rogue employees who can tap into the network, for example.
- xyzzyz 10y ago
  Anyone with access to the hypervisor at the service provider will have access to plaintext. This is mostly true with today's state of the industry, but with upcoming technologies like Intel SGX[1], the hypervisor will not be able to access the plaintext anymore. [1] - https://software.intel.com/en-us/blogs/2013/09/26/protecting-application-secrets-with-intel-sgx https://software.intel.com/en-us/blogs/2013/09/26/protecting...

dwheeler 10y ago

This is great to see. For those who don't know, this is an "assurance case" (definition: "a body of evidence organized into an argument demonstrating that some claim about a system holds, i.e., is assured") - https://www.us-cert.gov/bsi/articles/knowledge/assurance-cases/arguing-security-creating-security-assurance-cases https://www.us-cert.gov/bsi/articles/knowledge/assurance-cas... I'm glad to see more assurance cases. You can't just do one thing and have a secure system. And if you want people to trust you, you need to give them a reason to trust. The CII best practices badge ( https://bestpractices.coreinfrastructure.org https://bestpractices.coreinfrastructure.org ) also has an assurance case; details at https://github.com/linuxfoundation/cii-best-practices-badge/blob/master/doc/security.md https://github.com/linuxfoundation/cii-best-practices-badge/... . If you want to help us make that better, let us know!