5 ms·

I can't agree with that, this was on Google's on fiber connections between their own data centers, right? And no other company with multiple data centers encryp

by bo1024 13y ago

I can't agree with that, this was on Google's on fiber connections between their own data centers, right? And no other company with multiple data centers encrypts all traffic between them, right? (maybe you'll find a small counterexample but no big one.) So I don't think this is "security 101".

lsh123 13y ago
Well, I do :) Moreover, I encrypt all the traffic even inside the same data center.
- cryptbe 13y ago
  could you share which technology are you using to encrypt all the traffic?
  - lsh123 13y ago
    * Mid-tier servers: standard HTTPS with nginx * Database: SSL connections for MySQL * Memcached, Gearmand, and other tools that don't have built-in SSL support: simple home grown message level encryption (AES256) And of course, there are VPN tunnels between data centers in addition to the above.
    - cryptbe 13y ago
      Thanks. > And of course, there are VPN tunnels between data centers in addition to the above. Could you please be more specific on the VPN solution that you are using? How do you manage the shared keys? How do you make sure 'system administrators can't easily read the traffic?"
      - lsh123 13y ago
        We use Cisco appliances for VPN (a few different models) and indeed there is a shared key that we have to input manually. However, after the key is entered (and configs saved) in order to decrypt the traffic one would need to print Cisco configs which is a very unusual operation that would be logged and then alerts will fire, audits will catch it, etc.
        josephlord 13y ago
        Just out of interest how do you transfer the key between datacentres for setup? Same person travels between them? PGP encrypted email? Or over the phone? Phone is I think an obvious (and now clearly wrong choice) although maybe always suspect if you are concerned with dark fibre . The endpoint security of a device generating and transmitting the key now also being a risk. How far up the chain do you worry? An airgapped device to generate the key and a single person travelling between datacentres seems the secure (although costly) solution. Obviously if TSA/customs remove device from them for inspection or connect it to anything it needs to be thrown away (or moved to insecure duties) and the setup process restarted.
        lsh123 13y ago
        I think public key encryption with long enough key is a pretty safe bet these days. Of course, NSA might have new non-public discoveries in math/crypto that might make public encryption obsolete. Or they might have a device form Area 51 that breaks any encryption. However, I haven't seen any evidences of this yet.
        stevenrace 13y ago
        >> in order to decrypt the traffic one would need to print Cisco configs Which could be done legally via 'Cisco Service Independent Intercept (SII)' built into IOS to comply with CALEA (Communications Assistance for Law Enforcement Act). And not so legally via user-escalation exploits within the same service. Anyway, props for making the effort. I too am interested in your key exchange methods.
- deleted 13y ago
  [deleted]
dsl 13y ago
I work for a company bigger than Google, and we encrypt everything in flight between datacenters. It is security 101.
- rurounijones 13y ago
  Does your company have dedicated, unshared, fibre between those datacenters?
  - dsl 13y ago
    Yes we do. Still use encryption.
  - lsh123 13y ago
    Consider the recent passwords leak from Adobe: they stored passwords in a dedicated unshared datacenter. Does this make a good security decision to encrypt passwords instead of using a hash because nobody should have been able to access these encrypted passwords? I really don't think so.
    - rurounijones 13y ago
      There are problems with your analogy. 1. Data at rest (Adobe) vs data in travel (Google). 2. Software Hack vs Hardware hack The Adobe data was sitting on a server in a datacenter, it was accessible from the internet on some level. The Google data was taken, apparently, from a dedicated, google owned, unshared link (quite likely a fibre-optic tap) The methodologies, skill levels and required hardware for the penetrating the above two types of setup are wildly different. I blame adobe for getting a server hacked, it happens a lot and and they ignored a lot of body of knowledge built up over the years. I do not blame Google for getting their inter-datacentre links physically compromised by a security agency of the US government. Nor do I blame them for (incorrectly, as it turns out) deeming that an unlikely scenario and therefore giving it low priority. I would blame them for not doing anything about it now that they know it is happening but that does not seem to be the case. (I fully expect companies to encrypt data between datacentres if they are not on dedicated unshared links)
dingaling 13y ago
> And no other company with multiple data centers encrypts all traffic between them, right? Indeed they do! From personal experience, Cisco was hawking its TrustSec inter-DC encryption solution five or six years ago, even over dark fibre.
- fpgeek 13y ago
  If you believe the threat is a government agency splicing private, unshared fiber to capture your traffic between data centers, why in the world would you trust equipment from Cisco (who lists "Government" as one of the industries they sell to) to protect you from that?
  - dingaling 13y ago
    Well I certainly understand your point, but the question was 'are big companies encrypting their inter-DC traffic' and the answer is 'yes', even if it's backdoored without their knowledge.
- thrownaway2424 13y ago
  Google's inter-dc links are way too big for any appliance type of thing to encrypt. Like most things at Google the scale of their network is incomprehensible to most people.
  - dingaling 13y ago
    > Google's inter-dc links are way too big for any appliance type of thing to encrypt Frankly, no. There are numerous network devices that can handle AES-256 on 10 Gbps links, as a matter of routine, whilst doing 'mundane' switching for the day job. If you have the money there are dedicated hardware that can handle the same at 100 Gbps. IP Cores is one from memory that produces the circuitry for that. They can throw compression in there as well if you like. Encrypting data links isn't magic. Google just didn't do it.
    - thrownaway2424 13y ago
Cookingboy 13y ago
Amazon does that.

by bo1024 13y ago

lsh123 13y ago

Well, I do :) Moreover, I encrypt all the traffic even inside the same data center.

cryptbe 13y ago
could you share which technology are you using to encrypt all the traffic?
- lsh123 13y ago
  * Mid-tier servers: standard HTTPS with nginx * Database: SSL connections for MySQL * Memcached, Gearmand, and other tools that don't have built-in SSL support: simple home grown message level encryption (AES256) And of course, there are VPN tunnels between data centers in addition to the above.
  - cryptbe 13y ago
    Thanks. > And of course, there are VPN tunnels between data centers in addition to the above. Could you please be more specific on the VPN solution that you are using? How do you manage the shared keys? How do you make sure 'system administrators can't easily read the traffic?"
    - lsh123 13y ago
      We use Cisco appliances for VPN (a few different models) and indeed there is a shared key that we have to input manually. However, after the key is entered (and configs saved) in order to decrypt the traffic one would need to print Cisco configs which is a very unusual operation that would be logged and then alerts will fire, audits will catch it, etc.
      - josephlord 13y ago
        Just out of interest how do you transfer the key between datacentres for setup? Same person travels between them? PGP encrypted email? Or over the phone? Phone is I think an obvious (and now clearly wrong choice) although maybe always suspect if you are concerned with dark fibre . The endpoint security of a device generating and transmitting the key now also being a risk. How far up the chain do you worry? An airgapped device to generate the key and a single person travelling between datacentres seems the secure (although costly) solution. Obviously if TSA/customs remove device from them for inspection or connect it to anything it needs to be thrown away (or moved to insecure duties) and the setup process restarted.
        lsh123 13y ago
        I think public key encryption with long enough key is a pretty safe bet these days. Of course, NSA might have new non-public discoveries in math/crypto that might make public encryption obsolete. Or they might have a device form Area 51 that breaks any encryption. However, I haven't seen any evidences of this yet.
      - stevenrace 13y ago
        >> in order to decrypt the traffic one would need to print Cisco configs Which could be done legally via 'Cisco Service Independent Intercept (SII)' built into IOS to comply with CALEA (Communications Assistance for Law Enforcement Act). And not so legally via user-escalation exploits within the same service. Anyway, props for making the effort. I too am interested in your key exchange methods.
deleted 13y ago
[deleted]

dsl 13y ago

I work for a company bigger than Google, and we encrypt everything in flight between datacenters. It is security 101.

rurounijones 13y ago
Does your company have dedicated, unshared, fibre between those datacenters?
- dsl 13y ago
  Yes we do. Still use encryption.
- lsh123 13y ago
  Consider the recent passwords leak from Adobe: they stored passwords in a dedicated unshared datacenter. Does this make a good security decision to encrypt passwords instead of using a hash because nobody should have been able to access these encrypted passwords? I really don't think so.
  - rurounijones 13y ago
    There are problems with your analogy. 1. Data at rest (Adobe) vs data in travel (Google). 2. Software Hack vs Hardware hack The Adobe data was sitting on a server in a datacenter, it was accessible from the internet on some level. The Google data was taken, apparently, from a dedicated, google owned, unshared link (quite likely a fibre-optic tap) The methodologies, skill levels and required hardware for the penetrating the above two types of setup are wildly different. I blame adobe for getting a server hacked, it happens a lot and and they ignored a lot of body of knowledge built up over the years. I do not blame Google for getting their inter-datacentre links physically compromised by a security agency of the US government. Nor do I blame them for (incorrectly, as it turns out) deeming that an unlikely scenario and therefore giving it low priority. I would blame them for not doing anything about it now that they know it is happening but that does not seem to be the case. (I fully expect companies to encrypt data between datacentres if they are not on dedicated unshared links)

dingaling 13y ago

> And no other company with multiple data centers encrypts all traffic between them, right? Indeed they do! From personal experience, Cisco was hawking its TrustSec inter-DC encryption solution five or six years ago, even over dark fibre.

fpgeek 13y ago
If you believe the threat is a government agency splicing private, unshared fiber to capture your traffic between data centers, why in the world would you trust equipment from Cisco (who lists "Government" as one of the industries they sell to) to protect you from that?
- dingaling 13y ago
  Well I certainly understand your point, but the question was 'are big companies encrypting their inter-DC traffic' and the answer is 'yes', even if it's backdoored without their knowledge.
thrownaway2424 13y ago
Google's inter-dc links are way too big for any appliance type of thing to encrypt. Like most things at Google the scale of their network is incomprehensible to most people.
- dingaling 13y ago
  > Google's inter-dc links are way too big for any appliance type of thing to encrypt Frankly, no. There are numerous network devices that can handle AES-256 on 10 Gbps links, as a matter of routine, whilst doing 'mundane' switching for the day job. If you have the money there are dedicated hardware that can handle the same at 100 Gbps. IP Cores is one from memory that produces the circuitry for that. They can throw compression in there as well if you like. Encrypting data links isn't magic. Google just didn't do it.
  - thrownaway2424 13y ago

Cookingboy 13y ago

Amazon does that.