6 ms·

I'm still quite astonished that (according to some of the comments) it's fairly standard that traffic between datacenters is not encrypted. Granted we're talki

by eksith 13y ago

I'm still quite astonished that (according to some of the comments) it's fairly standard that traffic between datacenters is not encrypted. Granted we're talking about extremely large throughput, but I thought Google of all companies would have invested in routers capable of doing this or at least gone to the trouble of designing their own hardware that does it; since Google is no stranger to this already.

revelation 13y ago
Why would routers do this? IPSec didn't exactly take off (this weeks news reminded us why), so end-to-end encryption wouldn't really happen on their level. That said, I certainly expected and assumed Google would already encrypt traffic between data centers. Whats the point of forcing HTTPS on gmail when you constantly backup my complete email repository across the world over unencrypted connections? In this new context, the statements on Prism ("we do not give them direct access!") certainly seem misleading. Right, you do not give them direct access, you just sync your databases over fibers that you know they have access to, without encrypting the data.
- raldi 13y ago
  > Whats the point of forcing HTTPS on gmail when you constantly backup my complete email repository across the world over unencrypted connections? To protect users on questionable public WiFi, or traveling through a country known for spying on its telecommunications, or who might be using an ISP with untrustworthy employees, or working for a company with a nosy IT department. Until recently, the USA wasn't generally considered part of that second category, and private leased lines were generally considered secure -- at Google and elsewhere -- for the same reason a LAN transmission within a datacenter is / was generally considered secure.
  - devx 13y ago
    Speaking of untrustworthy employees, and HUMINT, the NSA/CIA could just have agents infiltrated at Google, to get access to a lot of that data. This is what's so striking about this. I thought Google already encrypted all data, and only a few people had access to it. Didn't they say this many years ago?
    - raldi 13y ago
      "Encrypt all data" is a vague term. For example, would that include encrypting it while a CPU is processing it? What about when the CPU is writing it to RAM? What if the RAM is on some other computer in the rack? What if it's in some computer on the other side of the world? I'm not sure what Google's public statements on this have been, but if you want to research it, be sure to distinguish statements regarding user data at rest from those regarding user data in transit.
    - some_googler 13y ago
      "the NSA/CIA could just have agents infiltrated at Google, to get access to a lot of that data" -- this would be really difficult, because no single person at Google can just sudo in some system and feast over sensitive data. Any access to user data or other highly sensitive data needs to go through a stack of authorizations, the granted access will be scoped, and we have obsessive amounts of logging in place even for trivial stuff let alone sensitive systems and data. One thing that surprised me with the Snowden leaks is that the NSA's "checks and balances" seems to be orders of magnitude inferior to what we have at Google; something like that is unimaginable here (esp. if you can believe the NSA in that they don't even know which documents were accessed). Even a conspiracy involving several infiltrated googlers at the right places couldn't do it without anyone noticing, because they'd need to sabotage code and configurations that live in repositories that thousands of engineers have access to (and have enforced code-review workflows); the internal openness of our systems is in fact among our best defenses against sabotage.
- abcd_f 13y ago
  > IPSec didn't exactly take off Oh, jezus. Did you read it on the Internets? IPsec (s is in lowercase) is the standard to securing L2 connectivity and it has been ubiquitously used for site-to-site and client-to-site connectivity for ages. In addition to several mature FOSS implementations, every network equipment vendor ships one. There is also a ton of client software - Windows supported it since Windows 2000, the SSH company (you know, the ssh creators) has been selling an IPsec Toolkit since as early as 1999, Cisco has a VPN client that is de-facto software in Cisco-based shops for remote workers, etc.
  - gonzo 13y ago
    IPsec, which runs at layer 3, secures IP (layer 3) traffic. To protect L2 connectivity with IPsec, you'll need to tunnel the l2 frames inside IP.
    - drdaeman 13y ago
      You're right. And he's right too. Standalone IPsec is supposedly nearly non-existent, but there's a plenty of L2TP/IPsec traffic out there.
  - acdha 13y ago
    > > IPSec didn't exactly take off > IPsec (s is in lowercase) is the standard to securing L2 connectivity and it has been ubiquitously used for site-to-site and client-to-site connectivity for ages The question wasn't whether the standard exists but whether it's widely used. Your claims don't sound plausible to me as I've yet to work on a network (corporate, .edu, .gov) where IPSec is used – everyone focused on protocol level security like SSH or SSL instead.
- nimrody 13y ago
  Mail messages may go through multiple relays. Even if they are encrypted between relays, all routing mailers see the message unencrypted. Your only option to secure your mail messages is end-to-end encryption. Do you really trust Google or any other company more than you trust the NSA?
  - some_googler 13y ago
    SMTP-TLS should fix that, but then the problem is that many email providers don't support that.
- some_googler 13y ago
  What raidi and other said. But also, do not assume that we didn't encrypt any of the backend backbone transfers, or that some specific kind of information was exposed before -- for one thing, when we make inter-DC backups/replication of data that's already stored in encrypted form (you gmail folders and such), it's likely that we just ship this data around without bothering to decrypt and re-encrypt it, which would be wasteful and pointless. (I'm not a intimate with netops here, just making educated guesses like anybody could do.) Also there's some significant data that needs no encryption, e.g. the gobs of public youtube content that we have to mirror and cache in a thousand places. We have tons of our own stuff moving through the same pipes (proprietary source code, all files in our corp network filesystems...), so it's our best interest to protect these. I suspect most of the unencrypted traffic is actually what the NSA would call "metadata" -- if valuable information can be mined from simple metadata like phone calls, I guess even better stuff can be derived from extremely rich RPCs/protobufs even if the core information was already in encrypted fields. Anyway the more comprehensive encryption support should turn our backbone into a wasteland for spooks.
philip1209 13y ago
Yes, this seems like a huge man-in-the-middle vulnerability.
- tekacs 13y ago
  In this context eavesdropping is probably far more likely than MITM. Quite apart from MITM being an active attack only needed (ish) for manipulation of the transferred data, at the throughputs likely involved here and the fact that leased lines are used, MITM would probably provide far too much hassle for its worth.
Gusfoo_2 13y ago
These are private circuits between datacentres, and as such have always been in the "assumed secure" category.
INTPenis 13y ago
Agreed, I first thought the title said that Google was decreasing the key size of the crypto to speed up the traffic, and that the story was a negative one. I work for TeliaSonera and I can tell you that it's very standard for us to use VPN connections between worldwide datacenters.

by eksith 13y ago

revelation 13y ago

Why would routers do this? IPSec didn't exactly take off (this weeks news reminded us why), so end-to-end encryption wouldn't really happen on their level. That said, I certainly expected and assumed Google would already encrypt traffic between data centers. Whats the point of forcing HTTPS on gmail when you constantly backup my complete email repository across the world over unencrypted connections? In this new context, the statements on Prism ("we do not give them direct access!") certainly seem misleading. Right, you do not give them direct access, you just sync your databases over fibers that you know they have access to, without encrypting the data.

raldi 13y ago
> Whats the point of forcing HTTPS on gmail when you constantly backup my complete email repository across the world over unencrypted connections? To protect users on questionable public WiFi, or traveling through a country known for spying on its telecommunications, or who might be using an ISP with untrustworthy employees, or working for a company with a nosy IT department. Until recently, the USA wasn't generally considered part of that second category, and private leased lines were generally considered secure -- at Google and elsewhere -- for the same reason a LAN transmission within a datacenter is / was generally considered secure.
- devx 13y ago
  Speaking of untrustworthy employees, and HUMINT, the NSA/CIA could just have agents infiltrated at Google, to get access to a lot of that data. This is what's so striking about this. I thought Google already encrypted all data, and only a few people had access to it. Didn't they say this many years ago?
  - raldi 13y ago
    "Encrypt all data" is a vague term. For example, would that include encrypting it while a CPU is processing it? What about when the CPU is writing it to RAM? What if the RAM is on some other computer in the rack? What if it's in some computer on the other side of the world? I'm not sure what Google's public statements on this have been, but if you want to research it, be sure to distinguish statements regarding user data at rest from those regarding user data in transit.
  - some_googler 13y ago
    "the NSA/CIA could just have agents infiltrated at Google, to get access to a lot of that data" -- this would be really difficult, because no single person at Google can just sudo in some system and feast over sensitive data. Any access to user data or other highly sensitive data needs to go through a stack of authorizations, the granted access will be scoped, and we have obsessive amounts of logging in place even for trivial stuff let alone sensitive systems and data. One thing that surprised me with the Snowden leaks is that the NSA's "checks and balances" seems to be orders of magnitude inferior to what we have at Google; something like that is unimaginable here (esp. if you can believe the NSA in that they don't even know which documents were accessed). Even a conspiracy involving several infiltrated googlers at the right places couldn't do it without anyone noticing, because they'd need to sabotage code and configurations that live in repositories that thousands of engineers have access to (and have enforced code-review workflows); the internal openness of our systems is in fact among our best defenses against sabotage.
abcd_f 13y ago
> IPSec didn't exactly take off Oh, jezus. Did you read it on the Internets? IPsec (s is in lowercase) is the standard to securing L2 connectivity and it has been ubiquitously used for site-to-site and client-to-site connectivity for ages. In addition to several mature FOSS implementations, every network equipment vendor ships one. There is also a ton of client software - Windows supported it since Windows 2000, the SSH company (you know, the ssh creators) has been selling an IPsec Toolkit since as early as 1999, Cisco has a VPN client that is de-facto software in Cisco-based shops for remote workers, etc.
- gonzo 13y ago
  IPsec, which runs at layer 3, secures IP (layer 3) traffic. To protect L2 connectivity with IPsec, you'll need to tunnel the l2 frames inside IP.
  - drdaeman 13y ago
    You're right. And he's right too. Standalone IPsec is supposedly nearly non-existent, but there's a plenty of L2TP/IPsec traffic out there.
- acdha 13y ago
  > > IPSec didn't exactly take off > IPsec (s is in lowercase) is the standard to securing L2 connectivity and it has been ubiquitously used for site-to-site and client-to-site connectivity for ages The question wasn't whether the standard exists but whether it's widely used. Your claims don't sound plausible to me as I've yet to work on a network (corporate, .edu, .gov) where IPSec is used – everyone focused on protocol level security like SSH or SSL instead.
nimrody 13y ago
Mail messages may go through multiple relays. Even if they are encrypted between relays, all routing mailers see the message unencrypted. Your only option to secure your mail messages is end-to-end encryption. Do you really trust Google or any other company more than you trust the NSA?
- some_googler 13y ago
  SMTP-TLS should fix that, but then the problem is that many email providers don't support that.
some_googler 13y ago
What raidi and other said. But also, do not assume that we didn't encrypt any of the backend backbone transfers, or that some specific kind of information was exposed before -- for one thing, when we make inter-DC backups/replication of data that's already stored in encrypted form (you gmail folders and such), it's likely that we just ship this data around without bothering to decrypt and re-encrypt it, which would be wasteful and pointless. (I'm not a intimate with netops here, just making educated guesses like anybody could do.) Also there's some significant data that needs no encryption, e.g. the gobs of public youtube content that we have to mirror and cache in a thousand places. We have tons of our own stuff moving through the same pipes (proprietary source code, all files in our corp network filesystems...), so it's our best interest to protect these. I suspect most of the unencrypted traffic is actually what the NSA would call "metadata" -- if valuable information can be mined from simple metadata like phone calls, I guess even better stuff can be derived from extremely rich RPCs/protobufs even if the core information was already in encrypted fields. Anyway the more comprehensive encryption support should turn our backbone into a wasteland for spooks.

philip1209 13y ago

Yes, this seems like a huge man-in-the-middle vulnerability.

tekacs 13y ago
In this context eavesdropping is probably far more likely than MITM. Quite apart from MITM being an active attack only needed (ish) for manipulation of the transferred data, at the throughputs likely involved here and the fact that leased lines are used, MITM would probably provide far too much hassle for its worth.

Gusfoo_2 13y ago

These are private circuits between datacentres, and as such have always been in the "assumed secure" category.

INTPenis 13y ago

Agreed, I first thought the title said that Google was decreasing the key size of the crypto to speed up the traffic, and that the story was a negative one. I work for TeliaSonera and I can tell you that it's very standard for us to use VPN connections between worldwide datacenters.