5 ms·

Google, and their "geniuses" in opsec, should not be given a pass at all for this. Even if this is a leased private line, non-Internet routed, whatever, it is

by rsync 13y ago

Google, and their "geniuses" in opsec, should not be given a pass at all for this. Even if this is a leased private line, non-Internet routed, whatever, it is trivially easy to encrypt the communications and is absolutely a best practice. I see this as great big egg on their face. In fact, it's such a cock-up that one wonders if this is the plausibly deniable ingress that they agreed to provide for the NSA, et. al This is akin to using telnet to access your home server because you're "on your own network". Nobody does that and I can't believe they would have either.

ChuckMcM 13y ago
You are of course entitled to your opinion, I know Brandon and I know how dedicated he is, I got to watch him and the OpSec team in action during the Chinese incursion. One of the side effects of being in the platforms group was that I had exceptional access to what was going on everywhere on Google's internal network and machines. And a lot changed after that event, both internally and externally. I can forgive someone for thinking that if they dug trenches in the street, bought some fiber, and ran it between a couple of buildings, in a country where the rule of law was in effect, they might consider it a reasonable assumption that the fiber is laying in the street unmolested. Even if the distance is such that they can't see the entire length of the ground above the conduit. Prior to Snowden's disclosures, it was the common belief amongst the security community that in 'safe' countries, the government in power would not subvert your infrastructure through physical access. They might do some network tricks, but not tap your fiber. In 'bad' countries counter measures were taken. And the network setup in say Russia or China was different than it was in the US and the UK. That your own government would illegally subvert your infrastructure [1] through the use of a technicality was not considered a "likely" threat [2]. Given that not it has been exploited it is rewriting a bunch of assumptions. I am not surprised in the least that they are now deploying the same hardening they use in hostile environments world wide. [1] The NSA cannot legally tap into communications infrastructure in the US (that is the FBI's job) and when the FBI does it they need a warrant. By doing this in the UK they sidestepped those constraints. [2] In classic vulnerability analysis you deploy your resources against both the probability and damage potential of a given threat. So for example datacenters are vulnerable to being bombed by aircraft, the probability of that is low enough that you don't defend against it, bombed by cars you put a security perimeter around the building.
- crucifiction 13y ago
  No, but they should take some precaution if someone unseemly (rogue employee, hacker, etc) has network access. You don't need a NSA level fiber taps under the ocean to do packet sniffing. Passing all of their service calls, database replications/connections, etc. in clear text over the wire is just plain lazy, bad security for an operation of their scale and (supposed) sophistication. I am sure it saved them some money on server/load balancers however.
packetslave 13y ago
Please explain how you would "trivially" encrypt an optical line system that is capable of pushing 19TB down a single fiber (for example, http://www.advaoptical.com/en/products/technology/dwdm.aspx http://www.advaoptical.com/en/products/technology/dwdm.aspx) The only people who think this is "trivially easy" are people who don't have to do it.
res0nat0r 13y ago
Tapping multi-mode dark fiber unnoticed is now considered trivially easy? I must have missed when the bar was shifted this high.
- XorNot 13y ago
  The fiber doesn't run straight into a Google datacenter. It's operated and managed by the company which lays the cable under the sea, which means it goes through their stations at the coast and probably through quite a bit of other non-exclusive routing hardware. The fact that routing hardware exists means you can pretty easily tap it.
  - res0nat0r 13y ago
    I believe my comment is pertaining to the use of the word "easily". James Bamford in The Shadow Factory was talking about how difficult this is, even for government institutions.
- crucifiction 13y ago
  You don't need to tap dark fiber for this to be poor security. They were passing all internal data around in clear text. It would have been trivial for any data center employee to gain significant amounts of open data this way. Why wouldn't you have services, db connections, etc. encrypted internally? Its certainly possible and done within many companies, it is surprising how lax google was in their assumption that once inside the network everything is going to be a-ok.
  - res0nat0r 13y ago
    This isn't something that is "easy" when you run one of the biggest infrastructures in the world.

by rsync 13y ago

ChuckMcM 13y ago

You are of course entitled to your opinion, I know Brandon and I know how dedicated he is, I got to watch him and the OpSec team in action during the Chinese incursion. One of the side effects of being in the platforms group was that I had exceptional access to what was going on everywhere on Google's internal network and machines. And a lot changed after that event, both internally and externally. I can forgive someone for thinking that if they dug trenches in the street, bought some fiber, and ran it between a couple of buildings, in a country where the rule of law was in effect, they might consider it a reasonable assumption that the fiber is laying in the street unmolested. Even if the distance is such that they can't see the entire length of the ground above the conduit. Prior to Snowden's disclosures, it was the common belief amongst the security community that in 'safe' countries, the government in power would not subvert your infrastructure through physical access. They might do some network tricks, but not tap your fiber. In 'bad' countries counter measures were taken. And the network setup in say Russia or China was different than it was in the US and the UK. That your own government would illegally subvert your infrastructure [1] through the use of a technicality was not considered a "likely" threat [2]. Given that not it has been exploited it is rewriting a bunch of assumptions. I am not surprised in the least that they are now deploying the same hardening they use in hostile environments world wide. [1] The NSA cannot legally tap into communications infrastructure in the US (that is the FBI's job) and when the FBI does it they need a warrant. By doing this in the UK they sidestepped those constraints. [2] In classic vulnerability analysis you deploy your resources against both the probability and damage potential of a given threat. So for example datacenters are vulnerable to being bombed by aircraft, the probability of that is low enough that you don't defend against it, bombed by cars you put a security perimeter around the building.

crucifiction 13y ago
No, but they should take some precaution if someone unseemly (rogue employee, hacker, etc) has network access. You don't need a NSA level fiber taps under the ocean to do packet sniffing. Passing all of their service calls, database replications/connections, etc. in clear text over the wire is just plain lazy, bad security for an operation of their scale and (supposed) sophistication. I am sure it saved them some money on server/load balancers however.

packetslave 13y ago

Please explain how you would "trivially" encrypt an optical line system that is capable of pushing 19TB down a single fiber (for example, http://www.advaoptical.com/en/products/technology/dwdm.aspx http://www.advaoptical.com/en/products/technology/dwdm.aspx) The only people who think this is "trivially easy" are people who don't have to do it.

res0nat0r 13y ago

Tapping multi-mode dark fiber unnoticed is now considered trivially easy? I must have missed when the bar was shifted this high.

XorNot 13y ago
The fiber doesn't run straight into a Google datacenter. It's operated and managed by the company which lays the cable under the sea, which means it goes through their stations at the coast and probably through quite a bit of other non-exclusive routing hardware. The fact that routing hardware exists means you can pretty easily tap it.
- res0nat0r 13y ago
  I believe my comment is pertaining to the use of the word "easily". James Bamford in The Shadow Factory was talking about how difficult this is, even for government institutions.
crucifiction 13y ago
You don't need to tap dark fiber for this to be poor security. They were passing all internal data around in clear text. It would have been trivial for any data center employee to gain significant amounts of open data this way. Why wouldn't you have services, db connections, etc. encrypted internally? Its certainly possible and done within many companies, it is surprising how lax google was in their assumption that once inside the network everything is going to be a-ok.
- res0nat0r 13y ago
  This isn't something that is "easy" when you run one of the biggest infrastructures in the world.