4 ms·

> Lets start from the beginning: the NSA "hack" became possible because Google (and its security team) made bad assumptions about the security of the connection

> Lets start from the beginning: the NSA "hack" became possible because Google (and its security team) made bad assumptions about the security of the connection between Google's data centers and did not encrypt the traffic. The assumption isn't bad - it's a private network line, not a public internet connection. Nobody else had access to that line, at least they weren't supposed to. Splicing a fiber line is a bit outside the scope of your random attacker. You can't blame Google for not anticipating a hostile break-in by the government. The discussion should absolutely, 100% be directed at the NSA here. To accept that a private network connection is open season for the government to tap is batshit insane. > Moreover, it is not clear if other governments or criminals also had access to the users' data (e.g. in Google's data centers located outside of the US). So far Google did not produce any public post-mortem thus we have no clue how bad was the problem. How is Google supposed to tell you if they themselves didn't know? Although from the leaks it sounds like everyone is fucked thanks to the GCHQ and the NSA getting friendly with each other.

lsh123 13y ago
Well, I feel that encrypting traffic inside the data center is not a bad idea (and we do it at WePay where I serve as CSO). The reasons is that you never know who is listening (big smile here). For example, I don't want our system administrators to have an easy way to look at the traffic: yes, it is still possible to do but it is harder and requires some very unusual actions that will trigger alerts everywhere. If indeed Google does not know then it's just another sign of security failures at the company. Nobody is perfect and security incidents do happen. A good security will have in-depth defense and built-in monitoring/audit measure that would at the very least allow you to determine what have happened post-factum.
- kllrnohj 13y ago
  > Well, I feel that encrypting traffic inside the data center is not a bad idea (and we do it at WePay where I serve as CSO). Do you have your own data center building? And if you don't have your own data center buildings, how are you guarding against physical attacks? Because just saying "encryption" doesn't actually mean anything. Encryption isn't free, and at Google scale that can add up. Useless encryption is just wasted power > For example, I don't want our system administrators to have an easy way to look at the traffic: yes, it is still possible to do but it is harder and requires some very unusual actions that will trigger alerts everywhere. That can be accomplished in many ways that don't involve encryption. And your servers are all capable of decrypting the data at some point, so you still have to trust your sys admins and/or have alternative systems in place as they still have access to the unencrypted data. > A good security will have in-depth defense and built-in monitoring/audit measure that would at the very least allow you to determine what have happened post-factum. How, exactly, do you detect cable splicing? Much less audit said splicing? You seem to be asking for a hell of a lot more than "good security"
  - lsh123 13y ago
    At WePay - no, we don't have our own data centers just yet. In a couple large companies I worked before - yes (and we did encrypt the traffic as much as possible). Some types of encryption are pretty cheap actually. I used to use special SSL cards in the servers 10-15 years ago but today my laptop would outperform these cards and wouldn't even get hot :) Plus you need to remember that relatively expensive public key encryption needs to be done only for key exchange. After that you run block or stream cyphers and those algorithms tend to be really fast. So far I haven't seen any evidences that there was cable splicing. Thus using occam's razor I would assume that the hack was much simpler than that. To detect the issue, I would start from reviewing the visitors log to the data center (assuming there is a visitor log). I'll re-iterate that security should be built on defense-in-depth principle. Every single protection layer will fail or someone will go around it. The assumption that a data center is "safe" is a bad assumption period. You have to play "what-if" game and think for the attacker.
    - kllrnohj 13y ago
      > So far I haven't seen any evidences that there was cable splicing. Thus using occam's razor I would assume that the hack was much simpler than that. What? The evidence totally points to cable splicing. What hack involves getting all the inter-DC packets but nothing else? Obviously the machines weren't compromised, or they wouldn't have cared about reverse-engineering the wire protocol. So what are you proposing was hacked? > I'll re-iterate that security should be built on defense-in-depth principle. Every single protection layer will fail or someone will go around it. The assumption that a data center is "safe" is a bad assumption period. You have to play "what-if" game and think for the attacker. And I'll re-iterate that you're asking for a goddamn magical pony. Side note, if your data center isn't safe go get a new one. Seriously. Most DCs have tons of security to make them safe. That's not an assumption.
      - lsh123 13y ago
        > The evidence totally points to cable splicing. I don't think there are any evidences at all. As far as I know, the only known thing is that NSA was able to obtain the un-encrypted google traffic. For example, it could have been backdoor in the router, one extra cable in the switch, or a few other similar low-tech options. > Most DCs have tons of security to make them safe. Don't disagree. But this doesn't make them invincible from other attack vectors (e.g. rogue employees). I actually heard the same argument from quite a few people during interviews and I usually don't hire them because you have to be paranoid to get security right :)
- mwytock 13y ago
  Seriously? If the NSA wanted to own WePay they would have even with your "security best practices". Sorry bud.
  - lsh123 13y ago
    Sure. A court order would do it no problem.
    - res0nat0r 13y ago
      Also: Having someone in your outsourced datacenter splice cables, SPAN ports, install trojaned hardware, etc etc etc...
lsh123 13y ago
BTW, "at least they weren't supposed to" is not a good enough argument in security :) You have to think about people who are not following the rules or your security is only protecting from a well-behaving 1st grade student.
- kllrnohj 13y ago
  There is no such thing as perfect security, only good enough security. At some point you have to accept risks, and the risk of physical network attacks is incredibly small compared to all the other attack vectors. Nobody was well prepared for the NSA's physical network attacks.
  - Daniel_Newby 13y ago
    Everybody who cared knew that the world's governments tap every fiber they can lay their hands on. It has been discussed on HN with great regularity for years before these NSA non-revelations. Physical attacks were and are a certainty. Anybody who ignores this fact has only themselves to blame. A good argument can even be made that they deserved to be pwned as punishment for their utter fecklessness.
    - kllrnohj 13y ago
      No, what was discussed was the NSA tapping in at ISP points, not digging up cables to splice them. And "discussed" is not accurate. It was proposed by a few but rejected by most as paranoid.
      - chris_mahan 13y ago
        Wait a minute... "It was proposed by a few but rejected by most as paranoid." and yet is most likely what happened? so the most in "rejected by most" were wrong. Wrong! So when someone says: "You're just being paranoid", my reply will be: "Better paranoid than wrong."
        13y ago
Daniel_Newby 13y ago
> You can't blame Google for not anticipating a hostile break-in by the government. "The" government? The lines were also being tapped by organized crime, China, France, etc. Google severely failed at data protection.
rsync 13y ago
"The assumption isn't bad - it's a private network line, not a public internet connection. Nobody else had access to that line, at least they weren't supposed to." You don't use telnet when you access your home server(s) from your laptop ... that's basically what they were doing. They skipped over a zero-cost, obvious best practice, and I think we should be suspicious. Either they've run that part of their network in a stunningly negligent fashion ... or this was the ingress they gave to the NSA which could be plausibly denied later.
- ajross 13y ago
  No, it's not. Telnet sends packets in the clear over many networks controlled directly by third parties and accessible to them as an inherent part of operating their business. This was a point-to-point cable. The only access possible was physical, by digging it up and splicing it. Obviously that attack was possible, but arguing that this is somehow "the same kind of attack" as running tcpdump on a router to sniff packets is just insane, sorry.
  - lsh123 13y ago
    Other attack vectors: rogue google employee, breakins into the data center, ...

lsh123 13y ago

Well, I feel that encrypting traffic inside the data center is not a bad idea (and we do it at WePay where I serve as CSO). The reasons is that you never know who is listening (big smile here). For example, I don't want our system administrators to have an easy way to look at the traffic: yes, it is still possible to do but it is harder and requires some very unusual actions that will trigger alerts everywhere. If indeed Google does not know then it's just another sign of security failures at the company. Nobody is perfect and security incidents do happen. A good security will have in-depth defense and built-in monitoring/audit measure that would at the very least allow you to determine what have happened post-factum.

kllrnohj 13y ago
> Well, I feel that encrypting traffic inside the data center is not a bad idea (and we do it at WePay where I serve as CSO). Do you have your own data center building? And if you don't have your own data center buildings, how are you guarding against physical attacks? Because just saying "encryption" doesn't actually mean anything. Encryption isn't free, and at Google scale that can add up. Useless encryption is just wasted power > For example, I don't want our system administrators to have an easy way to look at the traffic: yes, it is still possible to do but it is harder and requires some very unusual actions that will trigger alerts everywhere. That can be accomplished in many ways that don't involve encryption. And your servers are all capable of decrypting the data at some point, so you still have to trust your sys admins and/or have alternative systems in place as they still have access to the unencrypted data. > A good security will have in-depth defense and built-in monitoring/audit measure that would at the very least allow you to determine what have happened post-factum. How, exactly, do you detect cable splicing? Much less audit said splicing? You seem to be asking for a hell of a lot more than "good security"
- lsh123 13y ago
  At WePay - no, we don't have our own data centers just yet. In a couple large companies I worked before - yes (and we did encrypt the traffic as much as possible). Some types of encryption are pretty cheap actually. I used to use special SSL cards in the servers 10-15 years ago but today my laptop would outperform these cards and wouldn't even get hot :) Plus you need to remember that relatively expensive public key encryption needs to be done only for key exchange. After that you run block or stream cyphers and those algorithms tend to be really fast. So far I haven't seen any evidences that there was cable splicing. Thus using occam's razor I would assume that the hack was much simpler than that. To detect the issue, I would start from reviewing the visitors log to the data center (assuming there is a visitor log). I'll re-iterate that security should be built on defense-in-depth principle. Every single protection layer will fail or someone will go around it. The assumption that a data center is "safe" is a bad assumption period. You have to play "what-if" game and think for the attacker.
  - kllrnohj 13y ago
    > So far I haven't seen any evidences that there was cable splicing. Thus using occam's razor I would assume that the hack was much simpler than that. What? The evidence totally points to cable splicing. What hack involves getting all the inter-DC packets but nothing else? Obviously the machines weren't compromised, or they wouldn't have cared about reverse-engineering the wire protocol. So what are you proposing was hacked? > I'll re-iterate that security should be built on defense-in-depth principle. Every single protection layer will fail or someone will go around it. The assumption that a data center is "safe" is a bad assumption period. You have to play "what-if" game and think for the attacker. And I'll re-iterate that you're asking for a goddamn magical pony. Side note, if your data center isn't safe go get a new one. Seriously. Most DCs have tons of security to make them safe. That's not an assumption.
    - lsh123 13y ago
      > The evidence totally points to cable splicing. I don't think there are any evidences at all. As far as I know, the only known thing is that NSA was able to obtain the un-encrypted google traffic. For example, it could have been backdoor in the router, one extra cable in the switch, or a few other similar low-tech options. > Most DCs have tons of security to make them safe. Don't disagree. But this doesn't make them invincible from other attack vectors (e.g. rogue employees). I actually heard the same argument from quite a few people during interviews and I usually don't hire them because you have to be paranoid to get security right :)
mwytock 13y ago
Seriously? If the NSA wanted to own WePay they would have even with your "security best practices". Sorry bud.
- lsh123 13y ago
  Sure. A court order would do it no problem.
  - res0nat0r 13y ago
    Also: Having someone in your outsourced datacenter splice cables, SPAN ports, install trojaned hardware, etc etc etc...

lsh123 13y ago

BTW, "at least they weren't supposed to" is not a good enough argument in security :) You have to think about people who are not following the rules or your security is only protecting from a well-behaving 1st grade student.

kllrnohj 13y ago
There is no such thing as perfect security, only good enough security. At some point you have to accept risks, and the risk of physical network attacks is incredibly small compared to all the other attack vectors. Nobody was well prepared for the NSA's physical network attacks.
- Daniel_Newby 13y ago
  Everybody who cared knew that the world's governments tap every fiber they can lay their hands on. It has been discussed on HN with great regularity for years before these NSA non-revelations. Physical attacks were and are a certainty. Anybody who ignores this fact has only themselves to blame. A good argument can even be made that they deserved to be pwned as punishment for their utter fecklessness.
  - kllrnohj 13y ago
    No, what was discussed was the NSA tapping in at ISP points, not digging up cables to splice them. And "discussed" is not accurate. It was proposed by a few but rejected by most as paranoid.
    - chris_mahan 13y ago
      Wait a minute... "It was proposed by a few but rejected by most as paranoid." and yet is most likely what happened? so the most in "rejected by most" were wrong. Wrong! So when someone says: "You're just being paranoid", my reply will be: "Better paranoid than wrong."
      - 13y ago

Daniel_Newby 13y ago

> You can't blame Google for not anticipating a hostile break-in by the government. "The" government? The lines were also being tapped by organized crime, China, France, etc. Google severely failed at data protection.

rsync 13y ago

"The assumption isn't bad - it's a private network line, not a public internet connection. Nobody else had access to that line, at least they weren't supposed to." You don't use telnet when you access your home server(s) from your laptop ... that's basically what they were doing. They skipped over a zero-cost, obvious best practice, and I think we should be suspicious. Either they've run that part of their network in a stunningly negligent fashion ... or this was the ingress they gave to the NSA which could be plausibly denied later.

ajross 13y ago
No, it's not. Telnet sends packets in the clear over many networks controlled directly by third parties and accessible to them as an inherent part of operating their business. This was a point-to-point cable. The only access possible was physical, by digging it up and splicing it. Obviously that attack was possible, but arguing that this is somehow "the same kind of attack" as running tcpdump on a router to sniff packets is just insane, sorry.
- lsh123 13y ago
  Other attack vectors: rogue google employee, breakins into the data center, ...