7 ms·

Lets start from the beginning: the NSA "hack" became possible because Google (and its security team) made bad assumptions about the security of the connection b

by lsh123 13y ago

Lets start from the beginning: the NSA "hack" became possible because Google (and its security team) made bad assumptions about the security of the connection between Google's data centers and did not encrypt the traffic. Basically, this is security 101: protect data at rest and protect data in flight. So, sorry but I think the better subject for discussion would be how badly Google screwed up, not how evil is NSA. Moreover, it is not clear if other governments or criminals also had access to the users' data (e.g. in Google's data centers located outside of the US). So far Google did not produce any public post-mortem thus we have no clue how bad was the problem. P.S. I am sure I will get smashed in the comments, so let me say right away that NSA actions should be controlled and audited by the public (e.g. through our representatives in Congress). I think that the biggest "evil" here are the members of Congress who either approved NSA actions or failed to do their job and monitor/audit NSA properly. In particular, I would point my finger at Sen. Dianne Feinstein [D-CA] who should have been ousted from the office long time ago.

raldi 13y ago
Would you still feel Google had screwed up if the way the US government got the data was to burglarize one of their datacenters and tap directly into the machines' CPUs and memory buses?
- lsh123 13y ago
  Yes (search for SSAE16 or SAS70). However, I would not feel the same way if US government would have used Area 51 technology to hack 4096 public key encryption. The difference from my perspective is that in "burglary" scenario (and un-encrypted traffic scenario as well) Google failed to protect against well known threats. And in the "alien technology" case Google did everything you can at the known security/technology level and the failure came from aliens (aka an "unexpected technology advances"). Basically, I think Google's decision to do not encrypt the traffic is a gross negligence and I would love to see how someone would sue Google for it.
  - jonknee 13y ago
    > Basically, I think Google's decision to do not encrypt the traffic is a gross negligence and I would love to see how someone would sue Google for it. Wow. Just wow.
    - lsh123 13y ago
      I know that people don't like lawsuits. But for security to work there should be consequences for not doing security right. To give you an example, if a company X doesn't have any risks or damages from lack of security, then company X should not be investing in security to save money. However, if there is a monetary (or other) liability from a security breach, then the company X will have to make a choice and hopefully they will invest in security.
      - jonknee 13y ago
        That would make sense if you had a signed contract from Google that they were encrypting their internal traffic. I don't have that and I doubt anyone else does either. If the courts get involved it should 100% be at the hacking perpetrator, not to the victim.
        declan 13y ago
        Side note: Sec 702 of the FAA gives NSA complete immunity from all federal, state, and local laws, criminal prosecutions, and civil lawsuits when doing this kind of fiber tapping. "Notwithstanding" is an extremely powerful phrase. Wildcard. Trumps all other laws. (And this is why some of us were concerned about CISPA, which uses the identical language. Note CISPA's proponents have quietly faded into the woodwork post-Snowden revelations.)
  - raldi 13y ago
    What if the government kidnapped a Google engineer (or several) and hit them with a wrench until they retrieved the data? That's a known, low-tech threat too.
bo1024 13y ago
I can't agree with that, this was on Google's on fiber connections between their own data centers, right? And no other company with multiple data centers encrypts all traffic between them, right? (maybe you'll find a small counterexample but no big one.) So I don't think this is "security 101".
- lsh123 13y ago
  Well, I do :) Moreover, I encrypt all the traffic even inside the same data center.
  - cryptbe 13y ago
    could you share which technology are you using to encrypt all the traffic?
    - lsh123 13y ago
      * Mid-tier servers: standard HTTPS with nginx * Database: SSL connections for MySQL * Memcached, Gearmand, and other tools that don't have built-in SSL support: simple home grown message level encryption (AES256) And of course, there are VPN tunnels between data centers in addition to the above.
      - cryptbe 13y ago
        Thanks. > And of course, there are VPN tunnels between data centers in addition to the above. Could you please be more specific on the VPN solution that you are using? How do you manage the shared keys? How do you make sure 'system administrators can't easily read the traffic?"
        lsh123 13y ago
        We use Cisco appliances for VPN (a few different models) and indeed there is a shared key that we have to input manually. However, after the key is entered (and configs saved) in order to decrypt the traffic one would need to print Cisco configs which is a very unusual operation that would be logged and then alerts will fire, audits will catch it, etc.
kllrnohj 13y ago
> Lets start from the beginning: the NSA "hack" became possible because Google (and its security team) made bad assumptions about the security of the connection between Google's data centers and did not encrypt the traffic. The assumption isn't bad - it's a private network line, not a public internet connection. Nobody else had access to that line, at least they weren't supposed to. Splicing a fiber line is a bit outside the scope of your random attacker. You can't blame Google for not anticipating a hostile break-in by the government. The discussion should absolutely, 100% be directed at the NSA here. To accept that a private network connection is open season for the government to tap is batshit insane. > Moreover, it is not clear if other governments or criminals also had access to the users' data (e.g. in Google's data centers located outside of the US). So far Google did not produce any public post-mortem thus we have no clue how bad was the problem. How is Google supposed to tell you if they themselves didn't know? Although from the leaks it sounds like everyone is fucked thanks to the GCHQ and the NSA getting friendly with each other.
- lsh123 13y ago
  Well, I feel that encrypting traffic inside the data center is not a bad idea (and we do it at WePay where I serve as CSO). The reasons is that you never know who is listening (big smile here). For example, I don't want our system administrators to have an easy way to look at the traffic: yes, it is still possible to do but it is harder and requires some very unusual actions that will trigger alerts everywhere. If indeed Google does not know then it's just another sign of security failures at the company. Nobody is perfect and security incidents do happen. A good security will have in-depth defense and built-in monitoring/audit measure that would at the very least allow you to determine what have happened post-factum.
  - kllrnohj 13y ago
    > Well, I feel that encrypting traffic inside the data center is not a bad idea (and we do it at WePay where I serve as CSO). Do you have your own data center building? And if you don't have your own data center buildings, how are you guarding against physical attacks? Because just saying "encryption" doesn't actually mean anything. Encryption isn't free, and at Google scale that can add up. Useless encryption is just wasted power > For example, I don't want our system administrators to have an easy way to look at the traffic: yes, it is still possible to do but it is harder and requires some very unusual actions that will trigger alerts everywhere. That can be accomplished in many ways that don't involve encryption. And your servers are all capable of decrypting the data at some point, so you still have to trust your sys admins and/or have alternative systems in place as they still have access to the unencrypted data. > A good security will have in-depth defense and built-in monitoring/audit measure that would at the very least allow you to determine what have happened post-factum. How, exactly, do you detect cable splicing? Much less audit said splicing? You seem to be asking for a hell of a lot more than "good security"
lsh123 13y ago
I wrote my thoughts on the subject in a blog post as well: http://www.aleksey.com/2013/11/06/why-google-engineers-got-it-wrong/ http://www.aleksey.com/2013/11/06/why-google-engineers-got-i...
bas 13y ago
The moral of the story is that there is always a bigger fish.

by lsh123 13y ago

raldi 13y ago

Would you still feel Google had screwed up if the way the US government got the data was to burglarize one of their datacenters and tap directly into the machines' CPUs and memory buses?

lsh123 13y ago
Yes (search for SSAE16 or SAS70). However, I would not feel the same way if US government would have used Area 51 technology to hack 4096 public key encryption. The difference from my perspective is that in "burglary" scenario (and un-encrypted traffic scenario as well) Google failed to protect against well known threats. And in the "alien technology" case Google did everything you can at the known security/technology level and the failure came from aliens (aka an "unexpected technology advances"). Basically, I think Google's decision to do not encrypt the traffic is a gross negligence and I would love to see how someone would sue Google for it.
- jonknee 13y ago
  > Basically, I think Google's decision to do not encrypt the traffic is a gross negligence and I would love to see how someone would sue Google for it. Wow. Just wow.
  - lsh123 13y ago
    I know that people don't like lawsuits. But for security to work there should be consequences for not doing security right. To give you an example, if a company X doesn't have any risks or damages from lack of security, then company X should not be investing in security to save money. However, if there is a monetary (or other) liability from a security breach, then the company X will have to make a choice and hopefully they will invest in security.
    - jonknee 13y ago
      That would make sense if you had a signed contract from Google that they were encrypting their internal traffic. I don't have that and I doubt anyone else does either. If the courts get involved it should 100% be at the hacking perpetrator, not to the victim.
      - declan 13y ago
        Side note: Sec 702 of the FAA gives NSA complete immunity from all federal, state, and local laws, criminal prosecutions, and civil lawsuits when doing this kind of fiber tapping. "Notwithstanding" is an extremely powerful phrase. Wildcard. Trumps all other laws. (And this is why some of us were concerned about CISPA, which uses the identical language. Note CISPA's proponents have quietly faded into the woodwork post-Snowden revelations.)
- raldi 13y ago
  What if the government kidnapped a Google engineer (or several) and hit them with a wrench until they retrieved the data? That's a known, low-tech threat too.

bo1024 13y ago

I can't agree with that, this was on Google's on fiber connections between their own data centers, right? And no other company with multiple data centers encrypts all traffic between them, right? (maybe you'll find a small counterexample but no big one.) So I don't think this is "security 101".

lsh123 13y ago
Well, I do :) Moreover, I encrypt all the traffic even inside the same data center.
- cryptbe 13y ago
  could you share which technology are you using to encrypt all the traffic?
  - lsh123 13y ago
    * Mid-tier servers: standard HTTPS with nginx * Database: SSL connections for MySQL * Memcached, Gearmand, and other tools that don't have built-in SSL support: simple home grown message level encryption (AES256) And of course, there are VPN tunnels between data centers in addition to the above.
    - cryptbe 13y ago
      Thanks. > And of course, there are VPN tunnels between data centers in addition to the above. Could you please be more specific on the VPN solution that you are using? How do you manage the shared keys? How do you make sure 'system administrators can't easily read the traffic?"
      - lsh123 13y ago
        We use Cisco appliances for VPN (a few different models) and indeed there is a shared key that we have to input manually. However, after the key is entered (and configs saved) in order to decrypt the traffic one would need to print Cisco configs which is a very unusual operation that would be logged and then alerts will fire, audits will catch it, etc.

kllrnohj 13y ago

> Lets start from the beginning: the NSA "hack" became possible because Google (and its security team) made bad assumptions about the security of the connection between Google's data centers and did not encrypt the traffic. The assumption isn't bad - it's a private network line, not a public internet connection. Nobody else had access to that line, at least they weren't supposed to. Splicing a fiber line is a bit outside the scope of your random attacker. You can't blame Google for not anticipating a hostile break-in by the government. The discussion should absolutely, 100% be directed at the NSA here. To accept that a private network connection is open season for the government to tap is batshit insane. > Moreover, it is not clear if other governments or criminals also had access to the users' data (e.g. in Google's data centers located outside of the US). So far Google did not produce any public post-mortem thus we have no clue how bad was the problem. How is Google supposed to tell you if they themselves didn't know? Although from the leaks it sounds like everyone is fucked thanks to the GCHQ and the NSA getting friendly with each other.

lsh123 13y ago
Well, I feel that encrypting traffic inside the data center is not a bad idea (and we do it at WePay where I serve as CSO). The reasons is that you never know who is listening (big smile here). For example, I don't want our system administrators to have an easy way to look at the traffic: yes, it is still possible to do but it is harder and requires some very unusual actions that will trigger alerts everywhere. If indeed Google does not know then it's just another sign of security failures at the company. Nobody is perfect and security incidents do happen. A good security will have in-depth defense and built-in monitoring/audit measure that would at the very least allow you to determine what have happened post-factum.
- kllrnohj 13y ago
  > Well, I feel that encrypting traffic inside the data center is not a bad idea (and we do it at WePay where I serve as CSO). Do you have your own data center building? And if you don't have your own data center buildings, how are you guarding against physical attacks? Because just saying "encryption" doesn't actually mean anything. Encryption isn't free, and at Google scale that can add up. Useless encryption is just wasted power > For example, I don't want our system administrators to have an easy way to look at the traffic: yes, it is still possible to do but it is harder and requires some very unusual actions that will trigger alerts everywhere. That can be accomplished in many ways that don't involve encryption. And your servers are all capable of decrypting the data at some point, so you still have to trust your sys admins and/or have alternative systems in place as they still have access to the unencrypted data. > A good security will have in-depth defense and built-in monitoring/audit measure that would at the very least allow you to determine what have happened post-factum. How, exactly, do you detect cable splicing? Much less audit said splicing? You seem to be asking for a hell of a lot more than "good security"

lsh123 13y ago

I wrote my thoughts on the subject in a blog post as well: http://www.aleksey.com/2013/11/06/why-google-engineers-got-it-wrong/ http://www.aleksey.com/2013/11/06/why-google-engineers-got-i...

bas 13y ago

The moral of the story is that there is always a bigger fish.