4 ms·

Well, I feel that encrypting traffic inside the data center is not a bad idea (and we do it at WePay where I serve as CSO). The reasons is that you never know w

by lsh123 13y ago

Well, I feel that encrypting traffic inside the data center is not a bad idea (and we do it at WePay where I serve as CSO). The reasons is that you never know who is listening (big smile here). For example, I don't want our system administrators to have an easy way to look at the traffic: yes, it is still possible to do but it is harder and requires some very unusual actions that will trigger alerts everywhere. If indeed Google does not know then it's just another sign of security failures at the company. Nobody is perfect and security incidents do happen. A good security will have in-depth defense and built-in monitoring/audit measure that would at the very least allow you to determine what have happened post-factum.

kllrnohj 13y ago
> Well, I feel that encrypting traffic inside the data center is not a bad idea (and we do it at WePay where I serve as CSO). Do you have your own data center building? And if you don't have your own data center buildings, how are you guarding against physical attacks? Because just saying "encryption" doesn't actually mean anything. Encryption isn't free, and at Google scale that can add up. Useless encryption is just wasted power > For example, I don't want our system administrators to have an easy way to look at the traffic: yes, it is still possible to do but it is harder and requires some very unusual actions that will trigger alerts everywhere. That can be accomplished in many ways that don't involve encryption. And your servers are all capable of decrypting the data at some point, so you still have to trust your sys admins and/or have alternative systems in place as they still have access to the unencrypted data. > A good security will have in-depth defense and built-in monitoring/audit measure that would at the very least allow you to determine what have happened post-factum. How, exactly, do you detect cable splicing? Much less audit said splicing? You seem to be asking for a hell of a lot more than "good security"
- lsh123 13y ago
  At WePay - no, we don't have our own data centers just yet. In a couple large companies I worked before - yes (and we did encrypt the traffic as much as possible). Some types of encryption are pretty cheap actually. I used to use special SSL cards in the servers 10-15 years ago but today my laptop would outperform these cards and wouldn't even get hot :) Plus you need to remember that relatively expensive public key encryption needs to be done only for key exchange. After that you run block or stream cyphers and those algorithms tend to be really fast. So far I haven't seen any evidences that there was cable splicing. Thus using occam's razor I would assume that the hack was much simpler than that. To detect the issue, I would start from reviewing the visitors log to the data center (assuming there is a visitor log). I'll re-iterate that security should be built on defense-in-depth principle. Every single protection layer will fail or someone will go around it. The assumption that a data center is "safe" is a bad assumption period. You have to play "what-if" game and think for the attacker.
  - kllrnohj 13y ago
    > So far I haven't seen any evidences that there was cable splicing. Thus using occam's razor I would assume that the hack was much simpler than that. What? The evidence totally points to cable splicing. What hack involves getting all the inter-DC packets but nothing else? Obviously the machines weren't compromised, or they wouldn't have cared about reverse-engineering the wire protocol. So what are you proposing was hacked? > I'll re-iterate that security should be built on defense-in-depth principle. Every single protection layer will fail or someone will go around it. The assumption that a data center is "safe" is a bad assumption period. You have to play "what-if" game and think for the attacker. And I'll re-iterate that you're asking for a goddamn magical pony. Side note, if your data center isn't safe go get a new one. Seriously. Most DCs have tons of security to make them safe. That's not an assumption.
    - lsh123 13y ago
      > The evidence totally points to cable splicing. I don't think there are any evidences at all. As far as I know, the only known thing is that NSA was able to obtain the un-encrypted google traffic. For example, it could have been backdoor in the router, one extra cable in the switch, or a few other similar low-tech options. > Most DCs have tons of security to make them safe. Don't disagree. But this doesn't make them invincible from other attack vectors (e.g. rogue employees). I actually heard the same argument from quite a few people during interviews and I usually don't hire them because you have to be paranoid to get security right :)
mwytock 13y ago
Seriously? If the NSA wanted to own WePay they would have even with your "security best practices". Sorry bud.
- lsh123 13y ago
  Sure. A court order would do it no problem.
  - res0nat0r 13y ago
    Also: Having someone in your outsourced datacenter splice cables, SPAN ports, install trojaned hardware, etc etc etc...

by lsh123 13y ago

kllrnohj 13y ago

> Well, I feel that encrypting traffic inside the data center is not a bad idea (and we do it at WePay where I serve as CSO). Do you have your own data center building? And if you don't have your own data center buildings, how are you guarding against physical attacks? Because just saying "encryption" doesn't actually mean anything. Encryption isn't free, and at Google scale that can add up. Useless encryption is just wasted power > For example, I don't want our system administrators to have an easy way to look at the traffic: yes, it is still possible to do but it is harder and requires some very unusual actions that will trigger alerts everywhere. That can be accomplished in many ways that don't involve encryption. And your servers are all capable of decrypting the data at some point, so you still have to trust your sys admins and/or have alternative systems in place as they still have access to the unencrypted data. > A good security will have in-depth defense and built-in monitoring/audit measure that would at the very least allow you to determine what have happened post-factum. How, exactly, do you detect cable splicing? Much less audit said splicing? You seem to be asking for a hell of a lot more than "good security"

lsh123 13y ago
At WePay - no, we don't have our own data centers just yet. In a couple large companies I worked before - yes (and we did encrypt the traffic as much as possible). Some types of encryption are pretty cheap actually. I used to use special SSL cards in the servers 10-15 years ago but today my laptop would outperform these cards and wouldn't even get hot :) Plus you need to remember that relatively expensive public key encryption needs to be done only for key exchange. After that you run block or stream cyphers and those algorithms tend to be really fast. So far I haven't seen any evidences that there was cable splicing. Thus using occam's razor I would assume that the hack was much simpler than that. To detect the issue, I would start from reviewing the visitors log to the data center (assuming there is a visitor log). I'll re-iterate that security should be built on defense-in-depth principle. Every single protection layer will fail or someone will go around it. The assumption that a data center is "safe" is a bad assumption period. You have to play "what-if" game and think for the attacker.
- kllrnohj 13y ago
  > So far I haven't seen any evidences that there was cable splicing. Thus using occam's razor I would assume that the hack was much simpler than that. What? The evidence totally points to cable splicing. What hack involves getting all the inter-DC packets but nothing else? Obviously the machines weren't compromised, or they wouldn't have cared about reverse-engineering the wire protocol. So what are you proposing was hacked? > I'll re-iterate that security should be built on defense-in-depth principle. Every single protection layer will fail or someone will go around it. The assumption that a data center is "safe" is a bad assumption period. You have to play "what-if" game and think for the attacker. And I'll re-iterate that you're asking for a goddamn magical pony. Side note, if your data center isn't safe go get a new one. Seriously. Most DCs have tons of security to make them safe. That's not an assumption.
  - lsh123 13y ago
    > The evidence totally points to cable splicing. I don't think there are any evidences at all. As far as I know, the only known thing is that NSA was able to obtain the un-encrypted google traffic. For example, it could have been backdoor in the router, one extra cable in the switch, or a few other similar low-tech options. > Most DCs have tons of security to make them safe. Don't disagree. But this doesn't make them invincible from other attack vectors (e.g. rogue employees). I actually heard the same argument from quite a few people during interviews and I usually don't hire them because you have to be paranoid to get security right :)

mwytock 13y ago

Seriously? If the NSA wanted to own WePay they would have even with your "security best practices". Sorry bud.

lsh123 13y ago
Sure. A court order would do it no problem.
- res0nat0r 13y ago
  Also: Having someone in your outsourced datacenter splice cables, SPAN ports, install trojaned hardware, etc etc etc...