4 ms·

Correct. And now they encrypt all the inter-data center traffic. Question: Is that done on an end-to-end basis? Or do they encrypt the links between data ce

Correct. And now they encrypt all the inter-data center traffic. Question: Is that done on an end-to-end basis? Or do they encrypt the links between data centers? I want to encrypt a 10g ethernet and all the solutions look quite expensive. Has anyone done high speed encryption (i.e. 10gbps/1500 byte packets) with strongswan or similar?

acqq 11y ago
I believe end to end is simpler when you control all the ends and can tweak all the software, like they do. There's also an advantage of having very fast processors already provided.
nocarrier 11y ago
You have to encrypt the transport at the application layer when you're talking about the scale of major internet companies--there are so many links going out of each datacenter that it's not feasible to do IPsec or similar lower level encryption. There aren't many hardware devices sold that can do the encryption at the 10-100+ Gbps speeds that datacenter links use (if any), and it's much easier to amortize the encryption cost across all your application cpus versus having a small number of devices or dedicated hosts doing crypto at a lower layer.
- Sephr 11y ago
  > There aren't many hardware devices sold that can do the encryption at the 10-100+ Gbps speeds You are vastly underestimating AES-NI. Most modern Intel CPUs can handle that kind of encryption throughput (when paired with fast enough memory). Even my old Sandy Bridge-E CPU from 2011 gives me ~96 Gbps of AES throughput (with quad channel memory).
  - elipsey 11y ago
    Should we trust the implementation of AES-NI in this context? http://arstechnica.com/security/2013/12/we-cannot-trust-intel-and-vias-chip-based-crypto-freebsd-developers-say/ http://arstechnica.com/security/2013/12/we-cannot-trust-inte...
    - dlitz 11y ago
      AES-NI and RDRAND are different beasts. AES-NI is deterministic, and thus much more difficult to practically backdoor.
      - elipsey 11y ago
        Sorry if this is a dumb question, I'm just trying to think it through. It seems like it has to use the key you give it; it seems like the output stream has to be correct, or it just won't work. I'm not really sure how this would work for network traffic, I guess for disks I would expect CBC mode, which would require an IV? Is that right? So could the hardware be made to expose the key, or IV's, or leak information in some unexpected way?
        epistasis 11y ago
        Leaking an IV isn't a problem, but leaking the key would be a problem. If AES could be made to leak the key it would be a huge problem with the cipher itself. Otherwise leaking the key would be restricted to some extremely complex timing level behavior of the CPU, which would be difficult to make it all the way to the network level, without network card cooperation (though many 10gbps network cards are intel made). Perhaps feasible, but far far less likely than compromising the entropy of a random number generator.
  - nocarrier 11y ago
    I was replying to a question about whether or not you could encrypt the links between the datacenters, where you typically you have large core/cluster routers that handle traffic entering and leaving your datacenter. I was talking about the fact that these routers have dozens or hundreds of 10-100 Gbps ports on them, and are simply not built to do line rate IPsec on each port. So you can't just turn on IPsec for the routers at your datacenter border and magically get encrypted transport between your datacenters. You can setup IPsec tunnels on your application nodes and use that to get transparent encryption of all traffic, but that is pretty complicated to configure and manage. It's much easier to just have your different applications use an encrypted and authenticated transport when talking to each other over the network, whether they are in the same datacenter, or talking to a remote datacenter. This is how Google and Facebook do their cross datacenter encryption.
deleted 11y ago
[deleted]
Spooky23 11y ago
Companies like Ciena make line rate encryption cards that protect links. Not cheap, but not outrageously expensive for that level of gear.
Dylan16807 11y ago
With hardware acceleration, my quad core laptop can encrypt faster than 20gbps. Without it, it can still encrypt at at 4gbps. I was going to suggest you could place a couple servers at each end, but you only need one low end server. Anyone offering expensive solutions is doing you a disservice.
- epistasis 11y ago
  Is that at the network level? My experience is that packet handling in Linux and FreeBSD does not make high speed IPsec feasible.
  - Dylan16807 11y ago
    With things like kernel bypassing and bulk packet processing it's really not that hard to get to 10gbps on a single server. Heck, linux is getting close to ready for 100gbps https://lwn.net/Articles/629155/ https://lwn.net/Articles/629155/ I don't know exactly what resources IPSEC takes, but you can definitely get the "or similar" part of the question. AES-NI takes 3.5 cycles per byte. BLAKE2 takes 3.08 cycles per byte. That means that 2-2.5 cores can handle all the encryption and signing by themselves. Then you have 80% of the machine's resources free just to shovel packets in and out.
    - epistasis 11y ago
      It's easy to get 10Gbps packet handling on Linux or FreeBSD, it's easy to encrypt at >10Gbps on Intel, but I would love to know how to get both into a working situation IPsec solution at >10Gbps. Speculation isn't interesting, because clearly all the pieces should work, so I'm looking for anybody with actual experience with it working. I've tried, and been unsuccessful, and also unable to find anybody else on the internet that has been successful.
      - Dylan16807 11y ago
        Is IPsec a hard requirement? I would expect it to be easier to encrypt and sign a megabit at a time instead of individual packets. I'm having trouble picturing a situation where it has to be IPsec and it also has to be 10Gbps on a single box.
late2part 11y ago
Given the large interest and discussion, I've created a discussion forum here: https://groups.google.com/d/forum/highspeedencryption https://groups.google.com/d/forum/highspeedencryption It's public, but please do join if you're interested. I think it would benefit the community to have an open and central location for best practices of high speed encryption. Epistasis, Phil21, others, please do join and let's continue the discussion and progress there!

acqq 11y ago

I believe end to end is simpler when you control all the ends and can tweak all the software, like they do. There's also an advantage of having very fast processors already provided.

nocarrier 11y ago

You have to encrypt the transport at the application layer when you're talking about the scale of major internet companies--there are so many links going out of each datacenter that it's not feasible to do IPsec or similar lower level encryption. There aren't many hardware devices sold that can do the encryption at the 10-100+ Gbps speeds that datacenter links use (if any), and it's much easier to amortize the encryption cost across all your application cpus versus having a small number of devices or dedicated hosts doing crypto at a lower layer.

Sephr 11y ago
> There aren't many hardware devices sold that can do the encryption at the 10-100+ Gbps speeds You are vastly underestimating AES-NI. Most modern Intel CPUs can handle that kind of encryption throughput (when paired with fast enough memory). Even my old Sandy Bridge-E CPU from 2011 gives me ~96 Gbps of AES throughput (with quad channel memory).
- elipsey 11y ago
  Should we trust the implementation of AES-NI in this context? http://arstechnica.com/security/2013/12/we-cannot-trust-intel-and-vias-chip-based-crypto-freebsd-developers-say/ http://arstechnica.com/security/2013/12/we-cannot-trust-inte...
  - dlitz 11y ago
    AES-NI and RDRAND are different beasts. AES-NI is deterministic, and thus much more difficult to practically backdoor.
    - elipsey 11y ago
      Sorry if this is a dumb question, I'm just trying to think it through. It seems like it has to use the key you give it; it seems like the output stream has to be correct, or it just won't work. I'm not really sure how this would work for network traffic, I guess for disks I would expect CBC mode, which would require an IV? Is that right? So could the hardware be made to expose the key, or IV's, or leak information in some unexpected way?
      - epistasis 11y ago
        Leaking an IV isn't a problem, but leaking the key would be a problem. If AES could be made to leak the key it would be a huge problem with the cipher itself. Otherwise leaking the key would be restricted to some extremely complex timing level behavior of the CPU, which would be difficult to make it all the way to the network level, without network card cooperation (though many 10gbps network cards are intel made). Perhaps feasible, but far far less likely than compromising the entropy of a random number generator.
- nocarrier 11y ago
  I was replying to a question about whether or not you could encrypt the links between the datacenters, where you typically you have large core/cluster routers that handle traffic entering and leaving your datacenter. I was talking about the fact that these routers have dozens or hundreds of 10-100 Gbps ports on them, and are simply not built to do line rate IPsec on each port. So you can't just turn on IPsec for the routers at your datacenter border and magically get encrypted transport between your datacenters. You can setup IPsec tunnels on your application nodes and use that to get transparent encryption of all traffic, but that is pretty complicated to configure and manage. It's much easier to just have your different applications use an encrypted and authenticated transport when talking to each other over the network, whether they are in the same datacenter, or talking to a remote datacenter. This is how Google and Facebook do their cross datacenter encryption.

deleted 11y ago

[deleted]

Spooky23 11y ago

Companies like Ciena make line rate encryption cards that protect links. Not cheap, but not outrageously expensive for that level of gear.

Dylan16807 11y ago

With hardware acceleration, my quad core laptop can encrypt faster than 20gbps. Without it, it can still encrypt at at 4gbps. I was going to suggest you could place a couple servers at each end, but you only need one low end server. Anyone offering expensive solutions is doing you a disservice.

epistasis 11y ago
Is that at the network level? My experience is that packet handling in Linux and FreeBSD does not make high speed IPsec feasible.
- Dylan16807 11y ago
  With things like kernel bypassing and bulk packet processing it's really not that hard to get to 10gbps on a single server. Heck, linux is getting close to ready for 100gbps https://lwn.net/Articles/629155/ https://lwn.net/Articles/629155/ I don't know exactly what resources IPSEC takes, but you can definitely get the "or similar" part of the question. AES-NI takes 3.5 cycles per byte. BLAKE2 takes 3.08 cycles per byte. That means that 2-2.5 cores can handle all the encryption and signing by themselves. Then you have 80% of the machine's resources free just to shovel packets in and out.
  - epistasis 11y ago
    It's easy to get 10Gbps packet handling on Linux or FreeBSD, it's easy to encrypt at >10Gbps on Intel, but I would love to know how to get both into a working situation IPsec solution at >10Gbps. Speculation isn't interesting, because clearly all the pieces should work, so I'm looking for anybody with actual experience with it working. I've tried, and been unsuccessful, and also unable to find anybody else on the internet that has been successful.
    - Dylan16807 11y ago
      Is IPsec a hard requirement? I would expect it to be easier to encrypt and sign a megabit at a time instead of individual packets. I'm having trouble picturing a situation where it has to be IPsec and it also has to be 10Gbps on a single box.

late2part 11y ago

Given the large interest and discussion, I've created a discussion forum here: https://groups.google.com/d/forum/highspeedencryption https://groups.google.com/d/forum/highspeedencryption It's public, but please do join if you're interested. I think it would benefit the community to have an open and central location for best practices of high speed encryption. Epistasis, Phil21, others, please do join and let's continue the discussion and progress there!