3 ms·

Anyone had experience using this to setup up an ipsec VPN? I am looking for something that can do ipsec at 10Gbps which seems like it should be possible with a

Anyone had experience using this to setup up an ipsec VPN? I am looking for something that can do ipsec at 10Gbps which seems like it should be possible with an x86 processor that supports AES-NI but a lot of anecdotal benchmarks I've seen on randon websites seem to suggest it is difficult to achieve.

godzillabrennus 8y ago
YMMV with Pfsense. I've been using it since Scott and Chris were in Kentucky working on it together... that'll date me for sure... Anyway, I found that unless someone else has already done it and proven the hardware can handle it then it's all an experiment you conduct on your own dime. FYI - Netgate, the parent company of PFSense, went after OPNSense by scooping their domain and used it to spread FUD. Just dirty stuff. https://en.wikipedia.org/wiki/OPNsense https://en.wikipedia.org/wiki/OPNsense
- gonzo 8y ago
  > YMMV with Pfsense. I've been using it since Scott and Chris were in Kentucky working on it together... that'll date me for sure... You've been "using" it. We've been directly supporting it (and before it, m0n0wall) with cash on the barrel every month since before it was first released (Oct 2006). In Sep 2012, we bought out Scott, and started investing even more heavily in pfSense development.
  - godzillabrennus 8y ago
    I also “paid” for netgate hardware and recommended to many companies that they buy it. So in essence I helped you invest in the pfsense development. What you did was still dirty.
brynx97 8y ago
They have a new product called TNSR that looks like it can do this.
- gonzo 8y ago
  and more. See elsewhere in-thread for results.
Fnoord 8y ago
I don't know about FreeBSD, but on Linux WireGuard yields some 15-20% better performance (on both throughput and latency). OpenVPN is completely crap. And both OpenVPN and IPsec are much harder to set up than WireGuard.
- gonzo 8y ago
  > on Linux WireGuard yields some 15-20% better performance (on both throughput and latency). Assuming you're comparing to IPsec (and AES-NI accelerated AES-GCM on the P2s), and especially a routed IPsec (VTI) .vs the more normally found "policy-based" IPsec, then this isn't true, for at least 'throughput', and it's completely provable. I've not tested latency. Agreed with your statements on OpenVPN, and the advantages of setup for WireGuard .vs IPsec or OpenVPN.
paws 8y ago
Last summer I was looking into a building a little project that could do just that. TL, DR: It looks like those with the patience to compile a kernel+drivers can probably manage using Denverton boards [1]. I was blocked by no available drivers last year but hope to come back to my build project in the fall. NB: I'm just a casual tinkerer not an expert. Back story: Last summer when Intel shipped Denverton, QAT (which offloads crypto) apparently got a substantial speed boost and I saw claims that 10GbE IPSec should be possible. Denverton boards vary but some top end ones apparently ship with 4 10GbE NICs [2]. Back then the issue was that the QAT drivers weren't ready, so I put my project on hold. Today I see Intel has apparently shipped Linux and FreeBSD drivers. [3] Would of course be nice to see these drivers land upstream but no idea what the licensing story is like here. [1] https://www.servethehome.com/intel-atom-c3000-series-launch-skus-differentiation/ https://www.servethehome.com/intel-atom-c3000-series-launch-... [2] https://www.servethehome.com/supermicro-a2sdi-h-tp4f-review/ https://www.servethehome.com/supermicro-a2sdi-h-tp4f-review/ [2] https://01.org/intel-quick-assist-technology/downloads https://01.org/intel-quick-assist-technology/downloads
- gonzo 8y ago
  Depending on the SoC, C3000s have 0, 2 or 4 on-die MACs that will do 10Gbps. Also depending on the Doc, C3000s have no QAT or one that will run at "low", "medium" or "high" speed. The difference is in the number of "engines" that can be in-use. https://ark.intel.com/compare/97935,97928,97930,97929,97937,97934,97938,97926,97939,97932,97936,97931,97940,97933,97927 https://ark.intel.com/compare/97935,97928,97930,97929,97937,... Support for the C3000 NICs is included in pfSense 2.4.4. Support for the C3000 QAT will come later. I can say that we've tested to > 800Gbps on a C3550 using AES-GCM-128 between a pair of the C3558-based systems we sell. This is using AES-NI, not QAT.
gonzo 8y ago
If you have the traffic (many streams) it should be possible, yes. First thing you can do (if you haven't yet) is move from AES-CBC + HMAC-SHA1 to AES-GCM on the P2s, otherwise you're limited to the throughput of SHA1 (AES-NI accelerates AES, but the SHA1 will hold you back. AES-GCM doesn't need the HMAC (it's an AEAD algorithm), so it's faster. In theory, with AES-NI and AES-GCM, you should be able to get 2gbps per core. In fact, we've proven with with a different packet processing architecture (which I'll get to below). Problem: the GCM implementation we did for FreeBSD is slower than that, because I took the decision that the ability to resist side-channel attacks was more important that ultimate speed. It's also limited to a single core. There is some recent work in pfSense to use > 1 core for the AES encryption. So the second thing you can do is set net.inet.ipsec.async_crypto=1 Details are in https://reviews.freebsd.org/D10680 https://reviews.freebsd.org/D10680 We've exposed a knob for this in pfSense 2.4.4 https://redmine.pfsense.org/issues/8772#change-37604 https://redmine.pfsense.org/issues/8772#change-37604 But after that, you're going to be out of gas. Kernel networking will only go so fast, and the FreeBSD IPsec stack / cryptdev and AES-GCM implementation are all straight-forward packet-at-a-time. The FreeBSD (and linux) kernel stack(s) are inherently packet-at-a-time. This tends to limit available throughput. We are adding support for QAT to pfSense in 2.4.4, so there should be some additional gain there, as well. That all said, it is possible to get to 10Gbps IPsec. We did some testing on a pair of i7-6950X boxes equipped with 40Gbps NICs. (Details cam be found in https://www.netgate.com/blog/building-a-behemoth-router.html https://www.netgate.com/blog/building-a-behemoth-router.html) and achieved the following results. Remember these are on linux and, where noted, VPP. Measurements were to/from two more Xeon boxes equipped with 40gbps NICs as well. Lite details in the slides / video linked below. - kernel, AES-NI, AES-CBC-128 + HMAC-SHA1, 1 SA, 1 stream: 2.09 Gbps. - kernel, AES-NI, AES-CBC-128 + HMAC-SHA1, 1 SA, 4 streams: 2.07 Gbps. - kernel, AES-NI, AES-CBC-128 + HMAC-SHA1, 8 SAs, 8 streams: 10.85 Gbps. - kernel, AES-NI, AES-GCM-128-16, 1 SA, 1 stream: 5.06 Gbps. - kernel, AES-NI, AES-GCM-128-16, 1 SA, 4 streams: 5.06 Gbps. - kernel, AES-NI, AES-GCM-128-16, 8 SAs, 8 streams: 25.25 Gbps. Note in both the above that using > 1 SA and > 1 'stream' (TCP flow) allows the traffic to be spread across the cores. Remember also that this is linux. FreeBSD isn't able to turn in this type of result. Then we tested with a CPIC (https://www.netgate.com/products/cpic-8955.html https://www.netgate.com/products/cpic-8955.html) QuickAssist card in the tunnel endpoints: - kernel, QAT, AES-CBC-128 + HMAC-SHA1, 1 SA, 1 stream: 8.74 Gbps. - kernel, QAT, AES-CBC-128 + HMAC-SHA1, 1 SA, 4 streams: 8.74 Gbps. - kernel, QAT, AES-CBC-128 + HMAC-SHA1, 8 SAs, 8 streams: 27.08 Gbps. Note that 8.74 Gbps is quite close the the maximum you'll get through a 10 Gbps NIC, once the framing and packet overheads are accounted for. But look what happens when we move the processing to userspace with VPP: - vpp, AES-NI, AES-CBC-128 + HMAC-SHA1, 1 SA, 1 stream: 7.42 Gbps. - vpp, AES-NI, AES-CBC-128 + HMAC-SHA1, 1 SA, 4 streams: 8.28 Gbps. - vpp, AES-NI, AES-GCM-128-16, 1 SA, 1 stream: 13.70 Gbps. - vpp, AES-NI, AES-GCM-128-16, 1 SA, 4 streams: 15.93 Gbps. And with hardware offload, we can get to the maximum possible throughput on a 40Gbps NIC: - vpp, QAT, AES-CBC-128 + HMAC-SHA1, 1 SA, 1 stream: 32.68 Gbps. - vpp, QAT, AES-CBC-128 + HMAC-SHA1, 1 SA, 4 streams: 35.72 Gbps. - vpp, QAT, AES-CBC-128 + HMAC-SHA1, 8 SAs, 8 streams: 36.32 Gbps. - vpp, QAT, AES-GCM-128-16, 1 SA, 1 stream: 32.73 Gbps. - vpp, QAT, AES-GCM-128-16, 1 SA, 4 stream: 32.98 Gbps. Note that this was all with an internal build of VPP (as used in our TNSR product) in April 2017. Things have progressed since then. I talked about this some in 2017. Video https://www.safaribooksonline.com/library/view/oscon-2017-/9781491976227/video306685.html https://www.safaribooksonline.com/library/view/oscon-2017-/9... Slides https://wiki.fd.io/view/File:40_Gbps_IPsec_on_commodity_hardware.pdf https://wiki.fd.io/view/File:40_Gbps_IPsec_on_commodity_hard... https://www.netgate.com/products/tnsr/ https://www.netgate.com/products/tnsr/

godzillabrennus 8y ago

YMMV with Pfsense. I've been using it since Scott and Chris were in Kentucky working on it together... that'll date me for sure... Anyway, I found that unless someone else has already done it and proven the hardware can handle it then it's all an experiment you conduct on your own dime. FYI - Netgate, the parent company of PFSense, went after OPNSense by scooping their domain and used it to spread FUD. Just dirty stuff. https://en.wikipedia.org/wiki/OPNsense https://en.wikipedia.org/wiki/OPNsense

gonzo 8y ago
> YMMV with Pfsense. I've been using it since Scott and Chris were in Kentucky working on it together... that'll date me for sure... You've been "using" it. We've been directly supporting it (and before it, m0n0wall) with cash on the barrel every month since before it was first released (Oct 2006). In Sep 2012, we bought out Scott, and started investing even more heavily in pfSense development.
- godzillabrennus 8y ago
  I also “paid” for netgate hardware and recommended to many companies that they buy it. So in essence I helped you invest in the pfsense development. What you did was still dirty.

brynx97 8y ago

They have a new product called TNSR that looks like it can do this.

gonzo 8y ago
and more. See elsewhere in-thread for results.

Fnoord 8y ago

I don't know about FreeBSD, but on Linux WireGuard yields some 15-20% better performance (on both throughput and latency). OpenVPN is completely crap. And both OpenVPN and IPsec are much harder to set up than WireGuard.

gonzo 8y ago
> on Linux WireGuard yields some 15-20% better performance (on both throughput and latency). Assuming you're comparing to IPsec (and AES-NI accelerated AES-GCM on the P2s), and especially a routed IPsec (VTI) .vs the more normally found "policy-based" IPsec, then this isn't true, for at least 'throughput', and it's completely provable. I've not tested latency. Agreed with your statements on OpenVPN, and the advantages of setup for WireGuard .vs IPsec or OpenVPN.

paws 8y ago

Last summer I was looking into a building a little project that could do just that. TL, DR: It looks like those with the patience to compile a kernel+drivers can probably manage using Denverton boards [1]. I was blocked by no available drivers last year but hope to come back to my build project in the fall. NB: I'm just a casual tinkerer not an expert. Back story: Last summer when Intel shipped Denverton, QAT (which offloads crypto) apparently got a substantial speed boost and I saw claims that 10GbE IPSec should be possible. Denverton boards vary but some top end ones apparently ship with 4 10GbE NICs [2]. Back then the issue was that the QAT drivers weren't ready, so I put my project on hold. Today I see Intel has apparently shipped Linux and FreeBSD drivers. [3] Would of course be nice to see these drivers land upstream but no idea what the licensing story is like here. [1] https://www.servethehome.com/intel-atom-c3000-series-launch-skus-differentiation/ https://www.servethehome.com/intel-atom-c3000-series-launch-... [2] https://www.servethehome.com/supermicro-a2sdi-h-tp4f-review/ https://www.servethehome.com/supermicro-a2sdi-h-tp4f-review/ [2] https://01.org/intel-quick-assist-technology/downloads https://01.org/intel-quick-assist-technology/downloads

gonzo 8y ago
Depending on the SoC, C3000s have 0, 2 or 4 on-die MACs that will do 10Gbps. Also depending on the Doc, C3000s have no QAT or one that will run at "low", "medium" or "high" speed. The difference is in the number of "engines" that can be in-use. https://ark.intel.com/compare/97935,97928,97930,97929,97937,97934,97938,97926,97939,97932,97936,97931,97940,97933,97927 https://ark.intel.com/compare/97935,97928,97930,97929,97937,... Support for the C3000 NICs is included in pfSense 2.4.4. Support for the C3000 QAT will come later. I can say that we've tested to > 800Gbps on a C3550 using AES-GCM-128 between a pair of the C3558-based systems we sell. This is using AES-NI, not QAT.

gonzo 8y ago

If you have the traffic (many streams) it should be possible, yes. First thing you can do (if you haven't yet) is move from AES-CBC + HMAC-SHA1 to AES-GCM on the P2s, otherwise you're limited to the throughput of SHA1 (AES-NI accelerates AES, but the SHA1 will hold you back. AES-GCM doesn't need the HMAC (it's an AEAD algorithm), so it's faster. In theory, with AES-NI and AES-GCM, you should be able to get 2gbps per core. In fact, we've proven with with a different packet processing architecture (which I'll get to below). Problem: the GCM implementation we did for FreeBSD is slower than that, because I took the decision that the ability to resist side-channel attacks was more important that ultimate speed. It's also limited to a single core. There is some recent work in pfSense to use > 1 core for the AES encryption. So the second thing you can do is set net.inet.ipsec.async_crypto=1 Details are in https://reviews.freebsd.org/D10680 https://reviews.freebsd.org/D10680 We've exposed a knob for this in pfSense 2.4.4 https://redmine.pfsense.org/issues/8772#change-37604 https://redmine.pfsense.org/issues/8772#change-37604 But after that, you're going to be out of gas. Kernel networking will only go so fast, and the FreeBSD IPsec stack / cryptdev and AES-GCM implementation are all straight-forward packet-at-a-time. The FreeBSD (and linux) kernel stack(s) are inherently packet-at-a-time. This tends to limit available throughput. We are adding support for QAT to pfSense in 2.4.4, so there should be some additional gain there, as well. That all said, it is possible to get to 10Gbps IPsec. We did some testing on a pair of i7-6950X boxes equipped with 40Gbps NICs. (Details cam be found in https://www.netgate.com/blog/building-a-behemoth-router.html https://www.netgate.com/blog/building-a-behemoth-router.html) and achieved the following results. Remember these are on linux and, where noted, VPP. Measurements were to/from two more Xeon boxes equipped with 40gbps NICs as well. Lite details in the slides / video linked below. - kernel, AES-NI, AES-CBC-128 + HMAC-SHA1, 1 SA, 1 stream: 2.09 Gbps. - kernel, AES-NI, AES-CBC-128 + HMAC-SHA1, 1 SA, 4 streams: 2.07 Gbps. - kernel, AES-NI, AES-CBC-128 + HMAC-SHA1, 8 SAs, 8 streams: 10.85 Gbps. - kernel, AES-NI, AES-GCM-128-16, 1 SA, 1 stream: 5.06 Gbps. - kernel, AES-NI, AES-GCM-128-16, 1 SA, 4 streams: 5.06 Gbps. - kernel, AES-NI, AES-GCM-128-16, 8 SAs, 8 streams: 25.25 Gbps. Note in both the above that using > 1 SA and > 1 'stream' (TCP flow) allows the traffic to be spread across the cores. Remember also that this is linux. FreeBSD isn't able to turn in this type of result. Then we tested with a CPIC (https://www.netgate.com/products/cpic-8955.html https://www.netgate.com/products/cpic-8955.html) QuickAssist card in the tunnel endpoints: - kernel, QAT, AES-CBC-128 + HMAC-SHA1, 1 SA, 1 stream: 8.74 Gbps. - kernel, QAT, AES-CBC-128 + HMAC-SHA1, 1 SA, 4 streams: 8.74 Gbps. - kernel, QAT, AES-CBC-128 + HMAC-SHA1, 8 SAs, 8 streams: 27.08 Gbps. Note that 8.74 Gbps is quite close the the maximum you'll get through a 10 Gbps NIC, once the framing and packet overheads are accounted for. But look what happens when we move the processing to userspace with VPP: - vpp, AES-NI, AES-CBC-128 + HMAC-SHA1, 1 SA, 1 stream: 7.42 Gbps. - vpp, AES-NI, AES-CBC-128 + HMAC-SHA1, 1 SA, 4 streams: 8.28 Gbps. - vpp, AES-NI, AES-GCM-128-16, 1 SA, 1 stream: 13.70 Gbps. - vpp, AES-NI, AES-GCM-128-16, 1 SA, 4 streams: 15.93 Gbps. And with hardware offload, we can get to the maximum possible throughput on a 40Gbps NIC: - vpp, QAT, AES-CBC-128 + HMAC-SHA1, 1 SA, 1 stream: 32.68 Gbps. - vpp, QAT, AES-CBC-128 + HMAC-SHA1, 1 SA, 4 streams: 35.72 Gbps. - vpp, QAT, AES-CBC-128 + HMAC-SHA1, 8 SAs, 8 streams: 36.32 Gbps. - vpp, QAT, AES-GCM-128-16, 1 SA, 1 stream: 32.73 Gbps. - vpp, QAT, AES-GCM-128-16, 1 SA, 4 stream: 32.98 Gbps. Note that this was all with an internal build of VPP (as used in our TNSR product) in April 2017. Things have progressed since then. I talked about this some in 2017. Video https://www.safaribooksonline.com/library/view/oscon-2017-/9781491976227/video306685.html https://www.safaribooksonline.com/library/view/oscon-2017-/9... Slides https://wiki.fd.io/view/File:40_Gbps_IPsec_on_commodity_hardware.pdf https://wiki.fd.io/view/File:40_Gbps_IPsec_on_commodity_hard... https://www.netgate.com/products/tnsr/ https://www.netgate.com/products/tnsr/