6 ms·

Pretty amazing that you can achieve such a throughput in a Golang userspace program. I wonder if other UDP based protocols like QUIC can attain those numbers as

Pretty amazing that you can achieve such a throughput in a Golang userspace program. I wonder if other UDP based protocols like QUIC can attain those numbers as well.

kodah 4y ago
Slack has a system called Nebula that's pretty adjacent to userspace WireGuard.
- unixhero 4y ago
  Nebula is a Tailscale clone
  - psanford 4y ago
    Nebula predates tailscale.
    - unixhero 3y ago
      Oh it does? I stand corrected.
    - hardwaresofton 4y ago
      link for the lazy: https://github.com/slackhq/nebula https://github.com/slackhq/nebula Also with no ill intent looks like tailscale has the far more effective marketing organization :)
      - evulhotdog 4y ago
        The team that built nebula at Slack actually split off and are building a similar type of network, that they give you a single pane of glass to manage, so you don’t have to manage a bunch of separate lighthouse devices manually. I evaluated it when going through the many available options out there a year or two ago, and it was still pretty green feature wise. Still very cool and could work very well depending on your use cases. See: https://www.defined.net/ https://www.defined.net/
VWWHFSfQ 4y ago
is the go userspace program actually shoveling this data or are they in-kernel buffer copies a la sendfile and the like
- wmf 4y ago
  Yes, userspace has to touch the data to encrypt/decrypt it.
ffk 4y ago
Interestingly, the fastest CPU based network switches tend to do full kernel bypass. The kernel is generally slow compared to OVS and VPP, especially when they traverse over something like DPDK.
- arghwhat 4y ago
  Kernel bypass in DPDK grants the application direct access to DMA buffers so that the kernel is no longer involved. This is not because the kernel is slow, but because many small syscalls are expensive and putting your entire app in the kernel is a bad idea. There is no kernel bypass in wireguard-go, just a user-space implementation fast implementation with smart use of syscalls to minimize the overhead of being split between user-space and kernel-space. With io_uring, DPDK-style kernel bypass might stop making sense altogether.
  - ffk 4y ago
    It depends on what you are trying to do though. I don’t think the kernel has an easy path to operating on a set of packet headers as a vector at this point. Not saying it can’t happen, but it’s an area where user space is already ahead. For reference, there was a previous test that demonstrated 40gbps with ipsec between two pods on separate nodes in k8s where the encap/decap achieved 40gbps which was the line rate for the Intel NICs used. Details were published here: https://medium.com/fd-io-vpp/getting-to-40g-encrypted-container-networking-with-calico-vpp-on-commodity-hardware-d7144e52659a https://medium.com/fd-io-vpp/getting-to-40g-encrypted-contai... I do agree that io_uring will negate the need for DPDK for many use cases though, it will likely be a much simpler path and more secure path than DPDK.
- renewiltord 4y ago
  Are there consumer (<$2k) network switches that can do Wireguard in a very fast path?
  - wmf 4y ago
    By their nature as L2/L3 devices, I wouldn't expect switches to ever support Wireguard. I also haven't heard of any hardware Wireguard yet. The fastest implementation so far might be TNSR which just squeaks in under $2,000.
preseinger 4y ago
go is pretty fast in fact, i have a standing bet with some of my rustacean friends that they can't show me a typical HTTP service in rust, which has performance numbers (rps, latency, throughput) that i can't meet or beat in go of course lots of caveats there, what does normal-ish mean, well probably most of the work is gonna be i/o bound, it should run on normal server-class hardware, et cetera et cetera but nothing yet
- adam_arthur 4y ago
  Seems Rust places well in some composite benchmarks. Go is further down the list. Of course this depends on the quality of the implementation and doesn't account for UX/usability https://www.techempower.com/benchmarks/#section=data-r21&test=composite https://www.techempower.com/benchmarks/#section=data-r21&tes...
  - ThePhysicist 4y ago
    According to that benchmark, Javascript seems to be the way to go.
    - adam_arthur 4y ago
      Not familiar with Just, but seems its designed as a minimal wrapper over v8. The benchmark code is probably jumping directly from JS to native code... most of the github repo is C++ Edit: There's actually an article that explains how Just ranks so highly https://just.billywhizz.io/blog/on-javascript-performance-01/ https://just.billywhizz.io/blog/on-javascript-performance-01...
- _0w8t 4y ago
  Caddy (a web server written in Go) is like two times slower than Nginx on many benchmarks.
  - hnav 4y ago
    caddy is written in terms of net/http, nginx is written right on top of epoll/kqueue with bespoke HTTP/1x parser in a manually memory managed language. I think the point was not that go is "faster" than anything, it's that it makes it easy to write hard-to-beat network software once you get out of the realm of toy or highly specialized problems.
deleted 4y ago
[deleted]

Pretty amazing that you can achieve such a throughput in a Golang userspace program. I wonder if other UDP based protocols like QUIC can attain those numbers as well.

kodah 4y ago

Slack has a system called Nebula that's pretty adjacent to userspace WireGuard.

unixhero 4y ago
Nebula is a Tailscale clone
- psanford 4y ago
  Nebula predates tailscale.
  - unixhero 3y ago
    Oh it does? I stand corrected.
  - hardwaresofton 4y ago
    link for the lazy: https://github.com/slackhq/nebula https://github.com/slackhq/nebula Also with no ill intent looks like tailscale has the far more effective marketing organization :)
    - evulhotdog 4y ago
      The team that built nebula at Slack actually split off and are building a similar type of network, that they give you a single pane of glass to manage, so you don’t have to manage a bunch of separate lighthouse devices manually. I evaluated it when going through the many available options out there a year or two ago, and it was still pretty green feature wise. Still very cool and could work very well depending on your use cases. See: https://www.defined.net/ https://www.defined.net/

VWWHFSfQ 4y ago

is the go userspace program actually shoveling this data or are they in-kernel buffer copies a la sendfile and the like

wmf 4y ago
Yes, userspace has to touch the data to encrypt/decrypt it.

ffk 4y ago

Interestingly, the fastest CPU based network switches tend to do full kernel bypass. The kernel is generally slow compared to OVS and VPP, especially when they traverse over something like DPDK.

arghwhat 4y ago
Kernel bypass in DPDK grants the application direct access to DMA buffers so that the kernel is no longer involved. This is not because the kernel is slow, but because many small syscalls are expensive and putting your entire app in the kernel is a bad idea. There is no kernel bypass in wireguard-go, just a user-space implementation fast implementation with smart use of syscalls to minimize the overhead of being split between user-space and kernel-space. With io_uring, DPDK-style kernel bypass might stop making sense altogether.
- ffk 4y ago
  It depends on what you are trying to do though. I don’t think the kernel has an easy path to operating on a set of packet headers as a vector at this point. Not saying it can’t happen, but it’s an area where user space is already ahead. For reference, there was a previous test that demonstrated 40gbps with ipsec between two pods on separate nodes in k8s where the encap/decap achieved 40gbps which was the line rate for the Intel NICs used. Details were published here: https://medium.com/fd-io-vpp/getting-to-40g-encrypted-container-networking-with-calico-vpp-on-commodity-hardware-d7144e52659a https://medium.com/fd-io-vpp/getting-to-40g-encrypted-contai... I do agree that io_uring will negate the need for DPDK for many use cases though, it will likely be a much simpler path and more secure path than DPDK.
renewiltord 4y ago
Are there consumer (<$2k) network switches that can do Wireguard in a very fast path?
- wmf 4y ago
  By their nature as L2/L3 devices, I wouldn't expect switches to ever support Wireguard. I also haven't heard of any hardware Wireguard yet. The fastest implementation so far might be TNSR which just squeaks in under $2,000.

preseinger 4y ago

go is pretty fast in fact, i have a standing bet with some of my rustacean friends that they can't show me a typical HTTP service in rust, which has performance numbers (rps, latency, throughput) that i can't meet or beat in go of course lots of caveats there, what does normal-ish mean, well probably most of the work is gonna be i/o bound, it should run on normal server-class hardware, et cetera et cetera but nothing yet

adam_arthur 4y ago
Seems Rust places well in some composite benchmarks. Go is further down the list. Of course this depends on the quality of the implementation and doesn't account for UX/usability https://www.techempower.com/benchmarks/#section=data-r21&test=composite https://www.techempower.com/benchmarks/#section=data-r21&tes...
- ThePhysicist 4y ago
  According to that benchmark, Javascript seems to be the way to go.
  - adam_arthur 4y ago
    Not familiar with Just, but seems its designed as a minimal wrapper over v8. The benchmark code is probably jumping directly from JS to native code... most of the github repo is C++ Edit: There's actually an article that explains how Just ranks so highly https://just.billywhizz.io/blog/on-javascript-performance-01/ https://just.billywhizz.io/blog/on-javascript-performance-01...
_0w8t 4y ago
Caddy (a web server written in Go) is like two times slower than Nginx on many benchmarks.
- hnav 4y ago
  caddy is written in terms of net/http, nginx is written right on top of epoll/kqueue with bespoke HTTP/1x parser in a manually memory managed language. I think the point was not that go is "faster" than anything, it's that it makes it easy to write hard-to-beat network software once you get out of the realm of toy or highly specialized problems.

deleted 4y ago

[deleted]