6 ms·

Google NetStack: IPv4 and IPv6 userland network stack in Go

122 points by yarapavan 9y ago

deleted 9y ago
[deleted]
zbowling 9y ago
It's the same netstack used in Fuchsia OS
oh_sigh 9y ago
How does Google use this internally? Is it for cloud?
- deleted 9y ago
  [deleted]
- Rovanion 9y ago
  No, mist.
- unmole 9y ago
  This seems to have been initially started as personal project by some Googlers. Now it is used in Google's upcoming operating system Fuchsia: https://github.com/fuchsia-mirror/netstack https://github.com/fuchsia-mirror/netstack
ohnoesjmr 9y ago
I wonder if one could abstract tcp on top of udp using this.
- unmole 9y ago
  If by "abstract tcp", you mean implement reliability and in-order delivery then, yes you can. QUIC[0] is the most promising of such approaches. [0]: https://en.wikipedia.org/wiki/QUIC https://en.wikipedia.org/wiki/QUIC
cromulen 9y ago
As opposed to the fuchsia netstack written in Go by Google https://github.com/fuchsia-mirror/netstack https://github.com/fuchsia-mirror/netstack Why so many netstacks in Go? I would imagine even the tiny GC pauses Go has would be undesirable for a netstack.
- hugelgupf 9y ago
  Fuchsia's netstack is based off this netstack. Fuchsia's netstack is essentially a wrapper of this netstack with link endpoint implementations to access NICs via magenta.
  - deleted 9y ago
    [deleted]
- tybit 9y ago
  I would guess that Go's tiny pauses would be less of a concern than it's overhead compared to C, C++ and Rust where there is more room for low level optimizations not available in Go. Not particularly important, I'm just curious as to how much the GC is really responsible for slowness as opposed to just correlating with languages that don't allow controlling heap vs stack allocations.
  - 0xFFC 9y ago
    MIT disagrees with both of you : https://math.mit.edu/research/highschool/primes/materials/2015/Sheth-Welling.pdf https://math.mit.edu/research/highschool/primes/materials/20...
    - akavel 9y ago
      I'm somewhat sad they didn't include a few things: - a box/whisker plot in the latency comparison graph — esp. if we're to talk about Go's GC... - some discussion/arguments why the particular C implementation was chosen ("tapip") — I'm not an expert in this area so I don't have the slightest idea how notable it is; - how did they detect/measure the claimed memory leaks in C? also, some statistics about the claimed crashes? - isn't clear to me if they used some "well known" load testing tools, or some homemade framework? (e.g. "siege" is a tool I read about more than once?) - more details on how exactly the "correctness was determined by testing against Linux kernel [implementation]". That said, my initial loose conclusions from this seem to be: - it appears it may be easier to write a correct implementation in Go (the implementation seems to be written just ad hoc by the article authors?) than in C; - it appears it may be easier in Go than in C to write an implementation scaling well w.r.t. average latency & throughput, assuming the need is for a multi-threaded & user-space implementation.
gtirloni 9y ago
A nice article about kernel vs. userland network stacks at Cloudflare: https://blog.cloudflare.com/why-we-use-the-linux-kernels-tcp-stack/ https://blog.cloudflare.com/why-we-use-the-linux-kernels-tcp...
beagle3 9y ago
A search for "ledbat" in github doesn't find any reference, which is a shame. Apple are the only one with ledbat support in the kernel[0] afaik, and it's functionality that's sorely needed -- it's ridiculous that there's no way to say "data on this connection is not urgent, only give it bandwidth as long as it doesn't interfere with other connections". Some programs, e.g. rsync, let you limit bandwidth - but that's wasteful; I want this 2TB rsync job to be downloading at full speed at all time I'm not doing anything else. [0] yes, this is userland, but it's for fuchsia, so "kernel functionality" as far as its users are concerned.
- unmole 9y ago
  How would LEDBAT help for downloads unless the server supports it? Even then, how does the client indicate to the server that "data on this connection is not urgent, only give it bandwidth as long as it doesn't interfere with other connections"?
  - deleted 9y ago
    [deleted]
  - simcop2387 9y ago
    If it's TCP, adjust the congestion control to stop sending acks and slow the connection when others have priority. Can't do it for anything like udp though.
    - unmole 9y ago
      > adjust the congestion control to stop sending acks Huh? Congestion control is enforced on the sender side while ACKs are generated by the receiver. Apple has a patent on throttling lower priority TCP flows by advertising a reduced receive window based on inter-arrival jitter. I feel this is a cleaner way than delaying ACKs. Anyways, my point was that LEDBAT wouldn't help.
- diamondlovesyou 9y ago
  It's not implementable by local devices: your LAN is much faster than your WAN link, in most cases. Your router/modem does the packet dropping. But only your router/modem knows how much unused throughput is available at any moment, so rsync can't know how much it can use. Sure, rsync can look at TCP packet retransmits, but that will still cause other streams to experience an (assuming a lot of things) "equal" amount of drops too. So, you get a lot of complexity for not a lot of benefit. However, Fuchsia is destined for mobile devices, which may not have the LAN-WAN bottleneck, so this could see some benefit.
omarforgotpwd 9y ago
So i'm not an expert in kernel context switches... would running a network stack in userland improve performance in any meaningful way for your average network service written in go? or is there some other purpose for this?
- justincormack 9y ago
  No, tcp is unlikely to be the bottleneck for a typical Go network server.
slackingoff2017 9y ago
Yes! I've been waiting for a fast userland OSS network stack forever. There's things like DPDK for IP but nobody has taken up the monumental task of getting TCP working and reliable
- gonzo 9y ago
  f-stack, based on libuinet works over DPDK. https://github.com/F-Stack/f-stack https://github.com/F-Stack/f-stack Libuinet is the FreeBSD TCP stack in user space (including over netmap) https://github.com/pkelsey/libuinet https://github.com/pkelsey/libuinet There is also work in FD.io on userspace TCP & UDP. https://github.com/FDio/tldk https://github.com/FDio/tldk
jwn 9y ago
Can someone explain what the use cases for this are?
- convolvatron 9y ago
  a few things come to mind : - unikernel go - making better parallel abstractions over network streams without having to deal with the mess of a kernel interface - as a platform to implement more integrated network policy (like congestion control, or state management for ddos protection) - ultimately might be really helpful for portability by requiring only a raw packet interface from the host OS - much easier environment for tying in w/ SDN and QOS (which google seems to be pretty gung ho on) there are probably a lot more. I looked a little at the repo, and they went for the most readable version that leans very heavily on the rich go runtime...which I think is a great call if you want to play around. messing with the linux network stack is a bit fussy and painful, there is a lot of .. stuff.. in there
markonen 9y ago
Not sure to what extent keeping up with the Joneses is a factor here (or indeed which party is the one trying to keep up), but Apple recently moved to a userspace/in-process TCP/IP stack in iOS 11 (and in the corresponding watchOS and tvOS versions). Seeing that this is for fuchsia makes that seem relevant.

122 points by yarapavan 9y ago

deleted 9y ago

[deleted]

zbowling 9y ago

It's the same netstack used in Fuchsia OS

oh_sigh 9y ago

How does Google use this internally? Is it for cloud?

deleted 9y ago
[deleted]
Rovanion 9y ago
No, mist.
unmole 9y ago
This seems to have been initially started as personal project by some Googlers. Now it is used in Google's upcoming operating system Fuchsia: https://github.com/fuchsia-mirror/netstack https://github.com/fuchsia-mirror/netstack

ohnoesjmr 9y ago

I wonder if one could abstract tcp on top of udp using this.

unmole 9y ago
If by "abstract tcp", you mean implement reliability and in-order delivery then, yes you can. QUIC[0] is the most promising of such approaches. [0]: https://en.wikipedia.org/wiki/QUIC https://en.wikipedia.org/wiki/QUIC

cromulen 9y ago

As opposed to the fuchsia netstack written in Go by Google https://github.com/fuchsia-mirror/netstack https://github.com/fuchsia-mirror/netstack Why so many netstacks in Go? I would imagine even the tiny GC pauses Go has would be undesirable for a netstack.

hugelgupf 9y ago
Fuchsia's netstack is based off this netstack. Fuchsia's netstack is essentially a wrapper of this netstack with link endpoint implementations to access NICs via magenta.
- deleted 9y ago
  [deleted]
tybit 9y ago
I would guess that Go's tiny pauses would be less of a concern than it's overhead compared to C, C++ and Rust where there is more room for low level optimizations not available in Go. Not particularly important, I'm just curious as to how much the GC is really responsible for slowness as opposed to just correlating with languages that don't allow controlling heap vs stack allocations.
- 0xFFC 9y ago
  MIT disagrees with both of you : https://math.mit.edu/research/highschool/primes/materials/2015/Sheth-Welling.pdf https://math.mit.edu/research/highschool/primes/materials/20...
  - akavel 9y ago
    I'm somewhat sad they didn't include a few things: - a box/whisker plot in the latency comparison graph — esp. if we're to talk about Go's GC... - some discussion/arguments why the particular C implementation was chosen ("tapip") — I'm not an expert in this area so I don't have the slightest idea how notable it is; - how did they detect/measure the claimed memory leaks in C? also, some statistics about the claimed crashes? - isn't clear to me if they used some "well known" load testing tools, or some homemade framework? (e.g. "siege" is a tool I read about more than once?) - more details on how exactly the "correctness was determined by testing against Linux kernel [implementation]". That said, my initial loose conclusions from this seem to be: - it appears it may be easier to write a correct implementation in Go (the implementation seems to be written just ad hoc by the article authors?) than in C; - it appears it may be easier in Go than in C to write an implementation scaling well w.r.t. average latency & throughput, assuming the need is for a multi-threaded & user-space implementation.

gtirloni 9y ago

A nice article about kernel vs. userland network stacks at Cloudflare: https://blog.cloudflare.com/why-we-use-the-linux-kernels-tcp-stack/ https://blog.cloudflare.com/why-we-use-the-linux-kernels-tcp...

beagle3 9y ago

A search for "ledbat" in github doesn't find any reference, which is a shame. Apple are the only one with ledbat support in the kernel[0] afaik, and it's functionality that's sorely needed -- it's ridiculous that there's no way to say "data on this connection is not urgent, only give it bandwidth as long as it doesn't interfere with other connections". Some programs, e.g. rsync, let you limit bandwidth - but that's wasteful; I want this 2TB rsync job to be downloading at full speed at all time I'm not doing anything else. [0] yes, this is userland, but it's for fuchsia, so "kernel functionality" as far as its users are concerned.

unmole 9y ago
How would LEDBAT help for downloads unless the server supports it? Even then, how does the client indicate to the server that "data on this connection is not urgent, only give it bandwidth as long as it doesn't interfere with other connections"?
- deleted 9y ago
  [deleted]
- simcop2387 9y ago
  If it's TCP, adjust the congestion control to stop sending acks and slow the connection when others have priority. Can't do it for anything like udp though.
  - unmole 9y ago
    > adjust the congestion control to stop sending acks Huh? Congestion control is enforced on the sender side while ACKs are generated by the receiver. Apple has a patent on throttling lower priority TCP flows by advertising a reduced receive window based on inter-arrival jitter. I feel this is a cleaner way than delaying ACKs. Anyways, my point was that LEDBAT wouldn't help.
diamondlovesyou 9y ago
It's not implementable by local devices: your LAN is much faster than your WAN link, in most cases. Your router/modem does the packet dropping. But only your router/modem knows how much unused throughput is available at any moment, so rsync can't know how much it can use. Sure, rsync can look at TCP packet retransmits, but that will still cause other streams to experience an (assuming a lot of things) "equal" amount of drops too. So, you get a lot of complexity for not a lot of benefit. However, Fuchsia is destined for mobile devices, which may not have the LAN-WAN bottleneck, so this could see some benefit.

omarforgotpwd 9y ago

So i'm not an expert in kernel context switches... would running a network stack in userland improve performance in any meaningful way for your average network service written in go? or is there some other purpose for this?

justincormack 9y ago
No, tcp is unlikely to be the bottleneck for a typical Go network server.

slackingoff2017 9y ago

Yes! I've been waiting for a fast userland OSS network stack forever. There's things like DPDK for IP but nobody has taken up the monumental task of getting TCP working and reliable

gonzo 9y ago
f-stack, based on libuinet works over DPDK. https://github.com/F-Stack/f-stack https://github.com/F-Stack/f-stack Libuinet is the FreeBSD TCP stack in user space (including over netmap) https://github.com/pkelsey/libuinet https://github.com/pkelsey/libuinet There is also work in FD.io on userspace TCP & UDP. https://github.com/FDio/tldk https://github.com/FDio/tldk

jwn 9y ago

Can someone explain what the use cases for this are?

convolvatron 9y ago
a few things come to mind : - unikernel go - making better parallel abstractions over network streams without having to deal with the mess of a kernel interface - as a platform to implement more integrated network policy (like congestion control, or state management for ddos protection) - ultimately might be really helpful for portability by requiring only a raw packet interface from the host OS - much easier environment for tying in w/ SDN and QOS (which google seems to be pretty gung ho on) there are probably a lot more. I looked a little at the repo, and they went for the most readable version that leans very heavily on the rich go runtime...which I think is a great call if you want to play around. messing with the linux network stack is a bit fussy and painful, there is a lot of .. stuff.. in there

markonen 9y ago

Not sure to what extent keeping up with the Joneses is a factor here (or indeed which party is the one trying to keep up), but Apple recently moved to a userspace/in-process TCP/IP stack in iOS 11 (and in the corresponding watchOS and tvOS versions). Seeing that this is for fuchsia makes that seem relevant.