3 ms·

Golang has an excellent net stack, but not perfect. In particular, it appears to suffer from the ever-recurring issue of not paying enough attention to teardown

by klabb3 4y ago

Golang has an excellent net stack, but not perfect. In particular, it appears to suffer from the ever-recurring issue of not paying enough attention to teardown. Everything must return to 0, ideally gracefully, but even the best programmers like to gloss over the details, because it’s easier to just kill everything and call it a day. Tcp is full duplex, but net.Conn (the main std API that everyone is using), doesn’t have a “CloseWrite” method. That bubbles down to all downstream uses, such as tls. Another example would be how deadlines and timeouts are handled, and how they (don’t) interact with context, which is the blessed way to do timeouts for all non-IO purposes.. Still, Go is one of the best languages for network programming imo, because it allows you to check most edge cases (like timeouts) without wasting like 4 threads to do so. But that’s just like.. my opinion.

kevin_nisbet 4y ago
Also, TCP is more like 2 uni-directional streams bolted together than a single full-duplex connection, which I've seen trip up lots of systems.
- GauntletWizard 4y ago
  There's probably room for a socket option to automatically close connection on FIN; I'm surprised it isn't there already.
- sleepydog 4y ago
  Your analogy is better directed towards the bsd sockets API that 99% of programs use, rather than TCP itself. I still feel that the API should lean even further into the "two streams" abstraction, as described here: http://cr.yp.to/tcpip/twofd.html http://cr.yp.to/tcpip/twofd.html
  - klabb3 4y ago
    It really is logically two streams indeed. In an alternative universe…
bhawks 4y ago
net.Conn needs to cover non-duplex conns like UDP - so it's challenging to add halfclosable methods with reasonable semantics. You can always check for the presence of the CloseWrite method and cast in this case. One trick to unify the context and io timeouts is roughly: conn = ...// Wherever you get your file or con. ctx, cxl = context.WithCancel(ctx) defer cxl() go func(){ defer conn.Close() <-ctx.Done() }() // Do reads and writes knowing that if your context times out the fd will close and your io op will stop. The only caveat is if you need to be aware of partial data read/written when a timeout occurs - then you need to tell the os about your timeouts :(.
- klabb3 4y ago
  That’s correct! I do similar things EVERY time I deal with non-trivial operations on a conn, which is very tedious, and why I wish cancelation propagated down to IO. Although goroutines are cheap, something is wrong if you need in sequential code, imo. They’re also notoriously easy to leak. However, once you need idle timeouts or heartbeats you have to increase the complexity further. Setting deadlines before every op (which is recommended) is (a) super easy to forget and (b) a perf issue if you write a stream of data, due to the several syscalls involved, which can be noticeable if you transfer at bandwidth capacity. Even std (such as net/http, tls) is full of these workarounds. If Go could get the last remaining pieces right, it would be near perfect for userspace network programming imo, as well as benefits in other domains as well. I want this Pony.
rendaw 4y ago
Similarly there's no way to shut down an HTTP client, the connection pool sticks around for the duration of the program.
- lelanthran 4y ago
  > Similarly there's no way to shut down an HTTP client, the connection pool sticks around for the duration of the program. I don't understand this - isn't the pool supposed to stick around for the duration of the program? Did you mean that the connection sticks around for the duration of the program?
  - cormacrelf 4y ago
    If you make a one shot request to an HTTP server, and the system creates a connection pool for that hostname:port, and you never need it ever again, should the connection pool survive until the program terminates? What if you make hundreds of requests like this? I don’t know how Go buckets their HTTP connection pools, but ADO.NET (the .NET SQL API) does it by connection string, and has a function for freeing all pools and also for one specific pool.
    - lelanthran 4y ago
      > If you make a one shot request to an HTTP server, and the system creates a connection pool for that hostname:port, and you never need it ever again, should the connection pool survive until the program terminates? I did not know it did that, and scanning through the docs, they don't appear to mention it either. Is this only on client programs? Does it do the same for server programs too (create a pool for each incoming src-ip:src-port connections)?
      - cormacrelf 4y ago
        It does have connection pooling for clients: https://blog.cloudflare.com/whats-inside-net-http-socket-late-binding-in-the-go-standard-library/ https://blog.cloudflare.com/whats-inside-net-http-socket-lat.... The pools naturally clear themselves though, because they observe the keep-alive timeouts. As long as your servers are not setting insanely high timeouts you might never notice. Nevertheless it does seem to be impossible to manually clear the pool. This is more of a problem when you realise you can’t use this to temporarily fix a memory leak due to a missing Close() call on a request body or something like that. So in those cases yes it will just leak indefinitely unless you specify an idle timeout in the client constructor as well.

by klabb3 4y ago

kevin_nisbet 4y ago

Also, TCP is more like 2 uni-directional streams bolted together than a single full-duplex connection, which I've seen trip up lots of systems.

GauntletWizard 4y ago
There's probably room for a socket option to automatically close connection on FIN; I'm surprised it isn't there already.
sleepydog 4y ago
Your analogy is better directed towards the bsd sockets API that 99% of programs use, rather than TCP itself. I still feel that the API should lean even further into the "two streams" abstraction, as described here: http://cr.yp.to/tcpip/twofd.html http://cr.yp.to/tcpip/twofd.html
- klabb3 4y ago
  It really is logically two streams indeed. In an alternative universe…

bhawks 4y ago

net.Conn needs to cover non-duplex conns like UDP - so it's challenging to add halfclosable methods with reasonable semantics. You can always check for the presence of the CloseWrite method and cast in this case. One trick to unify the context and io timeouts is roughly: conn = ...// Wherever you get your file or con. ctx, cxl = context.WithCancel(ctx) defer cxl() go func(){ defer conn.Close() <-ctx.Done() }() // Do reads and writes knowing that if your context times out the fd will close and your io op will stop. The only caveat is if you need to be aware of partial data read/written when a timeout occurs - then you need to tell the os about your timeouts :(.

klabb3 4y ago
That’s correct! I do similar things EVERY time I deal with non-trivial operations on a conn, which is very tedious, and why I wish cancelation propagated down to IO. Although goroutines are cheap, something is wrong if you need in sequential code, imo. They’re also notoriously easy to leak. However, once you need idle timeouts or heartbeats you have to increase the complexity further. Setting deadlines before every op (which is recommended) is (a) super easy to forget and (b) a perf issue if you write a stream of data, due to the several syscalls involved, which can be noticeable if you transfer at bandwidth capacity. Even std (such as net/http, tls) is full of these workarounds. If Go could get the last remaining pieces right, it would be near perfect for userspace network programming imo, as well as benefits in other domains as well. I want this Pony.

rendaw 4y ago

Similarly there's no way to shut down an HTTP client, the connection pool sticks around for the duration of the program.

lelanthran 4y ago
> Similarly there's no way to shut down an HTTP client, the connection pool sticks around for the duration of the program. I don't understand this - isn't the pool supposed to stick around for the duration of the program? Did you mean that the connection sticks around for the duration of the program?
- cormacrelf 4y ago
  If you make a one shot request to an HTTP server, and the system creates a connection pool for that hostname:port, and you never need it ever again, should the connection pool survive until the program terminates? What if you make hundreds of requests like this? I don’t know how Go buckets their HTTP connection pools, but ADO.NET (the .NET SQL API) does it by connection string, and has a function for freeing all pools and also for one specific pool.
  - lelanthran 4y ago
    > If you make a one shot request to an HTTP server, and the system creates a connection pool for that hostname:port, and you never need it ever again, should the connection pool survive until the program terminates? I did not know it did that, and scanning through the docs, they don't appear to mention it either. Is this only on client programs? Does it do the same for server programs too (create a pool for each incoming src-ip:src-port connections)?
    - cormacrelf 4y ago
      It does have connection pooling for clients: https://blog.cloudflare.com/whats-inside-net-http-socket-late-binding-in-the-go-standard-library/ https://blog.cloudflare.com/whats-inside-net-http-socket-lat.... The pools naturally clear themselves though, because they observe the keep-alive timeouts. As long as your servers are not setting insanely high timeouts you might never notice. Nevertheless it does seem to be impossible to manually clear the pool. This is more of a problem when you realise you can’t use this to temporarily fix a memory leak due to a missing Close() call on a request body or something like that. So in those cases yes it will just leak indefinitely unless you specify an idle timeout in the client constructor as well.