7 ms·

I've been searching for a while for a good way to know whether a client has disconnected in the middle of a long-running HTTP request. (We do heavyweight SQL qu

by derefr 6y ago

I've been searching for a while for a good way to know whether a client has disconnected in the middle of a long-running HTTP request. (We do heavyweight SQL queries of indeterminate length in response to those requests, and we'd like to be able to cancel them, rather than wasting DB-server CPU cycles calculating reports nobody's going to consume.) You can't actually know whether the outgoing side of a TCP socket is closed, unless you write something to it. But it's hard to come up with something to write to an HTTP/1.1-over-TCP socket before you respond with anything, that would be a valid NOP according to all the protocol layers in play. (TCP keepalives would be perfect for this... if routers didn't silently drop them.) But I guess sending an HTTP 102 every second or two could be used for exactly this: prodding the socket with something that middleboxes will be sure to pass back to the client. If so, that's awesome! ...and also something I wish could be handled for me automatically by web frameworks, because getting that working sounds kind of ridiculous :)

comboy 6y ago
> You can't actually know whether the outgoing side of a TCP socket is closed, unless you write something to it Wouldn't setting appropriate net.ipv4.tcp_keepalive_* and trying to read work?
- derefr 6y ago
  > net.ipv4.tcp_keepalive_ Like I said: > TCP keepalives would be perfect for this... if routers didn't silently drop them. There are lots of middleboxes that don't pass along empty TCP packets. TCP keepalive is in a similar situation to IPsec: great for an Intranet, or for two public-Internet static peers with a clear layer-3 path between them; but everything falls apart in B2C scenarios. Plus, to add to this problem: HTTP has gateways (proxies et al.) Doing TCP keepalive on the server end, only tells you whether the last gateway in the chain before the server is still connected to the server, rather than whether the client is still connected to the server. Unless you can get every gateway in the chain to "propagate" keepalive (i.e. to push keepalive down to its client connection, iff the server pushes keepalive down onto it), silent undetected TCP disconnections will still happen—and even worse, you'll have false confidence that they aren't happening, as all your sockets will look like they're actively alive. For what I'm doing, the client end isn't likely to have any gateways, so TCP keepalives "would be" workable for my use-case if not for the middlebox thing. But in full generality, TCP keepalives aren't workable, because there's always those corporate L7 caching proxies + outbound WAFs messing things up, even when L4 middleboxes aren't. Keep your TCP keepalives for running connection-oriented stream protocols within your VPC. For HTTP on the open web, they're pretty unsuited. You need L7 keepalives. (If you've ever wondered, this is why websockets have their own L7 keepalives, a.k.a. "ping and pong" frames.) > and trying to read An HTTP client connection can legally half-close (i.e. close the output end) when it's done sending its last request; and this will result in a read(2) on the server's socket returning EOF. But this doesn't mean that the client's input end is closed! You have to do a write(2) to the server's socket to detect that. And, since empty TCP packets aren't guaranteed to make the trip, that means you need to write a nonzero number of bytes of ...something. Without that actually messing up the state-machine of your L7 protocol.
mleonhard 6y ago
You can do this with TCP keepalives [0]. Under Linux, a process can enable the SO_KEEPALIVE option on sockets to request that the Linux kernel send TCP keepalives periodically. A kernel option determines how frequently the kernel sends TCP keepalive packets. It is a single option that applies to all sockets of all processes. One can also configure the OS to enable SO_KEEPALIVE by default on all sockets of all processes. Golang's GRPC library implements keepalives at the GRPC protocol level [1]. It provides a `Context` value [2] that code can use to detect peer disconnect and cancel expensive operations. Golang's HTTP server API does not provide any way to detect peer disconnect before sending the final response [3]. Rust cannot set SO_KEEPALIVE [4]. One could possibly implement keepalives by writing zero-length chunks to the socket. Java's Netty server library can set SO_KEEPALIVE [5]. One can then code a request handler that periodically checks if the socket is connected [6] and cancels expensive operations. Unfortunately, there is standard tooling to do this. EDIT: You did mention TCP keepalives. I was not aware that some routers drop them. Can you link to any data on the prevalence of tcp-keepalive dropping for various kinds of client connections: home router, corporate wifi, mobile carrier-grade-NAT? [0] https://tldp.org/HOWTO/TCP-Keepalive-HOWTO/overview.html https://tldp.org/HOWTO/TCP-Keepalive-HOWTO/overview.html [1] https://pkg.go.dev/google.golang.org/grpc/keepalive https://pkg.go.dev/google.golang.org/grpc/keepalive [2] https://pkg.go.dev/google.golang.org/grpc#ServerStream https://pkg.go.dev/google.golang.org/grpc#ServerStream [3] https://pkg.go.dev/net/http#HandlerFunc https://pkg.go.dev/net/http#HandlerFunc [4] https://github.com/rust-lang/rust/issues/69774 https://github.com/rust-lang/rust/issues/69774 [5] https://netty.io/4.1/api/io/netty/channel/ChannelOption.html#SO_KEEPALIVE https://netty.io/4.1/api/io/netty/channel/ChannelOption.html... [6] https://netty.io/4.1/api/io/netty/channel/Channel.html#isOpen-- https://netty.io/4.1/api/io/netty/channel/Channel.html#isOpe...
- derefr 6y ago
  I can't offer any data myself, but I can suggest that chasing up the reason that more "modern" runtimes like Go's and Rust's don't bother to expose SO_KEEPALIVE support — i.e. the discussions that ensued when someone proposed adding this support, as they certainly did at some point — would be a good place to find that data cited. I can point out the obvious "analytical evidence", though: note how all the platform APIs that did expose SO_KEEPALIVE are from the 90s at the latest — i.e. before the proliferation of L4 middleboxes. And note how modern protocols like Websockets, gRPC, and even HTTP/2 (https://webconcepts.info/concepts/http2-frame-type/0x6 https://webconcepts.info/concepts/http2-frame-type/0x6) always do their own L7 keepalives, rather than relying on TCP keepalives — even when there's no technical obstacle to relying on TCP keepalives.
bawolff 6y ago
If using TLS, couldn't you send a zero-length application data fragment? The TLS 1.3 spec states "Zero-length fragments of Application Data MAY be sent, as they are potentially useful as a traffic analysis countermeasure." I guess that tls libraries wouldn't expose an api to do that which is problematic for this approach.
yencabulator 6y ago
Actually using HTTP 1xx will trigger hideous bugs, and worse those bugs will be timing-dependent. There is exactly one (non-websocket, which isn't really HTTP just pretends to be) codepath used in the wild: preapproval for a large file upload. This problem is one reason why success/error should NOT be the first thing to send. It should be a trailer. (HTTP/HTML tendency to substitute the response body for a human-visible error would require another mechanism to "reset" the response body.)
- ArchOversight 6y ago
  > codepath used in the wild: preapproval for a large file upload There is no current browser, or client that by default will send an Expect: 100-Continue. cURL removed it because it was too often broken. See https://curl.se/mail/lib-2017-07/0013.html https://curl.se/mail/lib-2017-07/0013.html As of right now, while server authors will continue to need to support it, it is unlikely that it is a well tested code path, and it will likely break in weird ways even trying to use it. So pre-approval for a large file upload is not even valid anymore.
  - yencabulator 6y ago
    Alright, make that the only codepath ever used in the wild. Even more reason to avoid 1xx.
deleted 6y ago
[deleted]
renonce 6y ago
Modern browsers reuse connections for subsequent requests so the connection may not be closed promptly, so this approach can't be relied upon anyway.

by derefr 6y ago

comboy 6y ago

> You can't actually know whether the outgoing side of a TCP socket is closed, unless you write something to it Wouldn't setting appropriate net.ipv4.tcp_keepalive_* and trying to read work?

derefr 6y ago
> net.ipv4.tcp_keepalive_ Like I said: > TCP keepalives would be perfect for this... if routers didn't silently drop them. There are lots of middleboxes that don't pass along empty TCP packets. TCP keepalive is in a similar situation to IPsec: great for an Intranet, or for two public-Internet static peers with a clear layer-3 path between them; but everything falls apart in B2C scenarios. Plus, to add to this problem: HTTP has gateways (proxies et al.) Doing TCP keepalive on the server end, only tells you whether the last gateway in the chain before the server is still connected to the server, rather than whether the client is still connected to the server. Unless you can get every gateway in the chain to "propagate" keepalive (i.e. to push keepalive down to its client connection, iff the server pushes keepalive down onto it), silent undetected TCP disconnections will still happen—and even worse, you'll have false confidence that they aren't happening, as all your sockets will look like they're actively alive. For what I'm doing, the client end isn't likely to have any gateways, so TCP keepalives "would be" workable for my use-case if not for the middlebox thing. But in full generality, TCP keepalives aren't workable, because there's always those corporate L7 caching proxies + outbound WAFs messing things up, even when L4 middleboxes aren't. Keep your TCP keepalives for running connection-oriented stream protocols within your VPC. For HTTP on the open web, they're pretty unsuited. You need L7 keepalives. (If you've ever wondered, this is why websockets have their own L7 keepalives, a.k.a. "ping and pong" frames.) > and trying to read An HTTP client connection can legally half-close (i.e. close the output end) when it's done sending its last request; and this will result in a read(2) on the server's socket returning EOF. But this doesn't mean that the client's input end is closed! You have to do a write(2) to the server's socket to detect that. And, since empty TCP packets aren't guaranteed to make the trip, that means you need to write a nonzero number of bytes of ...something. Without that actually messing up the state-machine of your L7 protocol.

mleonhard 6y ago

You can do this with TCP keepalives [0]. Under Linux, a process can enable the SO_KEEPALIVE option on sockets to request that the Linux kernel send TCP keepalives periodically. A kernel option determines how frequently the kernel sends TCP keepalive packets. It is a single option that applies to all sockets of all processes. One can also configure the OS to enable SO_KEEPALIVE by default on all sockets of all processes. Golang's GRPC library implements keepalives at the GRPC protocol level [1]. It provides a `Context` value [2] that code can use to detect peer disconnect and cancel expensive operations. Golang's HTTP server API does not provide any way to detect peer disconnect before sending the final response [3]. Rust cannot set SO_KEEPALIVE [4]. One could possibly implement keepalives by writing zero-length chunks to the socket. Java's Netty server library can set SO_KEEPALIVE [5]. One can then code a request handler that periodically checks if the socket is connected [6] and cancels expensive operations. Unfortunately, there is standard tooling to do this. EDIT: You did mention TCP keepalives. I was not aware that some routers drop them. Can you link to any data on the prevalence of tcp-keepalive dropping for various kinds of client connections: home router, corporate wifi, mobile carrier-grade-NAT? [0] https://tldp.org/HOWTO/TCP-Keepalive-HOWTO/overview.html https://tldp.org/HOWTO/TCP-Keepalive-HOWTO/overview.html [1] https://pkg.go.dev/google.golang.org/grpc/keepalive https://pkg.go.dev/google.golang.org/grpc/keepalive [2] https://pkg.go.dev/google.golang.org/grpc#ServerStream https://pkg.go.dev/google.golang.org/grpc#ServerStream [3] https://pkg.go.dev/net/http#HandlerFunc https://pkg.go.dev/net/http#HandlerFunc [4] https://github.com/rust-lang/rust/issues/69774 https://github.com/rust-lang/rust/issues/69774 [5] https://netty.io/4.1/api/io/netty/channel/ChannelOption.html#SO_KEEPALIVE https://netty.io/4.1/api/io/netty/channel/ChannelOption.html... [6] https://netty.io/4.1/api/io/netty/channel/Channel.html#isOpen-- https://netty.io/4.1/api/io/netty/channel/Channel.html#isOpe...

derefr 6y ago
I can't offer any data myself, but I can suggest that chasing up the reason that more "modern" runtimes like Go's and Rust's don't bother to expose SO_KEEPALIVE support — i.e. the discussions that ensued when someone proposed adding this support, as they certainly did at some point — would be a good place to find that data cited. I can point out the obvious "analytical evidence", though: note how all the platform APIs that did expose SO_KEEPALIVE are from the 90s at the latest — i.e. before the proliferation of L4 middleboxes. And note how modern protocols like Websockets, gRPC, and even HTTP/2 (https://webconcepts.info/concepts/http2-frame-type/0x6 https://webconcepts.info/concepts/http2-frame-type/0x6) always do their own L7 keepalives, rather than relying on TCP keepalives — even when there's no technical obstacle to relying on TCP keepalives.

bawolff 6y ago

If using TLS, couldn't you send a zero-length application data fragment? The TLS 1.3 spec states "Zero-length fragments of Application Data MAY be sent, as they are potentially useful as a traffic analysis countermeasure." I guess that tls libraries wouldn't expose an api to do that which is problematic for this approach.

yencabulator 6y ago

Actually using HTTP 1xx will trigger hideous bugs, and worse those bugs will be timing-dependent. There is exactly one (non-websocket, which isn't really HTTP just pretends to be) codepath used in the wild: preapproval for a large file upload. This problem is one reason why success/error should NOT be the first thing to send. It should be a trailer. (HTTP/HTML tendency to substitute the response body for a human-visible error would require another mechanism to "reset" the response body.)

ArchOversight 6y ago
> codepath used in the wild: preapproval for a large file upload There is no current browser, or client that by default will send an Expect: 100-Continue. cURL removed it because it was too often broken. See https://curl.se/mail/lib-2017-07/0013.html https://curl.se/mail/lib-2017-07/0013.html As of right now, while server authors will continue to need to support it, it is unlikely that it is a well tested code path, and it will likely break in weird ways even trying to use it. So pre-approval for a large file upload is not even valid anymore.
- yencabulator 6y ago
  Alright, make that the only codepath ever used in the wild. Even more reason to avoid 1xx.

deleted 6y ago

[deleted]

renonce 6y ago

Modern browsers reuse connections for subsequent requests so the connection may not be closed promptly, so this approach can't be relied upon anyway.