4 ms·
Original title doesn't mention "NGINX".
by speps 10y ago
Original title doesn't mention "NGINX".
- Dylan16807 10y agoContext in a title is useful.
- oarsinsync 10y ago> Not an NGINX problem either
- Dylan16807 10y agoThere is a header that says that, but it actually was an NGINX problem.
- masklinn 10y agoNot really. nginx's implemented behaviour seems more or less what anyone writing socket code would do: write to buffer, then wait with timeout to write more, if timeout is hit consider the write failed. The problem here is that (TIL) on linux the "writable socket" state behaves completely differently than the "readable socket" one, and a socket can be effectively writeable even though poll(2) reports that it isn't.
- Dylan16807 10y agoThe TCP layer is already perfectly capable of detecting if a connection has failed. NGINX wanted to do something extra, and did it incorrectly.
- masklinn 10y ago> NGINX wanted to do something extra, and did it incorrectly. Having a timeout on a poll (or select) is not "something extra". it's something entirely normal and necessary for any non-trivial software, especially public-facing ones.
- Dylan16807 10y agoWhat? They wanted a timeout on how long it takes X bytes to send on an active TCP connection. That has nothing to do with poll/select timeouts.
- masklinn 10y ago> What? They wanted a timeout on how long it takes X bytes to send on an active TCP connection. That has nothing to do with poll/select timeouts. nginx called send(file), then polled the socket for write with the timeout, closing the connection if the timeout was hit. It has everything to do with poll/select timeouts, and especially with Linux letting poll hit timeouts when waiting on effectively writable sockets.
- Dylan16807 10y ago> writable sockets The goal of NGINX here is not to measure whether a socket can consume any data whatsoever. A connection that consumes 1 byte every 15 seconds is still considered stalled, and should be killed. Using the time between sendfile invocations is not the goal, it's a means toward implementing a minimum rate, and they implemented a minimum rate the wrong way. It's an X Y problem and that's not the kernel's fault.
- pjc50 10y ago.. and the correct way to do this is SO_SNDTIMEO.
- masklinn 10y agoThat's for blocking IO, it doesn't work on evented IO: > [SO_RCVTIMEO and SO_SNDTIMEO] only have effect for system calls that perform socket I/O (e.g., read(2), recvmsg(2), send(2), sendmsg(2)); timeouts have no effect for select(2), poll(2), epoll_wait(2), and so on. And from my understanding would have the same effect as a timeout on select/poll, so it's not the correct way to do anything.
- zzzcpan 10y ago
- scurvy 10y agonginx is also resetting the connection instead of closing it. That's also a bug (separately). The use of a TCP reset here violates RFC and Postel's robustness maxim. Be liberal on what you accept and be conservative in what you send.
- NicoJuicy 10y agoThis was a unix stack problem, but it could be fixed in nginx :)
- Dylan16807 10y agoNGINX decided to have a maximum time between calls to sendfile, no matter whether the data is actually transferring or not. In configuring this setup, someone had a bad intuition about how often NGINX would end up calling sendfile. It is a problem entirely of NGINX's devising, not a unix stack problem.
- gruez 10y agoYou're correct, but the title is way less helpful without the "nginx". Unless the reader knows that cloudflare uses nginx, he won't know the article pertains to nginx
- masklinn 10y agoIs it now? nginx seems to have pretty normal behaviour here, so the core issue is write-polling buffers on linux rather than nginx itself. It seems to me nginx was the messenger of sorts, but the issue doesn't really pertain to nginx. In a nearby comment, LoSboccacc notes that they've hit that issue [removed: on S3].
- ParadoxOryx 10y agoIt sounded to me like LoSboccacc's solution was to serve files from S3, not that they experienced the same issue on S3.
- masklinn 10y agoYou're right, I misread it.