8 ms·

The curious case of slow downloads

207 points by jgrahamc 11y ago

speps 11y ago
Original title doesn't mention "NGINX".
- Dylan16807 11y ago
  Context in a title is useful.
  - oarsinsync 11y ago
    > Not an NGINX problem either
    - Dylan16807 11y ago
      There is a header that says that, but it actually was an NGINX problem.
      - masklinn 11y ago
        Not really. nginx's implemented behaviour seems more or less what anyone writing socket code would do: write to buffer, then wait with timeout to write more, if timeout is hit consider the write failed. The problem here is that (TIL) on linux the "writable socket" state behaves completely differently than the "readable socket" one, and a socket can be effectively writeable even though poll(2) reports that it isn't.
        Dylan16807 11y ago
        The TCP layer is already perfectly capable of detecting if a connection has failed. NGINX wanted to do something extra, and did it incorrectly.
        masklinn 11y ago
        > NGINX wanted to do something extra, and did it incorrectly. Having a timeout on a poll (or select) is not "something extra". it's something entirely normal and necessary for any non-trivial software, especially public-facing ones.
        Dylan16807 11y ago
        What? They wanted a timeout on how long it takes X bytes to send on an active TCP connection. That has nothing to do with poll/select timeouts.
        masklinn 11y ago
        > What? They wanted a timeout on how long it takes X bytes to send on an active TCP connection. That has nothing to do with poll/select timeouts. nginx called send(file), then polled the socket for write with the timeout, closing the connection if the timeout was hit. It has everything to do with poll/select timeouts, and especially with Linux letting poll hit timeouts when waiting on effectively writable sockets.
        Dylan16807 11y ago
        > writable sockets The goal of NGINX here is not to measure whether a socket can consume any data whatsoever. A connection that consumes 1 byte every 15 seconds is still considered stalled, and should be killed. Using the time between sendfile invocations is not the goal, it's a means toward implementing a minimum rate, and they implemented a minimum rate the wrong way. It's an X Y problem and that's not the kernel's fault.
        pjc50 11y ago
        .. and the correct way to do this is SO_SNDTIMEO.
        masklinn 11y ago
        That's for blocking IO, it doesn't work on evented IO: > [SO_RCVTIMEO and SO_SNDTIMEO] only have effect for system calls that perform socket I/O (e.g., read(2), recvmsg(2), send(2), sendmsg(2)); timeouts have no effect for select(2), poll(2), epoll_wait(2), and so on. And from my understanding would have the same effect as a timeout on select/poll, so it's not the correct way to do anything.
      - NicoJuicy 11y ago
        This was a unix stack problem, but it could be fixed in nginx :)
        Dylan16807 11y ago
        NGINX decided to have a maximum time between calls to sendfile, no matter whether the data is actually transferring or not. In configuring this setup, someone had a bad intuition about how often NGINX would end up calling sendfile. It is a problem entirely of NGINX's devising, not a unix stack problem.
- gruez 11y ago
  You're correct, but the title is way less helpful without the "nginx". Unless the reader knows that cloudflare uses nginx, he won't know the article pertains to nginx
  - masklinn 11y ago
    Is it now? nginx seems to have pretty normal behaviour here, so the core issue is write-polling buffers on linux rather than nginx itself. It seems to me nginx was the messenger of sorts, but the issue doesn't really pertain to nginx. In a nearby comment, LoSboccacc notes that they've hit that issue [removed: on S3].
    - ParadoxOryx 11y ago
      It sounded to me like LoSboccacc's solution was to serve files from S3, not that they experienced the same issue on S3.
      - masklinn 11y ago
        You're right, I misread it.
bigethan 11y ago
This is why open source (and controlling your whole stack) matters in big business. For example, Microsoft may be changing their image, but their core software is closed source. <cloud provider> may be great, but can you pull off something like this when you've got an issue? The importance of being able to debug and patch your mission critical systems is hard to overstate. Please encourage your employer to fiscally support the open source code that they rely upon!
- masklinn 11y ago
  > For example, Microsoft may be changing their image, but their core software is closed source. <cloud provider> may be great, but can you pull off something like this when you've got an issue? The importance of being able to debug and patch your mission critical systems is hard to overstate. Not OSS != no source access. A cloud provider on top of the MS stack would most likely have Shared Source Initiative licenses.
  - bigethan 11y ago
    What's that cost? How quickly can it be done? If I was a developer would I have to ask my manager to look into the details of our support plan? What about the ability to patch the source? `git clone` is an incredible power. Companies should reward those who are generous with their code.
    - masklinn 11y ago
      > What's that cost? How quickly can it be done? Careful with those goalposts, you're tearing up the pavement. > Companies should reward those who are generous with their code. So companies "should reward those who are generous with their code" but not compensate them for it?
      - lmm 11y ago
        > So companies "should reward those who are generous with their code" but not compensate them for it? No, they absolutely should compensate them for it. Buy OSS.
LoSboccacc 11y ago
ah! got bit by this and had to workaround serving stuff from s3. nice to see there was an actual problem and wasn't just some issue with our stack.
BugsBunnySan 11y ago
Related story time: At university, we ran a local quakeworld server (yes, this was in the stone age XD). And I wanted to write a tool to allow users to control to server. It would listen on a network port and pipe its input into the quakeworld server running as a child subprocess. Sounds simple enough, right? Disregarding accept(2)ing connections,etc... it's just going to sit in an endless loop, select(2) on the sockets then read what comes in, and write that to the quakeworld server. To debug this, everytime someone sent some data over the network, it would write a simple message ("blah") into a file on my home drive, called "~/blah" (very sophisticated debugging, I know). Now, the semantics of select(2), cousin of poll(2), are interesting. My understanding: select(2) blocks until you can read data from the socket. Actual behaviour: select(2) blocks until a subsequent read on the socket does not block. Fun fact: if there's not data available on the socket, read does not block, but returns with an error immediatly. Result: Over the weekend, the process was spinning in an endless loop, writing 'blah' into a file on the nfs-home drive as quickly as the filesystem allowed. This being a university scale solaris NFS server, that's pretty fast. End Result: A test file called 'blah', filled with about 800 GB of 'blah'. Good thing the admins where nice people and didn't shoot me when I went to explain the next monday why the home drive was filled with "blah" XD EDIT: read(2) doesn't actually error on EOF, just returns 0, effect is the same though.
- ajnin 11y ago
  Read will return 0 when the peer has closed the connection, that's how you detect it. When there is no data, the call will block instead until some data arrives (unless the socket is set to non-block mode).
  - BugsBunnySan 11y ago
    Hm yeah, it could've been that I disconnected from the server when I went home for the weekend and didn't check for that in the code. It was a long time ago :)
- brazzy 11y ago
  > Good thing the admins where nice people and didn't shoot me when I went to explain the next monday why the home drive was filled with "blah" XD They should be the ones explaining why they don't have disk quotas set up on a shared system.
bluesign 11y ago
I guess dropping very slow connections are also kind of protection against DoS or DDOS attacks, where attacker can drain resources.
- scurvy 11y ago
  Slow reads are not a great attack vector on modern OS releases. Slow requests (like slowloris) were much more effective.
scurvy 11y ago
I really wish people would stop misusing TCP resets. It's not just a fast way to close a connection. That's not what it's for at all. If you're ever thinking about sending a reset, please read RFC 793 and RFC 3360 first. I'm looking at you Arbor.
- Dylan16807 11y ago
  Well the goal here seems to be aborting a connection, not just closing it. They specifically don't want to empty the buffer, because time is up. Is that a misuse of reset?
  - scurvy 11y ago
    Yes, it is a misuse of a TCP reset. The RFCs are very clear about when to use a TCP reset. You can't use a reset just because you want to be fast or don't feel like sending any more data. Resets are for something abnormal happening. A timeout isn't abnormal to a TCP connection. It's part of the process.
    - Dylan16807 11y ago
      So what are you supposed to do?
    - dsp1234 11y ago
      RFC 1122[0] says: A TCP connection may terminate in two ways: (1) the normal TCP close sequence using a FIN handshake, and (2) an "abort" in which one or more RST segments are sent and the connection state is immediately discarded. and A host MAY implement a "half-duplex" TCP close sequence, so that an application that has called CLOSE cannot continue to read data from the connection. If such a host issues a CLOSE call while received data is still pending in TCP, or if new data is received after CLOSE is called, its TCP SHOULD send a RST to show that data was lost. So this behavior seems fine unless another RFC specifically prevents it. [0] - https://tools.ietf.org/html/rfc1122#section-4.2.2.13 https://tools.ietf.org/html/rfc1122#section-4.2.2.13
    - DanielDent 11y ago
dredmorbius 11y ago
Meanwhile, would CloudFlare care to address its nonfuntional Tor captchas?
Yrlec 11y ago
Great that they solved it! My company (degoo.com) had to use CloudFront instead of CloudFlare for our binaries because of this bug. Our conversion rate went down about 10% whenever we used CloudFlare instead. Perhaps time to switch back.
Animats 11y ago
They just rediscovered silly window syndrome, but at the application level rather than the TCP level. In early TCP implementations, if you read from a TCP connection one byte at a time, and the sender was writing faster than the receiver, the sender TCP would get an ACK opening one byte of window. The sender TCP would then transmit a packet with only one byte. This often came up if a TCP connection was driving a slow device. The fix was to not send a window update until there was at least one full packet worth of window available. This is the same problem, but at the OS buffering level. Linux has a similar solution. The problem is 1) servers today have to protect themselves against readers that are "too slow", because that can be used as a denial of service attack, and 2) the approach to doing this was not well designed. This is a tough one for Cloudflare, because they're in the DDOS-prevention business. They have to decide whether a slow reader is an attacker or just someone on a slow connection. Still, insisting that a reader consume at least 5MB/minute is a bit much. Some people are still on dialup.

207 points by jgrahamc 11y ago

speps 11y ago

Original title doesn't mention "NGINX".

Dylan16807 11y ago
Context in a title is useful.
- oarsinsync 11y ago
  > Not an NGINX problem either
  - Dylan16807 11y ago
    There is a header that says that, but it actually was an NGINX problem.
    - masklinn 11y ago
      Not really. nginx's implemented behaviour seems more or less what anyone writing socket code would do: write to buffer, then wait with timeout to write more, if timeout is hit consider the write failed. The problem here is that (TIL) on linux the "writable socket" state behaves completely differently than the "readable socket" one, and a socket can be effectively writeable even though poll(2) reports that it isn't.
      - Dylan16807 11y ago
        The TCP layer is already perfectly capable of detecting if a connection has failed. NGINX wanted to do something extra, and did it incorrectly.
        masklinn 11y ago
        > NGINX wanted to do something extra, and did it incorrectly. Having a timeout on a poll (or select) is not "something extra". it's something entirely normal and necessary for any non-trivial software, especially public-facing ones.
        Dylan16807 11y ago
        What? They wanted a timeout on how long it takes X bytes to send on an active TCP connection. That has nothing to do with poll/select timeouts.
        masklinn 11y ago
        > What? They wanted a timeout on how long it takes X bytes to send on an active TCP connection. That has nothing to do with poll/select timeouts. nginx called send(file), then polled the socket for write with the timeout, closing the connection if the timeout was hit. It has everything to do with poll/select timeouts, and especially with Linux letting poll hit timeouts when waiting on effectively writable sockets.
        Dylan16807 11y ago
        > writable sockets The goal of NGINX here is not to measure whether a socket can consume any data whatsoever. A connection that consumes 1 byte every 15 seconds is still considered stalled, and should be killed. Using the time between sendfile invocations is not the goal, it's a means toward implementing a minimum rate, and they implemented a minimum rate the wrong way. It's an X Y problem and that's not the kernel's fault.
        pjc50 11y ago
        .. and the correct way to do this is SO_SNDTIMEO.
        masklinn 11y ago
        That's for blocking IO, it doesn't work on evented IO: > [SO_RCVTIMEO and SO_SNDTIMEO] only have effect for system calls that perform socket I/O (e.g., read(2), recvmsg(2), send(2), sendmsg(2)); timeouts have no effect for select(2), poll(2), epoll_wait(2), and so on. And from my understanding would have the same effect as a timeout on select/poll, so it's not the correct way to do anything.
    - NicoJuicy 11y ago
      This was a unix stack problem, but it could be fixed in nginx :)
      - Dylan16807 11y ago
        NGINX decided to have a maximum time between calls to sendfile, no matter whether the data is actually transferring or not. In configuring this setup, someone had a bad intuition about how often NGINX would end up calling sendfile. It is a problem entirely of NGINX's devising, not a unix stack problem.
gruez 11y ago
You're correct, but the title is way less helpful without the "nginx". Unless the reader knows that cloudflare uses nginx, he won't know the article pertains to nginx
- masklinn 11y ago
  Is it now? nginx seems to have pretty normal behaviour here, so the core issue is write-polling buffers on linux rather than nginx itself. It seems to me nginx was the messenger of sorts, but the issue doesn't really pertain to nginx. In a nearby comment, LoSboccacc notes that they've hit that issue [removed: on S3].
  - ParadoxOryx 11y ago
    It sounded to me like LoSboccacc's solution was to serve files from S3, not that they experienced the same issue on S3.
    - masklinn 11y ago
      You're right, I misread it.

bigethan 11y ago

This is why open source (and controlling your whole stack) matters in big business. For example, Microsoft may be changing their image, but their core software is closed source. <cloud provider> may be great, but can you pull off something like this when you've got an issue? The importance of being able to debug and patch your mission critical systems is hard to overstate. Please encourage your employer to fiscally support the open source code that they rely upon!

masklinn 11y ago
> For example, Microsoft may be changing their image, but their core software is closed source. <cloud provider> may be great, but can you pull off something like this when you've got an issue? The importance of being able to debug and patch your mission critical systems is hard to overstate. Not OSS != no source access. A cloud provider on top of the MS stack would most likely have Shared Source Initiative licenses.
- bigethan 11y ago
  What's that cost? How quickly can it be done? If I was a developer would I have to ask my manager to look into the details of our support plan? What about the ability to patch the source? `git clone` is an incredible power. Companies should reward those who are generous with their code.
  - masklinn 11y ago
    > What's that cost? How quickly can it be done? Careful with those goalposts, you're tearing up the pavement. > Companies should reward those who are generous with their code. So companies "should reward those who are generous with their code" but not compensate them for it?
    - lmm 11y ago
      > So companies "should reward those who are generous with their code" but not compensate them for it? No, they absolutely should compensate them for it. Buy OSS.

LoSboccacc 11y ago

ah! got bit by this and had to workaround serving stuff from s3. nice to see there was an actual problem and wasn't just some issue with our stack.

BugsBunnySan 11y ago

Related story time: At university, we ran a local quakeworld server (yes, this was in the stone age XD). And I wanted to write a tool to allow users to control to server. It would listen on a network port and pipe its input into the quakeworld server running as a child subprocess. Sounds simple enough, right? Disregarding accept(2)ing connections,etc... it's just going to sit in an endless loop, select(2) on the sockets then read what comes in, and write that to the quakeworld server. To debug this, everytime someone sent some data over the network, it would write a simple message ("blah") into a file on my home drive, called "~/blah" (very sophisticated debugging, I know). Now, the semantics of select(2), cousin of poll(2), are interesting. My understanding: select(2) blocks until you can read data from the socket. Actual behaviour: select(2) blocks until a subsequent read on the socket does not block. Fun fact: if there's not data available on the socket, read does not block, but returns with an error immediatly. Result: Over the weekend, the process was spinning in an endless loop, writing 'blah' into a file on the nfs-home drive as quickly as the filesystem allowed. This being a university scale solaris NFS server, that's pretty fast. End Result: A test file called 'blah', filled with about 800 GB of 'blah'. Good thing the admins where nice people and didn't shoot me when I went to explain the next monday why the home drive was filled with "blah" XD EDIT: read(2) doesn't actually error on EOF, just returns 0, effect is the same though.

ajnin 11y ago
Read will return 0 when the peer has closed the connection, that's how you detect it. When there is no data, the call will block instead until some data arrives (unless the socket is set to non-block mode).
- BugsBunnySan 11y ago
  Hm yeah, it could've been that I disconnected from the server when I went home for the weekend and didn't check for that in the code. It was a long time ago :)
brazzy 11y ago
> Good thing the admins where nice people and didn't shoot me when I went to explain the next monday why the home drive was filled with "blah" XD They should be the ones explaining why they don't have disk quotas set up on a shared system.

bluesign 11y ago

I guess dropping very slow connections are also kind of protection against DoS or DDOS attacks, where attacker can drain resources.

scurvy 11y ago
Slow reads are not a great attack vector on modern OS releases. Slow requests (like slowloris) were much more effective.

scurvy 11y ago

I really wish people would stop misusing TCP resets. It's not just a fast way to close a connection. That's not what it's for at all. If you're ever thinking about sending a reset, please read RFC 793 and RFC 3360 first. I'm looking at you Arbor.

Dylan16807 11y ago
Well the goal here seems to be aborting a connection, not just closing it. They specifically don't want to empty the buffer, because time is up. Is that a misuse of reset?
- scurvy 11y ago
  Yes, it is a misuse of a TCP reset. The RFCs are very clear about when to use a TCP reset. You can't use a reset just because you want to be fast or don't feel like sending any more data. Resets are for something abnormal happening. A timeout isn't abnormal to a TCP connection. It's part of the process.
  - Dylan16807 11y ago
    So what are you supposed to do?
  - dsp1234 11y ago
    RFC 1122[0] says: A TCP connection may terminate in two ways: (1) the normal TCP close sequence using a FIN handshake, and (2) an "abort" in which one or more RST segments are sent and the connection state is immediately discarded. and A host MAY implement a "half-duplex" TCP close sequence, so that an application that has called CLOSE cannot continue to read data from the connection. If such a host issues a CLOSE call while received data is still pending in TCP, or if new data is received after CLOSE is called, its TCP SHOULD send a RST to show that data was lost. So this behavior seems fine unless another RFC specifically prevents it. [0] - https://tools.ietf.org/html/rfc1122#section-4.2.2.13 https://tools.ietf.org/html/rfc1122#section-4.2.2.13
  - DanielDent 11y ago

dredmorbius 11y ago

Meanwhile, would CloudFlare care to address its nonfuntional Tor captchas?

Yrlec 11y ago

Great that they solved it! My company (degoo.com) had to use CloudFront instead of CloudFlare for our binaries because of this bug. Our conversion rate went down about 10% whenever we used CloudFlare instead. Perhaps time to switch back.

Animats 11y ago

They just rediscovered silly window syndrome, but at the application level rather than the TCP level. In early TCP implementations, if you read from a TCP connection one byte at a time, and the sender was writing faster than the receiver, the sender TCP would get an ACK opening one byte of window. The sender TCP would then transmit a packet with only one byte. This often came up if a TCP connection was driving a slow device. The fix was to not send a window update until there was at least one full packet worth of window available. This is the same problem, but at the OS buffering level. Linux has a similar solution. The problem is 1) servers today have to protect themselves against readers that are "too slow", because that can be used as a denial of service attack, and 2) the approach to doing this was not well designed. This is a tough one for Cloudflare, because they're in the DDOS-prevention business. They have to decide whether a slow reader is an attacker or just someone on a slow connection. Still, insisting that a reader consume at least 5MB/minute is a bit much. Some people are still on dialup.