7 ms·

Google fixes nearly decade-old Linux kernel TCP bug

355 points by tim_sw 11y ago

eridius 11y ago
I'm confused by the last sentence: > The whole web, including Firefox users, will benefit. Why is Firefox thrown in here? And the post is even tagged with "firefox". But the contents of the post seems to have absolutely nothing to do with Firefox, except inasmuch as Firefox is an application that interacts with the web.
- TimothyFitz 11y ago
  The author works for mozilla, and firefox is a common theme on his blog. I think it's for regular readers of his blog to understand the practical impact (readers he assumes care more than average about firefox).
- stuaxo 11y ago
  I'd guess that because it was a bugfix by google - people might assume it was something to do with chrome or android, when it should help any webuser that talks to a linux based server, including firefox users.
- Manishearth 11y ago
  The tag is probably so that the post will show up in Planet Mozilla (http://planet.mozilla.org http://planet.mozilla.org). I often tag non-Mozilla posts as "mozilla" if they are still relevant to Planet's scope on my blog too. I've submitted a category feed instead of the full RSS blog feed so that I can exclude things which are physicsy or otherwise irrelevant.
userbinator 11y ago
Remember that TCP is robust enough that it seems to work anyhow - even at the cost of reduced network throughput in this case. That to me sounds like this is not really a bug, but a performance enhancement; a bug would be something like corrupted/missing/extra data at the application level. Tuning the TCP algorithms beyond the basics is really more of an art.
- wfunction 11y ago
  It's a bug if you consider performance to be a feature.
- plorg 11y ago
  Presumably it is considered to be a bug because it does not accomplish what the code is meant to accomplish - namely, the code does not properly characterize a congested state and thus does not operate as if it were congested.
  - nathanb 11y ago
    Given your definition, and viewed from a high enough level, every feature or enhancement is a bug fix. I think nitpicking the wording (while an interesting philosophical discussion) detracts from the point of the article: an unnamed Google engineer waded into the Linux kernel code and made the web slightly better for many people.
    - eru 11y ago
      I think it was actually multiple people.
    - nuxi7 11y ago
      You've hit on a fine point I often make about how the term "security bug" is used. I would call this a bug fix since someone has found what is probably an unintended behavior in the cubic algorithm. The cubic algorithm itself though would be a feature improving on the original tcp algorithm (nagle, if I remember right)
    - merb 11y ago
      No a feature or enhancement mostly introduces something new, like a new algorithm for cubic congestion control, which this patch of course doesn't to, it just fixes the old one. ;)
- smegel 11y ago
  I thought that too, but then > that an endpoint that is not moving any traffic cannot use the lack of errors as information in its feedback loop. I think the bug is that it was counting something when it should have not been, which would have violated both the design and intent even at the time it was written. But I agree that code is only a bug if it does not do what was originally intended, if the original design was bad, and the code correctly implemented that bad design, there is no bug.
thecaviardoer 11y ago
Honest question: How is it that Google is the company that's always doing all of this involved technical work? Or is it just that their work gets publicity?
- adventured 11y ago
  They have tens of thousands of engineers and work with vast amounts of open source software. Only a few other companies are of comparable scale in terms of sheer software engineering. I think it's inevitable that they'd have a very large footprint across the industry.
- vezzy-fnord 11y ago
  At least in the case of the Linux kernel, Google is actually around the 9th or 10th largest corporate contributor. [1] I'm not sure if they deserve to be particularly specially exalted, given the vast body of kernel work unrelated to them. [1] http://www.linuxfoundation.org/publications/linux-foundation/who-writes-linux-2015 http://www.linuxfoundation.org/publications/linux-foundation...
  - tga_d 11y ago
    Looking at the list here: [1] Most of the top employers are hardware companies, and therefore likely to contribute mostly driver patches. For example, the rankings by LoC for this release list AMD as over a third of the contributions, almost certainly because of their new AMDGPU driver. Drivers are important, of course, but hardly news-worthy. Red Hat is always near the top of the list, and they do a lot of both the drudge work and the interesting stuff, but I feel as though everyone in the Linux community already knows that. [1] https://lwn.net/Articles/654633/ https://lwn.net/Articles/654633/
    - digi_owl 11y ago
      I really do wonder what kind of numbers we would get if we did the same on a number of other large projects. I keep having the feeling that RH have a massive amount of say in the Linux ecosystem, for better or worse...
nathanb 11y ago
I would love to see a characterization of this bug's impact. Reading the description, it doesn't seem to be nearly as serious as the author of this post is making it out to be. But there's no information about how fast the client will step up the transmit rate after a period of quiescence and no data from traffic analysis to determine how frequently real-world servers encounter this condition and the overall effect in terms of dropped packets, increased queue depths, and latency spikes. (Google are actually in a good position to provide these data; a number of Google's own services will doubtless benefit. I suspect the scale of Google's operations means that even a barely-measurable decrease in latency or queue depth compounds to a big gain pretty quickly.)
- cornholio 11y ago
  Strictly speaking, it can't be too bad, since it would immediately stand out if Linux boxes have decreased network performance in a mixed shop. Maybe a few percent in the average case, with maybe some specialized applications more affected.
toolslive 11y ago
There is a talk https://www.youtube.com/watch?v=gQsOo_skjzk https://www.youtube.com/watch?v=gQsOo_skjzk Where they show there are plenty of similar bugs left in TCP implementations. It's a mess.
- mtkd 11y ago
  But less of a mess today than it was 17 days ago.
Supersaiyan_IV 11y ago
The reason this is good is because older wifi cards supported by iwlwifi get hardware buffer underruns and hangs when said bug occurs together with WPA2 enabled.
- tjoff 11y ago
  I'd say that is a reason for why this is bad. If I've got bad hardware I want to know about it and bad hardware should be flushed from the market. Not function "okaish" enough to not annoy users to the point of boycott.
  - Supersaiyan_IV 11y ago
    Allow me to clarify. "This" means: "The fact that Google fixed a decade old bug", meaning the topic itself. The "this" you use in your comment is entirely different. The above misuse of "this" causes me to be confused, because it allows your comment to be interpreted as if you're promoting the inference of bugs, where older hardware is affected, such that people lose faith in that hardware faster due to frustration, consequently affecting the market causing it relieve itself of older hardware faster. Which is why I'm asking you to clarify your argument.
    - tjoff 11y ago
      I only commented on the aspect of this bug that benefits crappy hardware. That is bad. Because your comment seemed to imply that the benefits for crappy hardware is the only reason for why this is good. Which, depending on your viewpoint, can be seen as: the only reason this bug fix is good is because some company can still sell crappy hardware and get away with it. Then we have the user perspective of course, which is "hey, my device doesn't crash as often" - which is awesome. But given the context the hardware is still buggy and it will still crash, albeit not as often. Which really isn't much comfort in the long run. Rare hiccups are worse than daily hiccups. Because daily hiccups you learn how to handle, a rare hiccup can really bite you. So you really haven't gained that much anyway, and in the long run the rare hiccup is rare enough for you not knowing who to attribute it to so you might buy the exact same thing next time which of course is very, very, unfortunate.
zurn 11y ago
The commit also says "This particularly shows up when slow_start_after_idle is disabled as a dangerous cwnd inflation (1.5 x RTT) after few seconds of idle time." And the slow_start_after_idle sysctl is enabled by default. The bug fix is in the CUBIC congestion control algorithm, which has been the default in Linux TCP since kernel 2.6.19 (released in November 2007). So impact in distros has been ~ 2008-2015.
zaroth 11y ago
Speaking of Linux kernel network stack enhancements, there's one I'm interested in but not sure where to start. I was trying to setup ECMP on Ubuntu 14 and found I was constantly getting TCP RST. Turns out after digging into the code I found the path selection algorithm is not a consistent hash of a packet header tuple but rather pseudo-randomly chosen. In older kernels there was a route cache so a path would be chosen and cached and so you would get a stable ECMP route per source. The route cache was removed so now the random selection runs on every packet leading to unstable routing. It seems a simple fix of choosing the right hash function and maybe adding some configuration flags to determine which fields are hashes on (typically 5-tuple SrcIP, DstIP, Ip Protocol, TCP Src Port, TCP Dst Port). But what would be incredibly cool is if my patch could one day possibly even merge to mainline. Is it reasonable to just blast a 'here's what I'm trying' out to netdev mailing list to get some feedback? I must admit I'm a bit intimidated to post...
- jfoks 11y ago
  Absolutely go for it! Post it even if you feel uncertain about it, just be clear about that, and about what your patch is trying to achieve, when the unexpected behaviour happens, and try to include a way for others to reproduce/investigate (as simple as possible, perhaps a program that demonstrates the bug). Also be clear about how you feel about whether or not the patch is the right approach and about what you would like list members to do with it (are you looking to confirm that what you're seeing is a real kernel bug, or a misunderstanding, or an application bug, and/or are you looking for help solving the bug/issue you're seeing, and/or are you looking to get it merged, etc). Can you demonstrate/quantify how your patch improves/fixes things, etc.
- Obi_Juan_Kenobi 11y ago
  Open source is generally quite kind to newbies, provided you provide that important context. Try to explain not just what you're doing, but also be very clear about why. Explain the problem as well as you can, and then discuss what solutions you've considered thus far. It sounds like you have a good understanding of the problem. If anything, I'd look into why the route cache was removed; perhaps there was a good reason for this? If nothing else, you can ask and learn something.

355 points by tim_sw 11y ago

I'm confused by the last sentence: > The whole web, including Firefox users, will benefit. Why is Firefox thrown in here? And the post is even tagged with "firefox". But the contents of the post seems to have absolutely nothing to do with Firefox, except inasmuch as Firefox is an application that interacts with the web.

TimothyFitz 11y ago
The author works for mozilla, and firefox is a common theme on his blog. I think it's for regular readers of his blog to understand the practical impact (readers he assumes care more than average about firefox).
stuaxo 11y ago
I'd guess that because it was a bugfix by google - people might assume it was something to do with chrome or android, when it should help any webuser that talks to a linux based server, including firefox users.
Manishearth 11y ago
The tag is probably so that the post will show up in Planet Mozilla (http://planet.mozilla.org http://planet.mozilla.org). I often tag non-Mozilla posts as "mozilla" if they are still relevant to Planet's scope on my blog too. I've submitted a category feed instead of the full RSS blog feed so that I can exclude things which are physicsy or otherwise irrelevant.

userbinator 11y ago

Remember that TCP is robust enough that it seems to work anyhow - even at the cost of reduced network throughput in this case. That to me sounds like this is not really a bug, but a performance enhancement; a bug would be something like corrupted/missing/extra data at the application level. Tuning the TCP algorithms beyond the basics is really more of an art.

wfunction 11y ago
It's a bug if you consider performance to be a feature.
plorg 11y ago
Presumably it is considered to be a bug because it does not accomplish what the code is meant to accomplish - namely, the code does not properly characterize a congested state and thus does not operate as if it were congested.
- nathanb 11y ago
  Given your definition, and viewed from a high enough level, every feature or enhancement is a bug fix. I think nitpicking the wording (while an interesting philosophical discussion) detracts from the point of the article: an unnamed Google engineer waded into the Linux kernel code and made the web slightly better for many people.
  - eru 11y ago
    I think it was actually multiple people.
  - nuxi7 11y ago
    You've hit on a fine point I often make about how the term "security bug" is used. I would call this a bug fix since someone has found what is probably an unintended behavior in the cubic algorithm. The cubic algorithm itself though would be a feature improving on the original tcp algorithm (nagle, if I remember right)
  - merb 11y ago
    No a feature or enhancement mostly introduces something new, like a new algorithm for cubic congestion control, which this patch of course doesn't to, it just fixes the old one. ;)
smegel 11y ago
I thought that too, but then > that an endpoint that is not moving any traffic cannot use the lack of errors as information in its feedback loop. I think the bug is that it was counting something when it should have not been, which would have violated both the design and intent even at the time it was written. But I agree that code is only a bug if it does not do what was originally intended, if the original design was bad, and the code correctly implemented that bad design, there is no bug.

thecaviardoer 11y ago

Honest question: How is it that Google is the company that's always doing all of this involved technical work? Or is it just that their work gets publicity?

adventured 11y ago
They have tens of thousands of engineers and work with vast amounts of open source software. Only a few other companies are of comparable scale in terms of sheer software engineering. I think it's inevitable that they'd have a very large footprint across the industry.
vezzy-fnord 11y ago
At least in the case of the Linux kernel, Google is actually around the 9th or 10th largest corporate contributor. [1] I'm not sure if they deserve to be particularly specially exalted, given the vast body of kernel work unrelated to them. [1] http://www.linuxfoundation.org/publications/linux-foundation/who-writes-linux-2015 http://www.linuxfoundation.org/publications/linux-foundation...
- tga_d 11y ago
  Looking at the list here: [1] Most of the top employers are hardware companies, and therefore likely to contribute mostly driver patches. For example, the rankings by LoC for this release list AMD as over a third of the contributions, almost certainly because of their new AMDGPU driver. Drivers are important, of course, but hardly news-worthy. Red Hat is always near the top of the list, and they do a lot of both the drudge work and the interesting stuff, but I feel as though everyone in the Linux community already knows that. [1] https://lwn.net/Articles/654633/ https://lwn.net/Articles/654633/
  - digi_owl 11y ago
    I really do wonder what kind of numbers we would get if we did the same on a number of other large projects. I keep having the feeling that RH have a massive amount of say in the Linux ecosystem, for better or worse...

nathanb 11y ago

I would love to see a characterization of this bug's impact. Reading the description, it doesn't seem to be nearly as serious as the author of this post is making it out to be. But there's no information about how fast the client will step up the transmit rate after a period of quiescence and no data from traffic analysis to determine how frequently real-world servers encounter this condition and the overall effect in terms of dropped packets, increased queue depths, and latency spikes. (Google are actually in a good position to provide these data; a number of Google's own services will doubtless benefit. I suspect the scale of Google's operations means that even a barely-measurable decrease in latency or queue depth compounds to a big gain pretty quickly.)

cornholio 11y ago
Strictly speaking, it can't be too bad, since it would immediately stand out if Linux boxes have decreased network performance in a mixed shop. Maybe a few percent in the average case, with maybe some specialized applications more affected.

toolslive 11y ago

There is a talk https://www.youtube.com/watch?v=gQsOo_skjzk https://www.youtube.com/watch?v=gQsOo_skjzk Where they show there are plenty of similar bugs left in TCP implementations. It's a mess.

mtkd 11y ago
But less of a mess today than it was 17 days ago.

Supersaiyan_IV 11y ago

The reason this is good is because older wifi cards supported by iwlwifi get hardware buffer underruns and hangs when said bug occurs together with WPA2 enabled.

tjoff 11y ago
I'd say that is a reason for why this is bad. If I've got bad hardware I want to know about it and bad hardware should be flushed from the market. Not function "okaish" enough to not annoy users to the point of boycott.
- Supersaiyan_IV 11y ago
  Allow me to clarify. "This" means: "The fact that Google fixed a decade old bug", meaning the topic itself. The "this" you use in your comment is entirely different. The above misuse of "this" causes me to be confused, because it allows your comment to be interpreted as if you're promoting the inference of bugs, where older hardware is affected, such that people lose faith in that hardware faster due to frustration, consequently affecting the market causing it relieve itself of older hardware faster. Which is why I'm asking you to clarify your argument.
  - tjoff 11y ago
    I only commented on the aspect of this bug that benefits crappy hardware. That is bad. Because your comment seemed to imply that the benefits for crappy hardware is the only reason for why this is good. Which, depending on your viewpoint, can be seen as: the only reason this bug fix is good is because some company can still sell crappy hardware and get away with it. Then we have the user perspective of course, which is "hey, my device doesn't crash as often" - which is awesome. But given the context the hardware is still buggy and it will still crash, albeit not as often. Which really isn't much comfort in the long run. Rare hiccups are worse than daily hiccups. Because daily hiccups you learn how to handle, a rare hiccup can really bite you. So you really haven't gained that much anyway, and in the long run the rare hiccup is rare enough for you not knowing who to attribute it to so you might buy the exact same thing next time which of course is very, very, unfortunate.

zurn 11y ago

The commit also says "This particularly shows up when slow_start_after_idle is disabled as a dangerous cwnd inflation (1.5 x RTT) after few seconds of idle time." And the slow_start_after_idle sysctl is enabled by default. The bug fix is in the CUBIC congestion control algorithm, which has been the default in Linux TCP since kernel 2.6.19 (released in November 2007). So impact in distros has been ~ 2008-2015.

zaroth 11y ago

Speaking of Linux kernel network stack enhancements, there's one I'm interested in but not sure where to start. I was trying to setup ECMP on Ubuntu 14 and found I was constantly getting TCP RST. Turns out after digging into the code I found the path selection algorithm is not a consistent hash of a packet header tuple but rather pseudo-randomly chosen. In older kernels there was a route cache so a path would be chosen and cached and so you would get a stable ECMP route per source. The route cache was removed so now the random selection runs on every packet leading to unstable routing. It seems a simple fix of choosing the right hash function and maybe adding some configuration flags to determine which fields are hashes on (typically 5-tuple SrcIP, DstIP, Ip Protocol, TCP Src Port, TCP Dst Port). But what would be incredibly cool is if my patch could one day possibly even merge to mainline. Is it reasonable to just blast a 'here's what I'm trying' out to netdev mailing list to get some feedback? I must admit I'm a bit intimidated to post...

jfoks 11y ago
Absolutely go for it! Post it even if you feel uncertain about it, just be clear about that, and about what your patch is trying to achieve, when the unexpected behaviour happens, and try to include a way for others to reproduce/investigate (as simple as possible, perhaps a program that demonstrates the bug). Also be clear about how you feel about whether or not the patch is the right approach and about what you would like list members to do with it (are you looking to confirm that what you're seeing is a real kernel bug, or a misunderstanding, or an application bug, and/or are you looking for help solving the bug/issue you're seeing, and/or are you looking to get it merged, etc). Can you demonstrate/quantify how your patch improves/fixes things, etc.
Obi_Juan_Kenobi 11y ago
Open source is generally quite kind to newbies, provided you provide that important context. Try to explain not just what you're doing, but also be very clear about why. Explain the problem as well as you can, and then discuss what solutions you've considered thus far. It sounds like you have a good understanding of the problem. If anything, I'd look into why the route cache was removed; perhaps there was a good reason for this? If nothing else, you can ask and learn something.