9 ms·

Remotely send Chrome and Node.js into infinite loops via OS X kernel bug

334 points by kentonv 12y ago

Dylan16807 12y ago
I wonder if OOB data is ever going to be fixed to allow more than one byte. I can't quite comprehend how the implementation got so screwed up in the first place. A two byte field in every header, supposed to be a pointer to designate a range of data, OBVIOUSLY when it's set it should transmit exactly one byte of information.
- duskwuff 12y ago
  Unlikely. Almost nothing uses OOB data (and it's unclear what it's even useful for!), so there's no demand to fix it. More likely, I suspect, would be a gradual move to deprecate OOB data. It's a poorly designed wart on TCP.
  - toast0 12y ago
    While I agree it is likely to be deprecated (or already is), I think being able to signal that side data which was transmitted later should be processed earlier has value in situations where processing delay is more significant than transmission delay. You could do this with a single stream, but the receiver would have to read ahead and check for flags etc. Had early implementations got this consistently usable, it might see more use.
  - ketralnis 12y ago
    I thought that SSH used it to pass ^C up faster than the rest of the stream so that you can quickly kill programs that flood the terminal. I've never verified this though.
    - derefr 12y ago
      SSH uses the URG flag, IIRC.
      - caf 12y ago
        The TCP URG flag and the OOB data described here are the same thing.
    - MBCook 12y ago
      The article says Telnet works that way, it seems reasonable to think SSH might use it for the same purpose.
      - deleted 12y ago
        [deleted]
      - 0x0 12y ago
        But the ^C should be encrypted. If SSH sets the URG flag on a packet containing an encrypted ^C, it would be leaking plain-text - even for just one byte :)
        krakensden 12y ago
        Not just encrypted, authenticated- it would be bad if a MITM could send ^Cs out to fuck with you at any time.
    - ambrop7 12y ago
      I don't know how it actually works, but I can't see how this would be needed in that situation. The ^C signal goes from client to server, while the program is flooding in the opposite direction. So the flood of data shouldn't delay the ^C transmission.
      - kentonv 12y ago
        The real problem is not with programs flooding the terminal, but rather that if the foreground program isn't accepting input (perfectly normal for many programs) then by design the socket buffer will fill up, TCP flow control will kick in, and the client machine will stop even sending the bytes to the server. In order for a ctrl+C to get around the backed up buffer, it has to bypass the regular queue.
  - burke 12y ago
    It's necessary to share file handles between processes using TCP over UNIX Domain Sockets. I'm sure some other mechanism could be devised for that though.
    - phs2501 12y ago
      Minor nit: While Unix Domain Sockets (AF_UNIX) can send open file descriptors between processes, and it uses the msg_control field of sendmsg's struct msghdr ("ancillary data"), that's not the same as TCP's OOB data and in fact TCP is not involved at all in unix sockets, even when used as SOCK_STREAM. sendmsg() even has a MSG_OOB flag that is used to trigger TCP's OOB mechanism that (as far as I can tell) doesn't use the msg_control field at all, as the same option is available for plain send() and sendto(); the OOB data should be sent through the normal msg_iov field. Some places where msg_control is used for TCP/IP sockets (at least in Linux) include kernel timestamping of messages. This is "out of band" data, but it is out of band to the kernel, not the network peer. It's pretty confusing.
      - burke 12y ago
        Weird. I guess I built a mental model that felt like it made sense. Turns out computers are crazier than common sense. Thanks for the detailed reply :)
  - m_eiman 12y ago
    I use OOB to force an early ACK to speed up communication with a request-reply style protocol.
- syncsynchalt 12y ago
  Well, there's no reason to fix it unless someone has a compelling use case. And there's never going to be a compelling use case built around it if it's not fixed. And regarding the fix, which do you propose, change the spec or change the majority of implementations? It may offend the sensibilities but we're stuck with the current state of affairs for the life of TCP I think.
vbcr 12y ago
The last statement reminds me of Linus saying we do not break user space If an application is utilizing a bug, it is not a bug but a feature
- MBCook 12y ago
  Yeah I was kind of wondering what his take on something like this would be. On the one hand you're changing an API, which is a promise to userspace. On the other hand, no one was using it right leading to most networking apps being vulnerable to a DOS. Given the proven risk and how little the feature is used, changing the API to match what people THINK it does seems sane. I have a vague memory or Linus making a similar 'change it' decision once.
  - UnoriginalGuy 12y ago
    Also according to this article it was either undocumented or extremely poorly documented depending on which doc's you look at. So maybe they're removing a feature a lot of developers either didn't know about or didn't know they COULD utilise.
  - aidenn0 12y ago
    If no examples could be found of someone relying on the behavior, then a change would be okay. Otherwise the solution would be to implement a modified API. Alternatively glibc could change it, as glibc almost seems to want to break existing programs.
- mkj 12y ago
  Apple's general approach is "if it isn't documented, we can break it", a bit different to Linux.
  - Aloha 12y ago
    I actually think it's much the same, judging from the quote from linus in the parent.
  - joakleaf 12y ago
adamnemecek 12y ago
> Linux has epoll. BSD has kqueue. Windows has... well, about five different mechanisms that cover differing subsets of usecases and you can only choose one. Aren't I/O completion ports fairly analogous to kqueue?
- kentonv 12y ago
  Only if the only event type you care about is stream I/O. If you also need to handle GUI events or wait on mutex handles or whatnot, you get to use MsgWaitForMultipleObjectsEx, which is as ugly as it sounds.
  - adamnemecek 12y ago
    You said that this was in the context of async I/O.
    - kentonv 12y ago
      UI events are I/O...
      - adamnemecek 12y ago
        You are not talking about UI events in the blog post. You are saying that there isn't a unified Windows API similar to kqueue or epoll that can be used for async file/stream I/O with which I disagree. AFAIK you can't use epoll to do UI events either.
        kentonv 12y ago
        X events are delivered via a socket. You can absolutely epoll that, and people in fact do so, and this is considered an essential feature.
        jmgao 12y ago
        One of the best features of all of the epoll/poll/select functions is that they let you listen on arbitrary file descriptors uniformly. You can send events over a unix domain socket or a pipe, and just treat it as if it were a regular file. It's especially useful with stuff like signalfd or timerfd, where you can listen on sockets, handle signals, and have a timeout, with barely any additional code.
maqr 12y ago
Anybody remember this gem? http://en.wikipedia.org/wiki/WinNuke http://en.wikipedia.org/wiki/WinNuke
- Throwaway90283 12y ago
  I think my favorite is the bug where you could delete any folder from a user's computer, just by linking them to a site with a few lines of code inserted in the source. I think it was on XP, and it would trigger Windows help to open at the same time.
  - euyyn 12y ago
    Yeah, you'd trigger a very specific help:// URL that happened to accept as a query parameter the path to a folder. That folder was created by the help page, which would put some temporary stuff in there and, when the user closed the help window, would remove the folder and its contents (because you have to clean after yourself!). The obvious thought is... "what if I point it to a folder that already exists?" Yep. It was definitely there before XP, at least since 2002.
    - przemoc 12y ago
      XP was RTMed on August 24, 2001 and GA was October 25, 2001. But I can understand why you wrote what you wrote. Early XP experience for many remains repressed memory, because it was quite buggy (be it OS itself or drivers delivered with it, BSODs were a norm), far from stable 2000 SP4, which I kept using for a long time. XP around SP2 (August 25, 2004) got usable.
    - yuhong 12y ago
      AFAIK it was fixed in XP SP1 and after user backlash they also released a patch for RTM.
- eli 12y ago
  I remember patching it and then scripting mirc to seek out the handle of the person who sent it via the source IP.
wmf 12y ago
I'm kind of disappointed that this bug doesn't have branding. :-)
- thomasjudge 12y ago
  "oob-killer?" "oob-o'-death?"
  - deleted 12y ago
    [deleted]
  - xorcist 12y ago
    Both the other brands have been clever and played on what's being exploited. (Heartbleed is bleeding data thorugh heartbeat, for example.) How about "reURGitate"? It plays on URG, and regurgitate is something you can do forever.
tinco 12y ago
I was a bit confused about at what layer this OOB byte would be sent, I think this article cleared it up for me[0]. TCP has a feature called TCP Urgent Data which can be enabled by flipping a bit in the TCP header. BSD maps this to the OOB feature as described. This means that if your Node.JS app was behind a http proxy that's not vulnerable, like nginx+passenger, you would've been safe. Just a TCP pass-through would leave you vulnerable though. All theory of course, since no one is really running a Node.JS server on OSX.. right? 0] http://www.serverframework.com/asynchronousevents/2011/10/out-of-band-data-and-overlapped-io.html http://www.serverframework.com/asynchronousevents/2011/10/ou...
- nostrademons 12y ago
  It's a bigger problem for Chrome. Imagine if you ran an ad on a major ad network that loaded a banner from your server that sent a byte as TCP urgent. It would lock up any Mac who visited a significant fraction of the Internet. (Incidentally, a bug like this was what got me to switch from Netscape to Internet Explorer back in 2000. Doubleclick ran an ad that included some Javascript which locked up Netscape and hung the browser entirely, blocking roughly 40% of the Internet for me. At that point I was like "This is ridiculous, fix your damn software Netscape" and switched.)
saool 12y ago
Such a twisted forewarning about the importance of net neutrality...
caf 12y ago
I think this would apply to a few common ircds running on affected versions of OS X too.
shakeel_mohamed 12y ago
I wonder if this is related to a wacky clipboard bug I hit today while VMWare Fusion was open. Basically I copied & pasted a file within a VM, then copied some text from Chrome on OSX and Chrome froze instantly.
deleted 12y ago
[deleted]
krishicks 12y ago
They seem to have only fixed this on Yosemite. I can reproduce the bug on Mavericks and Chrome 41.0.2272.118 after the 2014-004 update for Mavericks.
- brazzledazzle 12y ago
  If true that strikes me as kind of shitty. I know Apple doesn't care much for maintaining old versions and backwards compatibility but Mavericks isn't all that old.
  - kentonv 12y ago
    Ehhh. Maybe. Keep in mind that this bug is probably one of the least serious of the dozens of bugs patched today, as there's very little damage you can really do with it. Hardly anyone runs OSX servers, so you're not going to take down any major web sites with this. You can crash someone's browser, but that's actually already pretty easy to do with some javascript that eats a bunch of resources. Chrome doesn't even consider DoS to be a security issue because there's just nothing they'll ever be able to do about it anyhow. If the bug in any way threatened data integrity or confidentiality, then yeah, they should backport it. But for a DoS, I can see the case for not really caring. FWIW, for many of the bugs patched today, Apple did in fact backport to Mavericks and even Mountain Lion, so it seems like they haven't completely abandoned old versions.
    - 72deluxe 12y ago
      Judging from the many tabs I have open, it seems that you don't need an exploit to eat resources in Chrome! That thing eats my battery and loves CPU feasts without any prompting. (Disabling Flash helped)
    - userbinator 12y ago
      You can crash someone's browser, but that's actually already pretty easy to do with some javascript that eats a bunch of resources. Chrome doesn't even consider DoS to be a security issue because there's just nothing they'll ever be able to do about it anyhow They could implement resource limits and stop consuming more resources once those limits are reached.
danatkinson 12y ago
As an FYI, the Chromium bug has now been unrestricted for public viewing - https://code.google.com/p/chromium/issues/detail?id=437642 https://code.google.com/p/chromium/issues/detail?id=437642
chrisdotcode 12y ago
I think the conclusion warrants repeating: "The moral of the story? Confusing APIs are a security problem. If many users of your API get it wrong in a way that introduces a security bug, that’s a bug in your API, not their code."

334 points by kentonv 12y ago

I wonder if OOB data is ever going to be fixed to allow more than one byte. I can't quite comprehend how the implementation got so screwed up in the first place. A two byte field in every header, supposed to be a pointer to designate a range of data, OBVIOUSLY when it's set it should transmit exactly one byte of information.

duskwuff 12y ago
Unlikely. Almost nothing uses OOB data (and it's unclear what it's even useful for!), so there's no demand to fix it. More likely, I suspect, would be a gradual move to deprecate OOB data. It's a poorly designed wart on TCP.
- toast0 12y ago
  While I agree it is likely to be deprecated (or already is), I think being able to signal that side data which was transmitted later should be processed earlier has value in situations where processing delay is more significant than transmission delay. You could do this with a single stream, but the receiver would have to read ahead and check for flags etc. Had early implementations got this consistently usable, it might see more use.
- ketralnis 12y ago
  I thought that SSH used it to pass ^C up faster than the rest of the stream so that you can quickly kill programs that flood the terminal. I've never verified this though.
  - derefr 12y ago
    SSH uses the URG flag, IIRC.
    - caf 12y ago
      The TCP URG flag and the OOB data described here are the same thing.
  - MBCook 12y ago
    The article says Telnet works that way, it seems reasonable to think SSH might use it for the same purpose.
    - deleted 12y ago
      [deleted]
    - 0x0 12y ago
      But the ^C should be encrypted. If SSH sets the URG flag on a packet containing an encrypted ^C, it would be leaking plain-text - even for just one byte :)
      - krakensden 12y ago
        Not just encrypted, authenticated- it would be bad if a MITM could send ^Cs out to fuck with you at any time.
  - ambrop7 12y ago
    I don't know how it actually works, but I can't see how this would be needed in that situation. The ^C signal goes from client to server, while the program is flooding in the opposite direction. So the flood of data shouldn't delay the ^C transmission.
    - kentonv 12y ago
      The real problem is not with programs flooding the terminal, but rather that if the foreground program isn't accepting input (perfectly normal for many programs) then by design the socket buffer will fill up, TCP flow control will kick in, and the client machine will stop even sending the bytes to the server. In order for a ctrl+C to get around the backed up buffer, it has to bypass the regular queue.
- burke 12y ago
  It's necessary to share file handles between processes using TCP over UNIX Domain Sockets. I'm sure some other mechanism could be devised for that though.
  - phs2501 12y ago
    Minor nit: While Unix Domain Sockets (AF_UNIX) can send open file descriptors between processes, and it uses the msg_control field of sendmsg's struct msghdr ("ancillary data"), that's not the same as TCP's OOB data and in fact TCP is not involved at all in unix sockets, even when used as SOCK_STREAM. sendmsg() even has a MSG_OOB flag that is used to trigger TCP's OOB mechanism that (as far as I can tell) doesn't use the msg_control field at all, as the same option is available for plain send() and sendto(); the OOB data should be sent through the normal msg_iov field. Some places where msg_control is used for TCP/IP sockets (at least in Linux) include kernel timestamping of messages. This is "out of band" data, but it is out of band to the kernel, not the network peer. It's pretty confusing.
    - burke 12y ago
      Weird. I guess I built a mental model that felt like it made sense. Turns out computers are crazier than common sense. Thanks for the detailed reply :)
- m_eiman 12y ago
  I use OOB to force an early ACK to speed up communication with a request-reply style protocol.
syncsynchalt 12y ago
Well, there's no reason to fix it unless someone has a compelling use case. And there's never going to be a compelling use case built around it if it's not fixed. And regarding the fix, which do you propose, change the spec or change the majority of implementations? It may offend the sensibilities but we're stuck with the current state of affairs for the life of TCP I think.

vbcr 12y ago

The last statement reminds me of Linus saying we do not break user space If an application is utilizing a bug, it is not a bug but a feature

MBCook 12y ago
Yeah I was kind of wondering what his take on something like this would be. On the one hand you're changing an API, which is a promise to userspace. On the other hand, no one was using it right leading to most networking apps being vulnerable to a DOS. Given the proven risk and how little the feature is used, changing the API to match what people THINK it does seems sane. I have a vague memory or Linus making a similar 'change it' decision once.
- UnoriginalGuy 12y ago
  Also according to this article it was either undocumented or extremely poorly documented depending on which doc's you look at. So maybe they're removing a feature a lot of developers either didn't know about or didn't know they COULD utilise.
- aidenn0 12y ago
  If no examples could be found of someone relying on the behavior, then a change would be okay. Otherwise the solution would be to implement a modified API. Alternatively glibc could change it, as glibc almost seems to want to break existing programs.
mkj 12y ago
Apple's general approach is "if it isn't documented, we can break it", a bit different to Linux.
- Aloha 12y ago
  I actually think it's much the same, judging from the quote from linus in the parent.
- joakleaf 12y ago

adamnemecek 12y ago

> Linux has epoll. BSD has kqueue. Windows has... well, about five different mechanisms that cover differing subsets of usecases and you can only choose one. Aren't I/O completion ports fairly analogous to kqueue?

kentonv 12y ago
Only if the only event type you care about is stream I/O. If you also need to handle GUI events or wait on mutex handles or whatnot, you get to use MsgWaitForMultipleObjectsEx, which is as ugly as it sounds.
- adamnemecek 12y ago
  You said that this was in the context of async I/O.
  - kentonv 12y ago
    UI events are I/O...
    - adamnemecek 12y ago
      You are not talking about UI events in the blog post. You are saying that there isn't a unified Windows API similar to kqueue or epoll that can be used for async file/stream I/O with which I disagree. AFAIK you can't use epoll to do UI events either.
      - kentonv 12y ago
        X events are delivered via a socket. You can absolutely epoll that, and people in fact do so, and this is considered an essential feature.
      - jmgao 12y ago
        One of the best features of all of the epoll/poll/select functions is that they let you listen on arbitrary file descriptors uniformly. You can send events over a unix domain socket or a pipe, and just treat it as if it were a regular file. It's especially useful with stuff like signalfd or timerfd, where you can listen on sockets, handle signals, and have a timeout, with barely any additional code.

maqr 12y ago

Anybody remember this gem? http://en.wikipedia.org/wiki/WinNuke http://en.wikipedia.org/wiki/WinNuke

Throwaway90283 12y ago
I think my favorite is the bug where you could delete any folder from a user's computer, just by linking them to a site with a few lines of code inserted in the source. I think it was on XP, and it would trigger Windows help to open at the same time.
- euyyn 12y ago
  Yeah, you'd trigger a very specific help:// URL that happened to accept as a query parameter the path to a folder. That folder was created by the help page, which would put some temporary stuff in there and, when the user closed the help window, would remove the folder and its contents (because you have to clean after yourself!). The obvious thought is... "what if I point it to a folder that already exists?" Yep. It was definitely there before XP, at least since 2002.
  - przemoc 12y ago
    XP was RTMed on August 24, 2001 and GA was October 25, 2001. But I can understand why you wrote what you wrote. Early XP experience for many remains repressed memory, because it was quite buggy (be it OS itself or drivers delivered with it, BSODs were a norm), far from stable 2000 SP4, which I kept using for a long time. XP around SP2 (August 25, 2004) got usable.
  - yuhong 12y ago
    AFAIK it was fixed in XP SP1 and after user backlash they also released a patch for RTM.
eli 12y ago
I remember patching it and then scripting mirc to seek out the handle of the person who sent it via the source IP.

wmf 12y ago

I'm kind of disappointed that this bug doesn't have branding. :-)

thomasjudge 12y ago
"oob-killer?" "oob-o'-death?"
- deleted 12y ago
  [deleted]
- xorcist 12y ago
  Both the other brands have been clever and played on what's being exploited. (Heartbleed is bleeding data thorugh heartbeat, for example.) How about "reURGitate"? It plays on URG, and regurgitate is something you can do forever.

tinco 12y ago

I was a bit confused about at what layer this OOB byte would be sent, I think this article cleared it up for me[0]. TCP has a feature called TCP Urgent Data which can be enabled by flipping a bit in the TCP header. BSD maps this to the OOB feature as described. This means that if your Node.JS app was behind a http proxy that's not vulnerable, like nginx+passenger, you would've been safe. Just a TCP pass-through would leave you vulnerable though. All theory of course, since no one is really running a Node.JS server on OSX.. right? 0] http://www.serverframework.com/asynchronousevents/2011/10/out-of-band-data-and-overlapped-io.html http://www.serverframework.com/asynchronousevents/2011/10/ou...

nostrademons 12y ago
It's a bigger problem for Chrome. Imagine if you ran an ad on a major ad network that loaded a banner from your server that sent a byte as TCP urgent. It would lock up any Mac who visited a significant fraction of the Internet. (Incidentally, a bug like this was what got me to switch from Netscape to Internet Explorer back in 2000. Doubleclick ran an ad that included some Javascript which locked up Netscape and hung the browser entirely, blocking roughly 40% of the Internet for me. At that point I was like "This is ridiculous, fix your damn software Netscape" and switched.)

saool 12y ago

Such a twisted forewarning about the importance of net neutrality...

caf 12y ago

I think this would apply to a few common ircds running on affected versions of OS X too.

shakeel_mohamed 12y ago

I wonder if this is related to a wacky clipboard bug I hit today while VMWare Fusion was open. Basically I copied & pasted a file within a VM, then copied some text from Chrome on OSX and Chrome froze instantly.

deleted 12y ago

[deleted]

krishicks 12y ago

They seem to have only fixed this on Yosemite. I can reproduce the bug on Mavericks and Chrome 41.0.2272.118 after the 2014-004 update for Mavericks.

brazzledazzle 12y ago
If true that strikes me as kind of shitty. I know Apple doesn't care much for maintaining old versions and backwards compatibility but Mavericks isn't all that old.
- kentonv 12y ago
  Ehhh. Maybe. Keep in mind that this bug is probably one of the least serious of the dozens of bugs patched today, as there's very little damage you can really do with it. Hardly anyone runs OSX servers, so you're not going to take down any major web sites with this. You can crash someone's browser, but that's actually already pretty easy to do with some javascript that eats a bunch of resources. Chrome doesn't even consider DoS to be a security issue because there's just nothing they'll ever be able to do about it anyhow. If the bug in any way threatened data integrity or confidentiality, then yeah, they should backport it. But for a DoS, I can see the case for not really caring. FWIW, for many of the bugs patched today, Apple did in fact backport to Mavericks and even Mountain Lion, so it seems like they haven't completely abandoned old versions.
  - 72deluxe 12y ago
    Judging from the many tabs I have open, it seems that you don't need an exploit to eat resources in Chrome! That thing eats my battery and loves CPU feasts without any prompting. (Disabling Flash helped)
  - userbinator 12y ago
    You can crash someone's browser, but that's actually already pretty easy to do with some javascript that eats a bunch of resources. Chrome doesn't even consider DoS to be a security issue because there's just nothing they'll ever be able to do about it anyhow They could implement resource limits and stop consuming more resources once those limits are reached.

danatkinson 12y ago

As an FYI, the Chromium bug has now been unrestricted for public viewing - https://code.google.com/p/chromium/issues/detail?id=437642 https://code.google.com/p/chromium/issues/detail?id=437642

chrisdotcode 12y ago

I think the conclusion warrants repeating: "The moral of the story? Confusing APIs are a security problem. If many users of your API get it wrong in a way that introduces a security bug, that’s a bug in your API, not their code."