6 ms·

This vulnerability went from disclosure to exploit extremely quickly. The CVE was published on the 8th. I can't speak for them, but in the process of trying to

This vulnerability went from disclosure to exploit extremely quickly. The CVE was published on the 8th. I can't speak for them, but in the process of trying to update, we ran in to some issues with therubyracer (a core component of the Rails asset pipeline) and libv8 (the library that therubyracer uses to embed the V8 engine). It was extraordinarily bad timing, and it slowed down our update process by almost a day while waiting on a new version of therubyracer and libv8 to hit. Having said all that, if our upgrade holdup had taken any longer than it did, we would have implemented one of the mitigation strategies.

tptacek 14y ago
You cannot leave your site open to a pre-auth remote code execution vulnerability while you wait for fixes to the asset pipeline or to any other component of Rails. I don't know that that's what you're saying you did but we need to be glacier-blue-ice-clear about this. Nobody gets to wait on bugs like this. You patch or workaround immediately or, most probably, you shut your app down.
- jimktrains2 14y ago
  Weren't there patches for rails, and release for rails with the patches, really quickly from release?
  - tptacek 14y ago
    Yes, along with workarounds. I'm reacting to the inference that someone might wait to apply the patches because they broke the Asset Pipeline somehow.
- bradleyland 14y ago
  It slowed down the update (to 3.2.11), but it didn't prevent us from removing the XML parser from DEFAULT_PARSERS immediately. I'm not defending anyone here. I'm simply pointing out that the scenario wasn't exactly status quo. This has all moved very quickly. EDIT: I guess that qualifies as a mitigation strategy, but when I said that, I was talking more along the lines of the patches, or like another person I know, even more dramatic steps like forking Rails. There are regressions in the 3.2.x updates since 3.2.9 that affect some sites. Bottom line is that there was a lot of bad timing here that sucked up a lot of time in securing a Rails site.
  - tptacek 14y ago
    Right on. That was the right call to make. I wanted to be careful not to point a finger at you; it's just that this is exactly the kind of crazy mistake I can see a web startup making. Obviously, the timing sucked, but nobody had any control over that.
- bradleyland 14y ago
  Also, lest anyone think that they have time, there appear to be scanners up using the Metasploit code. This request came from an IP address on Road Runner's netblock from Ohio. The request hit a server that we don't publish, and the IP doesn't match any other requests in our logs. The greatest likelihood is that it is a computer locked up in a botnet. https://gist.github.com/4512579 https://gist.github.com/4512579 http://dev.metasploit.com/redmine/projects/framework/repository/entry/modules/auxiliary/scanner/http/rails_xml_yaml_scanner.rb http://dev.metasploit.com/redmine/projects/framework/reposit...
- moe 14y ago
  You also cannot publish little known pre-auth remote code execution vulnerabilities for your web-framework without first publishing a mitigation patch that doesn't paint a BIG FAT RED ARROW onto the attack vector. You also cannot leave 6 years of vulnerable Rails-versions up on rubygems.org without even backporting your patch (yes, they're still up there now). It makes no sense to blame the users of a web-framework, most of which are not security experts, for not responding instantly with the perfectly correct procedure to an event of this magnitude. Put the blame where it belongs, on the idiots who "handled" the issue by releasing a "recipe to exploit"-Advisory. The same idiots who ramble about "not breaking old apps" when asked why the vulnerable gems are still up on rubygems.org and continue to be quietly installed by any Gemfile referencing them...
  - danielweber 14y ago
    As a fan of responsible disclosure, the sad state of the universe is that telling people "upgrade to X now" on an open-source project makes it very easy for anyone to diff to find out what the fix is, which makes it very easy for people to figure out how to exploit it.
    - moe 14y ago
      No. A thousand times. A bit more creativity should be allowed when handling a vulnerability of this magnitude. For example, publish a patch that escapes all input in curious ways, presumably to prevent SQL injection. Pad it, obfuscate the actual fix with code-noise, make it annoying to read. Then release it as some handwavy, semi-plausible "follow-up" to the previous SQL-injection, urging everyone to upgrade in small caps. Yes, a handful of blackhats will see through the bluff. However, it will likely be the exact same blackhats that already had discovered the issue after the initial SQL injection advisory. This would have given the Rails-community a head-start, rather than unleashing every script-kiddy on the planet at once.
      - m0nastic 14y ago
        So you're advocating releasing a "decoy" patch that's intentionally obtuse and doesn't actually fix the issue? And then downplaying the severity by not encouraging developers to take the follow-up patch seriously? That's a flat out terrible idea.
pacala 14y ago
Perhaps this may sound harsh, but if you handle large amounts of money and are informed that your software stack allows arbitrary code execution then: You pull off the plug ASAP. Better have customers unable to use the service for one day than have the customers lose their money while you stumble trying to figure out how to patch it. This is elementary for any mission critical system. Just imagine your neighborhood nuclear plant delaying the insertion of the control rods during a meltdown because it has to wait for some shinier parts to arrive.
jrochkind1 14y ago
therubyracer thing was annoying, but there's no reason it had to delay your upgrades -- you could simply lock to the same version of therubyracer you were using previously without problems, it was just as compatible with the new version of rails. The fact remains though, that this exploit was _so_ severe, that, depending on how attractive a target you were and how disastrous it would be for you to be compromised (financial services? High on both scales) -- it would have been a better choice to _take your app down_ until you can fix it, rather than leave it up with the vulnerability. The vulnerability was "an attacker can run whatever code they want on your server, and they can discover that you are vulnerable by cheap automated port scan." It's as bad as it gets.
- bradleyland 14y ago
  I'm not saying that you shouldn't mitigate this vulnerability immediately (which is not the same as upgrading to 3.2.11). People are asking why/how this kind of thing happens, and I'm pointing out that there are a lot of opportunities to make the wrong decision in this situation.
kawsper 14y ago
Depends on what you do in your app. But if it was somewhat important, kill the application servers until you can be sure to have upgraded properly.

tptacek 14y ago

You cannot leave your site open to a pre-auth remote code execution vulnerability while you wait for fixes to the asset pipeline or to any other component of Rails. I don't know that that's what you're saying you did but we need to be glacier-blue-ice-clear about this. Nobody gets to wait on bugs like this. You patch or workaround immediately or, most probably, you shut your app down.

jimktrains2 14y ago
Weren't there patches for rails, and release for rails with the patches, really quickly from release?
- tptacek 14y ago
  Yes, along with workarounds. I'm reacting to the inference that someone might wait to apply the patches because they broke the Asset Pipeline somehow.
bradleyland 14y ago
It slowed down the update (to 3.2.11), but it didn't prevent us from removing the XML parser from DEFAULT_PARSERS immediately. I'm not defending anyone here. I'm simply pointing out that the scenario wasn't exactly status quo. This has all moved very quickly. EDIT: I guess that qualifies as a mitigation strategy, but when I said that, I was talking more along the lines of the patches, or like another person I know, even more dramatic steps like forking Rails. There are regressions in the 3.2.x updates since 3.2.9 that affect some sites. Bottom line is that there was a lot of bad timing here that sucked up a lot of time in securing a Rails site.
- tptacek 14y ago
  Right on. That was the right call to make. I wanted to be careful not to point a finger at you; it's just that this is exactly the kind of crazy mistake I can see a web startup making. Obviously, the timing sucked, but nobody had any control over that.
bradleyland 14y ago
Also, lest anyone think that they have time, there appear to be scanners up using the Metasploit code. This request came from an IP address on Road Runner's netblock from Ohio. The request hit a server that we don't publish, and the IP doesn't match any other requests in our logs. The greatest likelihood is that it is a computer locked up in a botnet. https://gist.github.com/4512579 https://gist.github.com/4512579 http://dev.metasploit.com/redmine/projects/framework/repository/entry/modules/auxiliary/scanner/http/rails_xml_yaml_scanner.rb http://dev.metasploit.com/redmine/projects/framework/reposit...
moe 14y ago
You also cannot publish little known pre-auth remote code execution vulnerabilities for your web-framework without first publishing a mitigation patch that doesn't paint a BIG FAT RED ARROW onto the attack vector. You also cannot leave 6 years of vulnerable Rails-versions up on rubygems.org without even backporting your patch (yes, they're still up there now). It makes no sense to blame the users of a web-framework, most of which are not security experts, for not responding instantly with the perfectly correct procedure to an event of this magnitude. Put the blame where it belongs, on the idiots who "handled" the issue by releasing a "recipe to exploit"-Advisory. The same idiots who ramble about "not breaking old apps" when asked why the vulnerable gems are still up on rubygems.org and continue to be quietly installed by any Gemfile referencing them...
- danielweber 14y ago
  As a fan of responsible disclosure, the sad state of the universe is that telling people "upgrade to X now" on an open-source project makes it very easy for anyone to diff to find out what the fix is, which makes it very easy for people to figure out how to exploit it.
  - moe 14y ago
    No. A thousand times. A bit more creativity should be allowed when handling a vulnerability of this magnitude. For example, publish a patch that escapes all input in curious ways, presumably to prevent SQL injection. Pad it, obfuscate the actual fix with code-noise, make it annoying to read. Then release it as some handwavy, semi-plausible "follow-up" to the previous SQL-injection, urging everyone to upgrade in small caps. Yes, a handful of blackhats will see through the bluff. However, it will likely be the exact same blackhats that already had discovered the issue after the initial SQL injection advisory. This would have given the Rails-community a head-start, rather than unleashing every script-kiddy on the planet at once.
    - m0nastic 14y ago
      So you're advocating releasing a "decoy" patch that's intentionally obtuse and doesn't actually fix the issue? And then downplaying the severity by not encouraging developers to take the follow-up patch seriously? That's a flat out terrible idea.

pacala 14y ago

Perhaps this may sound harsh, but if you handle large amounts of money and are informed that your software stack allows arbitrary code execution then: You pull off the plug ASAP. Better have customers unable to use the service for one day than have the customers lose their money while you stumble trying to figure out how to patch it. This is elementary for any mission critical system. Just imagine your neighborhood nuclear plant delaying the insertion of the control rods during a meltdown because it has to wait for some shinier parts to arrive.

jrochkind1 14y ago

therubyracer thing was annoying, but there's no reason it had to delay your upgrades -- you could simply lock to the same version of therubyracer you were using previously without problems, it was just as compatible with the new version of rails. The fact remains though, that this exploit was _so_ severe, that, depending on how attractive a target you were and how disastrous it would be for you to be compromised (financial services? High on both scales) -- it would have been a better choice to _take your app down_ until you can fix it, rather than leave it up with the vulnerability. The vulnerability was "an attacker can run whatever code they want on your server, and they can discover that you are vulnerable by cheap automated port scan." It's as bad as it gets.

bradleyland 14y ago
I'm not saying that you shouldn't mitigate this vulnerability immediately (which is not the same as upgrading to 3.2.11). People are asking why/how this kind of thing happens, and I'm pointing out that there are a lot of opportunities to make the wrong decision in this situation.

kawsper 14y ago

Depends on what you do in your app. But if it was somewhat important, kill the application servers until you can be sure to have upgraded properly.