7 ms·

Update Rails or not – security issues either way

42 points by kasparloog 14y ago

bradleyland 14y ago
Another option is to fork Rails and use the patch files (included with the CVE) that target only the vulnerabilities addressed in the CVE. The problem with upgrading to mitigate security issues is that the Rails team does not release security patches. They bump the minor-minor and do a release. That release almost always includes commits that are unrelated to the security issue. This is especially true when a lot of time passes between CVEs. Maintaining your own branch really isn't that difficult, because you can simply merge in from upstream. In most cases, you're really only interested in patching from Rails team issued security fixes, so you won't have conflicts. If you do have conflicts, you can safely overwrite anything in your fork, because you're not developing Rails, you're simply maintaining tighter control over the release that you use.
- locofacetwice 14y ago
  Or don't use Rails.
  - static_typed 14y ago
    ^^ This is good advice. Maybe they should question their choice of platform - there are other options, some of which seem to have had a bit more forethought in their architecture and engineering. Remember: Ruby/Rails to pose, Python for pros.
    - sbank 14y ago
      > Remember: Ruby/Rails to pose, Python for pros. How mature.
    - stephenhuey 14y ago
      ...and whitespacephiles. :)
  - tptacek 14y ago
    Or Linux, or MySQL.
    - bradleyland 14y ago
      Or Postgres.
      - tptacek 14y ago
        My subtext isn't very clear: there are other projects that haven't totally mastered handling vulnerabilities, but few people will fault you for using them. Rails is different because it has a personality cult, which makes it easy to personalize an issue that is a dry inconvenience for other popular packages. I'm not sure I'd put Postgres alongside Linux and Rails in terms of handling security issues.
  - myke_cameron 14y ago
    No matter your stack, you are occasionally going to have to deal with security vulnerabilities. Thankfully, rails at least is quick to issue releases which address security vulnerabilities. Back when I was working on java applets, I remember security issues in Java plugins which would persist for weeks, months, or even years.
- kasparloog 14y ago
  Well - that's enough work to maintain your own app. Maintainging a fork of Rails and drilling through all of the vulnerabilities - that's an extra overhead. It's simpler to regression test your own app, I suppose.
  - bradleyland 14y ago
    I disagree that it's simpler to regression test your own app. The recent minor-minor version upgrade passed Rails core tests, as well as test suites at high profile Rails users, yet it contained a regression that caused the disclosure of some sensitive issues at those same high profile shops. It's easier, from my viewpoint, to stick it out at a release that you know works for you, applying patches from the files contained in the CVEs. There's still a chance that the security patch will cause a regression, but at least you're not pulling in all the interim commits.
mratzloff 14y ago
So what was the issue from Rails, specifically?
- knowtheory 14y ago
  Specifically that the security releases included changes to the way that ActiveRecord works (which were unrelated to the security issues). As a consequence, their search queries were scoped differently than what they had intended. So their choice was roll back the security release, or modify their app to accommodate ActiveRecord's altered behavior.
  - mratzloff 14y ago
    I see. Thanks for the clarification.
  - jrochkind1 14y ago
    Actually, ironically, the _particular_ bug that changed how ActiveRecord works... which caused security problems... was actually an unintentional regression in a _security patch_. There were ALSO unrelated changes (quite many) in the patch release that included the latest security fixes. Which is a mess. But, the _particular_ problem here, the OP suggests it was the same problem as github reported [here](https://github.com/blog/1440-today-s-email-incident https://github.com/blog/1440-today-s-email-incident), where they suggest they isolated the introduction of the regression to [this commit](https://github.com/rails/rails/commit/f980289fd2c1b9073a94b5d49b780a49f5e2933c#L1L23 https://github.com/rails/rails/commit/f980289fd2c1b9073a94b5...), which in fact has a commit message as the fix to CVE-2013-1854. So, yes, a security fix unintentionally introduced a regression with _other_ security implications. Yeah, this is kind of ironic, and yeah, it means it's not so simple to say what could have been done to avoid it. (In this case, I'm surprised there wasn't an automated test already that caught the particular regression. It seems like something that should have been tested. But I haven't looked at the test source to see if it was an odd edge case or what have you.) (But it's STILL bad practice to release security patches only in releases bundled with a bunch of other changes).
tomkin 14y ago
Do you know how many Rails developers I've heard pissing on Flash or Java and its security vulnerabilities? Many - until recently. Now suddeningly these faults are an accepted aspect of Rails development. Really. I guess this more of a rant about how unfairly the Rails community (ie. dhh) has been on others, but now expects critics to look away while it happens in the Rails community on a weekly basis.
- tptacek 14y ago
  The security problems of Flash and Java are not comparable to those of Rails. They're different in magnitude, different in number, and different in circumstance and origin. I strongly agree with 'knowtheory that gloating about security vulnerabilities is a bad habit. But this Rails/Java comparison is even worse. Nobody personalizes Java insecurity. The Java applet plugin is a mess, responsible for a huge number of compromised desktops, but nobody I know would assume that a developer who worked in Java or on the JVM would be security-illiterate. That's not true of the Rails drama, which is really an opportunity for people to piss on DHH and his personality cult, as you can see in this subthread with 'static_typed's comment.
- static_typed 14y ago
  Rails became worse than the frameworks and ecosystems it laughed at in it's younger days. Remember - Rails is Omakase - meaning literally 'leave it to someeone else' - food for thought indeed.
  - knowtheory 14y ago
    Your point (trolling really) doesn't make any sense (and really the Omakase thing never made sense to begin with). The core value proposition for Rails has always been it incorporates enough of all of the things you need to get a web app up and running quickly, easily, and with sufficient power that your app can continue to grow into the future. That's what DHH's original blog post screen cast was about certainly. wycats and carllerche's contributions to Rails after the Rails/Merb merge have focused both on better code discipline and ease/simplicity of use for everyone, from beginners to advanced devs. On top of that the Rails security team has been totally on top of these disclosures and releasing patches that address them. Prominent members of the Rails community have been extraordinarily vocal in advocating that EVERYONE needs to upgrade their apps. So, please, tell me again how Rails leaves things to others. P.S. if you really want á la carte, use Sinatra, or Padrino. P.P.S. Ah, if you look at the actual meaning of omakase (http://en.wikipedia.org/wiki/Omakase http://en.wikipedia.org/wiki/Omakase ), it basically means, devs entrust the defaults to the Rails team, which is basically how things actually work w/ Rails.
sergiotapia 14y ago
I've shared my sentiment here before on why I'm not going to use Rails anymore. Imagine having your code working fine, and you update a MINOR version 3.2.x - and your ORM starts returning results that it didn't used to. Heh. Not worth the heartburn. I've switched to a saner, more tought out framework. Rails is gorgeous, but not safe to use for any serious systems where you're in a small team.
- superuser2 14y ago
  What did you switch to?
  - innguest 14y ago
    Probably ASP.Net MVC -> https://news.ycombinator.com/item?id=5410169 https://news.ycombinator.com/item?id=5410169
  - sergiotapia 14y ago
    ASP.Net MVC4.
- philwelch 14y ago
  If you're willing to trade off stability for features, the Rails 2.3 line still receives security patches to this day. You can upgrade to Rails 3 when Rails 4 comes out. If you're willing to make a slightly different trade-off, just apply the security patches as they come out and don't upgrade minor versions without integration testing.
  - zwily 14y ago
    I have heard, but can't find an official reference to it now, that Rails 2.3 will stop receiving security updates once Rails 4 is released. (I suspect that patches will still become available for serious issues even once official support is dropped.)
    - patio11 14y ago
static_typed 14y ago
Give a developer a Ruby-based web framework, and they can run for a day without patching, but give them anything else - even Prolog on Punchcards - and they run for so much longer! Stop today, and help recovering Ruby developers onto a better path!
- davesims 14y ago
  Your attempt to drive developers to Ruby and Rails by playing the role of a typical ignorant anti-rails fundamentalist crusader will fail.
- sbank 14y ago
  Enjoying your crusade?
- PetrolMan 14y ago
  I'm not really sure I understand this sentiment. I understand that the Rails community has had a reputation in the past of being arrogant but why take joy in the growing pains of a development community? Why foster an "us vs them" sentiment in the general development community? My understanding is that we are all doing the same things, we just have chosen different ways to get there.
websitescenes 14y ago
There is no such thing as an insurmountable security configuration. No matter what you use or how you use it, there will always be new methods to compromise your data. Hackers are creative and enjoy a challenge and will continually attack systems until the end. Whether it be Rails or another framework, you will have to upgrade and address security issues forever. Developers need be realistic and understand that any system can be compromised if one tries hard enough. I don't necessarily think that you should go from rails 2 to 3 or 3 to 4 just to address security features. Better off staying in the branch you started the project in and applying the applicable patches for said security concerns. With that said, RAILS RULES!!
jrochkind1 14y ago
Rails _needs_ to produce security-patch-only patch releases. This won't be foolproof; there is no such thing as foolproof in software development. There can still be unintentional regressions in security-patch-only releases (and even new security vulnerabilities), as well as new security vulnerabilities in other releases. Yes, nothing is foolproof. But you can work at improving your quality. Yes, there are other things Rails could do to improve quality, including trying to actually do some variation of semver. But the most obvious thing with the highest benefit/cost that Rails team could do is always release security patches in security patch only releases, so developers can apply security patch only releases and minimize their exposure to new regressions when doing so. Git should not make this hard to do, just cherry pick the security-patch commits into a branch created off the last release, and release it as a patch release seperately from any other changes. Am I missing something?
- jeremymcanally 14y ago
  Well, the release policy published by the Rails team states releases like this are supposed to only have security patches, but for some reason it wasn't followed here. I'm not sure why other than the frantic "zomg there's a security hole must fix it!" fray that may have clouded their judgement.
tptacek 14y ago
No question: update Rails. The security issues attributable to Rails upgrades are small in number and relatively minor. The security issues attributable to not upgrading Rails, and in particular in not upgrading during upgrade cycles where the Rails community was reassuring itself that everything was fine and that normal apps wouldn't be affected by security flaws, are much larger in number and extremely significant. If there was one lesson I'd hope people would take away from Rails Winter of Security Madness, it is be ready to patch at all times. That's a good lesson for every platform, not just Rails, but Rails teams now have specific reasons to be on top of this. When Rails announces security flaws, patch ASAP. If you're a professional Rails team, dry-run this well in advance; know you can patch at a moments notice, don't just hope.
- dasil003 14y ago
  How do you dry-run a Rails patch? I mean I know how to do `bundle update rails`. The problems with updates tend to be patch-specific.
  - tptacek 14y ago
    If the only mechanism you have to update Rails is to do a "bundle update", start by fixing that.
    - dasil003 14y ago
      Huh? No, I am familiar with Ruby development and use of git and patch files. I still don't understand your point. How do you a dry-run of a patch that doesn't exist?
      - tptacek 14y ago
        If you are seriously on top of Rails security, you're ready to deploy workarounds if you have to, instead of waiting for the "tested" fix to percolate into Ruby gems. So, start by making an innocuous but testable patch. Add a class to ActiveSupport or something. Also, when you drill this stuff (I think you should do it quarterly as soon as you have customers), spend as much time drilling how you'll profile your environment as you do in ensuring that your patch deployment is flawless. I worry more about hidden stale deployments than I do about production app servers.

42 points by kasparloog 14y ago

Another option is to fork Rails and use the patch files (included with the CVE) that target only the vulnerabilities addressed in the CVE. The problem with upgrading to mitigate security issues is that the Rails team does not release security patches. They bump the minor-minor and do a release. That release almost always includes commits that are unrelated to the security issue. This is especially true when a lot of time passes between CVEs. Maintaining your own branch really isn't that difficult, because you can simply merge in from upstream. In most cases, you're really only interested in patching from Rails team issued security fixes, so you won't have conflicts. If you do have conflicts, you can safely overwrite anything in your fork, because you're not developing Rails, you're simply maintaining tighter control over the release that you use.

locofacetwice 14y ago
Or don't use Rails.
- static_typed 14y ago
  ^^ This is good advice. Maybe they should question their choice of platform - there are other options, some of which seem to have had a bit more forethought in their architecture and engineering. Remember: Ruby/Rails to pose, Python for pros.
  - sbank 14y ago
    > Remember: Ruby/Rails to pose, Python for pros. How mature.
  - stephenhuey 14y ago
    ...and whitespacephiles. :)
- tptacek 14y ago
  Or Linux, or MySQL.
  - bradleyland 14y ago
    Or Postgres.
    - tptacek 14y ago
      My subtext isn't very clear: there are other projects that haven't totally mastered handling vulnerabilities, but few people will fault you for using them. Rails is different because it has a personality cult, which makes it easy to personalize an issue that is a dry inconvenience for other popular packages. I'm not sure I'd put Postgres alongside Linux and Rails in terms of handling security issues.
- myke_cameron 14y ago
  No matter your stack, you are occasionally going to have to deal with security vulnerabilities. Thankfully, rails at least is quick to issue releases which address security vulnerabilities. Back when I was working on java applets, I remember security issues in Java plugins which would persist for weeks, months, or even years.
kasparloog 14y ago
Well - that's enough work to maintain your own app. Maintainging a fork of Rails and drilling through all of the vulnerabilities - that's an extra overhead. It's simpler to regression test your own app, I suppose.
- bradleyland 14y ago
  I disagree that it's simpler to regression test your own app. The recent minor-minor version upgrade passed Rails core tests, as well as test suites at high profile Rails users, yet it contained a regression that caused the disclosure of some sensitive issues at those same high profile shops. It's easier, from my viewpoint, to stick it out at a release that you know works for you, applying patches from the files contained in the CVEs. There's still a chance that the security patch will cause a regression, but at least you're not pulling in all the interim commits.

mratzloff 14y ago

So what was the issue from Rails, specifically?

knowtheory 14y ago
Specifically that the security releases included changes to the way that ActiveRecord works (which were unrelated to the security issues). As a consequence, their search queries were scoped differently than what they had intended. So their choice was roll back the security release, or modify their app to accommodate ActiveRecord's altered behavior.
- mratzloff 14y ago
  I see. Thanks for the clarification.
- jrochkind1 14y ago
  Actually, ironically, the _particular_ bug that changed how ActiveRecord works... which caused security problems... was actually an unintentional regression in a _security patch_. There were ALSO unrelated changes (quite many) in the patch release that included the latest security fixes. Which is a mess. But, the _particular_ problem here, the OP suggests it was the same problem as github reported [here](https://github.com/blog/1440-today-s-email-incident https://github.com/blog/1440-today-s-email-incident), where they suggest they isolated the introduction of the regression to [this commit](https://github.com/rails/rails/commit/f980289fd2c1b9073a94b5d49b780a49f5e2933c#L1L23 https://github.com/rails/rails/commit/f980289fd2c1b9073a94b5...), which in fact has a commit message as the fix to CVE-2013-1854. So, yes, a security fix unintentionally introduced a regression with _other_ security implications. Yeah, this is kind of ironic, and yeah, it means it's not so simple to say what could have been done to avoid it. (In this case, I'm surprised there wasn't an automated test already that caught the particular regression. It seems like something that should have been tested. But I haven't looked at the test source to see if it was an odd edge case or what have you.) (But it's STILL bad practice to release security patches only in releases bundled with a bunch of other changes).

tomkin 14y ago

Do you know how many Rails developers I've heard pissing on Flash or Java and its security vulnerabilities? Many - until recently. Now suddeningly these faults are an accepted aspect of Rails development. Really. I guess this more of a rant about how unfairly the Rails community (ie. dhh) has been on others, but now expects critics to look away while it happens in the Rails community on a weekly basis.

tptacek 14y ago
The security problems of Flash and Java are not comparable to those of Rails. They're different in magnitude, different in number, and different in circumstance and origin. I strongly agree with 'knowtheory that gloating about security vulnerabilities is a bad habit. But this Rails/Java comparison is even worse. Nobody personalizes Java insecurity. The Java applet plugin is a mess, responsible for a huge number of compromised desktops, but nobody I know would assume that a developer who worked in Java or on the JVM would be security-illiterate. That's not true of the Rails drama, which is really an opportunity for people to piss on DHH and his personality cult, as you can see in this subthread with 'static_typed's comment.
static_typed 14y ago
Rails became worse than the frameworks and ecosystems it laughed at in it's younger days. Remember - Rails is Omakase - meaning literally 'leave it to someeone else' - food for thought indeed.
- knowtheory 14y ago
  Your point (trolling really) doesn't make any sense (and really the Omakase thing never made sense to begin with). The core value proposition for Rails has always been it incorporates enough of all of the things you need to get a web app up and running quickly, easily, and with sufficient power that your app can continue to grow into the future. That's what DHH's original blog post screen cast was about certainly. wycats and carllerche's contributions to Rails after the Rails/Merb merge have focused both on better code discipline and ease/simplicity of use for everyone, from beginners to advanced devs. On top of that the Rails security team has been totally on top of these disclosures and releasing patches that address them. Prominent members of the Rails community have been extraordinarily vocal in advocating that EVERYONE needs to upgrade their apps. So, please, tell me again how Rails leaves things to others. P.S. if you really want á la carte, use Sinatra, or Padrino. P.P.S. Ah, if you look at the actual meaning of omakase (http://en.wikipedia.org/wiki/Omakase http://en.wikipedia.org/wiki/Omakase ), it basically means, devs entrust the defaults to the Rails team, which is basically how things actually work w/ Rails.

sergiotapia 14y ago

I've shared my sentiment here before on why I'm not going to use Rails anymore. Imagine having your code working fine, and you update a MINOR version 3.2.x - and your ORM starts returning results that it didn't used to. Heh. Not worth the heartburn. I've switched to a saner, more tought out framework. Rails is gorgeous, but not safe to use for any serious systems where you're in a small team.

superuser2 14y ago
What did you switch to?
- innguest 14y ago
  Probably ASP.Net MVC -> https://news.ycombinator.com/item?id=5410169 https://news.ycombinator.com/item?id=5410169
- sergiotapia 14y ago
  ASP.Net MVC4.
philwelch 14y ago
If you're willing to trade off stability for features, the Rails 2.3 line still receives security patches to this day. You can upgrade to Rails 3 when Rails 4 comes out. If you're willing to make a slightly different trade-off, just apply the security patches as they come out and don't upgrade minor versions without integration testing.
- zwily 14y ago
  I have heard, but can't find an official reference to it now, that Rails 2.3 will stop receiving security updates once Rails 4 is released. (I suspect that patches will still become available for serious issues even once official support is dropped.)
  - patio11 14y ago

static_typed 14y ago

Give a developer a Ruby-based web framework, and they can run for a day without patching, but give them anything else - even Prolog on Punchcards - and they run for so much longer! Stop today, and help recovering Ruby developers onto a better path!

davesims 14y ago
Your attempt to drive developers to Ruby and Rails by playing the role of a typical ignorant anti-rails fundamentalist crusader will fail.
sbank 14y ago
Enjoying your crusade?
PetrolMan 14y ago
I'm not really sure I understand this sentiment. I understand that the Rails community has had a reputation in the past of being arrogant but why take joy in the growing pains of a development community? Why foster an "us vs them" sentiment in the general development community? My understanding is that we are all doing the same things, we just have chosen different ways to get there.

websitescenes 14y ago

There is no such thing as an insurmountable security configuration. No matter what you use or how you use it, there will always be new methods to compromise your data. Hackers are creative and enjoy a challenge and will continually attack systems until the end. Whether it be Rails or another framework, you will have to upgrade and address security issues forever. Developers need be realistic and understand that any system can be compromised if one tries hard enough. I don't necessarily think that you should go from rails 2 to 3 or 3 to 4 just to address security features. Better off staying in the branch you started the project in and applying the applicable patches for said security concerns. With that said, RAILS RULES!!

jrochkind1 14y ago

Rails _needs_ to produce security-patch-only patch releases. This won't be foolproof; there is no such thing as foolproof in software development. There can still be unintentional regressions in security-patch-only releases (and even new security vulnerabilities), as well as new security vulnerabilities in other releases. Yes, nothing is foolproof. But you can work at improving your quality. Yes, there are other things Rails could do to improve quality, including trying to actually do some variation of semver. But the most obvious thing with the highest benefit/cost that Rails team could do is always release security patches in security patch only releases, so developers can apply security patch only releases and minimize their exposure to new regressions when doing so. Git should not make this hard to do, just cherry pick the security-patch commits into a branch created off the last release, and release it as a patch release seperately from any other changes. Am I missing something?

jeremymcanally 14y ago
Well, the release policy published by the Rails team states releases like this are supposed to only have security patches, but for some reason it wasn't followed here. I'm not sure why other than the frantic "zomg there's a security hole must fix it!" fray that may have clouded their judgement.

tptacek 14y ago

No question: update Rails. The security issues attributable to Rails upgrades are small in number and relatively minor. The security issues attributable to not upgrading Rails, and in particular in not upgrading during upgrade cycles where the Rails community was reassuring itself that everything was fine and that normal apps wouldn't be affected by security flaws, are much larger in number and extremely significant. If there was one lesson I'd hope people would take away from Rails Winter of Security Madness, it is be ready to patch at all times. That's a good lesson for every platform, not just Rails, but Rails teams now have specific reasons to be on top of this. When Rails announces security flaws, patch ASAP. If you're a professional Rails team, dry-run this well in advance; know you can patch at a moments notice, don't just hope.

dasil003 14y ago
How do you dry-run a Rails patch? I mean I know how to do `bundle update rails`. The problems with updates tend to be patch-specific.
- tptacek 14y ago
  If the only mechanism you have to update Rails is to do a "bundle update", start by fixing that.
  - dasil003 14y ago
    Huh? No, I am familiar with Ruby development and use of git and patch files. I still don't understand your point. How do you a dry-run of a patch that doesn't exist?
    - tptacek 14y ago
      If you are seriously on top of Rails security, you're ready to deploy workarounds if you have to, instead of waiting for the "tested" fix to percolate into Ruby gems. So, start by making an innocuous but testable patch. Add a class to ActiveSupport or something. Also, when you drill this stuff (I think you should do it quarterly as soon as you have customers), spend as much time drilling how you'll profile your environment as you do in ensuring that your patch deployment is flawless. I worry more about hidden stale deployments than I do about production app servers.