4 ms·

Stop installing latest shiniest things manually and start using default package manager of your linux distribution. Then all the security maintenance you will p

by densh 14y ago

Stop installing latest shiniest things manually and start using default package manager of your linux distribution. Then all the security maintenance you will probably need is to do equivalent of "apt-get update; apt-get upgrade" once in a while. Yes, it was that easy before all these people who knew better came and decided that they really need latest ruby installed via rvm, latest gems etc. All these ruby/rails deployment "best practices" stroke me as advice given by people who had close to zero experience of maintaining a mission-critical application for a large period of time.

lengarvey 14y ago
And this common refrain is the shrill cry of someone who hasn't had to manage a Ruby on Rails application. Application libraries are the domain of the developer, not operations. Your job is to give me a stable system for me to deploy on top of, not to dictate what I can and cannot deploy. Rubygems and bundler makes updating your application stupidly easy. I blogged about it here: http://bottledup.net/2013/01/10/bundler-and-gemfile/ http://bottledup.net/2013/01/10/bundler-and-gemfile/ but if you have a good Gemfile and automated CI then all you need to do is `bundle update rails` and then deploy your code. This is at least as easy as using apt to update your stuff.
tptacek 14y ago
This is the worst possible option and I strongly recommend you minimize your dependencies on your OS package manager; if possible, build your web server from source too. You need to be prepared at all times to deploy workarounds to newly disclosed security problems. Package managers can and do delay fixes to accommodate the lowest common denominator of users. You cannot run a high profile application and be at the mercy of whoever is administrating your OS package manager.
- stouset 14y ago
  While I usually agree with your advice, I think this approach, while theoretically correct, is actually damaging to the majority of your audience here. The closest analogy I can think of is that it's like requiring users to change passwords every 30 days: great in theory, but in practice it's a disaster. The problem is that it is simply untenable for all but the highest-profile sites. Finding good ops people, even in the bay area, is extremely hard. Most sites are only going to realize that a security update has been released when their package manager tells them it has, and updating that package is usually a 30s process. The companies I've seen have a hard enough time keeping track of security updates with package management. For a small to medium team with two, one, or even zero dedicated ops people, asking them to custom-compile (for example) a webserver, ruby implementation, and other critical libraries (openssl, glibc, etc.), subscribe to the relevant security mailing lists, and follow along with updates and security patches is tantamount to having them leave unpatched vulnerabilities on their systems for months or even years. If you ask your users to change passwords every 30 days, there are inevitably going to be a few who take it seriously and generate and remember secure passwords every single time. But the vast majority are going to use weaker passwords than they otherwise would have, and duplicate those passwords across accounts as much as they can figure out how. Likewise, if you ask already overworked ops guys to manually compile and keep track of security vulnerabilities for their webserver and dozens of libraries, a few are inevitably going to keep on top of things and release fixes minutes after vulnerabilities are announced. But the vast majority are going to simply give up after a month or two and be significantly worse off than if they just use Ubuntu's automatic security package updates.
  - m0nastic 14y ago
    I'd like to think that if a company doesn't have the resources to properly maintain security patches on their deployed applications, they would use some app hosting platform (like Heroku), but I suspect that there are indeed many companies who are set up like you describe. That doesn't change the fact that relying on your OS package maintainers to properly update packages results in the same "having them leave unpatched vulnerabilities on their systems for months or even years" (at least the "months" part if Ubuntu is any indication). This would seem to be a "damned if you do, damned if you don't" scenario. Also, the 30 day password change thing isn't even something that's "technically correct". It's a "why do we cut the ends of the roast off" vestige from old DoD recommendations.
  - jfb 14y ago
    Perhaps if you can't afford to build a safe system, you should reconsider if you should be in the business of building unsafe ones?
  - ams6110 14y ago
    Hot-off-the-presses updates often contain serious bugs themselves as well. I lost count of the number of Drupal patches-to-patches-to-updates I've had to deploy, each coming within days or sometimes hours of the prior one. It got to the point where unless it was a critical remote vulnerability I would never apply a fix that had not been out for at least a week with no subsequent serious bug reports.
    - tptacek 14y ago
      How do you know what is or isn't critical? Do you just take it on faith that the vulnerability announcement is correct?
  - tptacek 14y ago
    I don't know what to tell you other than that this is something you actually have to get good at if you're going to run a high-profile app or hold sensitive information of any sort. I'm not saying you need to be able to find your own nginx vulnerabilities or even write your own patches. But reinstalling nginx or Apache from source shouldn't be a science project for your team; you should know that you can get your prod servers running on a from-source build. Sink in the time to make sure you can do that now, so you aren't caught totally flat-footed when an emergency happens.
rst 14y ago
Not if you want prompt fixes to the current spate of Rails bugs. As I write, Ubuntu currently has the fix for the last remote code execution vulnerability in Rails marked as being of "undecided" importance, and still "unassigned", even though rails-core shipped a patch weeks ago. This is remote code execution against any vulnerable app, which is pretty much as bad as it gets. Debian has a fix for the version of Rails 2.3 that they're shipping, but 2.3 is years obsolete, and officially unsupported upstream. Pointers on both: https://bugs.launchpad.net/ubuntu/+source/rails/+bug/1097643 https://bugs.launchpad.net/ubuntu/+source/rails/+bug/1097643 (comments in the Ubuntu issue reference the Debian fix.) There are things that the Debian and Ubuntu maintainers do well, but their packaging for Rails and the underlying Ruby language have both been problematic for quite some time, and their use is generally not recommended.
wglb 14y ago
This is not a good idea for certain packages. For example, if you are using something that is under seriously active development, such as Rails and Ruby, and in another case, SBCL Lisp, and even often Emacs. Up-to-date security is one seriously important reason.

by densh 14y ago

lengarvey 14y ago

And this common refrain is the shrill cry of someone who hasn't had to manage a Ruby on Rails application. Application libraries are the domain of the developer, not operations. Your job is to give me a stable system for me to deploy on top of, not to dictate what I can and cannot deploy. Rubygems and bundler makes updating your application stupidly easy. I blogged about it here: http://bottledup.net/2013/01/10/bundler-and-gemfile/ http://bottledup.net/2013/01/10/bundler-and-gemfile/ but if you have a good Gemfile and automated CI then all you need to do is `bundle update rails` and then deploy your code. This is at least as easy as using apt to update your stuff.

tptacek 14y ago

This is the worst possible option and I strongly recommend you minimize your dependencies on your OS package manager; if possible, build your web server from source too. You need to be prepared at all times to deploy workarounds to newly disclosed security problems. Package managers can and do delay fixes to accommodate the lowest common denominator of users. You cannot run a high profile application and be at the mercy of whoever is administrating your OS package manager.

stouset 14y ago
While I usually agree with your advice, I think this approach, while theoretically correct, is actually damaging to the majority of your audience here. The closest analogy I can think of is that it's like requiring users to change passwords every 30 days: great in theory, but in practice it's a disaster. The problem is that it is simply untenable for all but the highest-profile sites. Finding good ops people, even in the bay area, is extremely hard. Most sites are only going to realize that a security update has been released when their package manager tells them it has, and updating that package is usually a 30s process. The companies I've seen have a hard enough time keeping track of security updates with package management. For a small to medium team with two, one, or even zero dedicated ops people, asking them to custom-compile (for example) a webserver, ruby implementation, and other critical libraries (openssl, glibc, etc.), subscribe to the relevant security mailing lists, and follow along with updates and security patches is tantamount to having them leave unpatched vulnerabilities on their systems for months or even years. If you ask your users to change passwords every 30 days, there are inevitably going to be a few who take it seriously and generate and remember secure passwords every single time. But the vast majority are going to use weaker passwords than they otherwise would have, and duplicate those passwords across accounts as much as they can figure out how. Likewise, if you ask already overworked ops guys to manually compile and keep track of security vulnerabilities for their webserver and dozens of libraries, a few are inevitably going to keep on top of things and release fixes minutes after vulnerabilities are announced. But the vast majority are going to simply give up after a month or two and be significantly worse off than if they just use Ubuntu's automatic security package updates.
- m0nastic 14y ago
  I'd like to think that if a company doesn't have the resources to properly maintain security patches on their deployed applications, they would use some app hosting platform (like Heroku), but I suspect that there are indeed many companies who are set up like you describe. That doesn't change the fact that relying on your OS package maintainers to properly update packages results in the same "having them leave unpatched vulnerabilities on their systems for months or even years" (at least the "months" part if Ubuntu is any indication). This would seem to be a "damned if you do, damned if you don't" scenario. Also, the 30 day password change thing isn't even something that's "technically correct". It's a "why do we cut the ends of the roast off" vestige from old DoD recommendations.
- jfb 14y ago
  Perhaps if you can't afford to build a safe system, you should reconsider if you should be in the business of building unsafe ones?
- ams6110 14y ago
  Hot-off-the-presses updates often contain serious bugs themselves as well. I lost count of the number of Drupal patches-to-patches-to-updates I've had to deploy, each coming within days or sometimes hours of the prior one. It got to the point where unless it was a critical remote vulnerability I would never apply a fix that had not been out for at least a week with no subsequent serious bug reports.
  - tptacek 14y ago
    How do you know what is or isn't critical? Do you just take it on faith that the vulnerability announcement is correct?
- tptacek 14y ago
  I don't know what to tell you other than that this is something you actually have to get good at if you're going to run a high-profile app or hold sensitive information of any sort. I'm not saying you need to be able to find your own nginx vulnerabilities or even write your own patches. But reinstalling nginx or Apache from source shouldn't be a science project for your team; you should know that you can get your prod servers running on a from-source build. Sink in the time to make sure you can do that now, so you aren't caught totally flat-footed when an emergency happens.

rst 14y ago

Not if you want prompt fixes to the current spate of Rails bugs. As I write, Ubuntu currently has the fix for the last remote code execution vulnerability in Rails marked as being of "undecided" importance, and still "unassigned", even though rails-core shipped a patch weeks ago. This is remote code execution against any vulnerable app, which is pretty much as bad as it gets. Debian has a fix for the version of Rails 2.3 that they're shipping, but 2.3 is years obsolete, and officially unsupported upstream. Pointers on both: https://bugs.launchpad.net/ubuntu/+source/rails/+bug/1097643 https://bugs.launchpad.net/ubuntu/+source/rails/+bug/1097643 (comments in the Ubuntu issue reference the Debian fix.) There are things that the Debian and Ubuntu maintainers do well, but their packaging for Rails and the underlying Ruby language have both been problematic for quite some time, and their use is generally not recommended.

wglb 14y ago

This is not a good idea for certain packages. For example, if you are using something that is under seriously active development, such as Rails and Ruby, and in another case, SBCL Lisp, and even often Emacs. Up-to-date security is one seriously important reason.