6 ms·

Ruby deploys temporarily disabled

173 points by ujeezy 14y ago

taf2 14y ago
I thought at one point in time rubygems had a system in place to sign the contents of the gem? If not this might be an interesting addition. You could have a digest stored along side every gem file allowing you to validate the authenticity of the gem file... I'm sure others would have something to add to this idea...
- msmith 14y ago
  It is possible for the publisher to sign the gems [1], but it's not common. If rubygems.org is keeping fingerprints of each gem, then it still isn't sufficient, since those could have been compromised as well. If there's no other trustworthy source of fingerprints, then maybe we need to crowdsource it. Built a tool that will md5sum all the .gem files in your local cache directory, so that we can look for any files that were changed on rubygems.org [1] http://docs.rubygems.org/read/chapter/21 http://docs.rubygems.org/read/chapter/21
  - rmc 14y ago
    Sounds like a good reason to switch to "everything must be signed"
    - ikawe 14y ago
      Just to reiterate the parent: This is only valuable if we trust the signatures - which I wouldn't if they were, say, just held along side the "hacked" gems server.
      - rmc 14y ago
        I'm talking g about developers signing the archive on their local machine. Private key would be stored on developers laptop
        Xylakant 14y ago
        You still need the public key to validate the signature. If the attacker can change the public key, he can change the signature without you knowing - unless you explicitly want to trust each and every key for every gem you install.
zdgman 14y ago
Good to see them being proactive about it. Wonder how quickly they will be able to determine that Ruby/Gems are safe to deploy again.
whalesalad 14y ago
I always assumed that Heroku would have an internal proxy for gems. Seems like 80% of users would probably be fetching the same gems that another user might have just fetched. Perhaps something like this could be versioned or snapshotted so that in the event of something like this, you could roll your cache back to that snapshot and let people deploy who had gems in that cache. I'm just thinking out loud.
- rhizome 14y ago
  Which snapshot do you roll back to?
  - whalesalad 14y ago
    That's a good question. I do know one thing though: I deploy multiple times per day and typically none of my gems have changed. I guess it would depend on the folks doing the investigation. If an exact timestamp could be determined for when things could have been compromised, you just roll back to a short while before that time.
sergiotapia 14y ago
The DAY I manage to convince the big wigs where I work that we should switch from a typical shared environment to Heroku, this happens. Talk about luck. :( Hopefully I can spin this and not leave a bad taste in their mouths. We (engineers) understand what's happening, management doesn't and they don't give a shit.
- tomjen3 14y ago
  What is the alternative? You stay on the shared hosting and then what, you get hacked because you didn't verify a gem? Are the bigwigs going to authorize you time to look through all gems for potential backdoors or are they going to get i for free with their Heroku hosting?
- AlexMuir 14y ago
  Safe by default. That's your angle. It's additional effort to deploy dangerous code.
- sumone4life 14y ago
  they give you a straight forward workaround to still deploy. They just make you set a value explicitly so they know you are aware of the risk. Good to know someone is watching your back :)
- agotterer 14y ago
  The exploit still exists whether you are on Heroku, shared or bare metal hosting environment. It's not an issue specific to Heroku, it's an issue that affects ruby gems. Your situation would be worse if you convinced the big wigs to switch to Ruby today.
- rrikhy 14y ago
  Feel free to reach out to me if you'd like to have a conversation about how to support your case on Heroku or if you have any questions/concerns; raj@heroku.com
- pardner 14y ago
  From a manager perspective, this sort of alert (as well as recent Heroku emails telling you specifically which app(s) needed patching for the recent Rails CVE issues) is a great example of one of the extra benefits of Heroku. A team of engineers 'watching your back' at no extra charge is a good thing.
Pewpewarrows 14y ago
This should also be a reminder to everyone that you shouldn't be reliant on a single point of failure for your deploys. It's something that we in the Python community have already encountered (and hopefully learned from) due to the historical unreliability of our equivalent package repo, PyPI. Have an internal repo that's accessible by your deploy servers, which in turn locally caches anything that you might have previously needed to externally fetch.
- sumone4life 14y ago
  They still allow you to deploy you just have to explicitly set a variable in the deploy command so they know you are aware whats going on
  - tlrobinson 14y ago
    The point was unless you also previously cached all your gems somewhere you'd have to deploy using potentially compromised gems from rubygems.
  - splatcollision 14y ago
    Is this safe if you haven't changed any gems since the last deploy? I have a bugfix that I would like to deploy...
    - ikawe 14y ago
      Heroku runs bundle install on deploy, so it's not safe until all your gems (and their dependencies in gemfile.lock) are cleared.
- mapgrep 14y ago
  Indeed. I presume this is why Perl's package repository CPAN is actually a network of repositories ("Comprehensive Perl Archive Network"); Wikipedia says CPAN "is mirrored worldwide at more than 200 locations." Does anyone know why rubygems does not work this way? I had always just assumed it did (due to the historical intertwining of Ruby and Perl communities).
martingordon 14y ago
Between this hack and the recent Rails vulnerabilities, it seems like a perfect storm. I wonder if either the hack attempted to tamper with the Rails gems to catch late updaters or to remove the ability to use RubyGems to update to the latest versions and keep vulnerable sites vulnerable.
- hadees 14y ago
  I think this hack was related to the recent Rails vulnerability. Heroku is blaming this on a "YAML parsing vulnerability" which I think was the problem same issue with rails. I'm not sure if they are using rails or not but its surprising that if it's the same issue they didn't do anything about it before this happened.
- deleted 14y ago
  [deleted]
- awj 14y ago
  It more looks like this was a natural extension of (part of) the Rails vulnerabilities. People saw that YAML on Ruby has a giant gaping security hole in it and was commonly used to decode user-supplied data. I would not be surprised if we see even more of this as people feel out all of the other places that YAML is used as a user-facing data interchange format.
mildavw 14y ago
Survey: Do you depend on RubyGems for every deploy? Or do you have your own gem server? Or cache them at some point earlier in your pipeline? We rely on RubyGems and had a meeting yesterday about changing that when one of the gems we use had a version just disappear.
- pbiggar 14y ago
  Gems get yanked. Its a danger of this approach.
  - kawsper 14y ago
    And stuff get yanked for good reasons.
- scotje 14y ago
  If you use bundler, "bundle package" can help reduce or eliminate your dependency on external gem repositories. At least for deployments. I generally try to follow the "vendor everything" philosophy: http://ryan.mcgeary.org/2011/02/09/vendor-everything-still-applies/ http://ryan.mcgeary.org/2011/02/09/vendor-everything-still-a...
  - redeemedfadi 14y ago
    I couldn't get through this whole article due to the dude in the corner staring at me... :-/
    - Legion 14y ago
      Not that this line of discussion is particularly constructive, but I agree. I'm not sure why someone would think a giant head staring at the reader is a good idea, but it's not. On a 27" monitor, it's almost like a child sized head right up in your face. I can only imagine it being even worse on bigger screens.
    - xanido 14y ago
      document.getElementById('mugshot').remove()
freshfruit 14y ago
If you are not updating gems, does it hurt to continue deploying with a custom buildpack? Heroku shouldn't repull gems if the gemspec hasn't been altered. Is that logic correct? Does anybody have another suggestion for safely working around this issue? I don't have a clear sense for how long this will take to resolve and don't wish to slow down our release pace too much.
- kawsper 14y ago
  If you trust your buildpack (and its contents), and it contains all the gems necessary, you are safe to deploy.
benmmurphy 14y ago
This is the responsible thing to do. Going through the gems and verifying they aren't compromised is a lot of work. We should be thankful for all the effort the rubygems maintainers and other volunteers are putting into cleaning up this mess.
akavi 14y ago
A tangent, but I always thought "YAML" was pronounced /'jæm.ḷ/, however the post's use of "an YAML" suggests it's actually pronounced /waɪ.eɪ.ɛm.ɛl/. Weird.
- eric_the_read 14y ago
  Don't worry too much about this. A lot of people haven't internalized the correct rule (use "an" before a vowel sound, not a vowel letter), and instead just use "an" before a vowel, regardless of the sound it makes. I've certainly never heard anything but /'jæm.ḷ/ in the wild.
- deleted 14y ago
  [deleted]
- saurik 14y ago
  You still wouldn't say "an why", as "w" isn't a vowel. If Anything, I can understand "an yamel" as more legitimate (as "y" is at least sort of a vowel).
  - justsee 14y ago
    No, "an YAML" is grammatically incorrect [0]. It's "a YAML". What's more annoying: the odd (and wrong) belief that "An green apple" is grammatically correct. [1] [0] http://english.stackexchange.com/questions/1016/do-you-use-a-or-an-before-acronyms/11511#11511 http://english.stackexchange.com/questions/1016/do-you-use-a... [1] http://english.stackexchange.com/questions/152/when-should-i-use-a-vs-an/164#164 http://english.stackexchange.com/questions/152/when-should-i...
    - saurik 14y ago
      I didn't say that "an YAML" is correct; I was first correcting that "an YAML" would be correct if it were "an why aih ahm ell" (it wouldn't), and then went further into "if anything" land saying that "an yahmell" is at least something I can bring myself to say without feeling sick inside. ;P
- 14y ago
achalkley 14y ago
Heroku a great.
pardner 14y ago
Seems odd, however, that status.heroku.com lists the issue only on the development side, suggesting production apps are not affected? http://screencast.com/t/L36Hpx5dx http://screencast.com/t/L36Hpx5dx
- pvh 14y ago
  Running apps are not affected, but your ability to do development is. The threat vector affects app compilation, not app execution/scaling, which doesn't touch the tainted rubygems.org repositories.
  - pardner 14y ago
    I don't think that is correct. AFAIK the 'development' side of the heroku status panel is not related to developing, per se, it's the status for apps that are not running on production-level resources... for example, single-dyno apps, or apps not running with production flavor of database.
    - bgentry 14y ago
      You are incorrect: https://devcenter.heroku.com/articles/heroku-status#status-information https://devcenter.heroku.com/articles/heroku-status#status-i... I agree that the production/development split is not entirely clear without additional explanation. We've spent a lot of time thinking about how to communicate these things and have so far not come up with a way that we feel better describes the issues at hand.
      - pardner 14y ago
        Thx for correcting that... have used that screen a million times without realizing they lumped dev apps and prod+dev workflow together. 3 columns would eliminate any confusion (Production Apps, Development Apps, Development Workflow) but might not look at clean.
aneth4 14y ago
This is why you vendor gems, and the current accepted practice of not vendoring gems is dead wrong. I've been chastised before in rails irc for this. I strongly believe the source of all depndencies possible should be in your repository.
- delano 14y ago
  That's how Maven works but to a fault. The huge dependency repo tends to bog down deployments/releases. Today I use bundler but my solution back in the (java) day was to separate code releases from dependency releases (separate directories basically). e.g. /path/2/project /path/2/project-resources
- pmahoney 14y ago
  > source of all dependencies possible should be in your repository How far do you go? Do you include libxml for building nokogiri? Heck, do you include libc and gcc for building any gem with a C extension? Coming from Java, Maven and something like Nexus Sonatype make it easy (for certain values of "easy") to run a proxy repository. The equivalent of all "gem install <some_gem>" goes through the proxy, which continues to serve gems even if the original source goes away. I don't particularly like the inclusion of dependencies in a repository. Is this a custom version "some guy" long gone from the company created three years ago? Can I safely upgrade it to get security fix <X>? I suppose similar questions arise no matter the source... This is reason why you cryptographically sign your gems before publishing them. I (unfortunately) had not known this was supported by RubyGems, but it is: http://docs.rubygems.org/read/chapter/21 http://docs.rubygems.org/read/chapter/21 But I'll bet very few gems are signed. Rails does not appear to be: $ gem install rails -P HighSecurity ERROR: While executing gem ... (Gem::Exception) Unsigned gem
thinkbohemian 14y ago
Heroku engineer here: Ruby deploys are back online if you don't require any new gems, i.e. can deploy from the existing cache. We're still working on resolving the large problem with Ruby Gems.
pilif 14y ago
I'm surprised that rubygems.org of all places did not see fit to patch the vulnerability that's now known for multiple weeks and which has been declared to be incredibly dangerous and for which ready-made exploit kits exist. rubygems.org is a central distribution platform trusted by tons and tons of projects. As such, that site is the one site you probably do not ever want compromized. Imagine the damage an attacker could deal by uploading backdoored versions of various popular gems. I know - applying security patches is time-consuming and we are all afraid of breakage. But the moment rubygems.org stepped up to be a semi-official central distribution point for gems, I would have hoped they also took on the responsibility that goes along with that. If this was some new unknown 0day exploit, I would be much more understanding, but this was known to exist, known to be dangerous, known to be exploited.
- steveklabnik 14y ago
  This is not that vulnerability. Their Rails is just fine. This is related to the fact that gems use YAML to store their metadata.
  - 1qaz2wsx3edc 14y ago
    Semi related CVEs aside (CVE-2013-0333 and CVE-2013-0156): YAML's security problems have been known for years by the community. YAML aside, don't trust user input. This is egg on rubygems's face and the ruby community. I don't think http://rubycentral.org/ http://rubycentral.org/ has full-time staff for rubygems either. Gems, specifically should be signed. They are not, this type of exploit will continue to happen, hell, remember when github screwed up ssh keys? Who knows what's in the ecosystem. TL;DR; Ruby's security ecosystem is butter. Disclaimer: I love ruby and use it daily. It was two critical problems IMO: unsigned code and GIL. Yes, GIL. I'm looking at you ruby-core.
    - charliesome 14y ago
      The GIL is not really a problem. There are far more pressing issues with MRI, and many core developers are working hard to solve these.
djangolover301 14y ago
lmao Ruby and RoR shame PHP in terms of security flaws. Making you unknowingly write security holes, ridiculous flaws discovered on daily/weekly basis, package management hacked etc I have never seen as ridiculous holes as RoR even in the CodeIgniter framework. Where are the RoR-haters when we need them?
reybango 14y ago
Being totally new to RoR (trying to learn it), I'm trying to get my head around the scope of this. When did the compromise happen? Was it compromised yesterday or only found out yesterday? I have default gems installed on my system and haven't updated anything since the big Rails security issue that was reported a bit ago. It'd be great to get some guidance on what to do.

173 points by ujeezy 14y ago

taf2 14y ago

I thought at one point in time rubygems had a system in place to sign the contents of the gem? If not this might be an interesting addition. You could have a digest stored along side every gem file allowing you to validate the authenticity of the gem file... I'm sure others would have something to add to this idea...

msmith 14y ago
It is possible for the publisher to sign the gems [1], but it's not common. If rubygems.org is keeping fingerprints of each gem, then it still isn't sufficient, since those could have been compromised as well. If there's no other trustworthy source of fingerprints, then maybe we need to crowdsource it. Built a tool that will md5sum all the .gem files in your local cache directory, so that we can look for any files that were changed on rubygems.org [1] http://docs.rubygems.org/read/chapter/21 http://docs.rubygems.org/read/chapter/21
- rmc 14y ago
  Sounds like a good reason to switch to "everything must be signed"
  - ikawe 14y ago
    Just to reiterate the parent: This is only valuable if we trust the signatures - which I wouldn't if they were, say, just held along side the "hacked" gems server.
    - rmc 14y ago
      I'm talking g about developers signing the archive on their local machine. Private key would be stored on developers laptop
      - Xylakant 14y ago
        You still need the public key to validate the signature. If the attacker can change the public key, he can change the signature without you knowing - unless you explicitly want to trust each and every key for every gem you install.

zdgman 14y ago

Good to see them being proactive about it. Wonder how quickly they will be able to determine that Ruby/Gems are safe to deploy again.

whalesalad 14y ago

I always assumed that Heroku would have an internal proxy for gems. Seems like 80% of users would probably be fetching the same gems that another user might have just fetched. Perhaps something like this could be versioned or snapshotted so that in the event of something like this, you could roll your cache back to that snapshot and let people deploy who had gems in that cache. I'm just thinking out loud.

rhizome 14y ago
Which snapshot do you roll back to?
- whalesalad 14y ago
  That's a good question. I do know one thing though: I deploy multiple times per day and typically none of my gems have changed. I guess it would depend on the folks doing the investigation. If an exact timestamp could be determined for when things could have been compromised, you just roll back to a short while before that time.

sergiotapia 14y ago

The DAY I manage to convince the big wigs where I work that we should switch from a typical shared environment to Heroku, this happens. Talk about luck. :( Hopefully I can spin this and not leave a bad taste in their mouths. We (engineers) understand what's happening, management doesn't and they don't give a shit.

tomjen3 14y ago
What is the alternative? You stay on the shared hosting and then what, you get hacked because you didn't verify a gem? Are the bigwigs going to authorize you time to look through all gems for potential backdoors or are they going to get i for free with their Heroku hosting?
AlexMuir 14y ago
Safe by default. That's your angle. It's additional effort to deploy dangerous code.
sumone4life 14y ago
they give you a straight forward workaround to still deploy. They just make you set a value explicitly so they know you are aware of the risk. Good to know someone is watching your back :)
agotterer 14y ago
The exploit still exists whether you are on Heroku, shared or bare metal hosting environment. It's not an issue specific to Heroku, it's an issue that affects ruby gems. Your situation would be worse if you convinced the big wigs to switch to Ruby today.
rrikhy 14y ago
Feel free to reach out to me if you'd like to have a conversation about how to support your case on Heroku or if you have any questions/concerns; raj@heroku.com
pardner 14y ago
From a manager perspective, this sort of alert (as well as recent Heroku emails telling you specifically which app(s) needed patching for the recent Rails CVE issues) is a great example of one of the extra benefits of Heroku. A team of engineers 'watching your back' at no extra charge is a good thing.

Pewpewarrows 14y ago

This should also be a reminder to everyone that you shouldn't be reliant on a single point of failure for your deploys. It's something that we in the Python community have already encountered (and hopefully learned from) due to the historical unreliability of our equivalent package repo, PyPI. Have an internal repo that's accessible by your deploy servers, which in turn locally caches anything that you might have previously needed to externally fetch.

sumone4life 14y ago
They still allow you to deploy you just have to explicitly set a variable in the deploy command so they know you are aware whats going on
- tlrobinson 14y ago
  The point was unless you also previously cached all your gems somewhere you'd have to deploy using potentially compromised gems from rubygems.
- splatcollision 14y ago
  Is this safe if you haven't changed any gems since the last deploy? I have a bugfix that I would like to deploy...
  - ikawe 14y ago
    Heroku runs bundle install on deploy, so it's not safe until all your gems (and their dependencies in gemfile.lock) are cleared.
mapgrep 14y ago
Indeed. I presume this is why Perl's package repository CPAN is actually a network of repositories ("Comprehensive Perl Archive Network"); Wikipedia says CPAN "is mirrored worldwide at more than 200 locations." Does anyone know why rubygems does not work this way? I had always just assumed it did (due to the historical intertwining of Ruby and Perl communities).

martingordon 14y ago

Between this hack and the recent Rails vulnerabilities, it seems like a perfect storm. I wonder if either the hack attempted to tamper with the Rails gems to catch late updaters or to remove the ability to use RubyGems to update to the latest versions and keep vulnerable sites vulnerable.

hadees 14y ago
I think this hack was related to the recent Rails vulnerability. Heroku is blaming this on a "YAML parsing vulnerability" which I think was the problem same issue with rails. I'm not sure if they are using rails or not but its surprising that if it's the same issue they didn't do anything about it before this happened.
deleted 14y ago
[deleted]
awj 14y ago
It more looks like this was a natural extension of (part of) the Rails vulnerabilities. People saw that YAML on Ruby has a giant gaping security hole in it and was commonly used to decode user-supplied data. I would not be surprised if we see even more of this as people feel out all of the other places that YAML is used as a user-facing data interchange format.

mildavw 14y ago

Survey: Do you depend on RubyGems for every deploy? Or do you have your own gem server? Or cache them at some point earlier in your pipeline? We rely on RubyGems and had a meeting yesterday about changing that when one of the gems we use had a version just disappear.

pbiggar 14y ago
Gems get yanked. Its a danger of this approach.
- kawsper 14y ago
  And stuff get yanked for good reasons.
scotje 14y ago
If you use bundler, "bundle package" can help reduce or eliminate your dependency on external gem repositories. At least for deployments. I generally try to follow the "vendor everything" philosophy: http://ryan.mcgeary.org/2011/02/09/vendor-everything-still-applies/ http://ryan.mcgeary.org/2011/02/09/vendor-everything-still-a...
- redeemedfadi 14y ago
  I couldn't get through this whole article due to the dude in the corner staring at me... :-/
  - Legion 14y ago
    Not that this line of discussion is particularly constructive, but I agree. I'm not sure why someone would think a giant head staring at the reader is a good idea, but it's not. On a 27" monitor, it's almost like a child sized head right up in your face. I can only imagine it being even worse on bigger screens.
  - xanido 14y ago
    document.getElementById('mugshot').remove()

freshfruit 14y ago

If you are not updating gems, does it hurt to continue deploying with a custom buildpack? Heroku shouldn't repull gems if the gemspec hasn't been altered. Is that logic correct? Does anybody have another suggestion for safely working around this issue? I don't have a clear sense for how long this will take to resolve and don't wish to slow down our release pace too much.

kawsper 14y ago
If you trust your buildpack (and its contents), and it contains all the gems necessary, you are safe to deploy.

benmmurphy 14y ago

This is the responsible thing to do. Going through the gems and verifying they aren't compromised is a lot of work. We should be thankful for all the effort the rubygems maintainers and other volunteers are putting into cleaning up this mess.

akavi 14y ago

A tangent, but I always thought "YAML" was pronounced /'jæm.ḷ/, however the post's use of "an YAML" suggests it's actually pronounced /waɪ.eɪ.ɛm.ɛl/. Weird.

eric_the_read 14y ago
Don't worry too much about this. A lot of people haven't internalized the correct rule (use "an" before a vowel sound, not a vowel letter), and instead just use "an" before a vowel, regardless of the sound it makes. I've certainly never heard anything but /'jæm.ḷ/ in the wild.
deleted 14y ago
[deleted]
saurik 14y ago
You still wouldn't say "an why", as "w" isn't a vowel. If Anything, I can understand "an yamel" as more legitimate (as "y" is at least sort of a vowel).
- justsee 14y ago
  No, "an YAML" is grammatically incorrect [0]. It's "a YAML". What's more annoying: the odd (and wrong) belief that "An green apple" is grammatically correct. [1] [0] http://english.stackexchange.com/questions/1016/do-you-use-a-or-an-before-acronyms/11511#11511 http://english.stackexchange.com/questions/1016/do-you-use-a... [1] http://english.stackexchange.com/questions/152/when-should-i-use-a-vs-an/164#164 http://english.stackexchange.com/questions/152/when-should-i...
  - saurik 14y ago
    I didn't say that "an YAML" is correct; I was first correcting that "an YAML" would be correct if it were "an why aih ahm ell" (it wouldn't), and then went further into "if anything" land saying that "an yahmell" is at least something I can bring myself to say without feeling sick inside. ;P
14y ago

achalkley 14y ago

Heroku a great.

pardner 14y ago

Seems odd, however, that status.heroku.com lists the issue only on the development side, suggesting production apps are not affected? http://screencast.com/t/L36Hpx5dx http://screencast.com/t/L36Hpx5dx

pvh 14y ago
Running apps are not affected, but your ability to do development is. The threat vector affects app compilation, not app execution/scaling, which doesn't touch the tainted rubygems.org repositories.
- pardner 14y ago
  I don't think that is correct. AFAIK the 'development' side of the heroku status panel is not related to developing, per se, it's the status for apps that are not running on production-level resources... for example, single-dyno apps, or apps not running with production flavor of database.
  - bgentry 14y ago
    You are incorrect: https://devcenter.heroku.com/articles/heroku-status#status-information https://devcenter.heroku.com/articles/heroku-status#status-i... I agree that the production/development split is not entirely clear without additional explanation. We've spent a lot of time thinking about how to communicate these things and have so far not come up with a way that we feel better describes the issues at hand.
    - pardner 14y ago
      Thx for correcting that... have used that screen a million times without realizing they lumped dev apps and prod+dev workflow together. 3 columns would eliminate any confusion (Production Apps, Development Apps, Development Workflow) but might not look at clean.

aneth4 14y ago

This is why you vendor gems, and the current accepted practice of not vendoring gems is dead wrong. I've been chastised before in rails irc for this. I strongly believe the source of all depndencies possible should be in your repository.

delano 14y ago
That's how Maven works but to a fault. The huge dependency repo tends to bog down deployments/releases. Today I use bundler but my solution back in the (java) day was to separate code releases from dependency releases (separate directories basically). e.g. /path/2/project /path/2/project-resources
pmahoney 14y ago
> source of all dependencies possible should be in your repository How far do you go? Do you include libxml for building nokogiri? Heck, do you include libc and gcc for building any gem with a C extension? Coming from Java, Maven and something like Nexus Sonatype make it easy (for certain values of "easy") to run a proxy repository. The equivalent of all "gem install <some_gem>" goes through the proxy, which continues to serve gems even if the original source goes away. I don't particularly like the inclusion of dependencies in a repository. Is this a custom version "some guy" long gone from the company created three years ago? Can I safely upgrade it to get security fix <X>? I suppose similar questions arise no matter the source... This is reason why you cryptographically sign your gems before publishing them. I (unfortunately) had not known this was supported by RubyGems, but it is: http://docs.rubygems.org/read/chapter/21 http://docs.rubygems.org/read/chapter/21 But I'll bet very few gems are signed. Rails does not appear to be: $ gem install rails -P HighSecurity ERROR: While executing gem ... (Gem::Exception) Unsigned gem

thinkbohemian 14y ago

Heroku engineer here: Ruby deploys are back online if you don't require any new gems, i.e. can deploy from the existing cache. We're still working on resolving the large problem with Ruby Gems.

pilif 14y ago

I'm surprised that rubygems.org of all places did not see fit to patch the vulnerability that's now known for multiple weeks and which has been declared to be incredibly dangerous and for which ready-made exploit kits exist. rubygems.org is a central distribution platform trusted by tons and tons of projects. As such, that site is the one site you probably do not ever want compromized. Imagine the damage an attacker could deal by uploading backdoored versions of various popular gems. I know - applying security patches is time-consuming and we are all afraid of breakage. But the moment rubygems.org stepped up to be a semi-official central distribution point for gems, I would have hoped they also took on the responsibility that goes along with that. If this was some new unknown 0day exploit, I would be much more understanding, but this was known to exist, known to be dangerous, known to be exploited.

steveklabnik 14y ago
This is not that vulnerability. Their Rails is just fine. This is related to the fact that gems use YAML to store their metadata.
- 1qaz2wsx3edc 14y ago
  Semi related CVEs aside (CVE-2013-0333 and CVE-2013-0156): YAML's security problems have been known for years by the community. YAML aside, don't trust user input. This is egg on rubygems's face and the ruby community. I don't think http://rubycentral.org/ http://rubycentral.org/ has full-time staff for rubygems either. Gems, specifically should be signed. They are not, this type of exploit will continue to happen, hell, remember when github screwed up ssh keys? Who knows what's in the ecosystem. TL;DR; Ruby's security ecosystem is butter. Disclaimer: I love ruby and use it daily. It was two critical problems IMO: unsigned code and GIL. Yes, GIL. I'm looking at you ruby-core.
  - charliesome 14y ago
    The GIL is not really a problem. There are far more pressing issues with MRI, and many core developers are working hard to solve these.

djangolover301 14y ago

lmao Ruby and RoR shame PHP in terms of security flaws. Making you unknowingly write security holes, ridiculous flaws discovered on daily/weekly basis, package management hacked etc I have never seen as ridiculous holes as RoR even in the CodeIgniter framework. Where are the RoR-haters when we need them?

reybango 14y ago

Being totally new to RoR (trying to learn it), I'm trying to get my head around the scope of this. When did the compromise happen? Was it compromised yesterday or only found out yesterday? I have default gems installed on my system and haven't updated anything since the big Rails security issue that was reported a bit ago. It'd be great to get some guidance on what to do.