12 ms·
Rubygems.org compromised
- benmanns 14y agoThe current title is "Rubygems.org compromised." Isn't this just a gem that exploits the YAML bug, not an exploit on RubyGems.org? I believe this submission needs a title change. Edit: I am incorrect. It seems that this uploaded gem was used to compromise RubyGems.org.
- veloper 14y agoI think you're right, the title is inaccurate. Running `$ bundle update` will not inject this into your app. You'd have to intentionally add `gem "exploit", "~> 22.31.31"` to your Gemfile.
- kawsper 14y agoCouldn't all gems fetched from Rubygems be infected?
- timdorr 14y agoYes, but the attacker would need to inject code into every .gem file that's pushed to rubygems.org. Because that source code would be highly visible, it would be simple to figure out what they're trying to do. A more hidden attempt would be to simply alter the existing .gem files on S3. Once they have the access tokens from the configuration YAML files, they can do that from anywhere, not just the rubygems.org host. That keeps their activity under the radar and is slightly harder to track. In that case, only newly installed gems in your system that were uploaded to rubygems.org before this incident would be at risk. If you run bundle update and pull in gems built from today onward, there is not as much to worry about. If you're updating gems and the bundle is pulling in gems from a few days ago, then you are at high risk.
- nirvdrum 14y agoI'm not disputing anything here, but I just want to emphasize that if they have the S3 credentials, replacing any gem is trivial. And they wouldn't need to replace every gem. Picking the 10 most popular ones would have pretty big fallout. And, if like most people you don't verify your installed gems (not that you really can if they're not signed), you're going to have a very bad day. The "tracking" aspect is a situation where it doesn't matter because at that point, you're hosed anyway. Also, walking S3 is not particularly fast. So if the plan is to walk all gem files to verify them, expect that to take a while. Hopefully they have bucket logging enabled.
- benmmurphy 14y agoit looks like someone has tried to steal rubygems.org db and other credentials. bundle update is now dangerous because you might get trojaned files.
- benmanns 14y agoBut don't applications have to explicitly install this gem? If not, then I am incorrect.
- bradleyland 14y agoWe don't know the extent to which Rubygems was compromised yet. In a scenario like this, it's best to err on the safe side and avoid trusting the source for the time being.
- bittersweet 14y agoQrush, one of the guys behind Rubygems tweeted that the credentials of every yml file ended up on pastie: https://twitter.com/qrush/status/296638346283868160 https://twitter.com/qrush/status/296638346283868160 > Someone posted http://rubygems.org http://rubygems.org 's config/*.yml to pastie. Didn't include the S3 bucket secret key, but going to reset everything anyway.
- landr0id 14y agoMind explaining to me how this shows that rubygems was compromised? It looks to me like someone just uploaded a gem called "exploit" which takes advantage of a recent exploit.
- JohnBooty 14y agoI'd like to know this as well.
- kawsper 14y agoIt has been put into maintenance mode, so that is what you see now, until this is resolved.
- vinhboy 14y agoIt kind of says something when people can't tell the difference between your maintenance mode page and a hacked page...
- kawsper 14y agoShould he design a custom "We have been compromised" page before deactivating the site? It was put in maintenance to stop pushing possibly infected gems to users.
- DannyBee 14y agoIf https://twitter.com/qrush/status/296638346283868160 https://twitter.com/qrush/status/296638346283868160 is accurate (qrush is one of the rubygems maintainers), it worked.
- dantillberg 14y agoThe polite thing to do first would be to notify the maintainers of rubygems.org - not post the link to HN.
- benmmurphy 14y agoI didn't know who maintainers were :( I think i've notified one now.
- WestCoastJustin 14y ago$ whois Rubygems.org Tech Email: ...
- robotmay 14y agoIn Ben's defence, I doubt he's the only one who wouldn't have thought to check the whois data. I wouldn't have, for a start. There should have been an obvious contact email linked on the site itself for issues, not just a Twitter account.
- DannyBee 14y agoThis probably would be the first time in the past 5 years there was a non-spamtrap email address in whois
- WestCoastJustin 14y agoI am not sure what this is getting down voted? This was not a dig at Ben. I was trying to be helpful, so they he could get in touch with the admin contact?!
- yakiv 14y agoDon't sweat it. Getting the word as best you could was a good thing to do.
- tptacek 14y agoThere's evidence that Rubygems.org has been compromised and you think the best course of action might be to keep that secret for a little while?
- evanphx 14y agoIt's disappointing to see this show up on HN without contact myself or the other rubygems.org admins. We've put rubygems.org into maintenance mode until the issue is fixed.
- tptacek 14y agoWhat I saw was Ben twerping to Ruby people asking who to contact about this first, then posting to HN when he couldn't find the right person. Can I ask, where's the security page on Rubygems.org that tells people who to alert when stuff like this happens? I'm trying to find it on Archive.org and drawing a blank.
- benmmurphy 14y agoI looked for contact page first then panicked. I should have used whois :(
- tptacek 14y agoIf the whole world is pulling backdoored gems off a compromised Rubygems.org, panic seems like a reasonable first reaction. This isn't the Rails bug; there was nothing to be gained here from secrecy.
- benmmurphy 14y agoI became aware of this problem this morning when it was discussed on a public github issue. I think the problem has been known for a couple of weeks. I checked latest rack and active_support gem (would seem to be good targets...) to see if they had been backdoored but they were clean. Not sure if these gems are still clean or if any others have been compromised.
- vinhboy 14y agoconsider me a fan of your work!
- 14y ago
- Titanous 14y agoHere's what the payload looked like: https://gist.github.com/3e4829f79dbd1be11295 https://gist.github.com/3e4829f79dbd1be11295
- timdorr 14y agoDid you catch where was this payload injected? Did they have write access to the rubygems.org source? It would seem if they did, they didn't need to write this script at all and could simply read the configuration files directly.
- WestCoastJustin 14y agoAfter reading through the github.com issue posted in the comments, I get the impression this was targeted attack at rubygems.org (and not the user install base). There has been a tweet, also posted in the comments, suggesting that this was successful.
- Titanous 14y agoIt was in the gem metadata, which is YAML: https://gist.github.com/d891e876c53e55bf0920 https://gist.github.com/d891e876c53e55bf0920
- WestCoastJustin 14y agoLooks like this would create a pastie.org entry looking something like the this: e193256c9337b50b197f040e762dafcc745a66297c9db47ac30395d8022f94a8 encrypted "config/database.yml", "config/librato.yml", "config/newrelic.yml", "config/rubygems.yml" Searching pastie.org, I don't see any such entries.
- kawsper 14y agoThese params 'paste[authorization]' => 'burger', 'paste[access_key]' => '', is just hardcoded pastie.org arguments that gives them somekind of safety against the stupidest attempts. Maybe the post was removed, or the hash is different than the original so not all can find it.
- bttf 14y agorubygems.org is back up
- wereHamster 14y agoWhen will people start signing their gems? And gem/bundler warn if the signature is not present or invalid?
- mratzloff 14y agoI'd wager that most gem developers don't understand how it works, and there are basically no resources out there on it. Here's the chapter on it in the RubyGems manual: http://docs.rubygems.org/read/chapter/21 http://docs.rubygems.org/read/chapter/21 And that's all that I can find. Try to find something else, like a blog tutorial.
- Xylakant 14y agoEven if most gem developers would understand, there's still a problem with the trust model: If you choose "high security", you cannot install a single gem without signature - so things won't work unless _all_ authors sign their gems. Gem signing would have to be a requirement, similar to packages on ubuntu's ppa. If you choose a lower security setting, a gem may just be repackaged and signed again by the attacker or the signature removed.
- mratzloff 14y agoA solution that doesn't solve 100% of the problem is still not a bad solution. The most popular gems would all be signed, thereby dramatically reducing attack vectors.
- Xylakant 14y agoNo, as I said, this doesn't help. You would need to sign the most popular gems and all their dependencies and the dependencies dependencies. Given that a pretty much standard rails app can easily pull in hundred or more gems that's quite a bit of work. It's doable, but I only see that coming if for example rubygems requires a key and gem signatures. Then there's still the unsolved problem of providing key trust - the current implementation relies on self-signed certificates.
- adelevie 14y agoIs there anything gem authors should be doing?
- timdorr 14y agoSign your gems before pushing them: http://docs.rubygems.org/read/chapter/21 http://docs.rubygems.org/read/chapter/21
- kgo 14y agoThe trouble is self-signed X.509 certificates are pretty worthless. X.509 certs are built around the concept that they're issued by a trusted provider. I tried to get together a proposal to introduce OpenPGP signed gems and wrote some proof-of-concept code so there could be a proper Web-of-Trust for rubygems: https://github.com/grant-olson/rubygems-openpgp https://github.com/grant-olson/rubygems-openpgp But the developers were resistant (and understandably so) to having gpg as a dependency for gem verification, and I didn't trust myself to write the crypto and a decent keyring validation algorithm from scratch.
- Xylakant 14y agoEven with pgp you'd need a whole lot of infrastructure. Where to get the public key from etc. Ubuntu requires pgp keys for their ppa and provides a keyserver. Since a lot of gems seem to use github nowadays, maybe an option to provide a pgp key on github would be great. That would allow to trust at least the most interesting projects - rails, db-layers, rack, etc. which could limit the fallout of an attack against rubygems.org
- kgo 14y agoMy proposal was that rubygems would create a simulated certificate authority that operates similarly to the PGP Global Directory, which would allow registered users to have their keys signed by an authorized RubyGems.org signing key. (This is covered on the bottom of the page on the link above.) In addition more active developers who go to conferences and what not would do all the proper WoT stuff to ensure that the RubyGems.org key being used was indeed the authentic one, and that gems in their circle of friends/co-workers were signed by the correct keys.
- sch1zo 14y agocan someone explain what happened here to those like me who are a little late. seems like the issue is resolved already or at least temporarily fixed. All I can figure out from the url and the comments here is that there was a gem called exploit and that somehow the config files where posted to pasties.org
- runeb 14y agoThe exploit was written by @postmodern_mod3: https://twitter.com/judofyr/status/296649167189725184 https://twitter.com/judofyr/status/296649167189725184 Edit: At least a PoC. How someone other than him (since hes denying compromising rubygems) found his PoC and wrote the actual payload and pushed the exploit gem is unclear.
- judofyr 14y agoShort timeline: - 1 week ago: blambeau reported the issue to the rubygems people: https://github.com/tenderlove/psych/issues/119#issuecomment-12875715 https://github.com/tenderlove/psych/issues/119#issuecomment-... - Postmodern wrote a PoC - https://gist.github.com/4674219 https://gist.github.com/4674219 - Postmodern: "I posted it in a private chat room. Guess someone took it for a ride." - https://twitter.com/postmodern_mod3/status/296651922402840576 https://twitter.com/postmodern_mod3/status/29665192240284057... - A gem called `exploit-36.44.16` was pushed to RubyGems. This contains a payload that only sends `uname -r` to pastie. - A gem called `exploit-22.31.31` was pushed to RubyGems. This contains a payload that posted the config/database.yml to pastie. This gem was removed by the rubygems admins when they discovered it - http://news.ycombinator.com/item?id=5140109 http://news.ycombinator.com/item?id=5140109 & https://gist.github.com/d891e876c53e55bf0920 https://gist.github.com/d891e876c53e55bf0920 (that's the payload) Other exploit-gems that was pushed (but we don't know what did): - 16.17.49 - https://twitter.com/rubygems/status/296618422702309377 https://twitter.com/rubygems/status/296618422702309377 - 20.22.1 - https://twitter.com/rubygems/status/296617610622144512 https://twitter.com/rubygems/status/296617610622144512 - 12.5.4 - https://twitter.com/rubygems/status/296540884781109248 https://twitter.com/rubygems/status/296540884781109248 - 7.27.42 - https://twitter.com/rubygems/status/296537952320884736 https://twitter.com/rubygems/status/296537952320884736 My conclusion based on the current facts: 1 week is too long for such a severe security issue. A proof-of-concept gem seems to be a logical step to "force" rubygems.org to fix this issue. However, publishing database.yml on pastie is TOTALLY 100% NOT COOL, and turns this from "whitehat who wants to help" to "blackhat that wants to destroy (or doesn't understand social norms)".
- benmmurphy 14y agothis is the earliest version of exploit i've been able to find in the twitter timeline: https://twitter.com/rubygems/status/296537952320884736 https://twitter.com/rubygems/status/296537952320884736 (12:38) 7.27.42 ~ 7 hours ago :( but there was quite a few exploits pushed
- qrush 14y agoHi everyone - A full post-mortem is coming, but here's what happened from my perspective. I got mentioned on twitter about this thread while I was on the bus. I asked @evanphx to put the site into maintenance mode immediately. Once I got on wifi, I downloaded the "exploit" gems and found that they used the Psych hack to post config/initializers/* and config/*.yml to Pastie. Luckily, none of our API keys were actually in those files - our config is a bit different since you can still run RubyGems.org on Heroku, which needs to put secret keys in the ENV. I deleted the exploit gems permanently along with the throwaway account that posted them...that's why this post's URL is a 404. I've reset our S3 keys just to be sure and any other keys we had on our production site. From what we know right now, no other changes on our S3 bucket have taken place, and we're going to check the logs to make sure. Once we get a real fix for this issue out, pushing gems will be enabled again. Just a general PSA - Please, if you find an issue like this, be nice. Tell the maintainers privately. Don't post to Reddit, HN, or a public Gist. RubyGems.org is completely volunteer run. No one gets paid to work on it. Thanks for your patience everyone.
- blambeau 14y ago> Please, if you find an issue like this, be nice. Tell the maintainers privately. Nick. I dont support today's PoC, I don't, really. But I told you. Twice in a week. I feel deeply sad now.
- munin 14y agosomething to think about here is, how long has this bug been in rubygems.org? how long would someone have had to modify arbitrary gems? fun times!
- zalambar 14y agoWe have a great many people relying on a volunteer run project here. It would be great to have someone sponsored to spend time maintaining the site. Unfortunately I don't see any way to help fund such an effort. Is there an officially supported way to donate to RubyGems? Can we create a pool to help cover the cost of maintaining this resource?
- kawsper 14y agoThere is some larger organizations using Ruby. So not hiring a guy full-time, but they might be interested in chipping in, to get a gem-repo that they can trust.
- davexriess 14y agoWhat options do we have to work around this while the issue is being fully diagnosed? I'm considering re-installing my gems directly from the github repos. Thoughts?
- pjungwir 14y agoI also get an "invalid SSL certificate" warning when I try to visit rubygems.org. Here is the message in Firefox: rubygems.org uses an invalid security certificate. The certificate is not trusted because the issuer certificate is unknown. (Error code: sec_error_unknown_issuer) Any chance this is related?
- ilikepi 14y agoAccording to Certificate Patrol, I'm getting the same cert I first received on 2012-12-11: not valid before: 2012-03-03 serial: 00:E6:99:C4:B4:04:ED:07:D2:E5:BC:42:29:09:B8:17:D0 sha1 fingerprint: BD:2E:60:69:49:99:53:29:BD:E9:24:12:95:E1:5E:65:E0:64:79:2A
- jordanthoms 14y agoWhy is it that all these parsing issues with yaml and xml were ignored before it all blew up recently? Input parsing is usually the most venerable part of any application, and it's strange that these issues were ignored for so long. These are not super complicated exploits, they are using the _expected_ behavior of the yaml parser in terms of it constructing arbitrary objects.
- blambeau 14y agoBecause when you write "Rails vulnerabilities are not Rails'" (http://www.revision-zero.org/rails-vulnerabilities-are-not-rails http://www.revision-zero.org/rails-vulnerabilities-are-not-r...), most people respond "yes it is".
- jordanthoms 14y agoWell, a unsafe parser (which was designed only for parsing trusted input) was being used by lots of people (including Rails) for parsing untrusted input as if it was a safe parser. You can debate about whether there should be a safe parser for YAML, but that's a separate issue.
- blambeau 14y agoIs "which was designed only for parsing trusted input" written anywhere in Psych doc? Psych is shipped with Ruby, is it written in Ruby doc maybe?
- jordanthoms 14y agoAgreed, there should be a clear warning - in fact, the load method should be renamed unsafe_load. The root cause of this is probably unclear documentation and misunderstandings between the users and authors of psych.
- ruby_vuln_4eva 14y agoWhere are the adults? This is absolute amateurism. Why haven't Heroku or GitHub or some actual professionals in the Ruby community, stepped up to ensure that this kind of amateurism doesn't damage their ecosystem? [Throwaway as I'm a small-fish in the Ruby community. I expect I'll get to use this account again soon though]
- pifflesnort 14y agoThis is exactly what I'm wondering. Are there adults in the Ruby community? The vulnerability itself is simply ridiculous: https://github.com/tenderlove/psych/issues/119 https://github.com/tenderlove/psych/issues/119 The response from the Gem folks has betrayed a serious lack of experience. Has Rails been such an insular and young community for so long that there are truly no qualified engineers around with experience in this level of professional software development?
- dbenhur 14y agoThe problem is any YAML loader which supports tags that can name arbitrary classes and have existent classes whose initialization may `eval` the supplied values. Ruby is very publicly hurting, but equivalent vulnerabilities exist in other runtimes, such as Perl via [YAML::Syck](http://search.cpan.org/~toddr/YAML-Syck-1.22/lib/YAML/Syck.pm http://search.cpan.org/~toddr/YAML-Syck-1.22/lib/YAML/Syck.p...) or Python via [PyYAML](http://pyyaml.org/wiki/PyYAML http://pyyaml.org/wiki/PyYAML) and [LibYAML](http://pyyaml.org/wiki/LibYAML http://pyyaml.org/wiki/LibYAML). If your language can accept and load untrusted yaml and your language has `eval`, you're probably fucked and just don't know it yet.
- batiste 14y agoThe problem is that YAML is used by Rails & Co as a parser for untrusted user-provided data. It does not matter if there is buggy implementation in Python/Perl if nobody use them. If other framework were using YAML in this fashion, yesterday will be the time to look for security holes in those implementations.
- davebuster 14y agoWhy aren't gems protected by a trust chain all the way up like Debian does? Gems aren't even required to be cryptographically signed by the developer and from what I can tell are just thrown over http without any authentication. This seems terrible.
- rufugee 14y agoSo what source should app developers use until this is cleaned up? gemcutter.org and rubyforge.org?
- static_typed 14y agoOh the timing. So, there was a big meeting, where some of the devs pleaded the case that the recent wave of security issues in the Ruby world were a tipping point, and that we were over it. They presented a good case, and I think the architects were close to permitting the usage of Ruby and Rails in their current cases. Then we get an interruption, as a couple of projects had updated their gems today, and there was a concern about overall security of the apps. IT Security required the apps were offlined, and the green light was given to the PHP and Python guys in house (and a few retained developers) to start the process of rewriting the apps. It will be painful, but, I think, the business has lost it's appetite for the constant Ruby idea of "Let's run with Yaml", before firstly "Let's walk safely with Yaml".
- PommeDeTerre 14y agoI'm glad to hear that at least one organization has done the right thing, and started the process of moving completely away from Ruby and Ruby on Rails. It's just the responsible and sane thing to do, given how the software and the community apparently can't be trusted.
- sergiotapia 14y agoI see where you're coming from, it's a shame that so many bugs are available in Rails.
- railsblob 14y agoBugs is evrywher evn in javacs code. Rubby haz some of th finnest midns in comptuers and wil rzie agani.
- pifflesnort 14y agoIt's a tipping point, but not of the kind that they thought.
- Xylakant 14y agoI can understand it's bad timing, but the reaction seems ad-hoc. AFAIK neither python eggs nor PHPs packagist repository sign their packages, so in theory they're open to the same attack vector [1]. Judging from the docs, composer doesn't even allow signed packages. A better reaction would have been to fix the process: Allow only signed gems or cache and audit the packages you install. You don't need to depend on rubygems. [1] Even OS package repositories or mirrors were broken into. I faintly remember a break-in where debian(?) had to check all their mirrors.
- ErikRogneby 14y agostatus updates posted here: http://status.rubygems.org/ http://status.rubygems.org/
- prusswan 14y agoI never knew the gems were not subjected to version control at all on the server. Can't they put everything into a git repository somewhere?
- eric970 14y agoFor those that don't know how this exploit worked, I think the "exploit" gem description provides a pretty good explanation: 'A Proof-of-Concept PoC gem that exploits a vulnerability in the Psych YAML parser, which allows the #[]= method to be called on arbitrary Objects. If the #[]= method later calls eval() with the given arguments, this allows for arbitrary execution of code.'
- blackxored 14y agoJeeze, bad things happen (or could) to the greatest services ;) I'm late to the discussion, didn't have time to lurk around, nevertheless what I can agree the most is about gem signing. A gem is a package of software, and every single package distributor makes the author sign their work (think linux packages, being a Debian/Ubuntu developer I can tell you that if you don't get your package signed, it won't get uploaded, period, your key in some cases (Debian) needs to be verified in person to build a circle of trust. I think that besides the overhead (initially, I'm sure people will contribute scripts or integrate this so that the developer won't notice) this is really a needed step you guys should take.
- OffTheRails 14y agoDamn it! What a mess! This is some depressing press for the Ruby community.
- reybango 14y agoBeing totally new to RoR (trying to learn it), I'm trying to get my head around the scope of this. When did the compromise happen? Was it compromised yesterday or only found out yesterday? I have default gems installed on my system and haven't updated anything since the last big Rails security issue that was reported a bit ago. It'd be great to get some guidance on what to do.
- gravis 14y agoI'm sorry to mention some RG projects are still "potentially" dangerous: https://gemnasium.com/rubygems https://gemnasium.com/rubygems We'll do our best to help determining if these projects really need to be updated asap!