4 ms·
I've had deploys break because gems were removed from RubyGems.org. Using CI as a gate isn't sufficient because a gem can be removed in the window between a com
by nirvdrum 11y ago
I've had deploys break because gems were removed from RubyGems.org. Using CI as a gate isn't sufficient because a gem can be removed in the window between a completed CI run and the deploy. Making it harder, generally if you have a gem installed locally you won't notice the problem until you need to install it on a fresh system.
It used to be the case that a yanked gem was simply removed from the gem index, but the file was still published, so you'd still be able to install it. Now the gems are deleted, eliminating that option.
So, other systems are definitely susceptible and it does burn people. I just don't think we've seen an issue on this scale yet. The largest I've hit have been when the twitter gem was yanked in advance of Twitter's shutdown, a fog-XYZ gem being yanked as new versions were released, and at least one yanked version of net-ssh.
The only really effective way to avoid this is to mirror all your dependencies yourself (and not honor upstream yanks) or vendor all your gems. I know of exceedingly few people doing this.
I've yet to encounter a legitimate cause for a yanked gem. But I worry more that a bad actor gains credentials to a high profile account and just wreaks havoc on the ecosystem. On that note, I have found and reported users that've committed their RubyGems.org credentials to GitHub in a "dotfiles" repository.
- krakensden 11y agoDear Everyone Doing Devops: do not build gems on your production servers. Build a tarball, test the tarball through dev/staging, deploy the tarball to production. For any value of tarball- literal tarball, deb/rpm (actually CPIO under the hood, but who cares), prebaked AMI (not tar at all), docker image (tarball of JSON + nested tarballs. Seriously, it's tar all the way down). Everything will be better. Roll forwards. Roll backs. Speed. Attack surface. Reliance on 3rd party services that will rate limit you hard.
- softawre 11y agohttp://martinfowler.com/bliki/ImmutableServer.html http://martinfowler.com/bliki/ImmutableServer.html
- nirvdrum 11y agoYour point is well-taken. Broken deploys are just the extreme case. You'd run into the same problem if two devs on the team just happened to install libraries at different times. One would get it, one would not. And you don't get a remotely helpful error message in that case. So, you're back to mirroring or vendoring everything you need anyway. Arguably that's better, but why even bother with a centralized package management system at all then?