8 ms·
I just can't feel sorry for folks when I see comments like this one: > This destroyed 3 production server after a single deploy! I do think that the developer
by jlgaddis 9y ago
I just can't feel sorry for folks when I see comments like this one:
> This destroyed 3 production server after a single deploy!
I do think that the developers have a duty to do some testing of their software before putting out releases/updates. However, users also have a duty to perform sufficient testing before they push new versions to their production environments.
In my opinion, it's kinda like losing data because you didn't make and/or test your backups. It's a really crappy way to have to learn a lesson but at least they've finally learned it -- and if they haven't, well, then maybe they will the next time it happens.
- jguimont 9y agoI am the one who reported this ;) In fact that was a single production server that I tried to reinstall 3 times before catching it was not really one of the commits that was doing this. No data was lost or connectivity (as long as you do not reboot it), you just lose any ssh connection/login. Should I have done this on a staging server? Sure, but that does not change the fact that I would have had to rebuild the whole server there too. It is not expected that updating npm will kill the complete system it is on... It would be expected to have some deploy failure of some sort. As previously noted, `npm update -g npm` pulls in version 5.7.0. Version 5.6 is still the latest but for some obscure reason if you have thisupdate anywhere in your deploy script you are screwed.
- diggan 9y agoThe real fix is to not run npm with sudo. Why would you do that in the first place? npm runs install-scripts when you fetch packages, so you basically open up root access for all the packages you download.
- breatheoften 9y agoThis 1000 times. Running npm as sudo is a terrible terrible idea. I remember creating a slack channel in our team called 'never run npm with sudo' and ranting in dramatic fashion to try and overcome the effect of the printed advice which npm used to output in most failure situations to 'try re-running the command with sudo.' This tended to cause developers new to the ecosystem to re-run the command with sudo and create lots of problems for themselves -- in addition to being an extremely bad security practice. Honestly -- I think npm should be updated to exit without doing anything if it detects its run with root privileges ...
- amelius 9y agoTrue, but then npm can theoretically still e.g. install a keyboard logger for the current user, so you should remember to never run sudo as that user again. Of course, that's doable, but probably too much to ask from most users.
- deleted 9y ago[deleted]
- paulddraper 9y agoHow is this different than running apt or yum or pacman or most other package managers as sudo? Somebody has to install system software.
- diggan 9y agonpm is not for managing system software and has not been developed as such. It's a javascript package manager. apt and pacman (and probably yum but never used so can't speak about it) have active maintainers for most packages and the mirrors are well taken care of. npm is basically a giant array that anyone can add package to. I'm using them both accordingly.
- paulddraper 9y ago> npm is not for managing system software Debatable but irrelevant. I'm not saying npm is a good or bad system package manager, just that running arbitrary scripts for requested packages and their dependencies is hardly unique. It's oblivious to single out npm as a package manager that allows you to be pwned by packages in whatever repo you pull from.
- diggan 9y agoIt's not unique, but apt/pacman does not run arbitrary scripts. It runs what has been reviewed by others while npm packages are often not reviewed by anyone except the author, that's the difference.
- paulddraper 9y agoSo you're saying it's not the tool or packaging format. It's the curation of the repositories that npm/apt/pacman users tend to consume.
- jguimont 9y agoI believe that node is installed with npm and both are installed in system directories. I think they should by default point the npm global directory to the user dir and not a system dir.
- dagmx 9y agoBut you should have still tried it on the Staging server to begin with. That's the responsible method. The fact that you'd have to rebuild your staging server is exactly why you should have tested it there. Sure NPM shouldn't have broken this but any number of things can cause issues during deployment and it's your job to check for them before pushing it out
- jlgaddis 9y ago> It is not expected that updating npm will kill the complete system it is on... I don't disagree with you at all on that. The reality is, however, that sometimes "shit happens". I'm more of a sysadmin than a developer and I learned many, many years ago that even the smallest little updates can "go wrong" and take the rest of the system with it. After getting burned a few times, even a baby will learn to stop touching a hot stove. > Should I have done this on a staging server? Sure, but that does not change the fact that I would have had to rebuild the whole server there too. Yes, but in that case your production servers would still be humming along just fine, no?
- mbrumlow 9y ago> It is not expected that updating npm will kill the complete system it is on... Yeah, but that is why you test your deployments BEFORE deploying them. Hell would be had had any developer at my company ran any such command on a production sever. The notion of even running a command at the terminal on a production server is even scary. Things like should be done on build servers which are in general throwaway. Your build server should produce a artifact that can then be deployed to your staging servers and if all is well THEN productions servers. npm is a build tool and should not be installed or ran on production servers -- for many reasons more than just stupid stuff like this.
- Silhouette 9y agoThis response annoys me, because it's essentially victim-blaming. Yes, ideally you have some automation and staging in your server setup. We're grown-ups. We understand this. But it ignores many other dangerous possibilities here. Not everyone is blessed with working in a mature, well-funded environment full of experts. Maybe we're talking about a new or small organisation that simply doesn't have the resources and/or knowledge to isolate things with containers or VMs and related admin tools. Maybe even taking out a staging server is still going to waste significant time resetting everything, blocking other development/deployment jobs in the meantime. Maybe we're not talking about a server at all, but a developer's personal development workstation where they just use NPM to install a few Node-based tools. It's all very well saying npm shouldn't be run on production servers, but that doesn't really address the fundamental problem. Do we also ban system package managers, and say the only way to deploy anything is via some sort of imaging tool? What if there's an equivalent screw-up in that orchestration tool and it bricks all 100 servers at once?
- mbrumlow 9y agoI think there is a big difference here. I am responding to a reply that wanted remove any responsibility from running such non-since as npm on a productions server -- note I did not reply to "crap I hosed my stuff". But lets take a closer look at your comments. > Not everyone is blessed with working in a mature, well-funded environment full of experts. Maybe we're talking about a new or small organisation that simply doesn't have the resources and/or knowledge to isolate things with containers or VMs and related admin tools. These are not excuses to knowing your trade. And the size and funding of your environment should not stop you from practicing your trade well. > Maybe even taking out a staging server is still going to waste significant time resetting everything, blocking other development/deployment jobs in the meantime. I fundamentally disagree. Having a stating environment will always cut cost and can't EVER be considered to "waste significant time". It can only save time and improve your product. Its these types of attitudes that result in your service going down and lost of real revenue and ultimately the failure of the project. Taking time to setup proper staging environments always pays back in spades. > Maybe we're not talking about a server at all, but a developer's personal development workstation where they just use NPM to install a few Node-based tools. Yeah, maybe we are talking about a developers personal station, nope, we are talking about production servers. Nuking a developers station is not even on the same scale as nuking a production system. And had somebody complained about nuking their dev environment my reply would have been about not running tools as root. > Maybe we're not talking about a server at all, but a developer's personal development workstation where they just use NPM to install a few Node-based tools. Again, I am okay with a developers system being nuked, at least it was not production! > It's all very well saying npm shouldn't be run on production servers, but that doesn't really address the fundamental problem. Do we also ban system package managers, and say the only way to deploy anything is via some sort of imaging tool? What if there's an equivalent screw-up in that orchestration tool and it bricks all 100 servers at once? I would not advise running system package managers on production servers either -- not unless your staging environment had passed such test first. But that being said I am a big fan of fresh install and migrate -- where the migration code is something I own and can test to ensure it works before using it. If you have 100 servers then you should have the resources to handle setting up testing environments to ensure your production rolls out. You should also not update 100 servers at the same time. Production is production is production is production is production! You don't run things the first time ever in production. If you want your product, company whatever to succeeded then there really is NO excuse for you not to have good practices when building and deploying software. You can come up with 2^64 what if's but if you are running something that for the first time and it nukes your system you are at fault. Things like testing environments or staging environments were created to just dream about, and talk about when things go bad with deploying directly to production. These things came about because they bring real value to a project. The notion that these things are a waste, or cost too much just non-since. Anybody in this industry of deploying software to servers needs to stand-up to these ideas that these good practices are too costly. These are the ideas and notions I expect from executive teams who have never coded a line of code, the accountants trying to save money, and managers who only care about the next quarter. I don't expect to find these ideas on sites like HN or from peers in the industry, but when I do I think it is important to take a hard line and not allow the notion of bad programming, and deployment practices to be unmet with rebuke for fear of hurting somebodies feelings. So while I clearly towed a hard line in this reply Silhouette, please do not take this as a personal rebuke or attack at you. I am upset with the ideas, and the notion hat we have to settle for less, and have results like production servers falling on their face when we as a industry already know the answers to the problem, and have the solutions to minimize downtime and provide truly awesome software to others.
- nolok 9y ago> It is not expected that updating npm will kill the complete system it is on... It would be expected to have some deploy failure of some sort. "It is not expected that" is the definition of unexpected behavior, which is the very reason why we use staging servers. So your message is essentially "I didn't use a server meant to check for unexpected behavior, because I didn't expect that behavior to happen". Well, yeah, that's the point. Also, I'm really not sure what your smiley is trying to convey here, and of all the possibilities I can't see one that's positive and contributive to the conversation. Really un-needed, please refrain from doing that.
- wejick 9y agoActually you're answering your own question, rebuilding staging server is like nothing compared to having issue on production. it's lesson and reminder to everyone out there, be careful. When dev env broken, alpha skipped, staging unusable, then test on production, you sure like to live on the hell.
- eropple 9y ago> Sure, but that does not change the fact that I would have had to rebuild the whole server there too. chef-client -z -r 'my-cookbook::npm_web_server' Obviously the behavior of NPM absolutely sucks and is a total mess here, but "I had to rebuild the server", in 2018, is not nearly the material complaint it was a decade ago. The tradeoff of using rapidly-evolving tools with minimal oversight from the people creating them is that sometimes stuff blows up and not even always for good reasons. It is incumbent upon you, as the recipient of this enormous, jaw-dropping raft of free stuff that occasionally explodes, to write code and operate your systems defensively. Part of which is implementing those systems to be repeatable and quickly reinstantiated. If you do not like this tradeoff, you have other options as well.
- fortythirteen 9y agoTo play devil's advocate, this was a dev version, without any UA or E2E testing, tagged as a major release. This after there have been major bugs for several releases. This isn't 0.7.0. Perhaps the problem is that the stability of the world's fastest growing development platform hangs on the implementation of best practices by a two person developer team. NPM needs to step up its game or we need to make something like yarn the standard.
- manigandham 9y agoYarn is already the standard in every large organization I know of that uses JS. It is still faster and more reliable than even the latest NPM versions so I dont see why anyone is waiting to switch.
- amelius 9y agoWhat strikes me as odd is that there are a lot of immature comments in that thread.
- deleted 9y ago[deleted]
- deleted 9y ago[deleted]
- mattbierner 9y agoMore than the clueless “+1”, I also have noticed an uptick of basically troll comments on GitHub. And while I understand that this particular bug is really bad, GitHub for me has always been about getting shit done and solving problems, a nice escape from the “normal” Internet with all its drama. The fact that people seem to be actively enjoying the drama and are even trying to fuel it—even if this is currently limited to super high profile bugs likes this—is concerning
- sametmax 9y agoThere is nothing odd about it. The JS community is today attracting most script kiddies. During the bubble, it was the same with PHP, and some of us were part of it. Youngsters must start to code somewhere.
- tzahola 9y agoI wish nobody will ever find the posts I’ve made on php.net as a teenager circa 2005
- kakwa_ 9y agoWhat you say is 100% true, but I would go one step further and not have npm installed on any production server. And it's not specific to npm, I would do the same with gem, pip, cpan, etc. Not to mention curl http://ex.io/install.sh http://ex.io/install.sh | sudo bash. Call me old school, but personally, I would avoid installing anything from language specific package managers. I would instead either build an rpm/deb package for every dependencies as a single package or if it's too complex, bundle the dependencies and the application in one package which deploys the bundle under /opt/. That way I only have one source to check in order to see what is installed on my systems. Also, rpm and dpkg tends to be far better at managing what is installed by each package, and far better at uninstalling everything during cleanups. Also, mixing a language specific package manager and a distribution package manager can have unforeseen side effects as the two can step on one another (for example, I ran into issues recently with a pip install python-consul overlapping with a yum install salt-minion as both of them download python-requests as a dependency).
- rootlocus 9y agoOr use a container.
- vhost- 9y agoThis is actually why I like working with C and Go. The compilers are on the build server and I just publish the resulting binary on a barebones production server after the build passes. In the past, working with Django, people always gave me shit for vendoring python package as part of the build, but it's to avoid having needlessly complex production environments and to remove the need for package managers.
- jmagoon 9y agoI like this idea. Can you elaborate a little bit on how/where you would fetch the gem package without the language based package manager, and how this is linked to system deps? For dynamic web application deploys the self-contained binary seems ideal, but I've fallen back on Ansible to properly configure the server dependencies, along with gems that may have system dependencies (e.g. psql).
- rad88 9y agoIf he'd tested it beforehand the comment might instead be "this destroyed my laptop". You have a responsibility to test before releasing to production, yes. But the amount of fucked up your program has to be for `sudo ___ --help` to wreck the operating system, the unexpectedness of that result... IMO attention should be focused here on the irresponsibility of the npm team, not their users.
- mrguyorama 9y agoThe fact that --help can actually DO something makes me quite upset. That means npm is causing side effects even before reading what the user wants, or is blatantly ignoring the user's request
- Humdeee 9y agoThis may be the only time I'll see accessing --help docs result in borderline malware (it's debatable that this 'sudo commandeering' was intentional by design, but as shortsighted as it gets...)
- zyztem 9y agoThat situation can be easily fixed in RPM-based Linux with rpm --setperms and rpm --setugids. Correction will be little harder on Debian derivatives and whole incident can be completely prevented on Solaris with file-mac-profile.