13 ms·
NPM 6.9.1 is broken due to .git folder in published tarball
- nullwasamistake 7y agoSoftware this popular doesn't even run a test that installs, upgrades, then uninstalls before deployment? Isn't that the most important thing the software needs to be able to do? Does NPM test anything at all before they release? Another reason to switch to yarn.
- deleted 7y ago[deleted]
- floatingatoll 7y agoAll autoupdate tests will eventually fail to prevent a self-breaking autoupdate from affecting one or more users. The trick is to prepare effectively for that scenario ahead of time. For example, homebrew’s “brew doctor” command has existed for a long time, but in recent years it now comes with a litany or helpful text and other commands (including autoupdate code logic) reference it when they encounter errors.
- fps_doug 7y agoWelp, not that our product is that popular, we're selling in one country only, but there are exactly zero tests. Everything is manual. And I think this is still how the majority of projects out there do it. Not that this is great, but management, and a lot of devs don't see too much value in it, more like "nice to have if there's extra time".
- not_kurt_godel 7y agoThat is not how the majority of professional software projects do things, and even if it were it doesn't matter because npm is not your average software project anyway. It is absolute clown status that they don't have integ tests to catch this sort of thing, and adds yet more doubt in my mind about the wisdom of using it (on top of an already considerable doubt pile).
- aprdm 7y agoHow do you know how majority of it isn't like this ? Having a few years of experience and having worked in 10+ companies across different industries I would be surprised if it wasn't the case.
- emerongi 7y agoPretty sure it is how the majority of software projects are doing it, but I agree that NPM is not your average software project.
- fps_doug 7y agoDefine "professional". If "professional" == "projects that have tests" you're probably right. ;-) Otherwise, you might've gotten lucky with your previous jobs/open source projects you contributed to. The best I've seen so far at the 4 jobs I had in my life were that some, mostly smaller, components had test suites, then others some half-arsed attempts that stopped being maintained a year or two ago, and as soon as some test stopped working because something got refactored, it was simply removed if it couldn't be fixed within a couple minutes. My previous job probably had the best test coverage, but still not integration tests. Maybe it's because people slowly become more conscious about the importance.
- mikekchar 7y agoI'm still amazed that there are companies that actually test diligently (I work at one now -- I sometimes complain about our tests, but in general they are pretty darn good). Literally only 20 years ago there were a lot of places you'd be laughed out of if you suggested writing automated tests for all of your code (or even any of your code ;-) ). The first time I did TDD seriously, I thought, "I'd better hang on to this job. I'll never get another one like it". But, it seems that there are groups that are doing it, and even people who think you aren't professional unless you are doing it. It's down right amazing!
- TeMPOraL 7y agoTo drop another n=4 anecdote, yes, all places I worked in were more-less like that. (One reason why I dislike the terms like "professional software $something" or "industry standard" is because while they connote quality, they're defined by "as done by people who are paid to do this" and "the popular thing among people who get paid for this work", respectively. The two viewpoints - quality vs. what professionals do - are almost completely opposite in practice, given the shit show our industry is.)
- bassman9000 7y agoHow old is the product? Have you done any decent refactoring?
- de_watcher 7y ago> Another reason to switch to yarn. Another reason to stop making dozens of different package managers. Or dozens copies of literally anything. Can you people just talk and work together? These package managers aint rocket science. Look at GNU/Linux: there are only 2 or 3 variations for everything.
- diggan 7y ago> Can you people just talk and work together This is a general problem in technology, not just about package managers > Look at GNU/Linux: there is only 2 or 3 variations for everything. Not true, there is probably hundreds of versions of everything (probably not bad in itself) but 2-3 popular ones that you know about. The reason why we keep seeing new package managers is because people think they can do it better than the existing stuff. And I think it's a necessary step to improvement, but some of the suggested solutions will be crap. But then sometimes something really good comes along and the ecosystem self-corrects.
- sime2009 7y ago> Look at GNU/Linux: there are only 2 or 3 variations for everything. Mostly false. For package management on Linux I can think of 4 just off the top of my head: * apt * rpm * apk * pacman There are bound to be more and this is not counting other approaches like AppImage and Snap.
- benbristow 7y agoDon't forget portage for Gentoo and zypper for OpenSUSE
- de_watcher 7y agoTotally different. He said "switch from npm to yarn". I can't imagine the phrase "switch from apt to rpm". The difference here is that there is the whole layer of distribution maintainers who do the actual work to ensure compatibility and quality. AppImage and Snap are different beasts emerged from the need to bundle proprietary software. There are around two of them like I've said.
- eeZah7Ux 7y ago> doesn't even run a test that installs, upgrades, then uninstalls before deployment That would be what Debian does - automatically. Not hipster enough.
- rolltiide 7y agoThe interesting thing about semi-decentralized technologies is that every issue seems like a clusterfuck but it actually gets fixed fairly quickly and nobody in the future has to think about it at all
- choeger 7y agoMore interesting question: How many hip startups could not build/test/ship their latest iteration yesterday because their whole CI pipeline depended on npm working?
- gridlockd 7y agoWhat CI pipeline depends on the latest release of any piece of software working? Updates can always break stuff.
- TeMPOraL 7y agoA dumb one that pulls fresh copies of all packages off the Internet every time a CI job is run? There seems to be a few of these around, judging by the noise made every time Github suffers a brief outage.
- y4mi 7y agoOh, No. That's a different issue. If GitHub is unavailable, the repository itself is unreachable. A lot verify hashes from the repo. If the latest release of one package is broken and your ci breaks, then you really have a problem... You'd have to disregard lock files and go out of your way to reinstall everything to latest
- oarsinsync 7y ago> If GitHub is unavailable, the repository itself is unreachable. If a central repository is unavailable, a decentralised version control system should continue to function. If developers are creating fragile tooling that centralises a by-design decentralised service, that feels like a flawed decision.
- z3t4 7y agoThese sort of bugs are hard to find with unit tests. But it's still easy to test, to make sure it never happen again. I think it's called integration tests. It can also be done with live testing, by checking all packages that lands on the public repository - if they include .git folder, passwords/keys, or other credentials.
- mrolla 7y agoBug aside, it baffles me that it's a real person publishing rather than CI after tagging.
- majewsky 7y agoI don't know about their process, but I always make my release tags myself. I wouldn't want to give the signing key to any CI system.
- WoahNoun 7y agoThat seems fundamentally backwards. The CI system should do the tagging. Allowing manual tagging introduces intentional or unintentional malfeasance in shared projects.
- javagram 7y agoManual tagging is the best way for most projects to do stuff like sign the package using an offline hardware key. Putting your keys on CI makes you vulnerable to your CI being hacked, which anecdotally seems to have happened to several projects.
- mrolla 7y agoI am very reluctant myself, but I think at some point you have to compromise. You can self host and go the hard way, trust some 3rd party CI (reputation is key) or go, if available, with the one from your cloud provider. This is exactly why we went with Codebuild at some point. It's not great, but hey we are not giving it anything more they don't already have (we host everything on AWS).
- deleted 7y ago[deleted]
- gigatexal 7y agoI think this probably happens more often in other package repositories than we hear about it’s just that NPM is gigantic and people like to hate on JavaScript. I’ve done stupid things too. I once committed the s3 keys to our public repo and had to explain why our s3 bill was so high. Never made that mistake again.
- thrwaway48373 7y agoOuch. How long did it take you to figure out what happened, and how high was your bill? How many people abused it, do you think (just one?) and any idea for what?
- gigatexal 7y agoJust one day. Someone uploaded thousands of junk and since we had versioning it took a long time to delete. And I think we were being billed extra on the high load being given to our bucket. So could have been multiple someones. I revoked the keys, deleted the bucket, even my own account, and reset everything. Lucky for me the CTO was understanding. I don’t remember the exact bill amount but it went up in relative terms by something like 2x
- thrwaway48373 7y agoOh, that sounds like you really avoided something much worse that could have happened. I thought you meant like a bill a couple of orders of magnitude larger than you ever paid normally. Thanks for the answers! Good on you for catching it relatively quickly.
- fs111 7y agoWhich other package repos are those supposed to be?
- gigatexal 7y agoMaybe github’s repo push is going to fix things.
- Sahhaese 7y agoI see it's been fixed by releasing 6.9.2. Does this mean you can now "brick" people's projects by sneaking npm@6.9.1 into their package.json?
- KhoomeiK 7y agoCertainly seems possible... Looks like I probably won't be installing anything with npm until they unpublish it.
- Ajedi32 7y agoI don't think so. NPM packages aren't installed globally by default, they're scoped to the local project via `node_modules`. So by default, the version of npm on the user's PATH wouldn't get updated.
- anc84 7y agoThe majority of npm based installation instructions I encounter "out there" are using "npm install -g".
- hombre_fatal 7y agoWell, that would be up to the user to decide. And you don't generally install project deps that way beyond some dev-deps like linters and test runners.
- WorldMaker 7y agoEven that guidance has changed with the cultural acceptance of `npx`. At this point the only thing that should be `npm install -g` is npm itself, and that only depending on your philosophy of how current you need to keep npm versus the version that ships with your NodeJS install.
- bassman9000 7y agoEveryone installs npm globally, because the whole model is broken. Which is why we keep all node env in a Docker image.
- MehdiHK 7y agoProbably relevant- This week's Node.js weekly [1] mentions: "Last week we mentioned the long awaited status of npm 6.9.1 and the possible ‘strike’ [2] in ongoing community work on the project, but npm’s Isaac Z. Schlueter has stepped up, got a release out" [1] https://nodeweekly.com/issues/294 https://nodeweekly.com/issues/294 [2] https://gist.github.com/aeschright/8ed09cbc2a4aee00fcb4ad35086d76a6 https://gist.github.com/aeschright/8ed09cbc2a4aee00fcb4ad350...
- ch4s3 7y agoWell, they did just fire a bunch of people for complaining about working conditions, so it's no surprise that they're stumbling into other mistakes.
- deleted 7y ago[deleted]
- kotxig 7y agoWhile this is clearly a bug, the default publish strategy is to publish all the contents of the package including dev/test files. If you look at the node_modules tree on a typical server-side node project it's full of garbage files. We still think it's a great idea to pull down hundreds if not thousands of copies of modules from a remote every time we generate a build. The community has always been lacking maturity around how it manages modules and releases code for production, preferring to preserve developer productivity and trinket features over best security practices and optimizing for the size and quality of production packages. I don't want to install hundreds of modules every time I create a new build. I don't want your tests or your README in my production tarball. I don't want your browser compatibility code in my server code. I don't want my node_modules tree to be 9 layers deep. I don't want to have to dedupe multiple copies of modules by hand. I don't want to have to debug where shrinkwrap isn't respected. I don't want to play roulette with the package manager version to figure out which one does the right thing for my package. I don't want to run your dubious pre/post install scripts. I don't want to use npm. The sad thing is, npm is a commercial entity that has its package manager published by default with an opensource community project. And this is why we can't have nice things.
- LeoNatan25 7y agoMaybe "we" should not be writing our server code using a terrible ecosystem? Just an idea.
- kotxig 7y agoIndeed. It's hard to avoid that conclusion, but you can't deprecate things overnight.
- halfmatthalfcat 7y agoNode has a great server-side ecosystem. I'd go as far as saying Node lends itself to certain layers of the backend stack because of it's history being more CPU-efficient then other server-side runtimes. Can you provide any examples where Node does not provide a good ecosystem for backend programming? Genuinely curious.
- _Marak_ 7y agoNPM still has bugs from v5 which haven't been addressed and are ignored by it's maintainers. Problems with `npm link` were never fixed. There is a discussion on the NPM tracker where the maintainers tell Substack that his ( very correct ) assessment of an issue was wrong and lock the thread. It's always the same usual suspects at NPM never accepting responsibility or blame for multiple incidents and ongoing quality issues. NPM is currently on it's last legs before acquisition or buyout.
- shados 7y agonpm link almost feels like a troll attempt. Running 2 packages locally together is one of the most basic features I can think of, and it works so poorly.
- mercurysmessage 7y agoFound this out not long ago on my own. Wish I would have known link was so bad before I tried it.
- samirm 7y agoIf they are on the verge of a buyout, I hope Microsoft gets them.
- msoad 7y agoYup! Even this "amazing" bug was reported and was ignored by the team before: https://npm.community/t/npm-6-9-1-is-broken-due-to-git-folder-in-published-tarball/8454/4 https://npm.community/t/npm-6-9-1-is-broken-due-to-git-folde...
- mustardo 7y agoFuckit.JS to the rescue? https://github.com/mattdiamond/fuckitjs/blob/master/README.md https://github.com/mattdiamond/fuckitjs/blob/master/README.m...