23 ms·

The tragedy of running an old Node project

272 points by abdisalan 2y ago

sgt 2y ago
I've actually had a node project go bad in a mere 4 months. It must be a new record. That was about 4-5 years ago though. Hopefully the ecosystem as improved since then, but it was nearly impossible to get going. Some packages had been changed and the version number overwritten with incompatible packages, and the conflicts were plenty.
- abdisalan 2y ago
  Wow, that’s honestly impressive. If there was an option to guarantee versions could exist for X amount of years (maybe even months?) then that would greatly help the stability of projects.
- jerf 2y ago
  One of the things I'm intrigued by is that JS people, and the other couple of ecosystems where this is a big problem, go out to learn another language (as a good T-shaped developer does), and then start posting frantic questions to the new language's communities about how this popular library hasn't had a commit in six weeks, is it dead, oh my gosh wtf aaaaaaaaaaa. It's OK. Not every language ecosystem is so busted that you can reliably expect a project not to work if someone isn't staring at it weekly and building it over and over again just in case. Now, it's always a risk, sure, no language anywhere is immune to the issue [1], but there's plenty of languages where you can encounter things from 5 years ago and your default presumption is that it's probably still working as well now as it did then. It may be wrong, but it's an OK default presumption. [1]: Well... no language in common use anyhow. There's some really fringe stuff that uses what is basically content-based references for code dependencies, but I'm not aware of anything that I'd call "production quality" that even remotely looks like that, and is immune to someone just plain making an error with the semantic versioning or whatever.
  - rootnod3 2y ago
    These JS developers would probably shiver at seeing many Common Lisp repos with a last commit like 12 years ago and still working like a charm.
    - kaptainscarlet 2y ago
      Node is bad but the worst I have seen is Android
      - telgareith 2y ago
        How about node on android?
        lpapez 2y ago
        Delete this comment right now, don't give them ideas.
        ian-g 2y ago
        Too late, we already have react native
        kaptainscarlet 2y ago
        It's a double whammy
        plaidphantom 2y ago
        And Nativescript.
    - abdisalan 2y ago
      I’m curious, how do you measure the pulse of a project that old? Do people still talk about it? Or that not even necessary — use it until it breaks and otherwise don’t think about it?
- int_19h 2y ago
  The ecosystem has not improved since then.
- meiraleal 2y ago
  the problem was your research before using the packages then. So much bad engineering and architectural planning is blamed on the tools, not the human using it.
Klonoar 2y ago
You know, I ran into something similar recently with a static site engine (Zola). Was moving to a new host and figured I'd just copy and run the binary, only to have it fail due to linking OpenSSL. I had customized the internals years ago and stupidly never committed it anywhere, and attempting to build it fresh ran into issues with yanked crates. Since it's just a binary though, I wound up grabbing the OpenSSL from the old box and patching the binary to just point to that instead. Thing runs fine after that. This is all, of course, still totally stupid - but I did find myself thinking how much worse comparable events in JS have been for me over the years. What would have been easily an entire afternoon ended up taking 15 minutes - and a chunk of that was just double checking commands I'd long forgotten.
- abdisalan 2y ago
  Hold on, you had to do binary surgery using an OpenSSL version from an old box you had? I salute the dedication.
  - getcrunk 2y ago
    How exactly does one do that. Sounds exciting!
    - kaptainscarlet 2y ago
      I think you copy the library file and add it to you load path
    - exe34 2y ago
      patchelf
      - ian-g 2y ago
        but don't forget to make sure your new path is fewer characters than the original one so you don't overwrite any of the library
    - pavel_lishin 2y ago
vessenes 2y ago
This will always be an issue for the node community - it’s endemic to the JavaScript shipping / speed culture and the package management philosophy. Go is much, much better on these terms, although not perfect. I’d venture a guess that Perl 5 is outstanding here, although it’s been a few years since I tried to run an old Perl project. CPAN was dog slow, but other than that, everything worked first try. I’d also bet Tcl is nearly perfect on the ‘try this 10 year old repo’ test
- tpm 2y ago
  Yeah I'd expect 20yo Perl5 stuff to work without issues. A few weeks ago I was experimenting with a sound generation dsl/runtime called Csound and even most 30yo sources were working as long as they didn't use some obsolete UI.
  - transcriptase 2y ago
    It’s the same with R. The only thing preventing many ancient packages from running under new versions of R and vice-versa is the fact that the package author simply set the minimum version to whatever they happened to be using at the time.
- skybrian 2y ago
  Go’s minimum version selection is the way and I don’t understand why other ecosystems haven’t adopted it. You’re be able to compile an old project with all the library dependencies it had at the time it was released. It might have security issues, but at least you start with a version that works and then can go about upgrading it. It also helps that if some library dependency generated Go code using a tool, the Go source code is checked in and you don’t have to run their tool.
  - vessenes 2y ago
    I truly think it's just because the engineers that started working with node were ... young. They wanted to rapidly iterate, and so crufty old habits like this weren't what they wanted or felt they needed. What's been interesting is watching these devs age 10 years, and still mostly decide it's better to start new frameworks rather than treat legacy code as an asset. That feels to me like a generational shift. And I'm not shaking my cane and saying they're wrong -- a modern LLM can parse an API document and get you 95% of the way to your goal most of the time pretty quickly -- but I propose it's truly a cultural difference, and I suspect it won't wash out as people age, just create different benefits and costs.
theandrewbailey 2y ago
Node.js (or more accurately, the entire Javascript ecosystem) changes, but the tropes don't. https://medium.com/hackernoon/how-it-feels-to-learn-javascript-in-2016-d3a717dd577f#.758uh588b https://medium.com/hackernoon/how-it-feels-to-learn-javascri... (beware the green background, I recommend reader mode.)
uludag 2y ago
The worst part isn't just that it's nearly impossible to run/update an outdated JS project, but that this process will repeat itself ad infinitum. On the flip side, anything that uses vanilla JS without a build will most likely run just fine, probably till the end of human civilization.
- graypegg 2y ago
  I truly believe some flavour of "Javascript Classic" (some future state of JS before some big shift in syntax/mass migration to something else), C and x86 instructions will follow humanity for the rest of time. There will be javascript somewhere aboard the interstellar spaceships of the future, and we will still complain about it.
  - xg15 2y ago
    Yeah, starting to believe the hacking scene in the Matrix where the machine city was still running on IPv4 wasn't a blunder but foresight. There will also be someone playing Tetris, Doom and Final Fantasy VI on their neural interface, long after all modern games have been lost to time (and DRM).
  - elzbardico 2y ago
    Well, even Fortran is still around us in some lapack code in numpy and in a lot of the stuff behing scipy, so, a lot closer than a lot of people can imagine. Basically a lot of AI depends on a bunch of absurdly optimized numeric libraries writen in Fortran. Fortran is well into the way at becoming a centenarian programing language at 74 years of age.
  - pavel_lishin 2y ago
    My grandchildren will live to see Vernor Vinge's programmer-archeologists troubleshooting PHP issues on the Wordpress install responsible for life-support around Alpha Centauri.
    - kwertyoowiyop 2y ago
      Just bobble some programmers now.
binarymax 2y ago
I always try to remember to put the node version in my package.json - but I do agree that the dependency chain on node-gyp has been a blight on node packages for awhile. Really wonder how that wart became such a critical tool used by so many packages.
- ramesh31 2y ago
  >Really wonder how that wart became such a critical tool used by so many packages. The original dream for Node was that it would simply be a glue wrapper around libuv that allowed for easy packaging/sharing of modules written in C++. But everyone just started writing everything in JS, and the ecosystem ended up as a mish-mash of native/non-native. Ryan Dahl stated this was indeed his biggest mistake/regret with Node, thus we have Deno now.
  - com2kid 2y ago
    > But everyone just started writing everything in JS, and the ecosystem ended up as a mish-mash of native/non-native. Because the native written stuff breaks all the darn time and it creates cross-plat nightmares. My stress levels are inversely proportional to how many native packages I have to try to get building within a project, be that project in Python, Java, or JS. JS+Node runs on everything. Prepackaged C++ libraries always seem to be missing at least one target platform that I need!
    - int_19h 2y ago
      Seamless native builds are quite doable, but the tooling needs to be very deliberately designed around that. For a good example of how far this can be taken, consider https://andrewkelley.me/post/zig-cc-powerful-drop-in-replacement-gcc-clang.html https://andrewkelley.me/post/zig-cc-powerful-drop-in-replace...
      - com2kid 2y ago
        I'm a huge Zig fan! Thank you for making native programming fun again! Zig is the exception to native build systems being painful. But even a great build system doesn't help when old native libraries don't support newer hardware or OSs. At some point the high level -> native abstractions break and then builds break. :(
alganet 2y ago
node-gyp was a mistake, building of native addons should have been an explicit separate step all along.
mvkel 2y ago
> time to run it after not touching it for 4 years > Two hours of my life gone... Two hours of work after 4 years sounds ... perfectly acceptable? And it would have run perfectly right away if the node version was specified, so a good learning, too This feels like making a mountain out of a mole hill
- mattgreenrocks 2y ago
  Other ecosystems usually do not have problems to the extent the author had.
  - hathawsh 2y ago
    I am deep in the Python ecosystem, and I love Python, but I have to admit that Python has the same issue. Reviving a medium-size project after 4 or more years usually means I have to catch up on a lot of new surprising deprecations. That's not because there's anything wrong with Python; it's more of an economic issue: the authors of active libraries have little or no economic incentive to support old, deprecated versions, so they just don't. That's life in the modern world. It is a deep problem that should theoretically affect every large software ecosystem because very few library authors can predict the future with great accuracy, and very few open source library authors have any significant incentive to support old ideas.
    - tmpz22 2y ago
      > That's life in the modern world. It is a deep problem that should theoretically affect every large software ecosystem because very few library authors can predict the future with great accuracy, and very few open source library authors have any significant incentive to support old ideas. I disagree. This is an easy problem to avoid with minimal due diligence, people just choose convenience and make unnecessary tradeoffs. * Use the standard library (ironically not available for Node projects). It will be built with better backwards compatibility almost every time. What deprecations do occur will likely be VERY WELL documented with much quicker adaptions. * Limit third party dependencies. Do you really need an ORM for your apps 40 sql queries? How long would it take you to scaffold it with GenerativeAI then make it production-worthy without the ORM? 1 hour? 5 hours? 20 hours? * Pick technologies with better track records. Maybe don't use Beta software like Swift Data for your iOS App. Maybe choose golang for your API even though it'll take a little bit longer to build it.
deleted 2y ago
[deleted]
beardyw 2y ago
I'm pretty sanguine about languages and frameworks, but I draw the line at node. I have seen so many horrors visited by dependencies, often to do just one thing where 2 or 3 lines of code would do the job anyway. When I was managing teams, whatever the language, I would ban any new dependencies which I didn't personally agree with. A lack of control just creates a nightmare.
- CognitiveLens 2y ago
  Was that kind of control well-received by your teams? Out of context, it sounds like it would be pretty rough to be an engineer on a team where your manager had sole control over what tools you could use - I suppose it might make sense for junior devs or a very small codebase, but I would caution against taking that stance in a team where you want to facilitate mutual trust
  - beardyw 2y ago
    Well back then there were fewer options, but the result was that completed products were easy to work with. Perhaps we live in different times.
  - mst 2y ago
    Provided the manager only rarely exercises the power, and is open to being persuaded not to, having somebody able to veto risky dependencies can be really quite useful. Normally when I'm the one with that power we rapidly get to a general understanding of what's small enough that I (a) probably won't care (b) will take responsibility for tweaking the schedule to makre time to get rid of it if I do. And 'big' dependencies are generally best discussed amongst the entire team until consensus is reached before introducing one anyway.
everythingishax 2y ago
This is literally every "hot new thing" since 2000. It is systemic. Part of it is due to too many people creating systems on the fly with too little forethought, but also because there aren't enough "really smart people" working on long term solutions. Just hacks, done by hacks. What did you expect when the people writing the systems don't have long term experience?
- int_19h 2y ago
  The problem itself is old, but the extent to which it pervades different ecosystems varies. It's largely a cultural thing, and the problem with JS/Node ecosystem specifically is that most of the community (or, perhaps, rather most of the library/framework authors) accepts this kind of thing as normal.
kylehotchkiss 2y ago
Dealing with node-gyp cost me at least 5 hours a month in the 2010s. I'm so very happy to not see those errors in my console anymore.
speedgoose 2y ago
I would heavily recommend to avoid NodeJS packages that depend on node-gyp. Node-gyp powered dependencies are very seldomly worth the hassle. If you must depend on node-gyp, perhaps use dev containers so at least every developer in your team can work most of the time.
- graypegg 2y ago
  So I'm pretty uninformed about the guts of node-gyp, and why it's used, but if people need to bring in dependancies from outside javascript... could WASM be a good fit there? Could store the binaries instead, and ship those... and in theory (correct me if I'm wrong) that shouldn't be much of a security issue due to the security model of WASM modules... or at least equal to the risk of running arbitrary build commands on your machine from a random node package.
  - int_19h 2y ago
    In principle, yes. In practice, the problem is that getting some random native library or tool compile with wasm as a target is not always easy. E.g. anything that relied on pthreads was out until fairly recently.
  - _fat_santa 2y ago
    In practice you're just kinda stuck with it because whatever NPM package you're using is using that under the hood. One of my project depends on it because of postgres DB bindings, there would be no easy way for me to get rid of it without either finding another binding (which that is the official one) or rebuilding it myself which will just take too much time and effort for what it's worth
- eterm 2y ago
  I don't even know what node-gyp is, but I know it appears regularly in error messages to know it causes problems. I don't even develop against Node, it has just crept into our front-end build toolchain.
  - 2y ago
melbourne_mat 2y ago
First thing I would have done is upgrade the version of Gatsby to latest. Did the author try that? If upgrading is difficult because of 4 years of breaking changes, blame Gatsby for not being backwards compatible. Also blame your original choice of going with a hokey framework. Speaking of hokey framework: 167 dependencies and 3000 versions of Gatsby in npm.
- baq 2y ago
  blaming anything or anyone gets you exactly zero seconds closer to getting the job done.
  - tomgp 2y ago
    You could perhaps reframe "blame" as identifying the source of the problem and iunderstand why it can be a useful excercise (aslo none of us here are trying to solve the problem really, just wasting time on the internet). In this case Node and it's atendant ecosystem are certainly a part of the problem but I would agree that Gatsby is a bigger part of the issue as they don't seem to have any interest in taming the Node dependecy management beast. I've had to dig into Gatsby projects mere months old and it really was like opening a can of worms.
    - Eric_WVGG 2y ago
      > In this case Node and it's atendant ecosystem are certainly a part of the problem but I would agree that Gatsby is a bigger part of the issue I disagree completely. Regardless of what you think of Gatsby, Node versioning is a simple problem that affects nearly every javascript project. It should always be the first thing you check. I made this dumb obvious mistake again just last week, looking for a little time to audit all my `package.json` files for `engines`.
- Eric_WVGG 2y ago
  The first thing I would have done is check the version of Node. That's a quick fix, upgrading a framework is a guaranteed min hour of poking around before the system is even running. I dunno why `engines` isn't in every `package.json` file, would certainly have saved me hours of nonsense.
adamtaylor_13 2y ago
Acknowledging this is absolutely awful, and also commenting that a project .nvmrc file is your friend!
- baq 2y ago
  ...until the node version you locked can't be downloaded anymore, or hasn't ever existed for your CPU arch.
  - lvturner 2y ago
    I see you too had to run node v14(? my memory fails me somewhat) on Apple Silicon hardware...
    - baq 2y ago
      shivers
    - rkharsan64 2y ago
      FWIW, I run Node 12 painlessly on Apple Silicon using fnm, so you might be thinking of a few versions before that.
hinkley 2y ago
I joined a node project that was stuck on 0.12 while 7.0 was being developed. It was a shit show to get us to 6. As I recall, 10 was a little tricky, 12 and 16 had a lot of head scratchers. I finished the 16 upgrade more than a year after the last person tried, and it was a dumb luck epiphany that kept it that short. I had a similar experience with emberJS when it was still young. Every time I picked the project up I had one to two hours of upgrade work to get it to run again, and I just had a couple hours to work on it. So half my time went to maintenance and it wasn’t sustainable. I’m trying a related idea now in elixir and José may be a saint. Though I fear a Java 5 moment in their future, where the levee breaks and a flood of changes come at once.
cynicalsecurity 2y ago
The tragedy of running an̶ ̶o̶l̶d̶ Node project.
- cookiengineer 2y ago
  I had to chuckle after I read your username. Kudos.
- mst 2y ago
  On the one hand, it's not that terrible and most *of* the terrible is from people making silly choices. On the other hand, there's a reason I regularly get annoyed enough at it to call it nope.js. On the gripping hand, I mostly write perl, which argues for a different but unique set of masochistic tendencies on my part. (you just have to remember that what 'perl' *really* stands for is 'Perenially Eclectic Rubbish Lister' and then you will have appropriate expectations and can settle back and have fun ;)
Hayabusaa 2y ago
Have you tried DevContainer before ?
em-bee 2y ago
i could not tell from the article whether this was a site with a backend using node.js or if it was just a frontend depending on node.js for the build tools. for the latter i get around the problem by avoiding build tools altogether. i use a frontend framework that i can load directly into the browser, and use without needing any tools to manage dependencies. the benefit from that is that it will ensure that my site will keep running for years to come, even if i leave it dormant for some time. the downside is that it is probably less optimized. but for smaller sites that aren't under continuous maintenance this is a reasonable tradeoff. i built all my recent sites that way using a prebuilt version of the aurelia framework. incidentally just today i tried to research if i could build a site with svelte that way. well, it turns out that although it should theoretically be possible, i was unable to find a prebuilt version to do so after a few hours of searching. for vuejs i found one within minutes. i'll be learning vuejs now. see this thread for a discussion on going buildless: https://news.ycombinator.com/item?id=41479365 https://news.ycombinator.com/item?id=41479365
- mst 2y ago
  I've been experimenting recently, with quite some success, with having a 'libs.js' file that pulls in and re-exports everything external I want, and providing a script that applies 'bun build' to just that. I haven't yet decided if/how I want to include a prebuilt version of it in the repo, I *think* I may go the approach of having a commit that modifies libs.js and/or the lockfile and then an immediately following one that commits an updated prebuild ... oh, huh, actually, I should probably also consider doing those two commits on a branch, then forcing a merge commit so they land on master atomically but it's easy to tease out the human changes and the regen changes by poking inside said merge commit ... yeah, like I say, still thinking about exactly how to do this, don't mind me. Also for even simpler cases I've been using the preact-htm prebuild directly, since htm gives a lit-style html() tagged literal consuming function that can produce vnodes for preact so I can mess around without needing something that understands jsx between my editor and my browser window. vue's component system is IIRC noticeably less nice to work with if you don't have a compile step, but it's still pretty nice even without that so please don't think I'm trying to dissuade you here :)
  - 2y ago
nasmorn 2y ago
Good old node-gyp. I have absolutely no idea what it even is but it has been giving me errors for what feels like a decade. Mostly via front end build stuff from various projects I have worked on
- mkoryak 2y ago
  i forgot all about node-gyp. The only memories I do have of it are the errors and thinking about gimps.
- pavel_lishin 2y ago
  Same. One day I'll find out what it is.
  - enqk 2y ago
    if you want to know, it's a fork of google gyp, which is a C/C++ project/build system generator. I.e. it's a bit similar to CMAKE, a tool to describe native code projects and what needs to be built in order to make executables and dlls. It's a python codebase, largely abandoned by google. They used to use it for building Chrome.
keybored 2y ago
I had to build some project that uses some Ruby package manager. I forgot already what the package manager is called. I got some error about "you don't have all the dev tools". So I installed what Google told me "dev tools" was. Then it still told me that I needed more dev tools. Stackoverflow had some question about this package manager. For Windows (Linux here). 20+ answers, mostly for Mac. All in the style of "this random thing worked for me". All with at least one upvote. Some answer about "I needed to symlink this system library". Gave up. Then I ran `devbox init` and installed whatever it told me that was needed. `devbox shell`.
pavel_lishin 2y ago
> node-gyp We're in ... let's call it a transitional period at work. I've got something like a dozen versions of node being managed by asdf. And in half of the projects I work on regularly, I consistently get warnings about this particular project failing to build. One day, I'll actually look up what it actually is, and what it does, and why it's being built, but is apparently optional.
- mst 2y ago
  It's basically a set of tools to make building native modules easier, that said modules then use to deal with their binding to C/C++/etc. code. Everybody complains about it, and understandably so, but if it didn't exist you'd probably instead have one set of potential similar problems *per* native module which has a good chance of not actually being better overall. The counterargument is, I guess, "well, only people who can write their own high quality build setup in-tree should be writing things that bind to external code," and I do sometimes dream of that, but it's not hard to see the downsides of living in *that* world instead either.
philipwhiuk 2y ago
Native code in an npm module should be regarded as a massive red flag.
conoro 2y ago
I had this exact problem with multiple Node blog engines in the past. Constant version breakage was incredibly frustrating. I eventually moved to Hugo. A single binary which I committed with the blog files. Zero issues even years later. I can build the blog on any new machine within seconds. Which was the other revelation of Hugo. 10 seconds to build an 800+ post blog vs minutes using Hexo or similar.
nwhnwh 2y ago
How is that pretty hard?
iamsaitam 2y ago
At first I thought it would be a decade old project, but 4 years isn't old by any standards is it? Anyways, npm ci should have been the first attempt, not npm install so that it installs the same package versions defined in the package-lock.json. Then as others have mentioned, pin your node versions. If you're afraid of this happening again, npm pack is your friend. In the end, op could have done a bit more. BUT I'll give it to him that when bindings are involved, these things take more time than they should
Aeolun 2y ago
You spent only two hours on this and you think it’s too much? Also, do not run shit on a node version that is years out of date and out of service. Also, update your damn packages. I know I sound cranky, but running anything internet facing with god knows how many vulnerabilities in is an exceedingly bad idea.
SeriousM 2y ago
We use package.lock and docker image with local folder binding to run legacy node projects. Eg. docker run -v local:inner node:12 command
nenadg 2y ago
yeah? now try running 4 years old React project, it's a hell on earth.
- molszanski 2y ago
  next.js user? :D
noplacelikehome 2y ago
I’ve started to adopt Nix devShells to help keep a record of each project’s dependencies. If Nix is too heavy, the learning curve for tools like asdf-vm and mise is much lower and offers similar benefits. I really wish there was a good equivalent for Windows.
0points 2y ago
Lately I have been revisiting some older golang tool I wrote since before they introduced go modules. "go mod init" + identify a working dependency version was all I had to do on any of those 10+ year old projects (5 minute work tops)
Etheryte 2y ago
Can't help but feel that this is a massive nothing-burger. You wouldn't generally expect your Java project to run if you use an incompatible version of the JVM, nor would you generally expect your C++ project to build if you swap one compiler for a different one. Etc, always specify what your project relies on, whether it's in the readme or in the dependency tree.
- shepherdjerred 2y ago
  People love to hate on frontend/Node
- jcelerier 2y ago
  > nor would you generally expect your C++ project to build if you swap one compiler for a different one. it's absolutely the expectation for C++ projects to support building on GCC, Clang and MSVC
  - Etheryte 2y ago
    Maybe these days this is the case, last I used C++ about a decade ago though, this was most definitely not guaranteed to be so. This thread [0] is a good quick overview of the type of issues you used to run into, and often you didn't even need anything fancy to hit those edge cases. If life has improved in the meanwhile, that's good of course. [0] https://stackoverflow.com/q/51866968/1470607 https://stackoverflow.com/q/51866968/1470607
incrudible 2y ago
This goes for both node and python: Avoid native extensions. For python this is less feasible due to its inherently poor performance, so limit yourself to the crucial ones like numpy. For node, there are few good reasons why you would need a native extension. Unless you have your node version pinned, it will try to find the binary for your node version, fail, then attempt to build it on your system, which will most likely fail as well.
jollyllama 2y ago
For sure. This is the number one reason I am switching as many projects as I can to HTMX. https://dubroy.com/blog/cold-blooded-software/ https://dubroy.com/blog/cold-blooded-software/ Sibling comments say in so many words, it's no big deal bro, just update. But it is a big deal over time if you have dozens of cold-blooded projects to deal with.
- shepherdjerred 2y ago
  Gatsby and htmx have approximately zero overlap
  - jollyllama 2y ago
    The point is NPM. HTMX helps you avoid NPM.
bsuvc 2y ago
Having CI would have avoided this problem.
- abejfehr 2y ago
  Not for running the project locally
  - bsuvc 2y ago
    I think you're missing the point. CI solves it because it proves that it can build in the pipeline, using a well defined environment. No guessing at which node version you need or any other dependencies that may be required.
- negus 2y ago
  What you mean here is actually deps pinning + vendoring. It's way out of scope of basic CI concept.
- jcelerier 2y ago
  CI doesn't help when all the dependencies aren't locked, or where some deps disappear, or ...
anonzzzies 2y ago
We run node code that's 10 year old. No one dares to touch it; we just run it in docker and hope nothing goes wrong.
JansjoFromIkea 2y ago
node-sass is to blame for like 95% of these node-gyp issues in my experience, it's not that much grief to deal with but it's hard to grasp how it was allowed to hang around so terribly for so long
- pier25 2y ago
  > it's hard to grasp how it was allowed to hang around so terribly for so long It's not hard. The explanation is simply that the dart version of SASS (the sass npm package) is much slower than node-sass. At one point it was like 20x slower and I believe right now it's about 2-3x slower. https://github.com/sass/dart-sass/issues/1534 https://github.com/sass/dart-sass/issues/1534
__MatrixMan__ 2y ago
> Two hours of my life gone, just to pick up where I left off. If I had only wasted two hours every time I had to use npm for some reason I'd be significantly ahead of where I am now.
- kiru_io 2y ago
  Doesn't that apply to any it-issues you don't know how to solve?
  - __MatrixMan__ 2y ago
    no, most of the time when a README says: > run command xyz I run xyz and I'm on to the next step in an average of ten or fifteen minutes (because yeah, stuff still goes wrong some times). But if the instruction is to invoke npm, there's always some crazy side quest involved. It doesn't get any better with time because next time around it's a totally different side quest. I can't even offer any criticism towards fixing it, it just sucks for unique new reason every time.
RadiozRadioz 2y ago
I call this phenomenon "node rot". Judging by the comments here, it seems like a universal experience. My favorite is the way that Python projects rot. Not only does Python's setuptools give you all the fun that node-gyp does, the common practice of versioning packages with packagename>=1.25.5 means you're almost guaranteed breakages as pip installs newer versions of packages than what the project was built with.
- gre 2y ago
  I just today tried reviving an old v12.4 node project and node-gyp is trying to use python2 which I don't even have on my Macbook anymore.
  - ziml77 2y ago
    That Python 2 crap hit me as well when working on an older project. It used node-sass which depended on node-gyp. Dealing with upgrading that was a pain because I had to upgrade so much more to get versions of everything that played nice together. What really pissed me off about it was that Python 2 was already known to be nearing EOL when our project was started, so node-gyp should have been upgraded to work with Python 3 by then. And even more annoying was that node-gyp already had Node to run on, so why in the world was it coded to depend on Python at all!?
- lexlambda 2y ago
  And project specifying "requires python 3.X+" instead of Version X to Y is also a major culprit I often encounter. Most of the times it will not work with the newest shiny python, which I only notice after already installing it and then having searched search the Github issues.
  - __MatrixMan__ 2y ago
    That happens to me all the time. It helped cement my habit of binding the python version to the project with direnv and a flake.nix so I end up switching to the right version when I cd to the project dir.
betimsl 2y ago
OP is tired after 2 hours of work.
- andai 2y ago
  OP is tired after two hours of stupid bullshit that should not ever occur. Stuff like that's enough to make a man quit a job, or indeed a career.
dmuth 2y ago
This is one reason why whenever I build a new project, I build it inside of a Docker container. That way, the project has just the dependencies it needs, and I know I can rebuild it at some point in the future and will be unlikely to run into problems when I do.
- brokencube 2y ago
  My first step now when trying to resurrect old projects is to create a Docker container for it - that way I can install any old versions of anything (like node or PHP) I need without having to worry about it polluting anything else on the system.
- Cu3PO42 2y ago
  This works as long as your Dockerfile is reasonably reproducible and does its best to lock dependencies. However, this approach has failed me a couple of times in the past. For example, I rebuilt a container some weeks later, in the meantime a new version of clang had been released that just so happened to break my build due to a bug. I personally use Nix these days, but the complexity is too high for me to recommend it to everyone for every software project.
  - microtonal 2y ago
    Yeah, Nix pretty much solves this problem. The other day I wanted to try a really old version of spaCy for fun/historic interest. spaCy 1.8.2 installed freshly from the binary cache on NixOS-unstable as if it was still April 2017.
andai 2y ago
Not defending node here (I had OP's experience almost verbatim), but I had a much worse experience with trying to compile PaulStretch (a C++ project). The dependencies were specified as a bunch of random third party URLs, half of which had gone offline. I ended up giving up after a few hours, and then finding a fork that Just Works.
hombre_fatal 2y ago
OP was trying to install an old dep tree of gatsby on a different node target. These kinds of massive libraries break all the time. Look how big its dependency tree is: https://npmgraph.js.org/?q=gatsby https://npmgraph.js.org/?q=gatsby Fortunately this mindset has been changing in the node ecosystem with projects like https://hono.dev/ https://hono.dev/ (koa/express successor) and https://github.com/porsager/postgres https://github.com/porsager/postgres having zero deps.
brunoarueira 2y ago
I think many issues goes with bad version or, in some edge cases, not vendor dependencies. I'd good and bad experientes on multiple programming languages, some bad examples: - bump patch or minor version from a react package but the maintainer rewrote the entire project breaking a lot of things, following semver it's bad to expect things don't break like that for such version; - another example, the ruby gem is removed/yanked from rubygems.org and you'd to find a fork available On the end, we need to ensure the good practices from software engineering about tests and good release management, the last btw is decades old
gr4vityWall 2y ago
I recently migrated a project to Node.js 8 (!) to Node.js 14 (hopefully just the beginning), and I can relate to this post. In the JS ecosystem, I'm aware that Meteor is one major framework that takes backwards-compatibility seriously. Updating a project on an ancient version to a less-ancient version usually is not too hard. They try to keep APIs the same and introduce compatibility packages where possible. Meteor 2.16 to Meteor 3 introduced major breaking changes due to an underlying technical issue that had no workaround. They had to refactor the whole project from using Fibers-based concurrency to typical async/await. node-gyp in general has also been a source of issues in the past for me as well. More recently, ESLint changed their configuration file format and all existing tutorials suddenly became outdated. I firmly believe the ecosystem does not have to be like this, and we would save a lot of man-hours by being more committed to API stability where possible.
- yawnxyz 2y ago
  > Meteor 2.16 to Meteor 3 introduced major breaking changes That's when I picked up the Node/React ecosystem... > node-gyp errors > downgrade to 12.2 This is what I did until Vercel decided to not support Node 12 anymore...
- forty 2y ago
  After node 10, things started to stabilize. These day a node upgrade is a tiny commit just changing the version number in the Dockerfile and package.json
Havoc 2y ago
Node/JS seems particularly fragile in this regard thanks to the complicated maze of dependencies and sub dependencies and flavour of the month framework syndrome
exabrial 2y ago
2020 is not "old". "Not Invented Here" is whats going on. Developers of this age need to learn this. A recent exmaple is the RPI Foundation nullifying thousands of internet tutorials renaming "/boot" vs "/bootfs". Ask yourself a serious question, did that actually improve anything? No it did not.
aronhegedus 2y ago
Cool that you managed to get it running after just 2 hours. Same thing applies to python projects, a little note in README saves so much time in the future, I always try and use virtual environments and specify a specific python version so I can just nuke and reinstall everything
mostlysimilar 2y ago
DHH has said this experience is a big reason Rails is pursuing a no-build approach.
- em-bee 2y ago
  can you elaborate or link to an article that explains what this means?
bastawhiz 2y ago
All of the problems here ultimately came down to packages that used the native Node API. You don't need python or deal with C++ to run JavaScript. Node is an active project. If you build against the native API and don't pin your version to avoid breaking changes between versions, this is what happens. In my experience, JS very rarely breaks between major Node versions, but almost every native package requires a new major update. This isn't a Node specific problem. Go ahead and upgrade your Go or Python version.
turnsout 2y ago
Running a new Node project is nearly as problematic… The ecosystem is broken
contingencies 2y ago
FWIW, I've mostly maintained long term PHP projects, and I've had nearly unaltered codebases running for ~25 years since php3. No frameworks, just core PHP. People dump on PHP, but it's a very good tool if you're focused on maintainable output and pick the right functional APIs to cede to mature unix tools. You can expect decades of solid, maintainable output. Experienced programmers will not pick up a "built on shifting sand" stack, because they can acutely perceive the pain and suffering before it happens, generally from past experience. With fast-crumbling stacks, you need to execute quickly and move on, and treat the whole codebase as an expiring entity. Stacks I personally try to avoid: anything node/javascripty, anything Androidy, anything iDevicey. Those who don't understand Unix are condemned to reinvent it, poorly. - Henry Spencer ... via https://github.com/globalcitizen/taoup https://github.com/globalcitizen/taoup
stall84 2y ago
It's not just personal blogs as many of y'all know.. this is a daily struggle in any SDE mainly working in web .. I am constantly lamenting the fact I spend like 70% of my time at my main job (large financial company) just trying to get my environment , or the application's environment in a position to actually develop on the application itself .. It's insane. I f'n love javascript .. it's allowed a lot of us a doorway into software engineering where a lot of us realize how ... 'special' js and related web development are ;) .. But man it can make you really want to smash the computer some days.
YuukiRey 2y ago
This is the reason I ripped out Gatsby from every project where I could. Every six months I’d spend an entire evening fixing obscure problems that shouldn’t even exist just to get things running. And that’s not even considering actual breaking changes of which there were plenty.
z3t4 2y ago
You should save your deps in your SCM! Microsoft is giving away ownership to existing packages if you tell them you will use it for a TypeScript project.
- shepherdjerred 2y ago
  What? Does Microsoft control npm?
  - switch007 2y ago
    Yes. npm is a subsidiary of GitHub, a subsidiary of Microsoft
kiettv 2y ago
I still have to maintain many old projects that are using Node 7, but many dependencies are no longer available. Every update needs to patch the docker image manually from the last runnable docker image.
sn0n 2y ago
Made a site using nanogen ( https://github.com/doug2k1/nanogen https://github.com/doug2k1/nanogen ) about 7 years ago ... Anytime I setup a new machine I do a npm install on whatever version of node I end up on, and it..... Just works. Beat SSG I've found, and all from a medium or dev article on a SSG in 40 lines or less.
deleted 2y ago
[deleted]
stuaxo 2y ago
This could equally be written about old Android projects.
foxyv 2y ago
I think this was the biggest mistake that Java made. Breaking reverse compatibility after Java 8 means that thousands of organizations will never leave that version. There is an entire industry based around maintaining Java 8. Eventually there will be two versions of Java, just like Python 2 and 3. There will be Java 8 and Java 698.
tacone 2y ago
My take is one should be especially wary of packages that depend on C libraries and need compliation. You get to be extremely bound to what the OS distribution has to offer. If that's the case, docker is probably the safest solution.
droptablemain 2y ago
As a rule of thumb if you are jumping into a legacy node project the first thing you should try is a sequentially older node version. Could've reduced his two hours to two minutes.
banku_brougham 2y ago
Anybody have experience with pulling/running a docker image after many years? I'd like to see the consensus on most durable software packaging method.

272 points by abdisalan 2y ago

sgt 2y ago

I've actually had a node project go bad in a mere 4 months. It must be a new record. That was about 4-5 years ago though. Hopefully the ecosystem as improved since then, but it was nearly impossible to get going. Some packages had been changed and the version number overwritten with incompatible packages, and the conflicts were plenty.

abdisalan 2y ago
Wow, that’s honestly impressive. If there was an option to guarantee versions could exist for X amount of years (maybe even months?) then that would greatly help the stability of projects.
jerf 2y ago
One of the things I'm intrigued by is that JS people, and the other couple of ecosystems where this is a big problem, go out to learn another language (as a good T-shaped developer does), and then start posting frantic questions to the new language's communities about how this popular library hasn't had a commit in six weeks, is it dead, oh my gosh wtf aaaaaaaaaaa. It's OK. Not every language ecosystem is so busted that you can reliably expect a project not to work if someone isn't staring at it weekly and building it over and over again just in case. Now, it's always a risk, sure, no language anywhere is immune to the issue [1], but there's plenty of languages where you can encounter things from 5 years ago and your default presumption is that it's probably still working as well now as it did then. It may be wrong, but it's an OK default presumption. [1]: Well... no language in common use anyhow. There's some really fringe stuff that uses what is basically content-based references for code dependencies, but I'm not aware of anything that I'd call "production quality" that even remotely looks like that, and is immune to someone just plain making an error with the semantic versioning or whatever.
- rootnod3 2y ago
  These JS developers would probably shiver at seeing many Common Lisp repos with a last commit like 12 years ago and still working like a charm.
  - kaptainscarlet 2y ago
    Node is bad but the worst I have seen is Android
    - telgareith 2y ago
      How about node on android?
      - lpapez 2y ago
        Delete this comment right now, don't give them ideas.
        ian-g 2y ago
        Too late, we already have react native
        kaptainscarlet 2y ago
        It's a double whammy
        plaidphantom 2y ago
        And Nativescript.
  - abdisalan 2y ago
    I’m curious, how do you measure the pulse of a project that old? Do people still talk about it? Or that not even necessary — use it until it breaks and otherwise don’t think about it?
int_19h 2y ago
The ecosystem has not improved since then.
meiraleal 2y ago
the problem was your research before using the packages then. So much bad engineering and architectural planning is blamed on the tools, not the human using it.

Klonoar 2y ago

You know, I ran into something similar recently with a static site engine (Zola). Was moving to a new host and figured I'd just copy and run the binary, only to have it fail due to linking OpenSSL. I had customized the internals years ago and stupidly never committed it anywhere, and attempting to build it fresh ran into issues with yanked crates. Since it's just a binary though, I wound up grabbing the OpenSSL from the old box and patching the binary to just point to that instead. Thing runs fine after that. This is all, of course, still totally stupid - but I did find myself thinking how much worse comparable events in JS have been for me over the years. What would have been easily an entire afternoon ended up taking 15 minutes - and a chunk of that was just double checking commands I'd long forgotten.

abdisalan 2y ago
Hold on, you had to do binary surgery using an OpenSSL version from an old box you had? I salute the dedication.
- getcrunk 2y ago
  How exactly does one do that. Sounds exciting!
  - kaptainscarlet 2y ago
    I think you copy the library file and add it to you load path
  - exe34 2y ago
    patchelf
    - ian-g 2y ago
      but don't forget to make sure your new path is fewer characters than the original one so you don't overwrite any of the library
  - pavel_lishin 2y ago

vessenes 2y ago

This will always be an issue for the node community - it’s endemic to the JavaScript shipping / speed culture and the package management philosophy. Go is much, much better on these terms, although not perfect. I’d venture a guess that Perl 5 is outstanding here, although it’s been a few years since I tried to run an old Perl project. CPAN was dog slow, but other than that, everything worked first try. I’d also bet Tcl is nearly perfect on the ‘try this 10 year old repo’ test

tpm 2y ago
Yeah I'd expect 20yo Perl5 stuff to work without issues. A few weeks ago I was experimenting with a sound generation dsl/runtime called Csound and even most 30yo sources were working as long as they didn't use some obsolete UI.
- transcriptase 2y ago
  It’s the same with R. The only thing preventing many ancient packages from running under new versions of R and vice-versa is the fact that the package author simply set the minimum version to whatever they happened to be using at the time.
skybrian 2y ago
Go’s minimum version selection is the way and I don’t understand why other ecosystems haven’t adopted it. You’re be able to compile an old project with all the library dependencies it had at the time it was released. It might have security issues, but at least you start with a version that works and then can go about upgrading it. It also helps that if some library dependency generated Go code using a tool, the Go source code is checked in and you don’t have to run their tool.
- vessenes 2y ago
  I truly think it's just because the engineers that started working with node were ... young. They wanted to rapidly iterate, and so crufty old habits like this weren't what they wanted or felt they needed. What's been interesting is watching these devs age 10 years, and still mostly decide it's better to start new frameworks rather than treat legacy code as an asset. That feels to me like a generational shift. And I'm not shaking my cane and saying they're wrong -- a modern LLM can parse an API document and get you 95% of the way to your goal most of the time pretty quickly -- but I propose it's truly a cultural difference, and I suspect it won't wash out as people age, just create different benefits and costs.

theandrewbailey 2y ago

Node.js (or more accurately, the entire Javascript ecosystem) changes, but the tropes don't. https://medium.com/hackernoon/how-it-feels-to-learn-javascript-in-2016-d3a717dd577f#.758uh588b https://medium.com/hackernoon/how-it-feels-to-learn-javascri... (beware the green background, I recommend reader mode.)

uludag 2y ago

The worst part isn't just that it's nearly impossible to run/update an outdated JS project, but that this process will repeat itself ad infinitum. On the flip side, anything that uses vanilla JS without a build will most likely run just fine, probably till the end of human civilization.

graypegg 2y ago
I truly believe some flavour of "Javascript Classic" (some future state of JS before some big shift in syntax/mass migration to something else), C and x86 instructions will follow humanity for the rest of time. There will be javascript somewhere aboard the interstellar spaceships of the future, and we will still complain about it.
- xg15 2y ago
  Yeah, starting to believe the hacking scene in the Matrix where the machine city was still running on IPv4 wasn't a blunder but foresight. There will also be someone playing Tetris, Doom and Final Fantasy VI on their neural interface, long after all modern games have been lost to time (and DRM).
- elzbardico 2y ago
  Well, even Fortran is still around us in some lapack code in numpy and in a lot of the stuff behing scipy, so, a lot closer than a lot of people can imagine. Basically a lot of AI depends on a bunch of absurdly optimized numeric libraries writen in Fortran. Fortran is well into the way at becoming a centenarian programing language at 74 years of age.
- pavel_lishin 2y ago
  My grandchildren will live to see Vernor Vinge's programmer-archeologists troubleshooting PHP issues on the Wordpress install responsible for life-support around Alpha Centauri.
  - kwertyoowiyop 2y ago
    Just bobble some programmers now.

binarymax 2y ago

I always try to remember to put the node version in my package.json - but I do agree that the dependency chain on node-gyp has been a blight on node packages for awhile. Really wonder how that wart became such a critical tool used by so many packages.

ramesh31 2y ago
>Really wonder how that wart became such a critical tool used by so many packages. The original dream for Node was that it would simply be a glue wrapper around libuv that allowed for easy packaging/sharing of modules written in C++. But everyone just started writing everything in JS, and the ecosystem ended up as a mish-mash of native/non-native. Ryan Dahl stated this was indeed his biggest mistake/regret with Node, thus we have Deno now.
- com2kid 2y ago
  > But everyone just started writing everything in JS, and the ecosystem ended up as a mish-mash of native/non-native. Because the native written stuff breaks all the darn time and it creates cross-plat nightmares. My stress levels are inversely proportional to how many native packages I have to try to get building within a project, be that project in Python, Java, or JS. JS+Node runs on everything. Prepackaged C++ libraries always seem to be missing at least one target platform that I need!
  - int_19h 2y ago
    Seamless native builds are quite doable, but the tooling needs to be very deliberately designed around that. For a good example of how far this can be taken, consider https://andrewkelley.me/post/zig-cc-powerful-drop-in-replacement-gcc-clang.html https://andrewkelley.me/post/zig-cc-powerful-drop-in-replace...
    - com2kid 2y ago
      I'm a huge Zig fan! Thank you for making native programming fun again! Zig is the exception to native build systems being painful. But even a great build system doesn't help when old native libraries don't support newer hardware or OSs. At some point the high level -> native abstractions break and then builds break. :(

alganet 2y ago

node-gyp was a mistake, building of native addons should have been an explicit separate step all along.

mvkel 2y ago

> time to run it after not touching it for 4 years > Two hours of my life gone... Two hours of work after 4 years sounds ... perfectly acceptable? And it would have run perfectly right away if the node version was specified, so a good learning, too This feels like making a mountain out of a mole hill

mattgreenrocks 2y ago
Other ecosystems usually do not have problems to the extent the author had.
- hathawsh 2y ago
  I am deep in the Python ecosystem, and I love Python, but I have to admit that Python has the same issue. Reviving a medium-size project after 4 or more years usually means I have to catch up on a lot of new surprising deprecations. That's not because there's anything wrong with Python; it's more of an economic issue: the authors of active libraries have little or no economic incentive to support old, deprecated versions, so they just don't. That's life in the modern world. It is a deep problem that should theoretically affect every large software ecosystem because very few library authors can predict the future with great accuracy, and very few open source library authors have any significant incentive to support old ideas.
  - tmpz22 2y ago
    > That's life in the modern world. It is a deep problem that should theoretically affect every large software ecosystem because very few library authors can predict the future with great accuracy, and very few open source library authors have any significant incentive to support old ideas. I disagree. This is an easy problem to avoid with minimal due diligence, people just choose convenience and make unnecessary tradeoffs. * Use the standard library (ironically not available for Node projects). It will be built with better backwards compatibility almost every time. What deprecations do occur will likely be VERY WELL documented with much quicker adaptions. * Limit third party dependencies. Do you really need an ORM for your apps 40 sql queries? How long would it take you to scaffold it with GenerativeAI then make it production-worthy without the ORM? 1 hour? 5 hours? 20 hours? * Pick technologies with better track records. Maybe don't use Beta software like Swift Data for your iOS App. Maybe choose golang for your API even though it'll take a little bit longer to build it.

deleted 2y ago

[deleted]

beardyw 2y ago

I'm pretty sanguine about languages and frameworks, but I draw the line at node. I have seen so many horrors visited by dependencies, often to do just one thing where 2 or 3 lines of code would do the job anyway. When I was managing teams, whatever the language, I would ban any new dependencies which I didn't personally agree with. A lack of control just creates a nightmare.

CognitiveLens 2y ago
Was that kind of control well-received by your teams? Out of context, it sounds like it would be pretty rough to be an engineer on a team where your manager had sole control over what tools you could use - I suppose it might make sense for junior devs or a very small codebase, but I would caution against taking that stance in a team where you want to facilitate mutual trust
- beardyw 2y ago
  Well back then there were fewer options, but the result was that completed products were easy to work with. Perhaps we live in different times.
- mst 2y ago
  Provided the manager only rarely exercises the power, and is open to being persuaded not to, having somebody able to veto risky dependencies can be really quite useful. Normally when I'm the one with that power we rapidly get to a general understanding of what's small enough that I (a) probably won't care (b) will take responsibility for tweaking the schedule to makre time to get rid of it if I do. And 'big' dependencies are generally best discussed amongst the entire team until consensus is reached before introducing one anyway.

everythingishax 2y ago

This is literally every "hot new thing" since 2000. It is systemic. Part of it is due to too many people creating systems on the fly with too little forethought, but also because there aren't enough "really smart people" working on long term solutions. Just hacks, done by hacks. What did you expect when the people writing the systems don't have long term experience?

int_19h 2y ago
The problem itself is old, but the extent to which it pervades different ecosystems varies. It's largely a cultural thing, and the problem with JS/Node ecosystem specifically is that most of the community (or, perhaps, rather most of the library/framework authors) accepts this kind of thing as normal.

kylehotchkiss 2y ago

Dealing with node-gyp cost me at least 5 hours a month in the 2010s. I'm so very happy to not see those errors in my console anymore.

speedgoose 2y ago

I would heavily recommend to avoid NodeJS packages that depend on node-gyp. Node-gyp powered dependencies are very seldomly worth the hassle. If you must depend on node-gyp, perhaps use dev containers so at least every developer in your team can work most of the time.

graypegg 2y ago
So I'm pretty uninformed about the guts of node-gyp, and why it's used, but if people need to bring in dependancies from outside javascript... could WASM be a good fit there? Could store the binaries instead, and ship those... and in theory (correct me if I'm wrong) that shouldn't be much of a security issue due to the security model of WASM modules... or at least equal to the risk of running arbitrary build commands on your machine from a random node package.
- int_19h 2y ago
  In principle, yes. In practice, the problem is that getting some random native library or tool compile with wasm as a target is not always easy. E.g. anything that relied on pthreads was out until fairly recently.
- _fat_santa 2y ago
  In practice you're just kinda stuck with it because whatever NPM package you're using is using that under the hood. One of my project depends on it because of postgres DB bindings, there would be no easy way for me to get rid of it without either finding another binding (which that is the official one) or rebuilding it myself which will just take too much time and effort for what it's worth
eterm 2y ago
I don't even know what node-gyp is, but I know it appears regularly in error messages to know it causes problems. I don't even develop against Node, it has just crept into our front-end build toolchain.
- 2y ago

melbourne_mat 2y ago

First thing I would have done is upgrade the version of Gatsby to latest. Did the author try that? If upgrading is difficult because of 4 years of breaking changes, blame Gatsby for not being backwards compatible. Also blame your original choice of going with a hokey framework. Speaking of hokey framework: 167 dependencies and 3000 versions of Gatsby in npm.

baq 2y ago
blaming anything or anyone gets you exactly zero seconds closer to getting the job done.
- tomgp 2y ago
  You could perhaps reframe "blame" as identifying the source of the problem and iunderstand why it can be a useful excercise (aslo none of us here are trying to solve the problem really, just wasting time on the internet). In this case Node and it's atendant ecosystem are certainly a part of the problem but I would agree that Gatsby is a bigger part of the issue as they don't seem to have any interest in taming the Node dependecy management beast. I've had to dig into Gatsby projects mere months old and it really was like opening a can of worms.
  - Eric_WVGG 2y ago
    > In this case Node and it's atendant ecosystem are certainly a part of the problem but I would agree that Gatsby is a bigger part of the issue I disagree completely. Regardless of what you think of Gatsby, Node versioning is a simple problem that affects nearly every javascript project. It should always be the first thing you check. I made this dumb obvious mistake again just last week, looking for a little time to audit all my `package.json` files for `engines`.
Eric_WVGG 2y ago
The first thing I would have done is check the version of Node. That's a quick fix, upgrading a framework is a guaranteed min hour of poking around before the system is even running. I dunno why `engines` isn't in every `package.json` file, would certainly have saved me hours of nonsense.

adamtaylor_13 2y ago

Acknowledging this is absolutely awful, and also commenting that a project .nvmrc file is your friend!

baq 2y ago
...until the node version you locked can't be downloaded anymore, or hasn't ever existed for your CPU arch.
- lvturner 2y ago
  I see you too had to run node v14(? my memory fails me somewhat) on Apple Silicon hardware...
  - baq 2y ago
    shivers
  - rkharsan64 2y ago
    FWIW, I run Node 12 painlessly on Apple Silicon using fnm, so you might be thinking of a few versions before that.

hinkley 2y ago

I joined a node project that was stuck on 0.12 while 7.0 was being developed. It was a shit show to get us to 6. As I recall, 10 was a little tricky, 12 and 16 had a lot of head scratchers. I finished the 16 upgrade more than a year after the last person tried, and it was a dumb luck epiphany that kept it that short. I had a similar experience with emberJS when it was still young. Every time I picked the project up I had one to two hours of upgrade work to get it to run again, and I just had a couple hours to work on it. So half my time went to maintenance and it wasn’t sustainable. I’m trying a related idea now in elixir and José may be a saint. Though I fear a Java 5 moment in their future, where the levee breaks and a flood of changes come at once.

cynicalsecurity 2y ago

The tragedy of running an̶ ̶o̶l̶d̶ Node project.

cookiengineer 2y ago
I had to chuckle after I read your username. Kudos.
mst 2y ago
On the one hand, it's not that terrible and most *of* the terrible is from people making silly choices. On the other hand, there's a reason I regularly get annoyed enough at it to call it nope.js. On the gripping hand, I mostly write perl, which argues for a different but unique set of masochistic tendencies on my part. (you just have to remember that what 'perl' *really* stands for is 'Perenially Eclectic Rubbish Lister' and then you will have appropriate expectations and can settle back and have fun ;)

Hayabusaa 2y ago

Have you tried DevContainer before ?

em-bee 2y ago

i could not tell from the article whether this was a site with a backend using node.js or if it was just a frontend depending on node.js for the build tools. for the latter i get around the problem by avoiding build tools altogether. i use a frontend framework that i can load directly into the browser, and use without needing any tools to manage dependencies. the benefit from that is that it will ensure that my site will keep running for years to come, even if i leave it dormant for some time. the downside is that it is probably less optimized. but for smaller sites that aren't under continuous maintenance this is a reasonable tradeoff. i built all my recent sites that way using a prebuilt version of the aurelia framework. incidentally just today i tried to research if i could build a site with svelte that way. well, it turns out that although it should theoretically be possible, i was unable to find a prebuilt version to do so after a few hours of searching. for vuejs i found one within minutes. i'll be learning vuejs now. see this thread for a discussion on going buildless: https://news.ycombinator.com/item?id=41479365 https://news.ycombinator.com/item?id=41479365

mst 2y ago
I've been experimenting recently, with quite some success, with having a 'libs.js' file that pulls in and re-exports everything external I want, and providing a script that applies 'bun build' to just that. I haven't yet decided if/how I want to include a prebuilt version of it in the repo, I *think* I may go the approach of having a commit that modifies libs.js and/or the lockfile and then an immediately following one that commits an updated prebuild ... oh, huh, actually, I should probably also consider doing those two commits on a branch, then forcing a merge commit so they land on master atomically but it's easy to tease out the human changes and the regen changes by poking inside said merge commit ... yeah, like I say, still thinking about exactly how to do this, don't mind me. Also for even simpler cases I've been using the preact-htm prebuild directly, since htm gives a lit-style html() tagged literal consuming function that can produce vnodes for preact so I can mess around without needing something that understands jsx between my editor and my browser window. vue's component system is IIRC noticeably less nice to work with if you don't have a compile step, but it's still pretty nice even without that so please don't think I'm trying to dissuade you here :)
- 2y ago

nasmorn 2y ago

Good old node-gyp. I have absolutely no idea what it even is but it has been giving me errors for what feels like a decade. Mostly via front end build stuff from various projects I have worked on

mkoryak 2y ago
i forgot all about node-gyp. The only memories I do have of it are the errors and thinking about gimps.
pavel_lishin 2y ago
Same. One day I'll find out what it is.
- enqk 2y ago
  if you want to know, it's a fork of google gyp, which is a C/C++ project/build system generator. I.e. it's a bit similar to CMAKE, a tool to describe native code projects and what needs to be built in order to make executables and dlls. It's a python codebase, largely abandoned by google. They used to use it for building Chrome.

keybored 2y ago

I had to build some project that uses some Ruby package manager. I forgot already what the package manager is called. I got some error about "you don't have all the dev tools". So I installed what Google told me "dev tools" was. Then it still told me that I needed more dev tools. Stackoverflow had some question about this package manager. For Windows (Linux here). 20+ answers, mostly for Mac. All in the style of "this random thing worked for me". All with at least one upvote. Some answer about "I needed to symlink this system library". Gave up. Then I ran `devbox init` and installed whatever it told me that was needed. `devbox shell`.

pavel_lishin 2y ago

> node-gyp We're in ... let's call it a transitional period at work. I've got something like a dozen versions of node being managed by asdf. And in half of the projects I work on regularly, I consistently get warnings about this particular project failing to build. One day, I'll actually look up what it actually is, and what it does, and why it's being built, but is apparently optional.

mst 2y ago
It's basically a set of tools to make building native modules easier, that said modules then use to deal with their binding to C/C++/etc. code. Everybody complains about it, and understandably so, but if it didn't exist you'd probably instead have one set of potential similar problems *per* native module which has a good chance of not actually being better overall. The counterargument is, I guess, "well, only people who can write their own high quality build setup in-tree should be writing things that bind to external code," and I do sometimes dream of that, but it's not hard to see the downsides of living in *that* world instead either.

philipwhiuk 2y ago

Native code in an npm module should be regarded as a massive red flag.

conoro 2y ago

I had this exact problem with multiple Node blog engines in the past. Constant version breakage was incredibly frustrating. I eventually moved to Hugo. A single binary which I committed with the blog files. Zero issues even years later. I can build the blog on any new machine within seconds. Which was the other revelation of Hugo. 10 seconds to build an 800+ post blog vs minutes using Hexo or similar.

nwhnwh 2y ago

How is that pretty hard?

iamsaitam 2y ago

At first I thought it would be a decade old project, but 4 years isn't old by any standards is it? Anyways, npm ci should have been the first attempt, not npm install so that it installs the same package versions defined in the package-lock.json. Then as others have mentioned, pin your node versions. If you're afraid of this happening again, npm pack is your friend. In the end, op could have done a bit more. BUT I'll give it to him that when bindings are involved, these things take more time than they should

Aeolun 2y ago

You spent only two hours on this and you think it’s too much? Also, do not run shit on a node version that is years out of date and out of service. Also, update your damn packages. I know I sound cranky, but running anything internet facing with god knows how many vulnerabilities in is an exceedingly bad idea.

SeriousM 2y ago

We use package.lock and docker image with local folder binding to run legacy node projects. Eg. docker run -v local:inner node:12 command

nenadg 2y ago

yeah? now try running 4 years old React project, it's a hell on earth.

molszanski 2y ago
next.js user? :D

noplacelikehome 2y ago

I’ve started to adopt Nix devShells to help keep a record of each project’s dependencies. If Nix is too heavy, the learning curve for tools like asdf-vm and mise is much lower and offers similar benefits. I really wish there was a good equivalent for Windows.

0points 2y ago

Lately I have been revisiting some older golang tool I wrote since before they introduced go modules. "go mod init" + identify a working dependency version was all I had to do on any of those 10+ year old projects (5 minute work tops)

Etheryte 2y ago

Can't help but feel that this is a massive nothing-burger. You wouldn't generally expect your Java project to run if you use an incompatible version of the JVM, nor would you generally expect your C++ project to build if you swap one compiler for a different one. Etc, always specify what your project relies on, whether it's in the readme or in the dependency tree.

shepherdjerred 2y ago
People love to hate on frontend/Node
jcelerier 2y ago
> nor would you generally expect your C++ project to build if you swap one compiler for a different one. it's absolutely the expectation for C++ projects to support building on GCC, Clang and MSVC
- Etheryte 2y ago
  Maybe these days this is the case, last I used C++ about a decade ago though, this was most definitely not guaranteed to be so. This thread [0] is a good quick overview of the type of issues you used to run into, and often you didn't even need anything fancy to hit those edge cases. If life has improved in the meanwhile, that's good of course. [0] https://stackoverflow.com/q/51866968/1470607 https://stackoverflow.com/q/51866968/1470607

incrudible 2y ago

This goes for both node and python: Avoid native extensions. For python this is less feasible due to its inherently poor performance, so limit yourself to the crucial ones like numpy. For node, there are few good reasons why you would need a native extension. Unless you have your node version pinned, it will try to find the binary for your node version, fail, then attempt to build it on your system, which will most likely fail as well.

jollyllama 2y ago

For sure. This is the number one reason I am switching as many projects as I can to HTMX. https://dubroy.com/blog/cold-blooded-software/ https://dubroy.com/blog/cold-blooded-software/ Sibling comments say in so many words, it's no big deal bro, just update. But it is a big deal over time if you have dozens of cold-blooded projects to deal with.

shepherdjerred 2y ago
Gatsby and htmx have approximately zero overlap
- jollyllama 2y ago
  The point is NPM. HTMX helps you avoid NPM.

bsuvc 2y ago

Having CI would have avoided this problem.

abejfehr 2y ago
Not for running the project locally
- bsuvc 2y ago
  I think you're missing the point. CI solves it because it proves that it can build in the pipeline, using a well defined environment. No guessing at which node version you need or any other dependencies that may be required.
negus 2y ago
What you mean here is actually deps pinning + vendoring. It's way out of scope of basic CI concept.
jcelerier 2y ago
CI doesn't help when all the dependencies aren't locked, or where some deps disappear, or ...

anonzzzies 2y ago

We run node code that's 10 year old. No one dares to touch it; we just run it in docker and hope nothing goes wrong.

JansjoFromIkea 2y ago

node-sass is to blame for like 95% of these node-gyp issues in my experience, it's not that much grief to deal with but it's hard to grasp how it was allowed to hang around so terribly for so long

pier25 2y ago
> it's hard to grasp how it was allowed to hang around so terribly for so long It's not hard. The explanation is simply that the dart version of SASS (the sass npm package) is much slower than node-sass. At one point it was like 20x slower and I believe right now it's about 2-3x slower. https://github.com/sass/dart-sass/issues/1534 https://github.com/sass/dart-sass/issues/1534

__MatrixMan__ 2y ago

> Two hours of my life gone, just to pick up where I left off. If I had only wasted two hours every time I had to use npm for some reason I'd be significantly ahead of where I am now.

kiru_io 2y ago
Doesn't that apply to any it-issues you don't know how to solve?
- __MatrixMan__ 2y ago
  no, most of the time when a README says: > run command xyz I run xyz and I'm on to the next step in an average of ten or fifteen minutes (because yeah, stuff still goes wrong some times). But if the instruction is to invoke npm, there's always some crazy side quest involved. It doesn't get any better with time because next time around it's a totally different side quest. I can't even offer any criticism towards fixing it, it just sucks for unique new reason every time.

RadiozRadioz 2y ago

I call this phenomenon "node rot". Judging by the comments here, it seems like a universal experience. My favorite is the way that Python projects rot. Not only does Python's setuptools give you all the fun that node-gyp does, the common practice of versioning packages with packagename>=1.25.5 means you're almost guaranteed breakages as pip installs newer versions of packages than what the project was built with.

gre 2y ago
I just today tried reviving an old v12.4 node project and node-gyp is trying to use python2 which I don't even have on my Macbook anymore.
- ziml77 2y ago
  That Python 2 crap hit me as well when working on an older project. It used node-sass which depended on node-gyp. Dealing with upgrading that was a pain because I had to upgrade so much more to get versions of everything that played nice together. What really pissed me off about it was that Python 2 was already known to be nearing EOL when our project was started, so node-gyp should have been upgraded to work with Python 3 by then. And even more annoying was that node-gyp already had Node to run on, so why in the world was it coded to depend on Python at all!?
lexlambda 2y ago
And project specifying "requires python 3.X+" instead of Version X to Y is also a major culprit I often encounter. Most of the times it will not work with the newest shiny python, which I only notice after already installing it and then having searched search the Github issues.
- __MatrixMan__ 2y ago
  That happens to me all the time. It helped cement my habit of binding the python version to the project with direnv and a flake.nix so I end up switching to the right version when I cd to the project dir.

betimsl 2y ago

OP is tired after 2 hours of work.

andai 2y ago
OP is tired after two hours of stupid bullshit that should not ever occur. Stuff like that's enough to make a man quit a job, or indeed a career.

dmuth 2y ago

This is one reason why whenever I build a new project, I build it inside of a Docker container. That way, the project has just the dependencies it needs, and I know I can rebuild it at some point in the future and will be unlikely to run into problems when I do.

brokencube 2y ago
My first step now when trying to resurrect old projects is to create a Docker container for it - that way I can install any old versions of anything (like node or PHP) I need without having to worry about it polluting anything else on the system.
Cu3PO42 2y ago
This works as long as your Dockerfile is reasonably reproducible and does its best to lock dependencies. However, this approach has failed me a couple of times in the past. For example, I rebuilt a container some weeks later, in the meantime a new version of clang had been released that just so happened to break my build due to a bug. I personally use Nix these days, but the complexity is too high for me to recommend it to everyone for every software project.
- microtonal 2y ago
  Yeah, Nix pretty much solves this problem. The other day I wanted to try a really old version of spaCy for fun/historic interest. spaCy 1.8.2 installed freshly from the binary cache on NixOS-unstable as if it was still April 2017.

andai 2y ago

Not defending node here (I had OP's experience almost verbatim), but I had a much worse experience with trying to compile PaulStretch (a C++ project). The dependencies were specified as a bunch of random third party URLs, half of which had gone offline. I ended up giving up after a few hours, and then finding a fork that Just Works.

hombre_fatal 2y ago

OP was trying to install an old dep tree of gatsby on a different node target. These kinds of massive libraries break all the time. Look how big its dependency tree is: https://npmgraph.js.org/?q=gatsby https://npmgraph.js.org/?q=gatsby Fortunately this mindset has been changing in the node ecosystem with projects like https://hono.dev/ https://hono.dev/ (koa/express successor) and https://github.com/porsager/postgres https://github.com/porsager/postgres having zero deps.

brunoarueira 2y ago

I think many issues goes with bad version or, in some edge cases, not vendor dependencies. I'd good and bad experientes on multiple programming languages, some bad examples: - bump patch or minor version from a react package but the maintainer rewrote the entire project breaking a lot of things, following semver it's bad to expect things don't break like that for such version; - another example, the ruby gem is removed/yanked from rubygems.org and you'd to find a fork available On the end, we need to ensure the good practices from software engineering about tests and good release management, the last btw is decades old

gr4vityWall 2y ago

I recently migrated a project to Node.js 8 (!) to Node.js 14 (hopefully just the beginning), and I can relate to this post. In the JS ecosystem, I'm aware that Meteor is one major framework that takes backwards-compatibility seriously. Updating a project on an ancient version to a less-ancient version usually is not too hard. They try to keep APIs the same and introduce compatibility packages where possible. Meteor 2.16 to Meteor 3 introduced major breaking changes due to an underlying technical issue that had no workaround. They had to refactor the whole project from using Fibers-based concurrency to typical async/await. node-gyp in general has also been a source of issues in the past for me as well. More recently, ESLint changed their configuration file format and all existing tutorials suddenly became outdated. I firmly believe the ecosystem does not have to be like this, and we would save a lot of man-hours by being more committed to API stability where possible.

yawnxyz 2y ago
> Meteor 2.16 to Meteor 3 introduced major breaking changes That's when I picked up the Node/React ecosystem... > node-gyp errors > downgrade to 12.2 This is what I did until Vercel decided to not support Node 12 anymore...
forty 2y ago
After node 10, things started to stabilize. These day a node upgrade is a tiny commit just changing the version number in the Dockerfile and package.json

Havoc 2y ago

Node/JS seems particularly fragile in this regard thanks to the complicated maze of dependencies and sub dependencies and flavour of the month framework syndrome

exabrial 2y ago

2020 is not "old". "Not Invented Here" is whats going on. Developers of this age need to learn this. A recent exmaple is the RPI Foundation nullifying thousands of internet tutorials renaming "/boot" vs "/bootfs". Ask yourself a serious question, did that actually improve anything? No it did not.

aronhegedus 2y ago

Cool that you managed to get it running after just 2 hours. Same thing applies to python projects, a little note in README saves so much time in the future, I always try and use virtual environments and specify a specific python version so I can just nuke and reinstall everything

mostlysimilar 2y ago

DHH has said this experience is a big reason Rails is pursuing a no-build approach.

em-bee 2y ago
can you elaborate or link to an article that explains what this means?

bastawhiz 2y ago

All of the problems here ultimately came down to packages that used the native Node API. You don't need python or deal with C++ to run JavaScript. Node is an active project. If you build against the native API and don't pin your version to avoid breaking changes between versions, this is what happens. In my experience, JS very rarely breaks between major Node versions, but almost every native package requires a new major update. This isn't a Node specific problem. Go ahead and upgrade your Go or Python version.

turnsout 2y ago

Running a new Node project is nearly as problematic… The ecosystem is broken

contingencies 2y ago

FWIW, I've mostly maintained long term PHP projects, and I've had nearly unaltered codebases running for ~25 years since php3. No frameworks, just core PHP. People dump on PHP, but it's a very good tool if you're focused on maintainable output and pick the right functional APIs to cede to mature unix tools. You can expect decades of solid, maintainable output. Experienced programmers will not pick up a "built on shifting sand" stack, because they can acutely perceive the pain and suffering before it happens, generally from past experience. With fast-crumbling stacks, you need to execute quickly and move on, and treat the whole codebase as an expiring entity. Stacks I personally try to avoid: anything node/javascripty, anything Androidy, anything iDevicey. Those who don't understand Unix are condemned to reinvent it, poorly. - Henry Spencer ... via https://github.com/globalcitizen/taoup https://github.com/globalcitizen/taoup

stall84 2y ago

It's not just personal blogs as many of y'all know.. this is a daily struggle in any SDE mainly working in web .. I am constantly lamenting the fact I spend like 70% of my time at my main job (large financial company) just trying to get my environment , or the application's environment in a position to actually develop on the application itself .. It's insane. I f'n love javascript .. it's allowed a lot of us a doorway into software engineering where a lot of us realize how ... 'special' js and related web development are ;) .. But man it can make you really want to smash the computer some days.

YuukiRey 2y ago

This is the reason I ripped out Gatsby from every project where I could. Every six months I’d spend an entire evening fixing obscure problems that shouldn’t even exist just to get things running. And that’s not even considering actual breaking changes of which there were plenty.

z3t4 2y ago

You should save your deps in your SCM! Microsoft is giving away ownership to existing packages if you tell them you will use it for a TypeScript project.

shepherdjerred 2y ago
What? Does Microsoft control npm?
- switch007 2y ago
  Yes. npm is a subsidiary of GitHub, a subsidiary of Microsoft

kiettv 2y ago

I still have to maintain many old projects that are using Node 7, but many dependencies are no longer available. Every update needs to patch the docker image manually from the last runnable docker image.

sn0n 2y ago

Made a site using nanogen ( https://github.com/doug2k1/nanogen https://github.com/doug2k1/nanogen ) about 7 years ago ... Anytime I setup a new machine I do a npm install on whatever version of node I end up on, and it..... Just works. Beat SSG I've found, and all from a medium or dev article on a SSG in 40 lines or less.

deleted 2y ago

[deleted]

stuaxo 2y ago

This could equally be written about old Android projects.

foxyv 2y ago

I think this was the biggest mistake that Java made. Breaking reverse compatibility after Java 8 means that thousands of organizations will never leave that version. There is an entire industry based around maintaining Java 8. Eventually there will be two versions of Java, just like Python 2 and 3. There will be Java 8 and Java 698.

tacone 2y ago

My take is one should be especially wary of packages that depend on C libraries and need compliation. You get to be extremely bound to what the OS distribution has to offer. If that's the case, docker is probably the safest solution.

droptablemain 2y ago

As a rule of thumb if you are jumping into a legacy node project the first thing you should try is a sequentially older node version. Could've reduced his two hours to two minutes.

banku_brougham 2y ago

Anybody have experience with pulling/running a docker image after many years? I'd like to see the consensus on most durable software packaging method.