10 ms·

The Node ecosystem still has tooling problems

84 points by MaxLeiter 4y ago

interactivecode 4y ago
Understandable frustrations but without a comparison to other major languages in the same domain its a bit moot. Is it just the Node ecosystem that has these types of problems?
- antonvs 4y ago
  In my experience, Node is worse than any of Java, Rust, or Haskell. I think a couple of reasons for that is that it's untyped, and not compiled. Both of those mean that basic correctness checking ends up happening on the user's machine, rather than the developer's.
  - jitl 4y ago
    > not compiled Ironic you say this, since the whole blogpost is complaining about setting up compiler pipeline to target NodeJS.
    - antonvs 4y ago
      That fits in perfectly with what I was saying, I just wasn't precise enough. The issue is that Node.js source code is not compiled to a target language like native code or bytecode. As such, there's no package repository from which you can retrieve compiled packages - you're forced to deal with the same build process as the developer of the package. If that weren't the case, the compiler pipeline that the post is complaining about would be eliminated for package users. In fact, the author of this package could have avoided some of his pain by publishing a library with the compiled JS code, which would eliminate all the config of higher layers. But the JS ecosystem doesn't support making that distinction, there's no source package repo vs. compiled package repo (afaik - I don't use it more than I have to, because it all sucks.)
      - wonnage 4y ago
        I think you misunderstand... you don't have to worry about building packages at all (excluding some edge cases like native C++ extensions) when you install stuff from npm.
      - Macha 4y ago
        Note the convention in npm of using a dist/x.min.js or x.dist.js to use NPM to ship prebuilt modules (usually UMD as the ES6 world still mostly just expects you to use a build took, but e.g. Vue at one point at least had prebuilt es6 modules too).
  - reirob 4y ago
    Ironically the existence of NodeJS is somewhat related to Haskell, as Stephen Diehl mentions in [1]: > As a funny historical quirk, back in 2011 there was an interview with Ryan Dahl, the creator of NodeJS, who mentioned that the perceived difficulty in writing a new IO manager for GHC was a factor in the development of a new language called NodeJS. When asked why he chose Javascript, for the project, he replied: >> Originally I didn’t. I had several failed private projects doing the same on C, Lua, and Haskell. Haskell is pretty ideal but I’m not smart enough to hack the GHC. The interview itself is in [2]. [1]: https://www.stephendiehl.com/posts/decade.html https://www.stephendiehl.com/posts/decade.html [2]: https://www.bizjournals.com/boston/inno/stories/news/2011/01/31/nodejs-interview-4-questions-with-creator-ryan.html https://www.bizjournals.com/boston/inno/stories/news/2011/01...
- josteink 4y ago
  Node succeeded by “simply” being plain JS on V8 with a really small standard library. Easy! No nonsense! Everyone knows this already! Unfortunately that left everything else up to the community to solve: establishing a “good” standard library, building, debugging, packaging, bundling, dealing with the Node/Browser duality in a meaningful way, creating usable network frameworks, UI frameworks and surely much more. And they’ve all ended up reinventing that 100 times (like the curse of lisp) in 200 different ways, all recursively based on Node, while standing on the shoulder of non-giants trying to solve one of the other problems mentioned above, possibly standing on the shoulder of someone else who has already tried to solve what you are trying to solve now. Calling it a clusterfuck probably isn’t sufficient, yet at the same time it seems to work, so it’s simultaneously a sort of modern day miracle. Disclaimer: have set up quite a few node build-pipelines.
  - yulaow 4y ago
    Of npm (the online database of packages) I hate the fact they never forced a clear distinction between package meant to be used only in the backend with node, package meant to be used only on the fe inside a browser, and package which can be used in both. I mean just having a flag for that would be useful, simple and solve a lot of headaches
    - andrew_ 4y ago
      The beauty is that those distinctions don't exist. There are no limitations. There are also no guardrails or training wheels.
      - wizofaus 4y ago
        In what case would an npm package for talking to an external database be actually useful for a frontend project? I mean, yes, if you're only ever running the tool inside a LAN where that access exists, I suppose it's conceivable... And plenty of frontend packages that rely on storing state in the browser's window object (Redux etc.) would presumably be a bad idea to use for most backend projects, but no doubt somebody's found a way to make it work. I suppose I'm not disagreeing with you, but the GP kinda has a point too.
- vbezhenar 4y ago
  I don't think that Java would be different if one would want to replicate all features that author felt necessary. I'll speak for Maven as it's the most popular build tool and most straightforward. With gradle it would be worse. To build two configurations you ideally want at least two modules. So multi-module projects. Weak point for maven. Doable, but with lots of quirks. Last time I checked, to publish Java library, you would need to GPG sign it. This fact alone is a serious stuff. Pushing library to internal repository is easier. Tooling in maven is terrible. Plugins are archaic, some of those seen last commits 10 years ago. I don't even know which tools to autoformat Java code are popular nowadays. I think that most people just use their Idea to format before commit with shared style or something like that. I think it's good enough. Github bot to update dependencies and deploying demo site - I don't know, never did that, but I couldn't imagine it being easier with Java either. So Java tooling has its share of quirks. Most sane tooling I ever saw was for go, as long as you're ready to adjust your expectations and requirements to happy path. What's unique for node is huge numbers of tooling attempts. Java has Ant, Maven, Gradle. I don't think I've ever heard about anything else. Bazel may be. Well, may be Makefiles for some old beards. JavaScript guys just can't stop trying to invent new tools. And this fragments ecosystem and user base. I don't know why is that. I've never heard of alternative Rust build system, for example.
  - Zardoz84 4y ago
    You know that maven have profiles that are tool to use when you need different configurations, not ?
  - kaba0 4y ago
    > So multi-module projects. Weak point for maven But not for gradle. > Last time I checked, to publish Java library, you would need to GPG sign it. This fact alone is a serious stuff Why is it a problem? This is partly the reason why the Java ecosystem wasn’t hit by all the malicious package fiasco. > Github bot to update dependencies and deploying demo site Never felt a need for this, but contrary to your misrepresentation, even maven plugins have a healthy ecosystem and being part of the 3rd biggest language, I would be honestly surprised if no such plugin would exist.
- ActorNightly 4y ago
  Its a fundamental issue with JS and DOM. Its understandable that there needs to be a need to manipulate elements on the webpage, but the issue is that with JS, you can basically create the webpage programmatically from the ground up, but you are doing it through essentially concatenating text into html/css. Even worse, JS can actually modify itself. You can take defined functions that have definite common behavior. and overwrite them to arbitrary ones. The equivalent to a low level language like C would be writing your code with half C, half custom macros that are #defined at random places in the code libraries that you pulled in that change core C instructions, and instead of regular C code you are concatenating strings with assembly instructions that would then be put into a file and ran. No other language that I know of allows for this.
klodolph 4y ago
This is really short. I think my complaints, compiled into an article, would probably fill out an article five or ten times as long. There’s just so much shit. Everyone made their own system. There’s .js .ts .mjs .cjs extensions. Theres’s tsc and jest and npx and create-react-app. These are just the tip of the iceberg. If I try to update the build system to use newer versions of the software, all sorts of stuff will break. The really shocking part, for me, is that I feel like it’s easier to set up my build system for C than for JS.
- ramesh31 4y ago
  It’s beyond repair. The future is Rust, Go, and Deno.
- ravenstine 4y ago
  Yes, there is a lot of shit, but stick to ES2015+ and you'll be mostly fine. No reason to continue with the .cjs .mjs crap. Either set "type": "module" or use Deno (the superior option) and don't look back. If others still want to use Common JS, that's their problem. Yeah, a lot of modules won't work out of the box because they're Common JS, and visa versa, but screw Common JS. That and goofy junkware like Gulp and Grunt which are overcomplicated, change their APIs frequently, and don't really do anything better than a Makefile. Ok, Gulp might be helpful if you're running Windows, but I pity those who still do web development on Windows without WSL2.
  - andrew_ 4y ago
    Gulp and Grunt? Are you a time traveler from 2012?
    - jeroenhd 4y ago
      Both still work fine to this day. I don't know what the exact (dis)advantages of the competing toolsets are, but if you know Gulp and Grunt then I don't see why you wouldn't use them.
      - 4y ago
jitl 4y ago
The transition to ES6 modules made this much worse. Before that, for a npm package you could basically just call tsc on your entry point and then type `npm publish`. Now with the ES6 situation various dependencies and common packages demand you upgrade to ES6, and customers also ask for import syntax so their builds can do “tree shaking”. I maintain a Emscripten-based typescript library and the Emscripten wrapper code is allergic to ES6 imports - everything breaks if you do that since it removes various node globals. I wasted hours trying to figure out a way around, or a fix in Emscripten itself before giving up.
moltar 4y ago
I think projen may solve the configuration and tooling woos. At least I hope it will.
- LelouBil 4y ago
  Thanks for talking about it, I just discovered it.
- jitl 4y ago
  I’ve seen enough of these config wranglers come and go (and built/maintain a few myself) to know that the mismatch between user needs and project templates cannot be bridged. To be happy with a tool like this, you need to use the vanilla output. If you need anything more you’ll be in a world of hurt. But unfortunately often real world apps begin to need something more. Perhaps this time will be different? I hope so but I’m not optimistic.
  - moltar 4y ago
    The major difference with projen is that configurations are not set and forget. They can be updated by a normal dependency update. So as tools evolve and APIs get deprecated, the projen component maintainer can adjust the output of the config, which updates your project config with one command. And as community we can all pitch in at keeping these components up to date. Rather than every individual trying to figure out for themselves what is the right combo of options to make something work.
weaksauce 4y ago
my big gripe with npm is that the global packages overshadow the local packages... I can't understand when that would be a good option other than maybe having an option to do so but not being the default. I can't think of another package manager that does this. oh that and installing thousands of packages in the project root instead of a sensible place that could be shared per version like bundler and rubygems.
- GauntletWizard 4y ago
  You can't do a package cache, because nasal demons erupt whenever you use a package that's not precisely equal and all packages use their own copies of all their dependencies. Sane package managers don't try to solve for this problem; they make it clear when you're using bad software.
  - weaksauce 4y ago
    Bundler does this just fine. installs every single exact dependency in a central location and then sets the relevant paths to them before running ruby. It works very well. some of the people behind bundler wrote the much loved package manager for rust. I'm not sure where you are getting the thought that other systems haven't done it better.
    - GauntletWizard 4y ago
      That's exactly the behavior that makes NodeJS unsuitable as a package manager. Some resources are, in fact, globals. You should have a consistent enough API to use those through code that is imported and singular, rather than through importing a dozen versions of the API for each dependency you have.
- andrew_ 4y ago
  The globally installed packages only take precedence over local packages when you run the binary name from the command line. To get that to stop, you'd have to add the local node_modules/.bin to the PATH. Additionally, if you run npx/yarn/pnpx in a package script, it will use the local binary. > oh that and installing thousands of packages in the project root instead of a sensible place that could be shared per version like bundler and rubygems. Look into pnpm. It does precisely this in a way that's compatible with node. Honestly the information is out there to address most people's gripes.
hugozap 4y ago
Tooling in the JS ecosystem (specially frontend) is very frustrating. It's just layers on top of layers with a weak foundation and a whole set of annoying incompatibilities between them.
- deleted 4y ago
  [deleted]
andrew_ 4y ago
The node ecosystem doesn't have a tooling problem. Frontend development has had a tooling problem since the introduction of React and Angular, and TypeScript. One might argue that goes back as far as CoffeeScript. But can we stop boosting these "OMG this was too hard" posts? TypeScript (tsc) should display a hide-able message on invocation that states: "TypeScript isn't straightforward. It's a vast ecosystem all its own that accommodates several million developer's individual needs. It can be overwhelming. You don't have to use it." I won't opine on the dumpster fire that is ESM support in Node - that's a completely separate topic and head-through-wall session.
- jeroenhd 4y ago
  What's the trouble with Typescript, exactly? The setup is relatively straighforward (at least, for Javascript tooling). Copy/paste a file or two and you're off to the races. In comparison any frontend framework is much more difficult to learn. Typescript itself is just Javascript with types and the fancy parts of the type system are rarely ever required.
  - andrew_ 4y ago
    I'm in agreement with you. But I'm also an old salt that's been using it for nearly 8 years. I completely understand how someone unfamiliar with the language and its nuances may be overwhelmed by the configuration options and the effects they have on compilation. The types system does take time to learn, especially for those that didn't come up on a strongly (or hell, even loosely) typed language. I also think there's enough resources out there for someone, who isn't in a hurry, to grok TS enough to put together a project that will compile. But skills and comprehension aren't uniform - I've seen many a junior fresh out of bootcamp look at a tsconfig.json file like it was voodoo. That's why I think it's fair to say "Hey, you don't have to use this."
    - mnahkies 4y ago
      One big advantage of typescript with respect to the configuration options is that you can look at the output js and see what the options are producing (with a low barrier to entry - the output is easily readable js aside from some of the older targets + async keyword) The type system is super flexible and I agree it takes time to learn, but you can get a lot of value out of the basics from day one. I've not actually tried this but I suspect even with the most lenient configuration and no type annotations you'd still get limited value over plain js through type inference and flagging of obvious bugs
alexashka 4y ago
You're describing all command line tools and config files. Maybe people will discover GUIs one day, it's only been 30 years.
- andrew_ 4y ago
  "Everything old is new again"
- antonvs 4y ago
  That's not even remotely true. For example, Rust has an excellent command line build pipeline experience. That has to do both with design decisions - a typed, compiled language has a big edge here - and differences in how the respective ecosystems were designed and developed. As for GUIs, I'll paraphrase an old quote I saw on Usenet once: if you need to point at something to communicate, you're at the cognitive level of a preverbal child.
  - chakkepolja 4y ago
    Even google languages (Go, Dart) and modern Java, no matter how people here hate them, have excellent tooling.
  - deleted 4y ago
    [deleted]
  - iamnotafreeman 4y ago
    hope that last bit was very much in jest otherwise "someone on the Usenet is wrong." visual stuff (vs. text) is about massive parallelism and compression. see Rudy Arnheim for example. or Tufte. or video games. or excel. or a good "foreign" film. or VR once it doesn't suck. https://www.ucpress.edu/book/9780520242265/visual-thinking https://www.ucpress.edu/book/9780520242265/visual-thinking
tannhaeuser 4y ago
T'was all fine until folks had to come up with useless typescript and modules and webpack churn and Es2015+ bloat. It's not like we didn't warn them to turn JavaScript into Java; for some of us getting away from Java bloat was the entire point of node.js. Well that, and because CommonJS was portable, or used to be at least. If you want a typed language, just use one, rather than foobaring JavaScript or Python into something they just aren't, a fool's errand.
- foepys 4y ago
  Since JavaScript is the only way to interact with the DOM in a web browser, you literally cannot use anything different than JavaScript, which makes your point to use other languages quite hard to follow.
  - Zardoz84 4y ago
    WASM ?
    - Tajnymag 4y ago
      WASM came up way later than TypeScript and Webpack did. Also, no direct DOM access. You still need a Javascript proxy for that.
- cercatrova 4y ago
  > useless typescript The only reason I started and continue to use JS is because of TypeScript. There is no world in which I'd use pure JS over TS.
  - tannhaeuser 4y ago
    Different folks, different strokes I guess: there's no world in which I'd use an MS-proprietary language over a broadly implemented one, let alone a proprietary language changing all the time and that has a Turing-complete type system, so I don't need to complain about bloat and churn, which is the whole point of this thread.
deleted 4y ago
[deleted]
rubyist5eva 4y ago
The node ecosystem is just a giant problem on top of problems.
trinovantes 4y ago
I just checked my most recent frontend project that uses Typescript as much as possible. Here's how it works: - src/.ts: Compile with esbuild in webpack - tests/.test.ts: Compiled with babel - jest.config.ts: Compiled with ts-node - webpack.config.ts: Compiled with ts-node In an ideal world, one tool should do everything but I doubt that will ever happen
- mikojan 4y ago
  I'm curious: Why use webpack with esbuild in a "most recent frontend project"? Not so long ago at $DAYJOB we even migrated a big legacy project from webpack to esbuild in no time.
  - trinovantes 4y ago
    I've used webpack for several years now and have written custom plugins to accomplish what I need. esbuild-loader gives great typescript performance almost on par with vite so I don't see the need to learn a new build system. Also, there's no official Vue 3 support for esbuild so that's a problem for some of my projects
spankalee 4y ago
The problems start with frameworks that require custom compilers and non-standard module systems. If you use vanilla JS in standard modules most of this goes away.
- jitl 4y ago
  Well then you need to use vanilla JS, which seems to be widely regarded as not great “at scale”. I certainly will never go back to JS from TS. Fearless refactoring is worth slogging through all the JSON config soup.
  - tobyhinloopen 4y ago
    Fearless refactoring? What about tests? Sure TS helps, but tests are “better”
    - Tajnymag 4y ago
      Tests and static typing are not excluding each other. Best would be to have both.
claytongulick 4y ago
I'm pretty happy with just vanilla JavaScript. I don't do typescript, I don't use frameworks (unless you consider express to be a framework). I use vanilla web components on the front end, if I need reactivity and frequent re-renders I might sprinkle in a bit of lit-html, depending on the component. Debugging and development are brain dead simple, just sticking to the standards. I had a new developer come into my code base and he said "I couldn't believe it at first, you just write something and it does it... Wow!". He was so used to piles of frameworks and build systems and stuff, he'd never been in a code base that just... ran.
exabrial 4y ago
> Node is a problem Fixed it for you. The node ecosystem embodies mediocrity. There is no design, no testing, no stability, no benchmarks, no RFCs, no plan to produce a coherent product. Instead, you're giving a half baked set of features that are abandoned within 8-12 months. Not invented here syndrome runs wild, and over-engineering any solution is seen as a mark of accomplishment. Simple things, like displaying static text, now require 25mb of js files for whatever framework the author is currently playing around with as a hobby. I miss the days of the web where you could just load a page in under 25kb.
- vbezhenar 4y ago
  Our node docker images are 600-1000 MB for relatively simple website (backend and frontend). 25 MB would be so good and lightweight.
  - interblag 4y ago
    Unless I'm massively misunderstanding your setup, that Docker image contains an entire OS and runtime, not just your website. Reasonable people can disagree about whether that entire paradigm is sensible but, having chosen to use Docker, complaining about image size in this way seems an odd choice. More to this point, most of what is in the image will be little to do with Node or its build systems..
    - toomanybeersies 4y ago
      The NodeJS alpine base image is only 40 MB.
      - lolive 4y ago
        Then you _npm install_ something, and BOOOM! 600MB !
        Defman 4y ago
        You can use multi-stage builds [1] so your result image would be as small as possible by just copying the compiled JS from the build stage. [1]: https://docs.docker.com/develop/develop-images/multistage-build/ https://docs.docker.com/develop/develop-images/multistage-bu...
deleted 4y ago
[deleted]
shadycuz 4y ago
I doubt anyone is interested but I'm building some tooling for node and npm in jenkins. Just standard stuff, nvm, nodenv, npm and npx wrapper.
joshxyz 4y ago
Keep in mind that most of these are just rants about the developer experience, which is sacrificed in exchange for faster iteration and deployments on majority of business use cases. As a Node.js developer myself I even noticed that some of the best solutions is nodejs plus other language. Examples uwebsockets.js uses some c++ code. it is faster and has better api than express, fastify, hapi, koa, and has builtin websockets support. esbuild uses golang code. it is faster than webpack and babel for bundling and minifying javascript code.
pfesenmeier 4y ago
Interested if anyone is using a jsconfig.json and jsdoc comments to do ts typechecking. I found it really easy to setup, I got a lot out of the extra checks, but the practice never really took off with the team of js devs. https://code.visualstudio.com/docs/languages/jsconfig https://code.visualstudio.com/docs/languages/jsconfig
- theteapot 4y ago
  How you gonna use it on a 3rd party module?
NonNefarious 4y ago
Node has been superseded by Deno, right? How about that?
johnnypangs 4y ago
I saw the author was using vite, they have a mode to help you build a library that would mitigate their woes. Check it out! https://vitejs.dev/guide/build.html#library-mode https://vitejs.dev/guide/build.html#library-mode
- MaxLeiter 4y ago
  I'm using library mode
kabdib 4y ago
I have lost whole consecutive weeks to WebPack. It made me wish I could go back to something wholesome and healthy, like template metaprogramming in C++. "I'm making progress, boss. The error output from the build is smaller than the input source code now."
hestefisk 4y ago
Rant: could we please stop referring to a collection of <X> as an ecosystem? Node has a collection or environment of tools, frameworks and libraries to support it. An ecosystem is something living, a biological system, typically with a set of symbiotic properties within it. The trend is the same in business. Stripe no longer has business partners; they have an “ecosystem.” The same with banks that provide wholesale banking services through a set of retail businesses - they are no longer wholesale banks but “ecosystem players”. This kind of jargon really dilutes the meaning of words and makes it hard to understand what is really meant. Rant over, sorry. Apart from my rant, it’s funny to see Node and JavaScript doing what almost killed Perl 20 years ago. There are so many ways of doing things, so many bespoke frameworks and packages (doing OO in Perl, anyone?), so many odd dialects that it fragmented the language and people got frustrated. The same is happening to JS right now, although with some important differences. There is hardly a fragmentation of usage via formal “dialects” in JS; rather, the problem is in the sprawl of libraries and “infrastructure” services, many of which are solved problems (e.g. Gulp reinventing what Make is really good at; Yarn reinventing what npm is supposed to do, etc). The supposed “ecosystem” is fragmenting day by day from its inherent complexity, and it’s exactly what caused web developers to flee from Perl to more simple, consistent alternatives.
- stevage 4y ago
  It's a perfectly apt metaphor which is clear and useful. Your rant seems unfounded.
- lozenge 4y ago
  An ecosystem isn't alive, it's a collection of independent species that have an emergent behaviour.
  - lioeters 4y ago
    By that line of thought, is life alive? It's just a collection of interdependent cells and molecules with emergent behavior. Seems like an ecosystem is alive collectively, in the same way that an individual is alive, just on a different scale of organization.
- 4y ago
mhoad 4y ago
It’s funny I’ve been looking at industrial strength build systems like Bazel (language independent build tool from Google) recently the past few months just to get an idea of how things work in that world. One of the things happening in that community at the moment was their second major attempt to bring the JavaScript / Typescript and Node ecosystem onboard as a semi official language. It has been a 4 year effort so far and they had to make a bunch of compromises along the way as you can see in the design section of the projects readme here https://github.com/aspect-build/rules_js https://github.com/aspect-build/rules_js Watching the walls they would continually run into where the answer would essentially be “Node is very unlike other language runtimes” was illuminating and I suddenly started to understand why projects like Deno want to actually leave Node behind as a fundamentally unfixable mess and start again with a bit more planning this time around to avoid all of those issues.
stoicjumbotron 4y ago
So what is the "recommended" way to build libraries using TypeScript? I've heard that tsdx https://tsdx.io/ https://tsdx.io/ is quite good
arealpromise 4y ago
erikpukinskis 4y ago
It’s still early days, but I have been working on this… There are scaffolding tools that help configure all the JavaScript frontend stuff you need, but the problem is you run them once and you can’t ever run them again to change/add stuff. So I built Confgen which is sort of like create-react-app except it’s idempotent: https://github.com/erikpukinskis/confgen https://github.com/erikpukinskis/confgen It’s very alpha, but I would love to get ideas/ bug reports on GitHub. It’s also currently Vite-only, but I’m open to a possible webpack/babel mode in the future.
phendrenad2 4y ago
I've started using plain old CDN-served React/jQuery/Lodash to do everything, with no build system, and daring my employer to fire me. So far so good.
FractalHQ 4y ago
I use pnpm and Sveltekit and have none of these problems. Everything just works, and it’s rare anything related to tooling takes longer than 1second to accomplish. Being on OSX helps a lot with the “just works” part though.

84 points by MaxLeiter 4y ago

Understandable frustrations but without a comparison to other major languages in the same domain its a bit moot. Is it just the Node ecosystem that has these types of problems?

antonvs 4y ago
In my experience, Node is worse than any of Java, Rust, or Haskell. I think a couple of reasons for that is that it's untyped, and not compiled. Both of those mean that basic correctness checking ends up happening on the user's machine, rather than the developer's.
- jitl 4y ago
  > not compiled Ironic you say this, since the whole blogpost is complaining about setting up compiler pipeline to target NodeJS.
  - antonvs 4y ago
    That fits in perfectly with what I was saying, I just wasn't precise enough. The issue is that Node.js source code is not compiled to a target language like native code or bytecode. As such, there's no package repository from which you can retrieve compiled packages - you're forced to deal with the same build process as the developer of the package. If that weren't the case, the compiler pipeline that the post is complaining about would be eliminated for package users. In fact, the author of this package could have avoided some of his pain by publishing a library with the compiled JS code, which would eliminate all the config of higher layers. But the JS ecosystem doesn't support making that distinction, there's no source package repo vs. compiled package repo (afaik - I don't use it more than I have to, because it all sucks.)
    - wonnage 4y ago
      I think you misunderstand... you don't have to worry about building packages at all (excluding some edge cases like native C++ extensions) when you install stuff from npm.
    - Macha 4y ago
      Note the convention in npm of using a dist/x.min.js or x.dist.js to use NPM to ship prebuilt modules (usually UMD as the ES6 world still mostly just expects you to use a build took, but e.g. Vue at one point at least had prebuilt es6 modules too).
- reirob 4y ago
  Ironically the existence of NodeJS is somewhat related to Haskell, as Stephen Diehl mentions in [1]: > As a funny historical quirk, back in 2011 there was an interview with Ryan Dahl, the creator of NodeJS, who mentioned that the perceived difficulty in writing a new IO manager for GHC was a factor in the development of a new language called NodeJS. When asked why he chose Javascript, for the project, he replied: >> Originally I didn’t. I had several failed private projects doing the same on C, Lua, and Haskell. Haskell is pretty ideal but I’m not smart enough to hack the GHC. The interview itself is in [2]. [1]: https://www.stephendiehl.com/posts/decade.html https://www.stephendiehl.com/posts/decade.html [2]: https://www.bizjournals.com/boston/inno/stories/news/2011/01/31/nodejs-interview-4-questions-with-creator-ryan.html https://www.bizjournals.com/boston/inno/stories/news/2011/01...
josteink 4y ago
Node succeeded by “simply” being plain JS on V8 with a really small standard library. Easy! No nonsense! Everyone knows this already! Unfortunately that left everything else up to the community to solve: establishing a “good” standard library, building, debugging, packaging, bundling, dealing with the Node/Browser duality in a meaningful way, creating usable network frameworks, UI frameworks and surely much more. And they’ve all ended up reinventing that 100 times (like the curse of lisp) in 200 different ways, all recursively based on Node, while standing on the shoulder of non-giants trying to solve one of the other problems mentioned above, possibly standing on the shoulder of someone else who has already tried to solve what you are trying to solve now. Calling it a clusterfuck probably isn’t sufficient, yet at the same time it seems to work, so it’s simultaneously a sort of modern day miracle. Disclaimer: have set up quite a few node build-pipelines.
- yulaow 4y ago
  Of npm (the online database of packages) I hate the fact they never forced a clear distinction between package meant to be used only in the backend with node, package meant to be used only on the fe inside a browser, and package which can be used in both. I mean just having a flag for that would be useful, simple and solve a lot of headaches
  - andrew_ 4y ago
    The beauty is that those distinctions don't exist. There are no limitations. There are also no guardrails or training wheels.
    - wizofaus 4y ago
      In what case would an npm package for talking to an external database be actually useful for a frontend project? I mean, yes, if you're only ever running the tool inside a LAN where that access exists, I suppose it's conceivable... And plenty of frontend packages that rely on storing state in the browser's window object (Redux etc.) would presumably be a bad idea to use for most backend projects, but no doubt somebody's found a way to make it work. I suppose I'm not disagreeing with you, but the GP kinda has a point too.
vbezhenar 4y ago
I don't think that Java would be different if one would want to replicate all features that author felt necessary. I'll speak for Maven as it's the most popular build tool and most straightforward. With gradle it would be worse. To build two configurations you ideally want at least two modules. So multi-module projects. Weak point for maven. Doable, but with lots of quirks. Last time I checked, to publish Java library, you would need to GPG sign it. This fact alone is a serious stuff. Pushing library to internal repository is easier. Tooling in maven is terrible. Plugins are archaic, some of those seen last commits 10 years ago. I don't even know which tools to autoformat Java code are popular nowadays. I think that most people just use their Idea to format before commit with shared style or something like that. I think it's good enough. Github bot to update dependencies and deploying demo site - I don't know, never did that, but I couldn't imagine it being easier with Java either. So Java tooling has its share of quirks. Most sane tooling I ever saw was for go, as long as you're ready to adjust your expectations and requirements to happy path. What's unique for node is huge numbers of tooling attempts. Java has Ant, Maven, Gradle. I don't think I've ever heard about anything else. Bazel may be. Well, may be Makefiles for some old beards. JavaScript guys just can't stop trying to invent new tools. And this fragments ecosystem and user base. I don't know why is that. I've never heard of alternative Rust build system, for example.
- Zardoz84 4y ago
  You know that maven have profiles that are tool to use when you need different configurations, not ?
- kaba0 4y ago
  > So multi-module projects. Weak point for maven But not for gradle. > Last time I checked, to publish Java library, you would need to GPG sign it. This fact alone is a serious stuff Why is it a problem? This is partly the reason why the Java ecosystem wasn’t hit by all the malicious package fiasco. > Github bot to update dependencies and deploying demo site Never felt a need for this, but contrary to your misrepresentation, even maven plugins have a healthy ecosystem and being part of the 3rd biggest language, I would be honestly surprised if no such plugin would exist.
ActorNightly 4y ago
Its a fundamental issue with JS and DOM. Its understandable that there needs to be a need to manipulate elements on the webpage, but the issue is that with JS, you can basically create the webpage programmatically from the ground up, but you are doing it through essentially concatenating text into html/css. Even worse, JS can actually modify itself. You can take defined functions that have definite common behavior. and overwrite them to arbitrary ones. The equivalent to a low level language like C would be writing your code with half C, half custom macros that are #defined at random places in the code libraries that you pulled in that change core C instructions, and instead of regular C code you are concatenating strings with assembly instructions that would then be put into a file and ran. No other language that I know of allows for this.

klodolph 4y ago

This is really short. I think my complaints, compiled into an article, would probably fill out an article five or ten times as long. There’s just so much shit. Everyone made their own system. There’s .js .ts .mjs .cjs extensions. Theres’s tsc and jest and npx and create-react-app. These are just the tip of the iceberg. If I try to update the build system to use newer versions of the software, all sorts of stuff will break. The really shocking part, for me, is that I feel like it’s easier to set up my build system for C than for JS.

ramesh31 4y ago
It’s beyond repair. The future is Rust, Go, and Deno.
ravenstine 4y ago
Yes, there is a lot of shit, but stick to ES2015+ and you'll be mostly fine. No reason to continue with the .cjs .mjs crap. Either set "type": "module" or use Deno (the superior option) and don't look back. If others still want to use Common JS, that's their problem. Yeah, a lot of modules won't work out of the box because they're Common JS, and visa versa, but screw Common JS. That and goofy junkware like Gulp and Grunt which are overcomplicated, change their APIs frequently, and don't really do anything better than a Makefile. Ok, Gulp might be helpful if you're running Windows, but I pity those who still do web development on Windows without WSL2.
- andrew_ 4y ago
  Gulp and Grunt? Are you a time traveler from 2012?
  - jeroenhd 4y ago
    Both still work fine to this day. I don't know what the exact (dis)advantages of the competing toolsets are, but if you know Gulp and Grunt then I don't see why you wouldn't use them.
    - 4y ago

jitl 4y ago

The transition to ES6 modules made this much worse. Before that, for a npm package you could basically just call tsc on your entry point and then type `npm publish`. Now with the ES6 situation various dependencies and common packages demand you upgrade to ES6, and customers also ask for import syntax so their builds can do “tree shaking”. I maintain a Emscripten-based typescript library and the Emscripten wrapper code is allergic to ES6 imports - everything breaks if you do that since it removes various node globals. I wasted hours trying to figure out a way around, or a fix in Emscripten itself before giving up.

moltar 4y ago

I think projen may solve the configuration and tooling woos. At least I hope it will.

LelouBil 4y ago
Thanks for talking about it, I just discovered it.
jitl 4y ago
I’ve seen enough of these config wranglers come and go (and built/maintain a few myself) to know that the mismatch between user needs and project templates cannot be bridged. To be happy with a tool like this, you need to use the vanilla output. If you need anything more you’ll be in a world of hurt. But unfortunately often real world apps begin to need something more. Perhaps this time will be different? I hope so but I’m not optimistic.
- moltar 4y ago
  The major difference with projen is that configurations are not set and forget. They can be updated by a normal dependency update. So as tools evolve and APIs get deprecated, the projen component maintainer can adjust the output of the config, which updates your project config with one command. And as community we can all pitch in at keeping these components up to date. Rather than every individual trying to figure out for themselves what is the right combo of options to make something work.

weaksauce 4y ago

my big gripe with npm is that the global packages overshadow the local packages... I can't understand when that would be a good option other than maybe having an option to do so but not being the default. I can't think of another package manager that does this. oh that and installing thousands of packages in the project root instead of a sensible place that could be shared per version like bundler and rubygems.

GauntletWizard 4y ago
You can't do a package cache, because nasal demons erupt whenever you use a package that's not precisely equal and all packages use their own copies of all their dependencies. Sane package managers don't try to solve for this problem; they make it clear when you're using bad software.
- weaksauce 4y ago
  Bundler does this just fine. installs every single exact dependency in a central location and then sets the relevant paths to them before running ruby. It works very well. some of the people behind bundler wrote the much loved package manager for rust. I'm not sure where you are getting the thought that other systems haven't done it better.
  - GauntletWizard 4y ago
    That's exactly the behavior that makes NodeJS unsuitable as a package manager. Some resources are, in fact, globals. You should have a consistent enough API to use those through code that is imported and singular, rather than through importing a dozen versions of the API for each dependency you have.
andrew_ 4y ago
The globally installed packages only take precedence over local packages when you run the binary name from the command line. To get that to stop, you'd have to add the local node_modules/.bin to the PATH. Additionally, if you run npx/yarn/pnpx in a package script, it will use the local binary. > oh that and installing thousands of packages in the project root instead of a sensible place that could be shared per version like bundler and rubygems. Look into pnpm. It does precisely this in a way that's compatible with node. Honestly the information is out there to address most people's gripes.

hugozap 4y ago

Tooling in the JS ecosystem (specially frontend) is very frustrating. It's just layers on top of layers with a weak foundation and a whole set of annoying incompatibilities between them.

deleted 4y ago
[deleted]

andrew_ 4y ago

The node ecosystem doesn't have a tooling problem. Frontend development has had a tooling problem since the introduction of React and Angular, and TypeScript. One might argue that goes back as far as CoffeeScript. But can we stop boosting these "OMG this was too hard" posts? TypeScript (tsc) should display a hide-able message on invocation that states: "TypeScript isn't straightforward. It's a vast ecosystem all its own that accommodates several million developer's individual needs. It can be overwhelming. You don't have to use it." I won't opine on the dumpster fire that is ESM support in Node - that's a completely separate topic and head-through-wall session.

jeroenhd 4y ago
What's the trouble with Typescript, exactly? The setup is relatively straighforward (at least, for Javascript tooling). Copy/paste a file or two and you're off to the races. In comparison any frontend framework is much more difficult to learn. Typescript itself is just Javascript with types and the fancy parts of the type system are rarely ever required.
- andrew_ 4y ago
  I'm in agreement with you. But I'm also an old salt that's been using it for nearly 8 years. I completely understand how someone unfamiliar with the language and its nuances may be overwhelmed by the configuration options and the effects they have on compilation. The types system does take time to learn, especially for those that didn't come up on a strongly (or hell, even loosely) typed language. I also think there's enough resources out there for someone, who isn't in a hurry, to grok TS enough to put together a project that will compile. But skills and comprehension aren't uniform - I've seen many a junior fresh out of bootcamp look at a tsconfig.json file like it was voodoo. That's why I think it's fair to say "Hey, you don't have to use this."
  - mnahkies 4y ago
    One big advantage of typescript with respect to the configuration options is that you can look at the output js and see what the options are producing (with a low barrier to entry - the output is easily readable js aside from some of the older targets + async keyword) The type system is super flexible and I agree it takes time to learn, but you can get a lot of value out of the basics from day one. I've not actually tried this but I suspect even with the most lenient configuration and no type annotations you'd still get limited value over plain js through type inference and flagging of obvious bugs

alexashka 4y ago

You're describing all command line tools and config files. Maybe people will discover GUIs one day, it's only been 30 years.

andrew_ 4y ago
"Everything old is new again"
antonvs 4y ago
That's not even remotely true. For example, Rust has an excellent command line build pipeline experience. That has to do both with design decisions - a typed, compiled language has a big edge here - and differences in how the respective ecosystems were designed and developed. As for GUIs, I'll paraphrase an old quote I saw on Usenet once: if you need to point at something to communicate, you're at the cognitive level of a preverbal child.
- chakkepolja 4y ago
  Even google languages (Go, Dart) and modern Java, no matter how people here hate them, have excellent tooling.
- deleted 4y ago
  [deleted]
- iamnotafreeman 4y ago
  hope that last bit was very much in jest otherwise "someone on the Usenet is wrong." visual stuff (vs. text) is about massive parallelism and compression. see Rudy Arnheim for example. or Tufte. or video games. or excel. or a good "foreign" film. or VR once it doesn't suck. https://www.ucpress.edu/book/9780520242265/visual-thinking https://www.ucpress.edu/book/9780520242265/visual-thinking

tannhaeuser 4y ago

T'was all fine until folks had to come up with useless typescript and modules and webpack churn and Es2015+ bloat. It's not like we didn't warn them to turn JavaScript into Java; for some of us getting away from Java bloat was the entire point of node.js. Well that, and because CommonJS was portable, or used to be at least. If you want a typed language, just use one, rather than foobaring JavaScript or Python into something they just aren't, a fool's errand.

foepys 4y ago
Since JavaScript is the only way to interact with the DOM in a web browser, you literally cannot use anything different than JavaScript, which makes your point to use other languages quite hard to follow.
- Zardoz84 4y ago
  WASM ?
  - Tajnymag 4y ago
    WASM came up way later than TypeScript and Webpack did. Also, no direct DOM access. You still need a Javascript proxy for that.
cercatrova 4y ago
> useless typescript The only reason I started and continue to use JS is because of TypeScript. There is no world in which I'd use pure JS over TS.
- tannhaeuser 4y ago
  Different folks, different strokes I guess: there's no world in which I'd use an MS-proprietary language over a broadly implemented one, let alone a proprietary language changing all the time and that has a Turing-complete type system, so I don't need to complain about bloat and churn, which is the whole point of this thread.

deleted 4y ago

[deleted]

rubyist5eva 4y ago

The node ecosystem is just a giant problem on top of problems.

trinovantes 4y ago

I just checked my most recent frontend project that uses Typescript as much as possible. Here's how it works: - src/.ts: Compile with esbuild in webpack - tests/.test.ts: Compiled with babel - jest.config.ts: Compiled with ts-node - webpack.config.ts: Compiled with ts-node In an ideal world, one tool should do everything but I doubt that will ever happen

mikojan 4y ago
I'm curious: Why use webpack with esbuild in a "most recent frontend project"? Not so long ago at $DAYJOB we even migrated a big legacy project from webpack to esbuild in no time.
- trinovantes 4y ago
  I've used webpack for several years now and have written custom plugins to accomplish what I need. esbuild-loader gives great typescript performance almost on par with vite so I don't see the need to learn a new build system. Also, there's no official Vue 3 support for esbuild so that's a problem for some of my projects

spankalee 4y ago

The problems start with frameworks that require custom compilers and non-standard module systems. If you use vanilla JS in standard modules most of this goes away.

jitl 4y ago
Well then you need to use vanilla JS, which seems to be widely regarded as not great “at scale”. I certainly will never go back to JS from TS. Fearless refactoring is worth slogging through all the JSON config soup.
- tobyhinloopen 4y ago
  Fearless refactoring? What about tests? Sure TS helps, but tests are “better”
  - Tajnymag 4y ago
    Tests and static typing are not excluding each other. Best would be to have both.

claytongulick 4y ago

I'm pretty happy with just vanilla JavaScript. I don't do typescript, I don't use frameworks (unless you consider express to be a framework). I use vanilla web components on the front end, if I need reactivity and frequent re-renders I might sprinkle in a bit of lit-html, depending on the component. Debugging and development are brain dead simple, just sticking to the standards. I had a new developer come into my code base and he said "I couldn't believe it at first, you just write something and it does it... Wow!". He was so used to piles of frameworks and build systems and stuff, he'd never been in a code base that just... ran.

exabrial 4y ago

> Node is a problem Fixed it for you. The node ecosystem embodies mediocrity. There is no design, no testing, no stability, no benchmarks, no RFCs, no plan to produce a coherent product. Instead, you're giving a half baked set of features that are abandoned within 8-12 months. Not invented here syndrome runs wild, and over-engineering any solution is seen as a mark of accomplishment. Simple things, like displaying static text, now require 25mb of js files for whatever framework the author is currently playing around with as a hobby. I miss the days of the web where you could just load a page in under 25kb.

vbezhenar 4y ago
Our node docker images are 600-1000 MB for relatively simple website (backend and frontend). 25 MB would be so good and lightweight.
- interblag 4y ago
  Unless I'm massively misunderstanding your setup, that Docker image contains an entire OS and runtime, not just your website. Reasonable people can disagree about whether that entire paradigm is sensible but, having chosen to use Docker, complaining about image size in this way seems an odd choice. More to this point, most of what is in the image will be little to do with Node or its build systems..
  - toomanybeersies 4y ago
    The NodeJS alpine base image is only 40 MB.
    - lolive 4y ago
      Then you _npm install_ something, and BOOOM! 600MB !
      - Defman 4y ago
        You can use multi-stage builds [1] so your result image would be as small as possible by just copying the compiled JS from the build stage. [1]: https://docs.docker.com/develop/develop-images/multistage-build/ https://docs.docker.com/develop/develop-images/multistage-bu...

deleted 4y ago

[deleted]

shadycuz 4y ago

I doubt anyone is interested but I'm building some tooling for node and npm in jenkins. Just standard stuff, nvm, nodenv, npm and npx wrapper.

joshxyz 4y ago

Keep in mind that most of these are just rants about the developer experience, which is sacrificed in exchange for faster iteration and deployments on majority of business use cases. As a Node.js developer myself I even noticed that some of the best solutions is nodejs plus other language. Examples uwebsockets.js uses some c++ code. it is faster and has better api than express, fastify, hapi, koa, and has builtin websockets support. esbuild uses golang code. it is faster than webpack and babel for bundling and minifying javascript code.

pfesenmeier 4y ago

Interested if anyone is using a jsconfig.json and jsdoc comments to do ts typechecking. I found it really easy to setup, I got a lot out of the extra checks, but the practice never really took off with the team of js devs. https://code.visualstudio.com/docs/languages/jsconfig https://code.visualstudio.com/docs/languages/jsconfig

theteapot 4y ago
How you gonna use it on a 3rd party module?

NonNefarious 4y ago

Node has been superseded by Deno, right? How about that?

johnnypangs 4y ago

I saw the author was using vite, they have a mode to help you build a library that would mitigate their woes. Check it out! https://vitejs.dev/guide/build.html#library-mode https://vitejs.dev/guide/build.html#library-mode

MaxLeiter 4y ago
I'm using library mode

kabdib 4y ago

I have lost whole consecutive weeks to WebPack. It made me wish I could go back to something wholesome and healthy, like template metaprogramming in C++. "I'm making progress, boss. The error output from the build is smaller than the input source code now."

hestefisk 4y ago

Rant: could we please stop referring to a collection of <X> as an ecosystem? Node has a collection or environment of tools, frameworks and libraries to support it. An ecosystem is something living, a biological system, typically with a set of symbiotic properties within it. The trend is the same in business. Stripe no longer has business partners; they have an “ecosystem.” The same with banks that provide wholesale banking services through a set of retail businesses - they are no longer wholesale banks but “ecosystem players”. This kind of jargon really dilutes the meaning of words and makes it hard to understand what is really meant. Rant over, sorry. Apart from my rant, it’s funny to see Node and JavaScript doing what almost killed Perl 20 years ago. There are so many ways of doing things, so many bespoke frameworks and packages (doing OO in Perl, anyone?), so many odd dialects that it fragmented the language and people got frustrated. The same is happening to JS right now, although with some important differences. There is hardly a fragmentation of usage via formal “dialects” in JS; rather, the problem is in the sprawl of libraries and “infrastructure” services, many of which are solved problems (e.g. Gulp reinventing what Make is really good at; Yarn reinventing what npm is supposed to do, etc). The supposed “ecosystem” is fragmenting day by day from its inherent complexity, and it’s exactly what caused web developers to flee from Perl to more simple, consistent alternatives.

stevage 4y ago
It's a perfectly apt metaphor which is clear and useful. Your rant seems unfounded.
lozenge 4y ago
An ecosystem isn't alive, it's a collection of independent species that have an emergent behaviour.
- lioeters 4y ago
  By that line of thought, is life alive? It's just a collection of interdependent cells and molecules with emergent behavior. Seems like an ecosystem is alive collectively, in the same way that an individual is alive, just on a different scale of organization.
4y ago

mhoad 4y ago

It’s funny I’ve been looking at industrial strength build systems like Bazel (language independent build tool from Google) recently the past few months just to get an idea of how things work in that world. One of the things happening in that community at the moment was their second major attempt to bring the JavaScript / Typescript and Node ecosystem onboard as a semi official language. It has been a 4 year effort so far and they had to make a bunch of compromises along the way as you can see in the design section of the projects readme here https://github.com/aspect-build/rules_js https://github.com/aspect-build/rules_js Watching the walls they would continually run into where the answer would essentially be “Node is very unlike other language runtimes” was illuminating and I suddenly started to understand why projects like Deno want to actually leave Node behind as a fundamentally unfixable mess and start again with a bit more planning this time around to avoid all of those issues.

stoicjumbotron 4y ago

So what is the "recommended" way to build libraries using TypeScript? I've heard that tsdx https://tsdx.io/ https://tsdx.io/ is quite good

arealpromise 4y ago

erikpukinskis 4y ago

It’s still early days, but I have been working on this… There are scaffolding tools that help configure all the JavaScript frontend stuff you need, but the problem is you run them once and you can’t ever run them again to change/add stuff. So I built Confgen which is sort of like create-react-app except it’s idempotent: https://github.com/erikpukinskis/confgen https://github.com/erikpukinskis/confgen It’s very alpha, but I would love to get ideas/ bug reports on GitHub. It’s also currently Vite-only, but I’m open to a possible webpack/babel mode in the future.

phendrenad2 4y ago

I've started using plain old CDN-served React/jQuery/Lodash to do everything, with no build system, and daring my employer to fire me. So far so good.

FractalHQ 4y ago

I use pnpm and Sveltekit and have none of these problems. Everything just works, and it’s rare anything related to tooling takes longer than 1second to accomplish. Being on OSX helps a lot with the “just works” part though.