5 ms·

Bloat is most libraries on npm. The authors don't know good design and instead try to make every library do everything. Oh, my library converts strings from one

by nox101 3y ago

Bloat is most libraries on npm. The authors don't know good design and instead try to make every library do everything. Oh, my library converts strings from one encoding to another, I'll make it load files for you, save files for you, download them over the internet, and give you a command line tool, all in the same repo. The library should just do its one thing. The rest should be left to the user. I get the impression it's no better in rust land. Go try to edit the rust docs. Watch it install ~1000 crates. The issue is not the language, it's that anyone can post a library (not complaining) and anyone does. People who "just want to get stuff done" choose the library with the most features and then beg for even more since they can't be bothered to write the 3 lines of code it would have been to solve it outside the library. "Can you add render to pdf?" I don't know how to solve it. My one idea is to start an "Low Dependency" advocacy group with a badge or something and push to get people to want that badge on their libraries and for people to look for that badge on libraries they choose.

crooked-v 3y ago
On this note, I wish more npm library authors would emulate sindresorhus's approach of making small packages that each do very specific things in a predictable and well-documented manner.
- whstl 3y ago
  Hell no. This person is probably the worst offender in terms of supply-chain problems and bloat in the NPM ecosystem. A terminal spinner ("ora") with 15 dependencies (including transitive) is not an example of good design. Inflating the numbers of downloads in your own packages is doing no good to the world of software.
  - brnt 3y ago
    What you hope for is shallow dependency trees, not too branchy, but probably also not branchless (an indicator of vendoring). Larger projects will have and likely require depper trees through, but the branchiness should be relatively independent. I wonder if this has ever been formalized.
    - whstl 3y ago
      What I hope for is for code that is actually auditable, and doesn't pull a lot of unused bloat just to pump up someone else's vanity metrics. This applies for large and small projects. Babel also doesn't need to do what it does: it pulls a couple hundreds of non-reusable sub-packages even though it's all written by the same people and maintained in the same monorepo.
- Aeolun 3y ago
  Honestly, I’ve often considered chunking all those libs into a single big one to eliminate 50% of my npm dependency chain.
charlie0 3y ago
By definition, the only languages where this isn't true are languages that have a high barrier to entry, and because of that, they would also be unpopular. Perhaps if the goverment mandated a standard set of programming tools....
amelius 3y ago
The "3 more lines" problem can be solved by LLMs.
- mewpmewp2 3y ago
  Yeah honestly one key thing of copilot has been that I can create a fn definition I need that is very common and it will do the boilerplate for me. It feels dirty not using a library, but in the end I save bytes and have clear control over the fn that otherwise is quite standard.
- nehal3m 3y ago
  A black box that turns a few megawatts and 60TB of data into a model that can write three lines is antithetical to lean anything
PH95VuimJjqBqy 3y ago
> People who "just want to get stuff done" choose the library with the most features and then beg for even more since they can't be bothered to write the 3 lines of code it would have been to solve it outside the library. I feel this so much. I once had a contract for ruby on rails work. They were experiencing severe performance issues, so bad that they were developing in release mode (server won't detect a file change and reload for you). One day I get tired of it and start digging into it. I don't remember how many gems they were pulling in but it was A LOT. I came across one particular gem which saved 3 lines of code. Seriously. I stayed away from the RoR community after that. I recently picked up a contract for more RoR work (after all these years, lol) and ... it's not nearly as bad but it's not good either. Some communities just do NOT respect the risk that dependencies bring.
- sgarland 3y ago
  My experience from having worked at a mostly Python shop (and loving Python myself), and working at a Node shop, is that the latter is by far the worst. Python probably has just as many shitty packages as Node, but Python’s stdlib is so vast that you often don’t need to, if you bother to read the docs. Today, I was marveling at a function someone wrote at my work to cast strings to Titlecase. I learned today that JS has no built-in for that. That’s a tiny example, but there are far more.
  - spyke112 3y ago
    That’s also just a one line function with template literals in JS, if by Titlecase you mean upper casing the first character. Maybe a bit overkill to bake into a stdlib.
    - charlie0 3y ago
      It's not about ease, it's about frequency of use. If it's used often, it should be in a stdlib.
- moritonal 3y ago
  Given you've seen the same effect in both Ruby and JS maybe the takeaway should instead be that there is a group of devs who will always reach for a package first, not that a specific language has a problem.
sweeter 3y ago
This is why I like Go and the stdlib. Its very much "do it yourself" unless we are talking about monumental libraries for very specific applications. Outside of that if we are talking about simple libraries that do one simple thing, its better to just roll your own using the stdlib. Almost all of the building blocks are there. On the other hand its really nice that any git repo can host a Go lib and anyone can use it from that url
- MrDresden 3y ago
  Never used Go, but I much more prefer a proper stdlib that covers most base cases so that there isn't a need for rolling-your-own each time, or worse using a badly made/maintained bloated third party lib. Examples of excellent stdlibs in my opinion would be for instance Kotlin and Python.
  - LamaOfRuin 3y ago
    I think that's what they were saying (and what I would agree with). The Go std lib is very complete for important things, especially in the context of networked services. So the default is that you don't have to pull in many libraries for common needs.
    - scubbo 3y ago
      > The Go std lib is very complete for important things Important thing like sets[0], or get-with-default from maps[1], or enums[2], or appending-to[3] or mapping-over[4] slices? [0] https://stackoverflow.com/questions/34018908/golang-why-dont-we-have-a-set-datastructure/34020023#34020023 https://stackoverflow.com/questions/34018908/golang-why-dont... [1] https://www.digitalocean.com/community/tutorials/understanding-maps-in-go#checking-existence https://www.digitalocean.com/community/tutorials/understandi... [2] https://stackoverflow.com/questions/14426366/what-is-an-idiomatic-way-of-representing-enums-in-go https://stackoverflow.com/questions/14426366/what-is-an-idio... [3] https://go.dev/tour/moretypes/15 https://go.dev/tour/moretypes/15 [4] https://stackoverflow.com/questions/71624828/is-there-a-way-to-map-an-array-of-objects-in-go/71624929#71624929 https://stackoverflow.com/questions/71624828/is-there-a-way-...
worble 3y ago
>Oh, my library converts strings from one encoding to another, I'll make it load files for you, save files for you, download them over the internet, and give you a command line tool, all in the same repo. The library should just do its one thing. The rest should be left to the user. >I don't know how to solve it. My one idea is to start an "Low Dependency" advocacy group with a badge or something and push to get people to want that badge on their libraries and for people to look for that badge on libraries they choose. It sounds like you're conflating low bloat and low dependency, which is like trying to have your cake and eat it too. If you want low bloat libraries, then it's likely you're going to be pulling a lot in that don't independently do much. If you want low dependency libraries, then you'll be pulling in a few that do do a lot. From my perspective I'd rather have slightly fatter libraries if they're low dependency and by authors I trust. Lodash for instance, sure it's big but the es6 module version supports tree shaking and it's basically the standard library JS never had. Same for something like date-fns for Date. I pretty much pull these two in by default into any project to fill those holes in JS's core library.
- trickstra 3y ago
  What I saw way too often is that a project will use a dependency only for one tiny function. That's how we get a calendar app with 2.5GB node_modules.
whstl 3y ago
The problem comes from two sides. First you have package makers that want to make this into a career, and the only way to generate buzz is by having thousands of them. So they focus on generating packages that depends on others created by them, and on trying to get their one or two useful packages into someone else's code. The second is people who believe that the only solution to problems involves including a new package, without caring about how many dependencies it has, or even if the actual problems is actually hard to solve. So instead of learning how to solve the problem, they must learn the API of some wrapper with 4 GitHub stars.
mike_ivanov 3y ago
Reminds me of that disk defragmenting utility with an embedded mp3 player from early 2000s.
iambvk 3y ago
> ...since they can't be bothered to write the 3 lines of code... They actually think saner devs have not-invented-here syndrome :0
dogcomplex 3y ago
While "add a black box complexity extremely resource intensive AI to sort out the dependency trees" is somewhat of a tragic answer to the problem, I think it's the real one we're gonna get - and we'll still be better off than this current clusterfuck. Hopefully these will also spend the tedious hours pruning unnecessary packages, minimizing responsibilities of each, and overall organizing the environment in a sensible way, then caching that standalone so there's no LLM dependency - just an update checker/curator to call in when things go to shit. Honestly this is one of the worst problems of modern software and makes 50+% projects unusable. It seems like a very tedious yet solvable problem - perfect for the right LLM agent. Would be a godsend.
deleted 3y ago
[deleted]
throwawayqqq11 3y ago
Libraries as pure functions where ever possible and a build system that does not only enforce this but warns you when an update introduces a new/changed syscall. Yes please! I know its not exactly low-dependency but at least lower attack surface.
MagicMoonlight 3y ago
And that’s why Java is the only non-meme language. Need a hashmap? Need a webserver? Need a UI? It’s all built in. It just works on any device with the same code. People who actually know what they’re doing wrote it and you can trust it. No random downloads or weird packages. If you want to download random things you still can, but realistically you don’t need to.
begueradj 3y ago
>"Bloat is most libraries on npm. The authors don't know good design and instead try to make every library do everything." That is true for all other programming languages.
freeopinion 3y ago
Ironically, solving for your crticism leads to the criticism in the comment next to yours. They complain about a text encoding library that also reads and writes files, makes network connections, etc. This is a form of bloat. But it minimizes my number of dependencies. So which is the worse approach?
eternityforest 3y ago
Libraries that do everything are better that 50 different tiny libraries, amy one of which could be the next left-pad. Even if the big library uses a bazillion small libraries, the big library team usually at least does some testing. Maybe we could move to having big teams maintain collections of small libraries... But with tree shaking I'm not sure why we'd really need to. I'm happy just sticking with things that have lots of GitHub stars. I think the solution is just don't use stuff posted by random people on npm if there's an alternative that's widely used, with lots of eyeballs on it. Which there usually is, but people probably choose the smaller one because it's simpler.
therealdrag0 3y ago
Perhaps increasing pressure for secure tool chains will cause a consolidation of fewer but more feature rich and well-done shared dependencies. Instead of using Joe blows module you’ll find what you need in “Apache common” module, etc.

by nox101 3y ago

crooked-v 3y ago

On this note, I wish more npm library authors would emulate sindresorhus's approach of making small packages that each do very specific things in a predictable and well-documented manner.

whstl 3y ago
Hell no. This person is probably the worst offender in terms of supply-chain problems and bloat in the NPM ecosystem. A terminal spinner ("ora") with 15 dependencies (including transitive) is not an example of good design. Inflating the numbers of downloads in your own packages is doing no good to the world of software.
- brnt 3y ago
  What you hope for is shallow dependency trees, not too branchy, but probably also not branchless (an indicator of vendoring). Larger projects will have and likely require depper trees through, but the branchiness should be relatively independent. I wonder if this has ever been formalized.
  - whstl 3y ago
    What I hope for is for code that is actually auditable, and doesn't pull a lot of unused bloat just to pump up someone else's vanity metrics. This applies for large and small projects. Babel also doesn't need to do what it does: it pulls a couple hundreds of non-reusable sub-packages even though it's all written by the same people and maintained in the same monorepo.
Aeolun 3y ago
Honestly, I’ve often considered chunking all those libs into a single big one to eliminate 50% of my npm dependency chain.

charlie0 3y ago

By definition, the only languages where this isn't true are languages that have a high barrier to entry, and because of that, they would also be unpopular. Perhaps if the goverment mandated a standard set of programming tools....

amelius 3y ago

The "3 more lines" problem can be solved by LLMs.

mewpmewp2 3y ago
Yeah honestly one key thing of copilot has been that I can create a fn definition I need that is very common and it will do the boilerplate for me. It feels dirty not using a library, but in the end I save bytes and have clear control over the fn that otherwise is quite standard.
nehal3m 3y ago
A black box that turns a few megawatts and 60TB of data into a model that can write three lines is antithetical to lean anything

PH95VuimJjqBqy 3y ago

> People who "just want to get stuff done" choose the library with the most features and then beg for even more since they can't be bothered to write the 3 lines of code it would have been to solve it outside the library. I feel this so much. I once had a contract for ruby on rails work. They were experiencing severe performance issues, so bad that they were developing in release mode (server won't detect a file change and reload for you). One day I get tired of it and start digging into it. I don't remember how many gems they were pulling in but it was A LOT. I came across one particular gem which saved 3 lines of code. Seriously. I stayed away from the RoR community after that. I recently picked up a contract for more RoR work (after all these years, lol) and ... it's not nearly as bad but it's not good either. Some communities just do NOT respect the risk that dependencies bring.

sgarland 3y ago
My experience from having worked at a mostly Python shop (and loving Python myself), and working at a Node shop, is that the latter is by far the worst. Python probably has just as many shitty packages as Node, but Python’s stdlib is so vast that you often don’t need to, if you bother to read the docs. Today, I was marveling at a function someone wrote at my work to cast strings to Titlecase. I learned today that JS has no built-in for that. That’s a tiny example, but there are far more.
- spyke112 3y ago
  That’s also just a one line function with template literals in JS, if by Titlecase you mean upper casing the first character. Maybe a bit overkill to bake into a stdlib.
  - charlie0 3y ago
    It's not about ease, it's about frequency of use. If it's used often, it should be in a stdlib.
moritonal 3y ago
Given you've seen the same effect in both Ruby and JS maybe the takeaway should instead be that there is a group of devs who will always reach for a package first, not that a specific language has a problem.

sweeter 3y ago

This is why I like Go and the stdlib. Its very much "do it yourself" unless we are talking about monumental libraries for very specific applications. Outside of that if we are talking about simple libraries that do one simple thing, its better to just roll your own using the stdlib. Almost all of the building blocks are there. On the other hand its really nice that any git repo can host a Go lib and anyone can use it from that url

MrDresden 3y ago
Never used Go, but I much more prefer a proper stdlib that covers most base cases so that there isn't a need for rolling-your-own each time, or worse using a badly made/maintained bloated third party lib. Examples of excellent stdlibs in my opinion would be for instance Kotlin and Python.
- LamaOfRuin 3y ago
  I think that's what they were saying (and what I would agree with). The Go std lib is very complete for important things, especially in the context of networked services. So the default is that you don't have to pull in many libraries for common needs.
  - scubbo 3y ago
    > The Go std lib is very complete for important things Important thing like sets[0], or get-with-default from maps[1], or enums[2], or appending-to[3] or mapping-over[4] slices? [0] https://stackoverflow.com/questions/34018908/golang-why-dont-we-have-a-set-datastructure/34020023#34020023 https://stackoverflow.com/questions/34018908/golang-why-dont... [1] https://www.digitalocean.com/community/tutorials/understanding-maps-in-go#checking-existence https://www.digitalocean.com/community/tutorials/understandi... [2] https://stackoverflow.com/questions/14426366/what-is-an-idiomatic-way-of-representing-enums-in-go https://stackoverflow.com/questions/14426366/what-is-an-idio... [3] https://go.dev/tour/moretypes/15 https://go.dev/tour/moretypes/15 [4] https://stackoverflow.com/questions/71624828/is-there-a-way-to-map-an-array-of-objects-in-go/71624929#71624929 https://stackoverflow.com/questions/71624828/is-there-a-way-...

worble 3y ago

>Oh, my library converts strings from one encoding to another, I'll make it load files for you, save files for you, download them over the internet, and give you a command line tool, all in the same repo. The library should just do its one thing. The rest should be left to the user. >I don't know how to solve it. My one idea is to start an "Low Dependency" advocacy group with a badge or something and push to get people to want that badge on their libraries and for people to look for that badge on libraries they choose. It sounds like you're conflating low bloat and low dependency, which is like trying to have your cake and eat it too. If you want low bloat libraries, then it's likely you're going to be pulling a lot in that don't independently do much. If you want low dependency libraries, then you'll be pulling in a few that do do a lot. From my perspective I'd rather have slightly fatter libraries if they're low dependency and by authors I trust. Lodash for instance, sure it's big but the es6 module version supports tree shaking and it's basically the standard library JS never had. Same for something like date-fns for Date. I pretty much pull these two in by default into any project to fill those holes in JS's core library.

trickstra 3y ago
What I saw way too often is that a project will use a dependency only for one tiny function. That's how we get a calendar app with 2.5GB node_modules.

whstl 3y ago

The problem comes from two sides. First you have package makers that want to make this into a career, and the only way to generate buzz is by having thousands of them. So they focus on generating packages that depends on others created by them, and on trying to get their one or two useful packages into someone else's code. The second is people who believe that the only solution to problems involves including a new package, without caring about how many dependencies it has, or even if the actual problems is actually hard to solve. So instead of learning how to solve the problem, they must learn the API of some wrapper with 4 GitHub stars.

mike_ivanov 3y ago

Reminds me of that disk defragmenting utility with an embedded mp3 player from early 2000s.

iambvk 3y ago

> ...since they can't be bothered to write the 3 lines of code... They actually think saner devs have not-invented-here syndrome :0

dogcomplex 3y ago

While "add a black box complexity extremely resource intensive AI to sort out the dependency trees" is somewhat of a tragic answer to the problem, I think it's the real one we're gonna get - and we'll still be better off than this current clusterfuck. Hopefully these will also spend the tedious hours pruning unnecessary packages, minimizing responsibilities of each, and overall organizing the environment in a sensible way, then caching that standalone so there's no LLM dependency - just an update checker/curator to call in when things go to shit. Honestly this is one of the worst problems of modern software and makes 50+% projects unusable. It seems like a very tedious yet solvable problem - perfect for the right LLM agent. Would be a godsend.

deleted 3y ago

[deleted]

throwawayqqq11 3y ago

Libraries as pure functions where ever possible and a build system that does not only enforce this but warns you when an update introduces a new/changed syscall. Yes please! I know its not exactly low-dependency but at least lower attack surface.

MagicMoonlight 3y ago

And that’s why Java is the only non-meme language. Need a hashmap? Need a webserver? Need a UI? It’s all built in. It just works on any device with the same code. People who actually know what they’re doing wrote it and you can trust it. No random downloads or weird packages. If you want to download random things you still can, but realistically you don’t need to.

begueradj 3y ago

>"Bloat is most libraries on npm. The authors don't know good design and instead try to make every library do everything." That is true for all other programming languages.

freeopinion 3y ago

Ironically, solving for your crticism leads to the criticism in the comment next to yours. They complain about a text encoding library that also reads and writes files, makes network connections, etc. This is a form of bloat. But it minimizes my number of dependencies. So which is the worse approach?

eternityforest 3y ago

Libraries that do everything are better that 50 different tiny libraries, amy one of which could be the next left-pad. Even if the big library uses a bazillion small libraries, the big library team usually at least does some testing. Maybe we could move to having big teams maintain collections of small libraries... But with tree shaking I'm not sure why we'd really need to. I'm happy just sticking with things that have lots of GitHub stars. I think the solution is just don't use stuff posted by random people on npm if there's an alternative that's widely used, with lots of eyeballs on it. Which there usually is, but people probably choose the smaller one because it's simpler.

therealdrag0 3y ago

Perhaps increasing pressure for secure tool chains will cause a consolidation of fewer but more feature rich and well-done shared dependencies. Instead of using Joe blows module you’ll find what you need in “Apache common” module, etc.