6 ms·

In favor of small modules and plumbing

30 points by NetOpWibby 7y ago

pubby 7y ago
Small modules are only useful when they're self-contained. Leftpad was mocked because it brought in three dependencies on top of itself. The more dependencies one has, the more breakage hell one must deal with. So please, keep your code self-contained when sharing with the world.
stareatgoats 7y ago
FWIW, I've taken the exact opposite route. Much as I respect sindresorhus, my projects nowadays have very few dependencies; I treat every dependency as an elevated "point of failure"-risk, and I would not recommend anyone having potentially thousands of "left-pads" sprinkled across a business-critical system.
- eropple 7y ago
  You shouldn't be relying upon NPM or any other public package registry for a "business-critical system" in the first place. When the left-pad guy had his snit, we didn't notice at all because we had a caching NPM mirror.
wereHamster 7y ago
Small packages are fine, if the maintainers keep them up to date! You can't publish packages and never touch them again. This leads to situations like the one with merge-deep that I recently discovered: http://npm.anvaka.com/#/view/2d/merge-deep http://npm.anvaka.com/#/view/2d/merge-deep All but one of those dependencies are from the same maintainer. Yet I see two major versions of kind-of, for-in, and lazy-cache. What's the point of creating reusable functions if you're not going to reuse them? This is a huge waste. First on the filesystem, no wonder our node_modules folders are hundreds of megabytes big. Second it wastes bytes in memory and in CPU caches since they don't share byte code and each function has to be parsed, compiled, optimised separately. I seem to be in the minority thinking this situation is insane…
- unlinked_dll 7y ago
  It's insanity all the way down to the copper on the board frankly. People only care about making things more sane when you can prove there's a dollar to be made or saved by doing so. Like we all know nodjs is slow(er than other things). The question is not how to make it faster by doing things like keeping dependencies up to date but how much there is to gain by doing so. I suspect very little.
z3t4 7y ago
Small focused modules dont need much maintenance or braking changes, so you basically only have to download them once, and treat them as if they where your own code.
mhd 7y ago
I get an allergic reaction to any rationale that includes the tired phrase "easier to reason about". That's on my bingo card right next to "promote synergy"... Now, I'm not inherently against having small modules for small tasks, but I'd much prefer if those are used like sprinkles on a more solid foundation. So a small Perl module on CPAN? Sure. Your whole web app consisting of small modules, turtles all the way down? No, sir, no. It also means that you've got plenty of build systems, it's incredibly hard to change the output of all your dependencies to not-ES5, it's way more frequent that one developer introduces new dependencies, possibly duplicates etc. Sadly there's no decent way out. JS is getting a new syntactic feature every second Monday, but no big std lib movement is out there that I'm aware of. Sure, you probably wouldn't want to fit the Go/Python/Java core lib into a browser, but for the backend it would be quite nice, and would make tooling for the frontend less fragile. (And to be honest, even frontends are approaching mid-tier desktop UI complexity; otherwise we could've and should've stayed with jquery) I don't need to be able to reason about single modules independent from each other often. I need to trust the whole system enough to not lose sleep over it.
- lmm 7y ago
  > It also means that you've got plenty of build systems, it's incredibly hard to change the output of all your dependencies to not-ES5, On any realistic sized project you already need tooling for handling dependencies in bulk. If you want to build everything for ES6, that should be a problem you solve once in your build/dependency manager, not per-depednency. > it's way more frequent that one developer introduces new dependencies, possibly duplicates etc. I actually disagree with this: if you have large frameworks then you're more likely to have duplication, because you'll have frameworks that overlap but don't completely cover each other. E.g. C++ projects tend to end up with 5 different string classes, because every framework you use will bring its own (because C++ relies on big heavy frameworks because it has a very high per-library overhead due to limited tooling). Small focused dependencies make that less likely.
  - mhd 7y ago
    I don't quite get the latter. First of all, if the foundations are sound and the core language provides a good enough module already, neither large frameworks nor small modules need to duplicate that. And if it isn't included, why would 16 small subsystems have fewer duplicates than 2 large frameworks? By the way, C++ is exactly the trajectory I see JS moving towards. A plethora of different sub-languages and library sets. Only with the difference that it's both easier from a tooling and culture perspective to bring in new tiny modules.
cousin_it 7y ago
I just looked at the source code of left-pad for the first time, and it's strange. They noticed that concatenating a bunch of single-space strings is O(n^2), and optimized it to O(n log n) by using repeated doubling. So in the end they take longer than O(n) to fill an array with a constant value. Why? https://github.com/left-pad/left-pad/blob/master/index.js https://github.com/left-pad/left-pad/blob/master/index.js
- rhardih 7y ago
  Why write code to left pad your strings, when you can get an O(n log n) implementation just like that. You only need a "minimal" amount of bundle tooling and a few hundred MB of node_modules. Count your blessings. This is the way.
asterisk_ 7y ago
This doesn't seem too controversial of an opinion, haven't small modules already been at the core of a Unix philosophy? Once usage patterns emerge, these smaller libraries are composed into frameworks which ultimately build end-user applications. The entirety of engineering is premised upon leveraging powerful abstractions; cf. the reason we're not still coding via electric charge. Edit: To expand on the analogy: "A two-person startup already uses twenty-eight other tools" [1]. [1] https://news.ycombinator.com/item?id=22449314 https://news.ycombinator.com/item?id=22449314
- jasode 7y ago
  >, haven't small modules already been at the core of a Unix philosophy? I assume you're talking about userland programs such as "ls", "cut", "grep", etc. ?[1] But C Language is also part of UNIX and programmers typically link against a large more comprehensive "standard library" such as glibc: libc.a, libc.so <-- includes printf ... instead linking of a hundred little object files specified explicitly like this: printf.o strcat.o strncat.o malloc.o leftpad.o ... <hundred or thousand more .o files> What happened with Javascript is the language didn't have a comprehensive "standard library" like C, C++, Java, C#, Python "batteries included", Go, etc. Therefore, NPM became an adhoc "standard library" via bottom-up crowdsourcing. There are both positives and negatives to that approach. One negative is that leftpad() isn't produced by a canonical entity like Netscape or Mozilla but a random programmer. When he wanted to take his ball and go home, he broke everyone's build that depended on it. That type of event didn't happen with "printf()" in the C Language world. [1] https://en.wikipedia.org/wiki/Unix_philosophy#Do_One_Thing_and_Do_It_Well https://en.wikipedia.org/wiki/Unix_philosophy#Do_One_Thing_a...
KaiserPro 7y ago
All of this seems to be justifying a lack of a decent standard library. However its appears to be written from the point of view of someone who's never had the joy/pain of a decent standard library. The problem with lots of tiny modules is consistency and power. Sure its "open source" but each one of these modules are in various states of decay. better to lump them together in sets of functions, so that more eyes can be concentrated in the same place.
osrec 7y ago
The article highlights a huge pitfall of having lots of small dependencies, provides no real remediation, and says small dependencies are easier to reason about, so they're better anyway. I disagree. I personally find that with large projects, small dependencies actually become much harder to reason about - they sort of get lost in the codebase. Not to mention the duplication of small packages on npm doing similar things - with a large team, it's easy to add both to your codebase and then we're in a new sort of duplicative mess.
jscholes 7y ago
> Regarding Jobs' quote, here's another way to look at it: the most beautiful hotel in the world wouldn't feel like such if all the toilets overflowed when flushed. But this would be a functional problem, irritating at best and a potential health hazard at worst. Using plywood for the back of a chest of drawers is no such thing.

30 points by NetOpWibby 7y ago

Small modules are only useful when they're self-contained. Leftpad was mocked because it brought in three dependencies on top of itself. The more dependencies one has, the more breakage hell one must deal with. So please, keep your code self-contained when sharing with the world.

stareatgoats 7y ago

FWIW, I've taken the exact opposite route. Much as I respect sindresorhus, my projects nowadays have very few dependencies; I treat every dependency as an elevated "point of failure"-risk, and I would not recommend anyone having potentially thousands of "left-pads" sprinkled across a business-critical system.

eropple 7y ago
You shouldn't be relying upon NPM or any other public package registry for a "business-critical system" in the first place. When the left-pad guy had his snit, we didn't notice at all because we had a caching NPM mirror.

wereHamster 7y ago

Small packages are fine, if the maintainers keep them up to date! You can't publish packages and never touch them again. This leads to situations like the one with merge-deep that I recently discovered: http://npm.anvaka.com/#/view/2d/merge-deep http://npm.anvaka.com/#/view/2d/merge-deep All but one of those dependencies are from the same maintainer. Yet I see two major versions of kind-of, for-in, and lazy-cache. What's the point of creating reusable functions if you're not going to reuse them? This is a huge waste. First on the filesystem, no wonder our node_modules folders are hundreds of megabytes big. Second it wastes bytes in memory and in CPU caches since they don't share byte code and each function has to be parsed, compiled, optimised separately. I seem to be in the minority thinking this situation is insane…

unlinked_dll 7y ago
It's insanity all the way down to the copper on the board frankly. People only care about making things more sane when you can prove there's a dollar to be made or saved by doing so. Like we all know nodjs is slow(er than other things). The question is not how to make it faster by doing things like keeping dependencies up to date but how much there is to gain by doing so. I suspect very little.

z3t4 7y ago

Small focused modules dont need much maintenance or braking changes, so you basically only have to download them once, and treat them as if they where your own code.

mhd 7y ago

I get an allergic reaction to any rationale that includes the tired phrase "easier to reason about". That's on my bingo card right next to "promote synergy"... Now, I'm not inherently against having small modules for small tasks, but I'd much prefer if those are used like sprinkles on a more solid foundation. So a small Perl module on CPAN? Sure. Your whole web app consisting of small modules, turtles all the way down? No, sir, no. It also means that you've got plenty of build systems, it's incredibly hard to change the output of all your dependencies to not-ES5, it's way more frequent that one developer introduces new dependencies, possibly duplicates etc. Sadly there's no decent way out. JS is getting a new syntactic feature every second Monday, but no big std lib movement is out there that I'm aware of. Sure, you probably wouldn't want to fit the Go/Python/Java core lib into a browser, but for the backend it would be quite nice, and would make tooling for the frontend less fragile. (And to be honest, even frontends are approaching mid-tier desktop UI complexity; otherwise we could've and should've stayed with jquery) I don't need to be able to reason about single modules independent from each other often. I need to trust the whole system enough to not lose sleep over it.

lmm 7y ago
> It also means that you've got plenty of build systems, it's incredibly hard to change the output of all your dependencies to not-ES5, On any realistic sized project you already need tooling for handling dependencies in bulk. If you want to build everything for ES6, that should be a problem you solve once in your build/dependency manager, not per-depednency. > it's way more frequent that one developer introduces new dependencies, possibly duplicates etc. I actually disagree with this: if you have large frameworks then you're more likely to have duplication, because you'll have frameworks that overlap but don't completely cover each other. E.g. C++ projects tend to end up with 5 different string classes, because every framework you use will bring its own (because C++ relies on big heavy frameworks because it has a very high per-library overhead due to limited tooling). Small focused dependencies make that less likely.
- mhd 7y ago
  I don't quite get the latter. First of all, if the foundations are sound and the core language provides a good enough module already, neither large frameworks nor small modules need to duplicate that. And if it isn't included, why would 16 small subsystems have fewer duplicates than 2 large frameworks? By the way, C++ is exactly the trajectory I see JS moving towards. A plethora of different sub-languages and library sets. Only with the difference that it's both easier from a tooling and culture perspective to bring in new tiny modules.

cousin_it 7y ago

I just looked at the source code of left-pad for the first time, and it's strange. They noticed that concatenating a bunch of single-space strings is O(n^2), and optimized it to O(n log n) by using repeated doubling. So in the end they take longer than O(n) to fill an array with a constant value. Why? https://github.com/left-pad/left-pad/blob/master/index.js https://github.com/left-pad/left-pad/blob/master/index.js

rhardih 7y ago
Why write code to left pad your strings, when you can get an O(n log n) implementation just like that. You only need a "minimal" amount of bundle tooling and a few hundred MB of node_modules. Count your blessings. This is the way.

asterisk_ 7y ago

This doesn't seem too controversial of an opinion, haven't small modules already been at the core of a Unix philosophy? Once usage patterns emerge, these smaller libraries are composed into frameworks which ultimately build end-user applications. The entirety of engineering is premised upon leveraging powerful abstractions; cf. the reason we're not still coding via electric charge. Edit: To expand on the analogy: "A two-person startup already uses twenty-eight other tools" [1]. [1] https://news.ycombinator.com/item?id=22449314 https://news.ycombinator.com/item?id=22449314

jasode 7y ago
>, haven't small modules already been at the core of a Unix philosophy? I assume you're talking about userland programs such as "ls", "cut", "grep", etc. ?[1] But C Language is also part of UNIX and programmers typically link against a large more comprehensive "standard library" such as glibc: libc.a, libc.so <-- includes printf ... instead linking of a hundred little object files specified explicitly like this: printf.o strcat.o strncat.o malloc.o leftpad.o ... <hundred or thousand more .o files> What happened with Javascript is the language didn't have a comprehensive "standard library" like C, C++, Java, C#, Python "batteries included", Go, etc. Therefore, NPM became an adhoc "standard library" via bottom-up crowdsourcing. There are both positives and negatives to that approach. One negative is that leftpad() isn't produced by a canonical entity like Netscape or Mozilla but a random programmer. When he wanted to take his ball and go home, he broke everyone's build that depended on it. That type of event didn't happen with "printf()" in the C Language world. [1] https://en.wikipedia.org/wiki/Unix_philosophy#Do_One_Thing_and_Do_It_Well https://en.wikipedia.org/wiki/Unix_philosophy#Do_One_Thing_a...

KaiserPro 7y ago

All of this seems to be justifying a lack of a decent standard library. However its appears to be written from the point of view of someone who's never had the joy/pain of a decent standard library. The problem with lots of tiny modules is consistency and power. Sure its "open source" but each one of these modules are in various states of decay. better to lump them together in sets of functions, so that more eyes can be concentrated in the same place.

osrec 7y ago

The article highlights a huge pitfall of having lots of small dependencies, provides no real remediation, and says small dependencies are easier to reason about, so they're better anyway. I disagree. I personally find that with large projects, small dependencies actually become much harder to reason about - they sort of get lost in the codebase. Not to mention the duplication of small packages on npm doing similar things - with a large team, it's easy to add both to your codebase and then we're in a new sort of duplicative mess.

jscholes 7y ago

> Regarding Jobs' quote, here's another way to look at it: the most beautiful hotel in the world wouldn't feel like such if all the toilets overflowed when flushed. But this would be a functional problem, irritating at best and a potential health hazard at worst. Using plywood for the back of a chest of drawers is no such thing.