8 ms·

Following the Unix philosophy without getting left-pad (2021)

78 points by goranmoomin 4y ago

Teongot 4y ago
What was the origin of the idea that the Unix philosophy is "do one thing and do it well"? If anything, the UNIX philosophy is more like "do one thing, and do it to a text file".
- TylerE 4y ago
  do one thing, and do it with a file handle.
  - jbverschoor 4y ago
    aka functional
    - marginalia_nu 4y ago
      Really though?
- dementis 4y ago
  If you look at the wiki on the UNIX philosophy it does say "Store data in flat text files" and "Write programs to handle text streams, because that is a universal interface." You can do a surprising amount with flat files. For decades HP / HPE has been using flat files for Serviceguard / SGLX which is a high-availability cluster software that gets used in some very large Enterprise environments.
  - a1369209993 4y ago
    > You can do a surprising amount with flat files. For example, tables in relational databases like sqlite are representationally isomorphic to tabular files like the output from `ls -al`. Sqlite has a lot of useful performance optimizations (including eg, not having to encode strings as "\x20" or the like if you want arbitrary bytes), but those come at the cost of a data format that you can't easily (not even notice that you needed to and did) reverse engineer, when either writing new software to consume or emit it, or even just visually inspecting it in a editor that doesn't already speak that format.
  - Avshalom 4y ago
    you can do anything with (flat|text) files, they are after all just an often inefficient binary encoding the same as any other binary blob. The main thing you can do with them though, is tell yourself that you don't really need a spec or documentation because you can just eyeball the output, and not having to follow a spec makes things feel simple.
- dctoedt 4y ago
  > What was the origin of the idea that the Unix philosophy is "do one thing and do it well"? Relatedly: Brian Kernighan's Unix: A History and a Memoir was an enjoyable read. https://www.amazon.com/UNIX-History-Memoir-Brian-Kernighan/dp/1695978552 https://www.amazon.com/UNIX-History-Memoir-Brian-Kernighan/d... (not an affiliate link).
deleted 4y ago
[deleted]
nonameiguess 4y ago
This is conflating two things: the granularity of system utilities and the granularity of packages. Those are not the same thing. GNU core-utils provides 24 separate utilities into a single package. Add in sed, grep, and bash, which has built-ins providing virtually everything else, and you're pretty close to full POSIX with only four packages, thus four dependencies. At the language level rather than system utilities, think of something like boost. You install one dependency, and you get hundreds of individual libraries, each of which does only one thing and one thing well, and you can link only what you need into your own program. UNIX philosophy followed, without turning your dependency tree into combinatorial explosion.
- bryanlarsen 4y ago
  Yes, that is one of two conclusions in the article.
- pjmlp 4y ago
  On top of that calling either man or info on those utilities will show how far beyond they are of doing just one thing.
kerblang 4y ago
- deleted 4y ago
  [deleted]
- pvg 4y ago
  If you want a flag to rally around, the 'no meta' one is better and a lot less effort.
- ummonk 4y ago
  It writes a 10 page essay for what can be described in one or two paragraphs - and is obvious to basically anyone who advocates for good standard libraries.
dementis 4y ago
The article seems to miss a very important factor of the Unix philosophy and that is "Make every program a filter" which means you also have to "Write programs to work together". Having a program that can do one thing well without left-pad is a worthwhile endeavor but if you can't leverage it for more than its original intended purpose has it truly reached its full potential?
niccl 4y ago
total nitpick, but it's unfortunate that he didn't credit XKCD for the image (https://xkcd.com/2347/ https://xkcd.com/2347/), since it's under creative commons with attribution licence (https://creativecommons.org/licenses/by-nc/2.5/ https://creativecommons.org/licenses/by-nc/2.5/)
- spullara 4y ago
  It is directly linked to the XKCD page.
Tainnor 4y ago
The article raises some interesting points but I still disagree that comparing the mess of JavaScript dependencies[0] to GNU utils is warranted - the level of trust is fundamentally different. I have some reasonable intuition that the authors of GNU packages as well as the distro maintainers do a good job writing, packaging and vetting these tools. That's not really the case with left-pad or is-array. I can trust that an update to cat or ls will not break my system because enough people have looked at it, but I really don't know what's gonna happen when I update left-pad (without even knowing it, because a it's a transitive dependency). The other point is that GNU utils are self-contained binaries and don't usually include transitive dependencies 20 levels deep. Even when the problems these packages solve are not 100% trivial, unless you trust that library to be continuously maintained, maybe consider not adding the dependency, instead taking inspiration from the code and writing your own, maybe even simplified version of it (careful with that, you don't want to write your own crypto, for example - but when we're talking about a three-file "compare XML files" test util that hasn't been updated since 2014, just copying that code into your project might be the better approach). [0]: And it really is JavaScript - I don't know of any other language with the same kind of explosion of micro-packages. I'm not saying that other languages never experience dependency hell - of course they do - only that it's usually not as horrifying as it is with JS.
- amyjess 4y ago
  > I can trust that an update to cat or ls will not break my system because enough people have looked at it, but I really don't know what's gonna happen when I update left-pad (without even knowing it, because a it's a transitive dependency). The biggest difference here is that distros vet their maintainers, often with a lengthy process similar to hiring someone at a company, while NPM and similar repos are entirely self-serve, where anyone can register an account and just publish whatever packages.
  - iudqnolq 4y ago
    > The biggest difference here is that distros vet their maintainers, often with a lengthy process similar to hiring someone at a company This is very surprising to me. During the debacle over Rust in the Python cryptography package maintainers were saying it's unreasonable to expect them to read dependency changelogs.
TrianguloY 4y ago
One thing missing in the article is the all-or-nothing of dependencies in most languages. A package with millions of micro functions is nice, but if your program only uses 10 of them it doesn't matter, you program has all. Some dependency managers allows you to "disable" components, but they need to be manually picked and usually you can only disable chunks, not individual functions. Wouldn't be nice to let the compiler package only the functions really used in the code and discard the rest? Do any language provide this? (C/C++?)
- dathanb82 4y ago
  It's available (usually as an optional feature) in lots of languages where dependencies are linked statically. The most standard terminology I'm aware of for it is "dead code elimination" In Javascript, webpack (as well as other module bundlers) supports it, though they call their dead code elimination mechanism "tree shaking".
dwohnitmok 4y ago
The three solutions the author lists at the end of the article are themselves often in tension with each other, that is an expressive language is at odds with having a single great standard library and having an agreed-upon single utility package. An expressive language almost inevitably leads to more than one way of doing things and those different ways of doing things are often fundamentally mutually incompatible (i.e. a better way of writing the code will not save you). That essentially fragments the standard library and any single utility packages. See e.g. the absolute profusion of alternative "standard libraries" for Common Lisp, Haskell, etc. Even ones that rein things in a little bit like Clojure still fragment when it comes to things like concurrency, with a lot of 3rd-party libraries purporting to replace core libraries (e.g. core.async vs manifold).
raggi 4y ago
the at&t unix video marketed this idea by way of composing a spell checker in a shell script. even at the time, many users did not follow this pattern because the ergonomics for spell checking in this manner were already sub-par. what you get if you follow this pattern is git, not left-pad. git high level commands were derivative higher order combinations of underlying programs that are quite conformant to this aspect of the unix philosophy. their newer replacements in many cases no longer compose from single purpose programs, instead importing aspects of their function as code, and specializing new behavior in order to better meet user interface demands. This is a direct repetition of the spell checker UX story.
zokier 4y ago
From my point of view the problem is not micropackages, but the constantly changing micropackages making the foundation of your software ever shifting around. The problem with leftpad was not any problem with the library itself, but the library changing (in this case getting deleted). I posit that embracing the culture of vendoring dependencies, and even fully absorbing them to your projects, leads in the longer term healthier ecosystem. Again leftpad, if everyone had vendored it then it would have been largely a non-event. Of course this way of thinking puts much higher demand on correctness and also probably some novel innovations regarding patch management when as a library maintainer you can not just rely on ability to push out new versions from your cathedralesque project.
kazinator 4y ago
Each one-job Unix program has to do about four things: 1. parse command line arguments 2. parse an input format 3. do its one job 4. format output Two or three out of these are superfluous and could be shared with a larger program if it were just a function in it. For instance, regarding (1) if the program just has a "dry run -n" and "verbose debugging -v" flag, and it becomes a function in the larger program which already has those flags, all the function has to do is refer to the configuration. Regarding (2), the function probably has access to some data structure constructed by the larger program rather than raw text. Regarding (4), the function may be able to put out a data structure for whose type the larger program already has an output format. Unix itself showed that the one-program-for-one-task is a weak idea when Awk was developed: a programmable utility that allows a single program running in one process to avoid using numerous Unix tools.
- bombolo 4y ago
  > Two or three out of these are superfluous and could be shared with a larger program if it were just a function in it. They don't "parse" command line. They use getopt/getopt_long, which does the parsing. Besides using library functions, GNU coreutils commands do share code.
fedeb95 4y ago
The point isn't big or small packages. Is unregulated public registries. Some quality/security should be enforced. Like some linux distributions do.
deleted 4y ago
[deleted]
foooobaba 4y ago
Programs/functions like left-pad do not follow the unix philosophy (which the author did say). The idea of the unix philosophy is to write less programs. Not write a ton of tiny programs, but a small number of composable programs. So instead of left-pad, right-pad, middle-pad, and anything else to get specific need you can use sprintf which can format a string in general, one program that can do implement the features of all of those. It is about “coding the permitter, not the area”, so if you have NxM features to build, don’t write N*M individual programs/functions, instead write N+M programs/functions that can be composed together to get all NxM features. This video explains it well: https://youtu.be/3Ea3pkTCYx4 https://youtu.be/3Ea3pkTCYx4
amadeuspagel 4y ago
The only problem with left-pad was that NPM let people delete packages that other packages depend on. That's literally it.
- crznp 4y ago
  I need a test runner, so I install "karma" and therefore "braces" and "fill-range" and "to-regex-range" and "is-number" and 100 other things. Dependencies come with risk and cognitive load: deleted like left-pad or loaded with malware like ua-parser-js. Sometimes that is worth it. But if you need 10 lines to pad a string or validate a number, copy some code off of stack overflow like a real developer. Nobody has to know.
akira2501 4y ago
It's a flawed comparison. The tools may be distinct and "do one thing well" but they all just compose functionality that's mostly already provided by the giant shared libc you have on the system. The micro-packages idea would require us to take glibc and break it out into thousands of different packages. Perhaps one packaged library for each individual system call? That's about the level that the "left pad as a package" idea operates at. If you want a larger javascript library that provides fills for many common string functions that are missing in the broader language and keep those separate from a group of array functions, that would be a reasonable middle ground and would approach more closely the actual reality of a glibc based distribution of utilities. It's either that, or you have to accept that these small cases really do need to be folded back into a "standard library" that gets distributed along side whichever JavaScript environment you're using. This is something that languages like ruby, python and go were much better positioned to solve while still having a large repository of third party dependencies you can easily incorporate into your project.
donatj 4y ago
I love love love ffmpeg but I'll be damned if it's not a horrible übertool that should probably be several hundred separate tools. It's flags are a language in and of themselves.
kasabali 4y ago
> left-pad is Unix philosophy Yeah. No.

78 points by goranmoomin 4y ago

What was the origin of the idea that the Unix philosophy is "do one thing and do it well"? If anything, the UNIX philosophy is more like "do one thing, and do it to a text file".

TylerE 4y ago
do one thing, and do it with a file handle.
- jbverschoor 4y ago
  aka functional
  - marginalia_nu 4y ago
    Really though?
dementis 4y ago
If you look at the wiki on the UNIX philosophy it does say "Store data in flat text files" and "Write programs to handle text streams, because that is a universal interface." You can do a surprising amount with flat files. For decades HP / HPE has been using flat files for Serviceguard / SGLX which is a high-availability cluster software that gets used in some very large Enterprise environments.
- a1369209993 4y ago
  > You can do a surprising amount with flat files. For example, tables in relational databases like sqlite are representationally isomorphic to tabular files like the output from `ls -al`. Sqlite has a lot of useful performance optimizations (including eg, not having to encode strings as "\x20" or the like if you want arbitrary bytes), but those come at the cost of a data format that you can't easily (not even notice that you needed to and did) reverse engineer, when either writing new software to consume or emit it, or even just visually inspecting it in a editor that doesn't already speak that format.
- Avshalom 4y ago
  you can do anything with (flat|text) files, they are after all just an often inefficient binary encoding the same as any other binary blob. The main thing you can do with them though, is tell yourself that you don't really need a spec or documentation because you can just eyeball the output, and not having to follow a spec makes things feel simple.
dctoedt 4y ago
> What was the origin of the idea that the Unix philosophy is "do one thing and do it well"? Relatedly: Brian Kernighan's Unix: A History and a Memoir was an enjoyable read. https://www.amazon.com/UNIX-History-Memoir-Brian-Kernighan/dp/1695978552 https://www.amazon.com/UNIX-History-Memoir-Brian-Kernighan/d... (not an affiliate link).

deleted 4y ago

[deleted]

nonameiguess 4y ago

This is conflating two things: the granularity of system utilities and the granularity of packages. Those are not the same thing. GNU core-utils provides 24 separate utilities into a single package. Add in sed, grep, and bash, which has built-ins providing virtually everything else, and you're pretty close to full POSIX with only four packages, thus four dependencies. At the language level rather than system utilities, think of something like boost. You install one dependency, and you get hundreds of individual libraries, each of which does only one thing and one thing well, and you can link only what you need into your own program. UNIX philosophy followed, without turning your dependency tree into combinatorial explosion.

bryanlarsen 4y ago
Yes, that is one of two conclusions in the article.
pjmlp 4y ago
On top of that calling either man or info on those utilities will show how far beyond they are of doing just one thing.

kerblang 4y ago

deleted 4y ago
[deleted]
pvg 4y ago
If you want a flag to rally around, the 'no meta' one is better and a lot less effort.
ummonk 4y ago
It writes a 10 page essay for what can be described in one or two paragraphs - and is obvious to basically anyone who advocates for good standard libraries.

dementis 4y ago

The article seems to miss a very important factor of the Unix philosophy and that is "Make every program a filter" which means you also have to "Write programs to work together". Having a program that can do one thing well without left-pad is a worthwhile endeavor but if you can't leverage it for more than its original intended purpose has it truly reached its full potential?

niccl 4y ago

total nitpick, but it's unfortunate that he didn't credit XKCD for the image (https://xkcd.com/2347/ https://xkcd.com/2347/), since it's under creative commons with attribution licence (https://creativecommons.org/licenses/by-nc/2.5/ https://creativecommons.org/licenses/by-nc/2.5/)

spullara 4y ago
It is directly linked to the XKCD page.

Tainnor 4y ago

The article raises some interesting points but I still disagree that comparing the mess of JavaScript dependencies[0] to GNU utils is warranted - the level of trust is fundamentally different. I have some reasonable intuition that the authors of GNU packages as well as the distro maintainers do a good job writing, packaging and vetting these tools. That's not really the case with left-pad or is-array. I can trust that an update to cat or ls will not break my system because enough people have looked at it, but I really don't know what's gonna happen when I update left-pad (without even knowing it, because a it's a transitive dependency). The other point is that GNU utils are self-contained binaries and don't usually include transitive dependencies 20 levels deep. Even when the problems these packages solve are not 100% trivial, unless you trust that library to be continuously maintained, maybe consider not adding the dependency, instead taking inspiration from the code and writing your own, maybe even simplified version of it (careful with that, you don't want to write your own crypto, for example - but when we're talking about a three-file "compare XML files" test util that hasn't been updated since 2014, just copying that code into your project might be the better approach). [0]: And it really is JavaScript - I don't know of any other language with the same kind of explosion of micro-packages. I'm not saying that other languages never experience dependency hell - of course they do - only that it's usually not as horrifying as it is with JS.

amyjess 4y ago
> I can trust that an update to cat or ls will not break my system because enough people have looked at it, but I really don't know what's gonna happen when I update left-pad (without even knowing it, because a it's a transitive dependency). The biggest difference here is that distros vet their maintainers, often with a lengthy process similar to hiring someone at a company, while NPM and similar repos are entirely self-serve, where anyone can register an account and just publish whatever packages.
- iudqnolq 4y ago
  > The biggest difference here is that distros vet their maintainers, often with a lengthy process similar to hiring someone at a company This is very surprising to me. During the debacle over Rust in the Python cryptography package maintainers were saying it's unreasonable to expect them to read dependency changelogs.

TrianguloY 4y ago

One thing missing in the article is the all-or-nothing of dependencies in most languages. A package with millions of micro functions is nice, but if your program only uses 10 of them it doesn't matter, you program has all. Some dependency managers allows you to "disable" components, but they need to be manually picked and usually you can only disable chunks, not individual functions. Wouldn't be nice to let the compiler package only the functions really used in the code and discard the rest? Do any language provide this? (C/C++?)

dathanb82 4y ago
It's available (usually as an optional feature) in lots of languages where dependencies are linked statically. The most standard terminology I'm aware of for it is "dead code elimination" In Javascript, webpack (as well as other module bundlers) supports it, though they call their dead code elimination mechanism "tree shaking".

dwohnitmok 4y ago

The three solutions the author lists at the end of the article are themselves often in tension with each other, that is an expressive language is at odds with having a single great standard library and having an agreed-upon single utility package. An expressive language almost inevitably leads to more than one way of doing things and those different ways of doing things are often fundamentally mutually incompatible (i.e. a better way of writing the code will not save you). That essentially fragments the standard library and any single utility packages. See e.g. the absolute profusion of alternative "standard libraries" for Common Lisp, Haskell, etc. Even ones that rein things in a little bit like Clojure still fragment when it comes to things like concurrency, with a lot of 3rd-party libraries purporting to replace core libraries (e.g. core.async vs manifold).

raggi 4y ago

the at&t unix video marketed this idea by way of composing a spell checker in a shell script. even at the time, many users did not follow this pattern because the ergonomics for spell checking in this manner were already sub-par. what you get if you follow this pattern is git, not left-pad. git high level commands were derivative higher order combinations of underlying programs that are quite conformant to this aspect of the unix philosophy. their newer replacements in many cases no longer compose from single purpose programs, instead importing aspects of their function as code, and specializing new behavior in order to better meet user interface demands. This is a direct repetition of the spell checker UX story.

zokier 4y ago

From my point of view the problem is not micropackages, but the constantly changing micropackages making the foundation of your software ever shifting around. The problem with leftpad was not any problem with the library itself, but the library changing (in this case getting deleted). I posit that embracing the culture of vendoring dependencies, and even fully absorbing them to your projects, leads in the longer term healthier ecosystem. Again leftpad, if everyone had vendored it then it would have been largely a non-event. Of course this way of thinking puts much higher demand on correctness and also probably some novel innovations regarding patch management when as a library maintainer you can not just rely on ability to push out new versions from your cathedralesque project.

kazinator 4y ago

Each one-job Unix program has to do about four things: 1. parse command line arguments 2. parse an input format 3. do its one job 4. format output Two or three out of these are superfluous and could be shared with a larger program if it were just a function in it. For instance, regarding (1) if the program just has a "dry run -n" and "verbose debugging -v" flag, and it becomes a function in the larger program which already has those flags, all the function has to do is refer to the configuration. Regarding (2), the function probably has access to some data structure constructed by the larger program rather than raw text. Regarding (4), the function may be able to put out a data structure for whose type the larger program already has an output format. Unix itself showed that the one-program-for-one-task is a weak idea when Awk was developed: a programmable utility that allows a single program running in one process to avoid using numerous Unix tools.

bombolo 4y ago
> Two or three out of these are superfluous and could be shared with a larger program if it were just a function in it. They don't "parse" command line. They use getopt/getopt_long, which does the parsing. Besides using library functions, GNU coreutils commands do share code.

fedeb95 4y ago

The point isn't big or small packages. Is unregulated public registries. Some quality/security should be enforced. Like some linux distributions do.

deleted 4y ago

[deleted]

foooobaba 4y ago

Programs/functions like left-pad do not follow the unix philosophy (which the author did say). The idea of the unix philosophy is to write less programs. Not write a ton of tiny programs, but a small number of composable programs. So instead of left-pad, right-pad, middle-pad, and anything else to get specific need you can use sprintf which can format a string in general, one program that can do implement the features of all of those. It is about “coding the permitter, not the area”, so if you have NxM features to build, don’t write N*M individual programs/functions, instead write N+M programs/functions that can be composed together to get all NxM features. This video explains it well: https://youtu.be/3Ea3pkTCYx4 https://youtu.be/3Ea3pkTCYx4

amadeuspagel 4y ago

The only problem with left-pad was that NPM let people delete packages that other packages depend on. That's literally it.

crznp 4y ago
I need a test runner, so I install "karma" and therefore "braces" and "fill-range" and "to-regex-range" and "is-number" and 100 other things. Dependencies come with risk and cognitive load: deleted like left-pad or loaded with malware like ua-parser-js. Sometimes that is worth it. But if you need 10 lines to pad a string or validate a number, copy some code off of stack overflow like a real developer. Nobody has to know.

akira2501 4y ago

It's a flawed comparison. The tools may be distinct and "do one thing well" but they all just compose functionality that's mostly already provided by the giant shared libc you have on the system. The micro-packages idea would require us to take glibc and break it out into thousands of different packages. Perhaps one packaged library for each individual system call? That's about the level that the "left pad as a package" idea operates at. If you want a larger javascript library that provides fills for many common string functions that are missing in the broader language and keep those separate from a group of array functions, that would be a reasonable middle ground and would approach more closely the actual reality of a glibc based distribution of utilities. It's either that, or you have to accept that these small cases really do need to be folded back into a "standard library" that gets distributed along side whichever JavaScript environment you're using. This is something that languages like ruby, python and go were much better positioned to solve while still having a large repository of third party dependencies you can easily incorporate into your project.

donatj 4y ago

I love love love ffmpeg but I'll be damned if it's not a horrible übertool that should probably be several hundred separate tools. It's flags are a language in and of themselves.

kasabali 4y ago

> left-pad is Unix philosophy Yeah. No.