5 ms·

> I mean the empirical evidence alone seems point at the Unix Philosophy not actually being popular in practice. GNU tools are much more widely used on boxes hu

> I mean the empirical evidence alone seems point at the Unix Philosophy not actually being popular in practice. GNU tools are much more widely used on boxes humans interact with than BSD/busybox. I think that I can present an even better piece of empirical evidence (not to diminish the poignancy of your own): Almost no large or successful piece of software is composed of Unix command-line utilities. Firefox, Linux, gcc, Visual Studio Code, Windows, Chrome, Google services, Discord, Spotify, Blender, GIMP, Apache, Slack, Office, Steam, Matrix/Element...almost every single non-trivial piece of software in existence is not composed out of Unix shell utility pipelines. "Do one thing well" allows you to do one thing well, tautologically - but it doesn't allow you to do many things well. The fact that Unix utilities compose at a very superficial level is irrelevant, because the use of text as an interchange format is such a heart-stoppingly bad idea (consider a programming language where the only type is a string - you wanna write a web server in that?) that it renders Unix tools near-useless for building larger tools (as evidenced by the fact that nothing is made with them), and only useful for immediate interactive use, and building small scripts with (that would already be better served by a real programming language). The Unix Philosophy is self-contradictory. "Do one thing well" is meant to enable composition, for building larger programs - but "Text is the universal interface" is in direct opposition to building larger programs. But, because tools are only supposed to do one thing well, they're feature-impoverished and not very useful on their own, so you have to compose them. You're screwed either way.

jmfldn 5y ago
Shell programming through small cli tools was never meant to scale into massive complex software so you're comparing apples and bananas. Do one thing well and text as the universal interface works well in the small for parsing files, doing sys admin tasks, gluing a few things together. That's what it's intended for. It's obviously the wrong tool to build a system in. Used in the right context they compose beautifully.
rustyminnow 5y ago
Different tools serve different purposes. Just because you wouldn't write a web browser/server/IDE/OS in a shell scripting language doesn't mean they don't have value or are failed paradigms. I've written tons of shell scripts for one-off jobs or simple work that would've taken 2, 3, maybe 4 times the amount the time to write in a "real" programming language. (E.g. throwaway code to generate 10,000 test files that are slightly different would take me 5-10 minutes in bash, but maybe half an hour in node or python.) Passing around strings isn't the BEST paradigm but it's extremely flexible when you need flexibility. There are times when you need something more solid and that's fine too. And fwiw I'd bet a majority of the software you listed uses bash somewhere in either their build process or SAAS stack.
- shatteredgate 5y ago
  On the projects I've hacked on, a lot of time has been spent trying to replace bash and autotools in the build process and reduce its usage. They are brittle and nobody really likes to maintain giant shell scripts filled with fragile regular expressions and other unreadable awk invocations. Plus I've lost count of the number of build scripts I've seen that do strange things like failing to handle spaces in paths because something is not escaped correctly. They are really not a good solution. This can be caught with CI, but if you use any kind of CI then you have even more incentive to get rid of it because bash and makefiles (which fork an additional shell on every command) are very slow.
- throw10920 5y ago
  Different tools serve different purposes, yes. However, some tools have much narrower uses than others. For instance, most developers agree that writing programs in assembly language is a very bad idea for about 99.9% of programs (1 out of 1000 - which is generous). And, from both empirical evidence, and reasoning from first principles (stringly-typed, quoting and escaping issues, lack of tooling, low performance, implementation-defined behaviors, poor ecosystem, try-to-continue instead of fail-quick design), we can see that shell is entirely unsuited for everything except interactive use. Moreover, "time to write script" by itself is a bad metric, for several reasons: (1) the cost of understanding and maintaining the script is ignored (and obviously higher for bash than Python) (2) you're not including the higher likelihood of bugs in your bash programs (and all it takes is a single bug that requires 20 minutes to find and fix...) (3) if you had more experience with Python (or with specific libraries useful for writing shell-script-like things - this often goes overlooked) then your development time would probably be shorter and (4) the pool of Python devs is far greater than the pool of bash devs. > Passing around strings...[is] extremely flexible when you need flexibility This isn't accurate. Passing around typed data is just as flexible as strings - except that it adds a bunch of extra safety and type-checks. All data in "string" format, with the singular exception of actual English text that is being passed to an NLP system or something, is actually in a structured text format (the operations that you perform on which are merely isomorphic to existing operations on structured data)...that is very likely under-specified relative and more brittle than actual typed data. All that using a string representation does is make errors easier to make and harder to find. > And fwiw I'd bet a majority of the software you listed uses bash somewhere in either their build process or SAAS stack. Yes, because bash is a (bad) habit for tooling devs, not because it scales well or adds safety/performance/maintainability - as evidenced by the fact that there aren't any large pieces of software written in it. That is - bash is used "somewhere" and not "everywhere" because you can only get away with it at small scales - at large scales, it just falls apart.
PaulDavisThe1st 5y ago
> Firefox, Linux, gcc, Visual Studio Code, Windows, Chrome, Google services, Discord, Spotify, Blender, GIMP, Apache, Slack, Office, Steam, Matrix/Element...almost every single non-trivial piece of software in existence is not composed out of Unix shell utility pipelines. All true. However, the development environment (and frequently the build systems) used for all of the above are likely substantially composed out of Unix shell utilities and pipelines, aided by very unix-y tools such as Perl or awk. That says something, I think. Just not sure what.
- Maursault 5y ago
  Nice point. Pretty sure GP's observation, while correct, is wrong-headed, and something akin to saying, "Honda, Toyota, Volkswagen, Audi, Porsche, Chevrolet, Ford and even Tesla... every single non-trivial transportation vehicle in existence is not composed out of garage tools."
  - throw10920 5y ago
    "Wrong-headed" is a dismissal, not a counter-argument. Do you have a specific point that you want to argue against?
    - deleted 5y ago
      [deleted]
    - Maursault 5y ago
      Thanks! I really appreciate your straw man and shameless baiting, but my point was made. To be perfectly clear, if I can dumb it down a bit, though the observation is correct and maybe even mildly interesting, maybe it is even a unique observation no one else has noticed, it is, regardless, not a valid criticism in the same way it is not a valid criticism of garage tools that cars are not made out of them. So criticizing UNIX commands because no major applications are made out of them is, if not wrong-headed, then it is barking up the wrong tree, because, at least my understanding is, major applications are developed with programming language. I suspect there are a myriad of programs that fail to rise to the level to be classified major applications that are basically ugly and slow and strung-together UNIX commands, so it can be done, if desired, just as I expect one could build a vehicle out of garage tools if one had the drive to do so rather than use the appropriate materials. So there is perhaps some question begging in your observation in the hidden and incorrect assumption "UNIX tools are intended to be used as elements of major applications, that's what they're for" so that you may criticize in the precise way that you did. There are plenty of valid criticism of UNIX tools, but yours is not one of them. Though, it seems obvious, you have an keen and interesting mind possibly with a touch of the inability to be disagreed with, nor can tolerate criticism of your observations. Maybe I'm wrong, it just seemed that way just then.
citrin_ru 5y ago
> use of text as an interchange format is such a heart-stoppingly bad idea ... consider a programming language where the only type is a string Posix shell is a programming language where the only type is a string. Text as an interchange format works reasonably well in practice for a wide range of tasks. To me shell, easy to combine unix commands and text as interchange is a clear example of New Jersey style from "Worse is better" [1] and their critique comes mostly from followers of MIT/Stanford style. In my carrier I wrote hundreds of small shell scripts which use unix commands and text as an an interchange format. Yes, I see that some bugs can be prevented/detected by using something more strongly typed than shell, but all my scripts worked reliably enough and didn't suffer from any significant bugs so why bother? (disclaimer: I'm old fashioned and like to read man pages; I imagine shell can be full of traps if you ignore documentation and use only trail and error approach or practice stackoverflow driven development). Of course for large projects and even some small ones other tools/languages fit better than shell but shell have its niche. [1] https://en.wikipedia.org/wiki/Worse_is_better https://en.wikipedia.org/wiki/Worse_is_better
- throw10920 5y ago
  > Posix shell is a programming language where the only type is a string. You got it! > Text as an interchange format works reasonably well in practice for a wide range of tasks. Solutions using real programming languages score better on almost every conceivable metric (robustness, tooling, debuggability, maintainability, extensibility, performance, library ecosystem, developer pool, language pitfalls) other than initial time-to-develop, where they only win by a small margin (e.g. 10 minutes to develop a script instead of 2) - so, unless you're developing software purely for your own enjoyment, it's somewhat irresponsible to use shell instead of Python (or some other real, but script-amenable, programming language). > To me shell, easy to combine unix commands and text as interchange is a clear example of New Jersey style from "Worse is better" [1] and their critique comes mostly from followers of MIT/Stanford style. The New Jersey style is not only flawed on a theoretical level (just go look at any of those characteristics - tell me that it's more important for your bank or X-ray machine software, or your compiler or firewall to be simple than correct - and it's pretty clear that software that prioritizes simplicity above all else doesn't actually usefully serve humans, which is the point of software in the first place) but also unpopular on a practical level for actually building useful pieces of software. The New Jersey style is a style guide that is only useful for making art projects (where the value is in beauty, not utility) that aren't required to be performant, maintainable, feature-complete, or have correct behavior. > all my scripts worked reliably enough and didn't suffer from any significant bugs so why bother Because of the above list of metrics that real languages score better than shell on - the only thing that shell wins on is initial time-to-development! It's going to be harder for others to maintain, debug, optimize, understand, and extend your scripts after you leave, harder to diagnose bugs, harder to find people who can write bash... > shell have its niche Yes - but it's an extremely narrow one where time-to-development is the most important metric and you're only working with small problems (as above a relatively low problem complexity, shell scripting is slower than using real languages anyway) - and these conditions are very rarely true at the same time (e.g. in infrastructure you might have a small problem, but correctness is far more important than TTS - while in a startup you want low TTS, but your problems are non-trivial).
akitzmiller 5y ago
I think the whole value of the Unix tool ecosystem is that you don't need to build large pieces of software. If you look at all the GUI tools that Microsoft had to build for Windows administration in the earlier days and how they eventually came out with PowerShell because of the inadequacy of those tools, you can see how successful Unix tools have been by the --lack-- of big software that does their job.
- nyanpasu64 5y ago
  True, Linux is easier to automate. Though I still find Process Monitor's GUI more intuitive than trying to learn strace.
- pjmlp 5y ago
  Windows automation did not require Powershell, we were doing all of that with VBScript and JScript before alongside Technet Resource Kit. Also Powershell doesn't get much love from UNIX heads.
bodhiandpysics1 5y ago
Git is probably the most notable example of a complex software program that is composed of a series of independant tools, though gcc is actually also architecturally a pipeline. Visual studio code might also not be a great example. It's an ide, whose ide features are based on seperate process servers which communicate over a textual protocol... it's very plan 9 esque.

jmfldn 5y ago

Shell programming through small cli tools was never meant to scale into massive complex software so you're comparing apples and bananas. Do one thing well and text as the universal interface works well in the small for parsing files, doing sys admin tasks, gluing a few things together. That's what it's intended for. It's obviously the wrong tool to build a system in. Used in the right context they compose beautifully.

rustyminnow 5y ago

Different tools serve different purposes. Just because you wouldn't write a web browser/server/IDE/OS in a shell scripting language doesn't mean they don't have value or are failed paradigms. I've written tons of shell scripts for one-off jobs or simple work that would've taken 2, 3, maybe 4 times the amount the time to write in a "real" programming language. (E.g. throwaway code to generate 10,000 test files that are slightly different would take me 5-10 minutes in bash, but maybe half an hour in node or python.) Passing around strings isn't the BEST paradigm but it's extremely flexible when you need flexibility. There are times when you need something more solid and that's fine too. And fwiw I'd bet a majority of the software you listed uses bash somewhere in either their build process or SAAS stack.

shatteredgate 5y ago
On the projects I've hacked on, a lot of time has been spent trying to replace bash and autotools in the build process and reduce its usage. They are brittle and nobody really likes to maintain giant shell scripts filled with fragile regular expressions and other unreadable awk invocations. Plus I've lost count of the number of build scripts I've seen that do strange things like failing to handle spaces in paths because something is not escaped correctly. They are really not a good solution. This can be caught with CI, but if you use any kind of CI then you have even more incentive to get rid of it because bash and makefiles (which fork an additional shell on every command) are very slow.
throw10920 5y ago
Different tools serve different purposes, yes. However, some tools have much narrower uses than others. For instance, most developers agree that writing programs in assembly language is a very bad idea for about 99.9% of programs (1 out of 1000 - which is generous). And, from both empirical evidence, and reasoning from first principles (stringly-typed, quoting and escaping issues, lack of tooling, low performance, implementation-defined behaviors, poor ecosystem, try-to-continue instead of fail-quick design), we can see that shell is entirely unsuited for everything except interactive use. Moreover, "time to write script" by itself is a bad metric, for several reasons: (1) the cost of understanding and maintaining the script is ignored (and obviously higher for bash than Python) (2) you're not including the higher likelihood of bugs in your bash programs (and all it takes is a single bug that requires 20 minutes to find and fix...) (3) if you had more experience with Python (or with specific libraries useful for writing shell-script-like things - this often goes overlooked) then your development time would probably be shorter and (4) the pool of Python devs is far greater than the pool of bash devs. > Passing around strings...[is] extremely flexible when you need flexibility This isn't accurate. Passing around typed data is just as flexible as strings - except that it adds a bunch of extra safety and type-checks. All data in "string" format, with the singular exception of actual English text that is being passed to an NLP system or something, is actually in a structured text format (the operations that you perform on which are merely isomorphic to existing operations on structured data)...that is very likely under-specified relative and more brittle than actual typed data. All that using a string representation does is make errors easier to make and harder to find. > And fwiw I'd bet a majority of the software you listed uses bash somewhere in either their build process or SAAS stack. Yes, because bash is a (bad) habit for tooling devs, not because it scales well or adds safety/performance/maintainability - as evidenced by the fact that there aren't any large pieces of software written in it. That is - bash is used "somewhere" and not "everywhere" because you can only get away with it at small scales - at large scales, it just falls apart.

PaulDavisThe1st 5y ago

> Firefox, Linux, gcc, Visual Studio Code, Windows, Chrome, Google services, Discord, Spotify, Blender, GIMP, Apache, Slack, Office, Steam, Matrix/Element...almost every single non-trivial piece of software in existence is not composed out of Unix shell utility pipelines. All true. However, the development environment (and frequently the build systems) used for all of the above are likely substantially composed out of Unix shell utilities and pipelines, aided by very unix-y tools such as Perl or awk. That says something, I think. Just not sure what.

Maursault 5y ago
Nice point. Pretty sure GP's observation, while correct, is wrong-headed, and something akin to saying, "Honda, Toyota, Volkswagen, Audi, Porsche, Chevrolet, Ford and even Tesla... every single non-trivial transportation vehicle in existence is not composed out of garage tools."
- throw10920 5y ago
  "Wrong-headed" is a dismissal, not a counter-argument. Do you have a specific point that you want to argue against?
  - deleted 5y ago
    [deleted]
  - Maursault 5y ago
    Thanks! I really appreciate your straw man and shameless baiting, but my point was made. To be perfectly clear, if I can dumb it down a bit, though the observation is correct and maybe even mildly interesting, maybe it is even a unique observation no one else has noticed, it is, regardless, not a valid criticism in the same way it is not a valid criticism of garage tools that cars are not made out of them. So criticizing UNIX commands because no major applications are made out of them is, if not wrong-headed, then it is barking up the wrong tree, because, at least my understanding is, major applications are developed with programming language. I suspect there are a myriad of programs that fail to rise to the level to be classified major applications that are basically ugly and slow and strung-together UNIX commands, so it can be done, if desired, just as I expect one could build a vehicle out of garage tools if one had the drive to do so rather than use the appropriate materials. So there is perhaps some question begging in your observation in the hidden and incorrect assumption "UNIX tools are intended to be used as elements of major applications, that's what they're for" so that you may criticize in the precise way that you did. There are plenty of valid criticism of UNIX tools, but yours is not one of them. Though, it seems obvious, you have an keen and interesting mind possibly with a touch of the inability to be disagreed with, nor can tolerate criticism of your observations. Maybe I'm wrong, it just seemed that way just then.

citrin_ru 5y ago

> use of text as an interchange format is such a heart-stoppingly bad idea ... consider a programming language where the only type is a string Posix shell is a programming language where the only type is a string. Text as an interchange format works reasonably well in practice for a wide range of tasks. To me shell, easy to combine unix commands and text as interchange is a clear example of New Jersey style from "Worse is better" [1] and their critique comes mostly from followers of MIT/Stanford style. In my carrier I wrote hundreds of small shell scripts which use unix commands and text as an an interchange format. Yes, I see that some bugs can be prevented/detected by using something more strongly typed than shell, but all my scripts worked reliably enough and didn't suffer from any significant bugs so why bother? (disclaimer: I'm old fashioned and like to read man pages; I imagine shell can be full of traps if you ignore documentation and use only trail and error approach or practice stackoverflow driven development). Of course for large projects and even some small ones other tools/languages fit better than shell but shell have its niche. [1] https://en.wikipedia.org/wiki/Worse_is_better https://en.wikipedia.org/wiki/Worse_is_better

throw10920 5y ago
> Posix shell is a programming language where the only type is a string. You got it! > Text as an interchange format works reasonably well in practice for a wide range of tasks. Solutions using real programming languages score better on almost every conceivable metric (robustness, tooling, debuggability, maintainability, extensibility, performance, library ecosystem, developer pool, language pitfalls) other than initial time-to-develop, where they only win by a small margin (e.g. 10 minutes to develop a script instead of 2) - so, unless you're developing software purely for your own enjoyment, it's somewhat irresponsible to use shell instead of Python (or some other real, but script-amenable, programming language). > To me shell, easy to combine unix commands and text as interchange is a clear example of New Jersey style from "Worse is better" [1] and their critique comes mostly from followers of MIT/Stanford style. The New Jersey style is not only flawed on a theoretical level (just go look at any of those characteristics - tell me that it's more important for your bank or X-ray machine software, or your compiler or firewall to be simple than correct - and it's pretty clear that software that prioritizes simplicity above all else doesn't actually usefully serve humans, which is the point of software in the first place) but also unpopular on a practical level for actually building useful pieces of software. The New Jersey style is a style guide that is only useful for making art projects (where the value is in beauty, not utility) that aren't required to be performant, maintainable, feature-complete, or have correct behavior. > all my scripts worked reliably enough and didn't suffer from any significant bugs so why bother Because of the above list of metrics that real languages score better than shell on - the only thing that shell wins on is initial time-to-development! It's going to be harder for others to maintain, debug, optimize, understand, and extend your scripts after you leave, harder to diagnose bugs, harder to find people who can write bash... > shell have its niche Yes - but it's an extremely narrow one where time-to-development is the most important metric and you're only working with small problems (as above a relatively low problem complexity, shell scripting is slower than using real languages anyway) - and these conditions are very rarely true at the same time (e.g. in infrastructure you might have a small problem, but correctness is far more important than TTS - while in a startup you want low TTS, but your problems are non-trivial).

akitzmiller 5y ago

I think the whole value of the Unix tool ecosystem is that you don't need to build large pieces of software. If you look at all the GUI tools that Microsoft had to build for Windows administration in the earlier days and how they eventually came out with PowerShell because of the inadequacy of those tools, you can see how successful Unix tools have been by the --lack-- of big software that does their job.

nyanpasu64 5y ago
True, Linux is easier to automate. Though I still find Process Monitor's GUI more intuitive than trying to learn strace.
pjmlp 5y ago
Windows automation did not require Powershell, we were doing all of that with VBScript and JScript before alongside Technet Resource Kit. Also Powershell doesn't get much love from UNIX heads.

bodhiandpysics1 5y ago

Git is probably the most notable example of a complex software program that is composed of a series of independant tools, though gcc is actually also architecturally a pipeline. Visual studio code might also not be a great example. It's an ide, whose ide features are based on seperate process servers which communicate over a textual protocol... it's very plan 9 esque.