14 ms·

The growth of command line options, 1979-Present (2020)

186 points by intrepidhero 5y ago

pjmlp 5y ago
Now there is a page I will glady bookmark for those "UNIX philosophy" arguments.
hulitu 5y ago
> Ironically, one of the reasons for the rise in the number of command line options is another McIlroy dictum, "Write programs to handle text streams, because that is a universal interface" (see ls for one example of this). That is unfortunately not the reason. There are 2 main reasons why today (GNU) command line utilities have a lot of options: 1). GNU. Apparrently they like to add features to programs 2). Herritage of SYSV/BSD. Traditionally BSD options were different from SYSV and the new GNU utilities were made to understand both.
- avgcorrection 5y ago
  I don’t see an argument from you that (1) wasn’t caused by the point that Luu raises. It’s not strange to end up adding features to programs if the effective IPC is so impoverished that you would need to sandwich a bunch of repetitive formatting commands if you didn’t add some convenient built-in formatting options.
- fsckboy 5y ago
  > 1). GNU. Apparrently they like to add features to programs unix comes out of "worse is better". GNU comes out of The Right Thing.
- seoaeu 5y ago
  Adding features isn't itself a bad thing. But when you cannot iterate or remove features (even the smallest change to existing functionality could break a script somewhere!) then you end up with the jumbled mess we have.
intrepidhero 5y ago
This got linked in another front page story. Uncomfortable Truths in Software Engineering. > The Unix philosophy of “do one thing well” doesn’t actually work that well. Dan Luu explains this better than I could. Usually The Unix Philosophy gets nothing but praise in these parts so I was interested to see a counterpoint.
- ReleaseCandidat 5y ago
  The 'Unix Haters Book' has some too: https://web.mit.edu/~simsong/www/ugh.pdf https://web.mit.edu/~simsong/www/ugh.pdf
  - marcodiego 5y ago
    The UHHB has very few points that are still valid. It is a humorous piece and was written about 3 decades ago. We shouldn't continue citing it as valid criticism.
    - ReleaseCandidat 5y ago
      > The UHHB has very few points that are still valid. I disagree. Yes, nobody uses sendmail anymore, filesystems vastly improved and X isn't a resource hog any more. I've reread it some months ago and the main points still hold. For example the part about `kill`: Most operating systems have a “kill” command. So does Unix. In most operating systems, the kill command kills a process. The Unix implementation is much more general: the “kill” command sends a process a message. This illustrates the first Unix design principle: • Give the user power by making operations fully general. The kill command is very powerful; it lets you send all sorts of messages to processes. For example, one message you can send to a process tells it to kill itself. This message is -9. -9 is, of course, the largest single-digit message, which illustrates another important Unix design principle: • Choose simple names that reflect function.
      - monroeclinton 5y ago
        I don't find the argument about kill convincing. You can use `kill pid` and it will send a SIGTERM signal to the process. The -9 option is only necessary when you want to send a SIGKILL signal. I don't think it is a bad thing that kill gives users the option to choose which signal to send. I only use the -9 option when I want to terminate a process without it being handled.
pif 5y ago
Nothing more than a load of garbage.
- simiones 5y ago
  The article, or the growing command line options list?
  - munk-a 5y ago
    The comment since it lacks any context and presents nothing but a vague opinion. It's actually a recursive comment because the poster heard that programmers love recursion.
mjw1007 5y ago
A gentle suggestion for bloggers: if you're going to write an article with a title like "The growth of command line options, 1979-Present", please also arrange to prominently display the date of composition.
- pugworthy 5y ago
  March 2020 for those of you who just had to find out after reading this comment.
  - HotHotLava 5y ago
    > The latest year in the table is 2017 because I wrote the first draft for this post in 2017 and didn't get around to cleaning it up until 2020.
- ghaff 5y ago
  Really, any blog post should have a prominent date on it. For a lot of things, it's essential context. Older posts can be useful/interesting but they can also be pretty much completely useless depending upon the topic.
- bo1024 5y ago
  Every article (really, every thing) posted on the web should be timestamped.
woodruffw 5y ago
Maybe it's because I'm not the biggest Unix purist, but I don't really mind the proliferation of command line options. The two things that do irk me are: * Using short options in scripts. Scripts should almost always use long options. As a corollary, tools should always provide long options unless the tool isn't intended for scripting. * Lack of discoverability. We still don't have a standard way to ask an arbitrary tool which flags it supports, or for a rough semantic mapping of those flags (e.g. the classic -v/-V verbose and version confusion). Developing a standard here would go a long way towards automating things like tab completion without each tool needing to maintain N scripts for N shells or having its own slightly-different `--completions=SHELL` generator.
- jamespwilliams 5y ago
  On the latter, an example that comes to mind is how to specify field numbers: -f in cut, -k in sort, -j for the most common option in join. And specifying field delimiters: -d in cut, -t in sort, -F in awk.
  - woodruffw 5y ago
    Ach, those always get me. Thankfully most (all?) of those also respect the `$IFS` variable, which is what I tend to use.
    - gnubison 5y ago
      None of them should respect IFS — that’s only for the shell.
      - woodruffw 5y ago
        Huh. I could have sworn that GNU Awk respects IFS, but I can't find that in the manual now. Maybe I dreamed it up.
- nauticacom 5y ago
  The last point would be amazing, imo. It's unfortunate that, because of history, commands accept and parse arbitrary strings as input instead of formally specifying it like a function signature + docblock. If I could rewrite the universe, commands would use a central API for specifying the names, data types, descriptions, etc. for all the input they take. In our timeline, maybe some file format could be standardized that describes the particular inputs and options a command takes, and e.g. shells would hook into it. Kind of like header files, but distributed either in a community repository or by each of the tools themselves.
munificent 5y ago
Two of the many joys in life are: 1. Having a problem and solving it with a small amount of effort. 2. Having a mental model of some external thing and delighting in seeing that your model is accurate with respect to the actual thing. Elegant, minimal systems excel at providing #2. Big sprawling systems with lots of options and special cases excel at providing #1. Many software engineers seem to be calibrated to extract a lot more joy from experiencing #2 than #1. It's probably something to do with the nature of the work. We work "backwards" by having a mental model of a thing we wish existed and then bring it into being. If we didn't find deep satisfaction in harmony between our mental model and software model, we wouldn't have the drive to keep fixing all of the bugs in our software. But there is nothing instrinsically superior to the joy of #2 compared to #1. It's just a psychological/emotional preference. Many people, if not most, outside of the world of software engineering seem to prefer #1. They just like to get stuff done and make stuff happening. You rarely see woodworkers lamenting about the variety of clamps, jigs, and tools they have. Instead, they delight in finding just the right specific tool to make a job easy. Fishermen love their tackleboxes. Of course, there is an appreciation of #2 in all of us, and you see minimalism everywhere. But I've never seen it raised to such a religious, moral level as I see in software engineering. It's important to remember that that's a psychological choice we're making—a personal preference we impose—and not something fundamental to systems.
- wyager 5y ago
  > Big sprawling systems with lots of options and special cases excel at providing #1. This is the sales pitch, but in reality you spend so much time configuring and debugging that you rarely actually save any time.
  - munificent 5y ago
    > in reality you spend so much time configuring and debugging that you rarely actually save any time. Conversely, that's the sales pitch for minimalism. But the reality is that well-intentioned, thoughtful designers and users seem to end up spending most of their time building and using systems with a lot of complexity. If it really ended up costing them more time to use those systems, it begs the question of why we keep building them.
tra3 5y ago
Interesting article, but I don't see how the increase in number of arguments invalidates the "do one thing well". Sure ls has a 58 (!) arguments but it's still a tool to list files ultimately. > I've heard people say that there isn't really any alternative to this kind of complexity for command line tools, but people who say that have never really tried the alternative, something like PowerShell. I've spent a lot of time passing around JSON and XML, not the same but similar. While it's true that structured data makes things easier, it's not without it's own faults. I subscribe to unix philosophy of "do one thing well" and pass text around because it's served me well. You have to question status quo to come up with something new and better, but so far I'm not seeing it.
- useerup 5y ago
  > Sure ls has a 58 (!) arguments but it's still a tool to list files ultimately. No, it has evolved into a tool to get files/directories of a directory and sort them and format them. Powershell separates those. `ls` in Powershell gets filesystem objects. It does not have any of the nix craziness for controlling the output and sorting, yet it is possible to do the exact same thing by piping output from ls into a format-command like format-table(ft), format-list(fl) or even convertto-json
AtlasBarfed 5y ago
AND YET LS STILL IS NOT RELIABLY PARSEABLE And yet there aren't --json (and/or --yaml) universal output option flags to structure your data in a reliably parseable format. Shouldn't every option for a major UNIX command (insert your definition for major) have an extensive list of useful examples/use cases? Man still fails me here. As does the AWS CLI too as a side complaint. If your option is important and useful, then it demands a good example.
- munk-a 5y ago
  I read in an article (this one - joke) that all lower case letters except `jvyz` are in use - it looks like `-j` and `-y` are just sitting there waiting for someone to add some output formats - then we just need to add a verbose flag and... maybe a switch that automatically gzips the output? We're so close to the 100% completion achievement - we can get there in our lifetimes.
- shatteredgate 5y ago
  It doesn't really make sense to have unix commands output json or yaml. They're already doing a form of parsing internally, you're then asking them to encode the output in another text format so you can parse it again. Why? You'd have better results just using the underlying API to get a list of files, and this is what every other program that needs to get a list of files actually does. From a scripting perspective this is one of the things that powershell actually got right.
  - AtlasBarfed 5y ago
    well, because, ls, lsblk, ps, and a host of other table-based outputs aren't reliably parseable. Their columns overflow. They aren't durable to line breaks in indentifiers and other UTF8 and control character oddities. You know what is? A decent data output format. And JSON is the best of the compromises that exist now. YAML would be nice because if you are outputting json, yaml output is easy and it's a bit more readable. That doesn't violate the UNIX philosophy of text output. It probably improves it. Underlying APIs aren't as universal, portable, or well-known. We are talking basically GNU cross-platform programs.
dang 5y ago
Discussed at the time (of the article): The growth of command-line options, 1979-Present - https://news.ycombinator.com/item?id=22473821 https://news.ycombinator.com/item?id=22473821 - March 2020 (171 comments)
marttt 5y ago
Interesting table. One more column would be great, though: number of flags for all these commands in Plan 9 or 9front.
marcodiego 5y ago
A graphic showing this growth would be super interesting.
908B64B197 5y ago
With Python installed everywhere I often question the logic behind writing long or non-trivial shell scripts. It always seems so brittle since the output of a command line tool can change (unless it's frozen in some standard) and relies way too much on regex and assumptions about what the data returned will look like. I much prefer to keep bash around only for interactive commands and ad-hoc one-liners.
- pjmlp 5y ago
  It is installed, but are you sure it is the version you will need? :)
WalterBright 5y ago
Every command line switch is a bug.
- WalterBright 5y ago
  Far too often, when people cannot agree on how a program should work, the "solution" is to add a switch and so avoid making a decision.
  - z3t4 5y ago
    A better option is to create a common library and then make many programs
  - kelnos 5y ago
    Or the program should legitimately work in both ways, at different times, depending on context and the user's current needs. Sometimes I want `ls -t`, sometimes I want `ls -S` and sometimes I just want plain `ls`. It would be silly to make them three different programs, and if those options didn't exist, that would be foolish. Sure, you could say that `ls -t` should be `ls | sort --time`, and if command output was structured rather than just text, and standardized, that would work and be easy to implement in a cross-tool manner. But now we have `sort --time`, which is also an option that you say shouldn't exist. So now what? `ls | timesort` and `ls | sizesort`? That's just silly. Ok, how about something that doesn't affect how the output is presented, but what output is presented. Most of the time I just want `ls`, but sometimes I want to see dotfiles, so I have `ls -a`. How do we fix that? `ls` and `lshidden`? Again: silly. Do we just say "dotfiles are stupid; we shouldn't have this artificial concept of hidden files"? No, that's silly too. I get that your follow-up post softens the absolutism with "far too often", but your original post did assert that "every" command line option is a bug, and I'm tired of this "options are bad" argument. Overall I would rather have the option to change behavior than not. Take those options away, and we get complaints that the maintainers are dumbing things down and removing choice. It's really a no-win situation: either your software is bloated, or it's not flexible enough. Few people agree on where the happy medium is. Discussions about this are, IMO, just tiring and pointless, and are some of the worst forms of bike-shedding. The only reason a potentially-useful option should be removed (or not added in the first place) is if the option creates a maintenance burden that the maintainer is not comfortable taking on.
kjellsbells 5y ago
Some observations: - Tool options increasing monotonically seems like the very definition of bloat. Probably since the marginal cost of adding an option is very low and the cost of removing an option is very high (angry users). - There's a cognitive cost to bloat. Personally, I know maybe 5-10 options to all the commands in the regular *nix tool set and that's about the limit my brain could hold. - Options are useless if they are not discoverable at lower mental effort than the alternatives. Man/info pages are great, but I'm not wading through multiple pages on the off chance that ls has an option that can output in JSON if it's a Wednesday and my POSIX locale is unset. Discoverability is a serious problem in large systems, and I'm not sure anyone knows what the solution is. I mean, we've had google, Stack Overflow, man X intro, GNU info, ... So what does that mean in practice? Is there a way to square modernity with the UNIX philosophy? I would suggest that 1. Tool authors ruthlessly restrict options to fit in the working set memory of the average human user. Let's say <10 options. If you need more options, you need a new tool. 2. Tools are pluggable, so that the base implementation is simple and known-to-be-present (shades of POSIX here), but that there is a expansion pack mechanism for people who need it. Imagine DLC for ls. You want extra magicka in your find(1)? Make sure you have the plugin and off you go. 3. Tools designers and users are clear about the decision criteria to *not\* use a specific tool. Sure I could do crypto in awk, or parse \0 in ls, but I shouldn't do so. There's no shame in reaching for a better solution, and, over time, solutions will coalesce into new tools. For example, everyone uses jq not awk to parse their JSON...right? Edit: I just returned from my kitchen with a IRL example: my dishwasher has 15 programs but only two buttons show any sign of wear: "quick wash" and "start"...
ape4 5y ago
Gotta love the `clear` command. Started with zero options and still has zero. Does one thing well. Would you want an option to clear half the screen ;)
- unhammer 5y ago
  SYNOPSIS clear [-Ttype] [-V] [-x] I suppose in GNU, <5 is basically zero.
1vuio0pswjnm7 5y ago
"Complaining that memory usage grew by a factor of one thousand when a (portable!) machine that's more than an order of magnitude cheaper has four million times more memory seems a bit ridiculous." Consider a different user from the the author who is designing a system where memory is the filesystem. "Disk" space is memory. The more "disk" a program occupies, the less memory available for everything else. (The same is true for non-memory backed storage space.) For this user, the complaint is not ridiculous at all. BusyBox commands do not have the full range of command line options. NetBSD has versions of common utilties that have reduced options, intended to be used on small-sized and/or memory-backed filesystems. There are many examples. The idea that today's computers have more memory/storage and therefore today's programs should be larger does not make sense if the user intended to use the increased memory/storage for something other than programs. Purchasing computers over a few decades with increases in resources for the same price ("bigger bang for the same buck"), the purchaser might expect an increase in the ratio of (a) memory or storage used by others' programs to (b) free memory or storage she can use for her own things. When the size and memory usage of software not written by the computer purchaser increases because of the increased availablity of storage and memory, then that takes away those resources from use by other things. Who do the resources belong to, who paid for them, and who should have the final say in how they are used. For example, consider programs the computer owner has written or wishes to write or files she wishes to store. Less resources are available for/to those things because someone else thinks that the bigger bang the purchaser is getting for her buck is not entirely the purchaser's. It belongs to anyone who can write programs. Under this theory, the purchaser does not get to decide the quantity of resources other programmers get to use. Other programmers declare they should be unconstrained to use as much resources as they think they need. There is more memory and storage thus they get to use it. Yet those programmers did not pay for that increased storage and memory. The computer purchaser did. How dare a user complain that a program grew by a factor of 1000. The computer has more resources. Therefore anyone can use them. Except the resources do not belong to anyone else besides the person who purchased the computer. Who should get to decide how they are used. McIlroy is just telling it like it is, it is strange that younger people today want to "disagree". Bloat is bloat. More resources does not make bloat cease to exist as a concept. If someone is eating too much food (more than their body requires), then the availability of more food does not change the fact they are eating too much. Not to mention when someone else is paying for it. "There is so much food available. I am only eating a tiny proportion of what is available. Therefore, I am not eating too much." Statements like "We were doing X in the 1970's" when the author was not even born or yet to have fully formed adult mind by the 1970's are peculiar. Is "we" the appropriate word. "They" were different people. IIRC, McIlroy once said something like, "The hero is the negative coder." Bloat has become so pervasive that IMO such coders really are heros, maybe not to other programmers, but to this user for sure. Write enormous, complex programs with 50+ options, have fun, be "productive", enjoy convenience and resource abundance, but trying to "disagree" with McIlroy's (accurate) observations is quite silly. For a user who only uses 20% of the commandline options of a program, the code and resulting size increase correspending to the other 80% of options she never uses is perhaps something to consider, especially when this applies to a large number of programs. The shell can keep track of utility usage so one can see which programs she is using the most and which programs she does not need. Perhaps it would be useful to measure options usage so she can assess what options are used the most and which options she does not need. Ideally unutilised options might be disabled at compile-time.
cmsefton 5y ago
> We used to sit around in the UNIX room saying "what can we throw out? Why is there this option?" This reminded me of Antoine de Saint-Exupéry's quote that "Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away."

186 points by intrepidhero 5y ago

Now there is a page I will glady bookmark for those "UNIX philosophy" arguments.

> Ironically, one of the reasons for the rise in the number of command line options is another McIlroy dictum, "Write programs to handle text streams, because that is a universal interface" (see ls for one example of this). That is unfortunately not the reason. There are 2 main reasons why today (GNU) command line utilities have a lot of options: 1). GNU. Apparrently they like to add features to programs 2). Herritage of SYSV/BSD. Traditionally BSD options were different from SYSV and the new GNU utilities were made to understand both.

avgcorrection 5y ago
I don’t see an argument from you that (1) wasn’t caused by the point that Luu raises. It’s not strange to end up adding features to programs if the effective IPC is so impoverished that you would need to sandwich a bunch of repetitive formatting commands if you didn’t add some convenient built-in formatting options.
fsckboy 5y ago
> 1). GNU. Apparrently they like to add features to programs unix comes out of "worse is better". GNU comes out of The Right Thing.
seoaeu 5y ago
Adding features isn't itself a bad thing. But when you cannot iterate or remove features (even the smallest change to existing functionality could break a script somewhere!) then you end up with the jumbled mess we have.

intrepidhero 5y ago

This got linked in another front page story. Uncomfortable Truths in Software Engineering. > The Unix philosophy of “do one thing well” doesn’t actually work that well. Dan Luu explains this better than I could. Usually The Unix Philosophy gets nothing but praise in these parts so I was interested to see a counterpoint.

ReleaseCandidat 5y ago
The 'Unix Haters Book' has some too: https://web.mit.edu/~simsong/www/ugh.pdf https://web.mit.edu/~simsong/www/ugh.pdf
- marcodiego 5y ago
  The UHHB has very few points that are still valid. It is a humorous piece and was written about 3 decades ago. We shouldn't continue citing it as valid criticism.
  - ReleaseCandidat 5y ago
    > The UHHB has very few points that are still valid. I disagree. Yes, nobody uses sendmail anymore, filesystems vastly improved and X isn't a resource hog any more. I've reread it some months ago and the main points still hold. For example the part about `kill`: Most operating systems have a “kill” command. So does Unix. In most operating systems, the kill command kills a process. The Unix implementation is much more general: the “kill” command sends a process a message. This illustrates the first Unix design principle: • Give the user power by making operations fully general. The kill command is very powerful; it lets you send all sorts of messages to processes. For example, one message you can send to a process tells it to kill itself. This message is -9. -9 is, of course, the largest single-digit message, which illustrates another important Unix design principle: • Choose simple names that reflect function.
    - monroeclinton 5y ago
      I don't find the argument about kill convincing. You can use `kill pid` and it will send a SIGTERM signal to the process. The -9 option is only necessary when you want to send a SIGKILL signal. I don't think it is a bad thing that kill gives users the option to choose which signal to send. I only use the -9 option when I want to terminate a process without it being handled.

pif 5y ago

Nothing more than a load of garbage.

simiones 5y ago
The article, or the growing command line options list?
- munk-a 5y ago
  The comment since it lacks any context and presents nothing but a vague opinion. It's actually a recursive comment because the poster heard that programmers love recursion.

mjw1007 5y ago

A gentle suggestion for bloggers: if you're going to write an article with a title like "The growth of command line options, 1979-Present", please also arrange to prominently display the date of composition.

pugworthy 5y ago
March 2020 for those of you who just had to find out after reading this comment.
- HotHotLava 5y ago
  > The latest year in the table is 2017 because I wrote the first draft for this post in 2017 and didn't get around to cleaning it up until 2020.
ghaff 5y ago
Really, any blog post should have a prominent date on it. For a lot of things, it's essential context. Older posts can be useful/interesting but they can also be pretty much completely useless depending upon the topic.
bo1024 5y ago
Every article (really, every thing) posted on the web should be timestamped.

woodruffw 5y ago

Maybe it's because I'm not the biggest Unix purist, but I don't really mind the proliferation of command line options. The two things that do irk me are: * Using short options in scripts. Scripts should almost always use long options. As a corollary, tools should always provide long options unless the tool isn't intended for scripting. * Lack of discoverability. We still don't have a standard way to ask an arbitrary tool which flags it supports, or for a rough semantic mapping of those flags (e.g. the classic -v/-V verbose and version confusion). Developing a standard here would go a long way towards automating things like tab completion without each tool needing to maintain N scripts for N shells or having its own slightly-different `--completions=SHELL` generator.

jamespwilliams 5y ago
On the latter, an example that comes to mind is how to specify field numbers: -f in cut, -k in sort, -j for the most common option in join. And specifying field delimiters: -d in cut, -t in sort, -F in awk.
- woodruffw 5y ago
  Ach, those always get me. Thankfully most (all?) of those also respect the `$IFS` variable, which is what I tend to use.
  - gnubison 5y ago
    None of them should respect IFS — that’s only for the shell.
    - woodruffw 5y ago
      Huh. I could have sworn that GNU Awk respects IFS, but I can't find that in the manual now. Maybe I dreamed it up.
nauticacom 5y ago
The last point would be amazing, imo. It's unfortunate that, because of history, commands accept and parse arbitrary strings as input instead of formally specifying it like a function signature + docblock. If I could rewrite the universe, commands would use a central API for specifying the names, data types, descriptions, etc. for all the input they take. In our timeline, maybe some file format could be standardized that describes the particular inputs and options a command takes, and e.g. shells would hook into it. Kind of like header files, but distributed either in a community repository or by each of the tools themselves.

munificent 5y ago

Two of the many joys in life are: 1. Having a problem and solving it with a small amount of effort. 2. Having a mental model of some external thing and delighting in seeing that your model is accurate with respect to the actual thing. Elegant, minimal systems excel at providing #2. Big sprawling systems with lots of options and special cases excel at providing #1. Many software engineers seem to be calibrated to extract a lot more joy from experiencing #2 than #1. It's probably something to do with the nature of the work. We work "backwards" by having a mental model of a thing we wish existed and then bring it into being. If we didn't find deep satisfaction in harmony between our mental model and software model, we wouldn't have the drive to keep fixing all of the bugs in our software. But there is nothing instrinsically superior to the joy of #2 compared to #1. It's just a psychological/emotional preference. Many people, if not most, outside of the world of software engineering seem to prefer #1. They just like to get stuff done and make stuff happening. You rarely see woodworkers lamenting about the variety of clamps, jigs, and tools they have. Instead, they delight in finding just the right specific tool to make a job easy. Fishermen love their tackleboxes. Of course, there is an appreciation of #2 in all of us, and you see minimalism everywhere. But I've never seen it raised to such a religious, moral level as I see in software engineering. It's important to remember that that's a psychological choice we're making—a personal preference we impose—and not something fundamental to systems.

wyager 5y ago
> Big sprawling systems with lots of options and special cases excel at providing #1. This is the sales pitch, but in reality you spend so much time configuring and debugging that you rarely actually save any time.
- munificent 5y ago
  > in reality you spend so much time configuring and debugging that you rarely actually save any time. Conversely, that's the sales pitch for minimalism. But the reality is that well-intentioned, thoughtful designers and users seem to end up spending most of their time building and using systems with a lot of complexity. If it really ended up costing them more time to use those systems, it begs the question of why we keep building them.

tra3 5y ago

Interesting article, but I don't see how the increase in number of arguments invalidates the "do one thing well". Sure ls has a 58 (!) arguments but it's still a tool to list files ultimately. > I've heard people say that there isn't really any alternative to this kind of complexity for command line tools, but people who say that have never really tried the alternative, something like PowerShell. I've spent a lot of time passing around JSON and XML, not the same but similar. While it's true that structured data makes things easier, it's not without it's own faults. I subscribe to unix philosophy of "do one thing well" and pass text around because it's served me well. You have to question status quo to come up with something new and better, but so far I'm not seeing it.

useerup 5y ago
> Sure ls has a 58 (!) arguments but it's still a tool to list files ultimately. No, it has evolved into a tool to get files/directories of a directory and sort them and format them. Powershell separates those. `ls` in Powershell gets filesystem objects. It does not have any of the nix craziness for controlling the output and sorting, yet it is possible to do the exact same thing by piping output from ls into a format-command like format-table(ft), format-list(fl) or even convertto-json

AtlasBarfed 5y ago

AND YET LS STILL IS NOT RELIABLY PARSEABLE And yet there aren't --json (and/or --yaml) universal output option flags to structure your data in a reliably parseable format. Shouldn't every option for a major UNIX command (insert your definition for major) have an extensive list of useful examples/use cases? Man still fails me here. As does the AWS CLI too as a side complaint. If your option is important and useful, then it demands a good example.

munk-a 5y ago
I read in an article (this one - joke) that all lower case letters except `jvyz` are in use - it looks like `-j` and `-y` are just sitting there waiting for someone to add some output formats - then we just need to add a verbose flag and... maybe a switch that automatically gzips the output? We're so close to the 100% completion achievement - we can get there in our lifetimes.
shatteredgate 5y ago
It doesn't really make sense to have unix commands output json or yaml. They're already doing a form of parsing internally, you're then asking them to encode the output in another text format so you can parse it again. Why? You'd have better results just using the underlying API to get a list of files, and this is what every other program that needs to get a list of files actually does. From a scripting perspective this is one of the things that powershell actually got right.
- AtlasBarfed 5y ago
  well, because, ls, lsblk, ps, and a host of other table-based outputs aren't reliably parseable. Their columns overflow. They aren't durable to line breaks in indentifiers and other UTF8 and control character oddities. You know what is? A decent data output format. And JSON is the best of the compromises that exist now. YAML would be nice because if you are outputting json, yaml output is easy and it's a bit more readable. That doesn't violate the UNIX philosophy of text output. It probably improves it. Underlying APIs aren't as universal, portable, or well-known. We are talking basically GNU cross-platform programs.

dang 5y ago

Discussed at the time (of the article): The growth of command-line options, 1979-Present - https://news.ycombinator.com/item?id=22473821 https://news.ycombinator.com/item?id=22473821 - March 2020 (171 comments)

marttt 5y ago

Interesting table. One more column would be great, though: number of flags for all these commands in Plan 9 or 9front.

marcodiego 5y ago

A graphic showing this growth would be super interesting.

908B64B197 5y ago

With Python installed everywhere I often question the logic behind writing long or non-trivial shell scripts. It always seems so brittle since the output of a command line tool can change (unless it's frozen in some standard) and relies way too much on regex and assumptions about what the data returned will look like. I much prefer to keep bash around only for interactive commands and ad-hoc one-liners.

pjmlp 5y ago
It is installed, but are you sure it is the version you will need? :)

WalterBright 5y ago

Every command line switch is a bug.

WalterBright 5y ago
Far too often, when people cannot agree on how a program should work, the "solution" is to add a switch and so avoid making a decision.
- z3t4 5y ago
  A better option is to create a common library and then make many programs
- kelnos 5y ago
  Or the program should legitimately work in both ways, at different times, depending on context and the user's current needs. Sometimes I want `ls -t`, sometimes I want `ls -S` and sometimes I just want plain `ls`. It would be silly to make them three different programs, and if those options didn't exist, that would be foolish. Sure, you could say that `ls -t` should be `ls | sort --time`, and if command output was structured rather than just text, and standardized, that would work and be easy to implement in a cross-tool manner. But now we have `sort --time`, which is also an option that you say shouldn't exist. So now what? `ls | timesort` and `ls | sizesort`? That's just silly. Ok, how about something that doesn't affect how the output is presented, but what output is presented. Most of the time I just want `ls`, but sometimes I want to see dotfiles, so I have `ls -a`. How do we fix that? `ls` and `lshidden`? Again: silly. Do we just say "dotfiles are stupid; we shouldn't have this artificial concept of hidden files"? No, that's silly too. I get that your follow-up post softens the absolutism with "far too often", but your original post did assert that "every" command line option is a bug, and I'm tired of this "options are bad" argument. Overall I would rather have the option to change behavior than not. Take those options away, and we get complaints that the maintainers are dumbing things down and removing choice. It's really a no-win situation: either your software is bloated, or it's not flexible enough. Few people agree on where the happy medium is. Discussions about this are, IMO, just tiring and pointless, and are some of the worst forms of bike-shedding. The only reason a potentially-useful option should be removed (or not added in the first place) is if the option creates a maintenance burden that the maintainer is not comfortable taking on.

kjellsbells 5y ago

Some observations: - Tool options increasing monotonically seems like the very definition of bloat. Probably since the marginal cost of adding an option is very low and the cost of removing an option is very high (angry users). - There's a cognitive cost to bloat. Personally, I know maybe 5-10 options to all the commands in the regular *nix tool set and that's about the limit my brain could hold. - Options are useless if they are not discoverable at lower mental effort than the alternatives. Man/info pages are great, but I'm not wading through multiple pages on the off chance that ls has an option that can output in JSON if it's a Wednesday and my POSIX locale is unset. Discoverability is a serious problem in large systems, and I'm not sure anyone knows what the solution is. I mean, we've had google, Stack Overflow, man X intro, GNU info, ... So what does that mean in practice? Is there a way to square modernity with the UNIX philosophy? I would suggest that 1. Tool authors ruthlessly restrict options to fit in the working set memory of the average human user. Let's say <10 options. If you need more options, you need a new tool. 2. Tools are pluggable, so that the base implementation is simple and known-to-be-present (shades of POSIX here), but that there is a expansion pack mechanism for people who need it. Imagine DLC for ls. You want extra magicka in your find(1)? Make sure you have the plugin and off you go. 3. Tools designers and users are clear about the decision criteria to *not\* use a specific tool. Sure I could do crypto in awk, or parse \0 in ls, but I shouldn't do so. There's no shame in reaching for a better solution, and, over time, solutions will coalesce into new tools. For example, everyone uses jq not awk to parse their JSON...right? Edit: I just returned from my kitchen with a IRL example: my dishwasher has 15 programs but only two buttons show any sign of wear: "quick wash" and "start"...

ape4 5y ago

Gotta love the `clear` command. Started with zero options and still has zero. Does one thing well. Would you want an option to clear half the screen ;)

unhammer 5y ago
SYNOPSIS clear [-Ttype] [-V] [-x] I suppose in GNU, <5 is basically zero.

1vuio0pswjnm7 5y ago

"Complaining that memory usage grew by a factor of one thousand when a (portable!) machine that's more than an order of magnitude cheaper has four million times more memory seems a bit ridiculous." Consider a different user from the the author who is designing a system where memory is the filesystem. "Disk" space is memory. The more "disk" a program occupies, the less memory available for everything else. (The same is true for non-memory backed storage space.) For this user, the complaint is not ridiculous at all. BusyBox commands do not have the full range of command line options. NetBSD has versions of common utilties that have reduced options, intended to be used on small-sized and/or memory-backed filesystems. There are many examples. The idea that today's computers have more memory/storage and therefore today's programs should be larger does not make sense if the user intended to use the increased memory/storage for something other than programs. Purchasing computers over a few decades with increases in resources for the same price ("bigger bang for the same buck"), the purchaser might expect an increase in the ratio of (a) memory or storage used by others' programs to (b) free memory or storage she can use for her own things. When the size and memory usage of software not written by the computer purchaser increases because of the increased availablity of storage and memory, then that takes away those resources from use by other things. Who do the resources belong to, who paid for them, and who should have the final say in how they are used. For example, consider programs the computer owner has written or wishes to write or files she wishes to store. Less resources are available for/to those things because someone else thinks that the bigger bang the purchaser is getting for her buck is not entirely the purchaser's. It belongs to anyone who can write programs. Under this theory, the purchaser does not get to decide the quantity of resources other programmers get to use. Other programmers declare they should be unconstrained to use as much resources as they think they need. There is more memory and storage thus they get to use it. Yet those programmers did not pay for that increased storage and memory. The computer purchaser did. How dare a user complain that a program grew by a factor of 1000. The computer has more resources. Therefore anyone can use them. Except the resources do not belong to anyone else besides the person who purchased the computer. Who should get to decide how they are used. McIlroy is just telling it like it is, it is strange that younger people today want to "disagree". Bloat is bloat. More resources does not make bloat cease to exist as a concept. If someone is eating too much food (more than their body requires), then the availability of more food does not change the fact they are eating too much. Not to mention when someone else is paying for it. "There is so much food available. I am only eating a tiny proportion of what is available. Therefore, I am not eating too much." Statements like "We were doing X in the 1970's" when the author was not even born or yet to have fully formed adult mind by the 1970's are peculiar. Is "we" the appropriate word. "They" were different people. IIRC, McIlroy once said something like, "The hero is the negative coder." Bloat has become so pervasive that IMO such coders really are heros, maybe not to other programmers, but to this user for sure. Write enormous, complex programs with 50+ options, have fun, be "productive", enjoy convenience and resource abundance, but trying to "disagree" with McIlroy's (accurate) observations is quite silly. For a user who only uses 20% of the commandline options of a program, the code and resulting size increase correspending to the other 80% of options she never uses is perhaps something to consider, especially when this applies to a large number of programs. The shell can keep track of utility usage so one can see which programs she is using the most and which programs she does not need. Perhaps it would be useful to measure options usage so she can assess what options are used the most and which options she does not need. Ideally unutilised options might be disabled at compile-time.

cmsefton 5y ago

> We used to sit around in the UNIX room saying "what can we throw out? Why is there this option?" This reminded me of Antoine de Saint-Exupéry's quote that "Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away."