5 ms·

The famous "each tool does just one thing well" mantra is also a depressingly overblown myth. Among the current Unix tools, there is a ridiculous amount of over

The famous "each tool does just one thing well" mantra is also a depressingly overblown myth. Among the current Unix tools, there is a ridiculous amount of overlap. That's why we have both "ls" and "find", even though they do the same thing conceptually. "ps" has column output, but the way it formats, sorts, selects etc. columns is reinvented and not transferable to other tools such as "lsof" and "netstat", not to mention "top", which of course is just a self-updating "ps". Every tool invents filtering, sorting, formatting etc. in their own unique way. The various flags are not translatable to other tools. I've never used Powershell, but it seems to get one thing right that Unix never bothered to. Tools should emit data, or consume data, or display data, but tools should never do all three, because they will invariably reimplement something poorly that has been done elsewhere.

int_19h 10y ago
Powershell did one thing wrong, in that its tools emit objects, not data. In other words, they emit data with behavior (methods). This, in turn, imposes the CLR object and memory model on the whole thing, and makes the pipeline impossible to use between distinct processes. The right way to do this is to pick some reasonable text-based structured interchange format - s-exprs, JSON, whatever. Actually, it wouldn't be a bad thing to have a binary protocol as well, so long as all that complexity is negotiable, and is implemented by the standard library.
- lobster_johnson 10y ago
  Agreed. One huge benefit to a text format is that you can generate small "ad hoc" snippets (e.g. echo) easily. Otherwise I'd love something like Protocol Buffers here.
- EdiX 10y ago
  The problem here is that most unix tools (ls being the exception) do double duty, they both emit output for other programs and output for the user. Which one you get and how the user output is presented being controlled by flags. If you want them to only emit data in a standardized format you are implicitly cutting out the user. Since direct use by a human is arguably the main use case, output formatting needs to be reindtroduced somehow. You could make some adapter that you always pipe into that handles most formatting concerns but it's unlikely that it will be truly general. The only general way to do that is for the tool to output, along with the data, also some code that can be used to properly format the data. So there are only three ways to solve this: * Unix solution, with tools doing double duty and each tool having its own incompatible set of flags * Actual programming language solution, tools emit pure data. Great for programming, bad for interactive use. * Powershell solution, programs emit data+code: you are locked into some VM.
  - skissane 10y ago
    Another option is to check what stdout is hooked up to – if it is hooked up to a terminal output something human readable, if it is hooked up to a pipe output something machine readable like JSON. (Actually, a number of utilities do this terminal-vs-nonterminal check already – e.g. GNU ls with --color=auto). One problem I've seen with this approach in practice, is sometimes you want the human readable output to go to a pipe. For example, when you use jq, it does syntax highlighting of JSON when talking to a terminal, and omits it when talking to a pipe or a file, but then you want to use a pager which understands ANSI escape sequences (e.g. less -R), so now you have to pass an option to tell the command to output ANSI escape sequences even though it is talking to a pipe not a terminal (e.g. jq -C). I wish there was a standard mechanism in Unix for the two ends of a pipe to negotiate about what data format goes down the pipe – MIME type, character set, etc. Then if I was piping jq to less, jq could ask less "do you understand ANSI color escapes?" And less could reply "yes I do please send them". Then I wouldn't need to remember to pass -C to jq and -R to less. (Maybe this could be implemented as ioctls at each end, by which the sender transmits a list of MIME types it supports, and the receiver replies by choosing one of them...)
    - fragmede 10y ago
      Less supports setting options in the LESS environment variable, so if you set LESS='-R' in your .bashrc/whatever, you don't have to pass -R to less. (If you do `jq foo | LESS='' less` for the rare times you need less to not behave that way.) Sadly, it doesn't seem like jq will read an environment setting to always specific -C
  - TeMPOraL 10y ago
    > most unix tools (ls being the exception) do double duty, they both emit output for other programs and output for the user. Which one you get and how the user output is presented being controlled by flags. Which is where they fundamentally break the principle of "doing one thing, and doing it well". Formatting output for user should be a separate step (even if implicitly handled by your shell).
    - 10y ago
- tonyedgecombe 10y ago
  You would have to be prepared for a flood of data then, if I type get-printer I get a handful of useful properties, if I add ConvertTo-Json then I get 385 lines.
dredmorbius 10y ago
"Each tool does just one thing well" doesn't mean "just one tool does one thing". In fact, ETDJOTW means that interoperable alternative tools -- loose coupling -- is more readily supported. So that, say, you can transparently migrate from unsecured rsh to secured ssh. Actually: you can even gain capabilities, since rsh is (misfeature) limited to the 512 low-numbered ports, and hence, a maximum of 512 outbound connections. SSH has no such limitation, and can scale far higher as a management tool. (That bug, and option, was what finally convinced one old-time shop to swap in SSH for RSH for such tasks, after I identified the problem and the solution.) A better example would be "one tools" that do many things. Rob Pike suggested Perl, though IMO that's kind of foolish: Perl is Yet Another Programming Language, and programming languages, rather by design, tend to do multiple things (not always, or not always easily: see sed as an example of both caveats -- text replacement or branching code). Other examples are generally nexus points of complex interactions. Editors (most especially Emacs). Email management (communications protocols, format protocols, ties to DNS, anti-spam, filtering, remote-access (POP and IMAP), extensions, authentication (at various levels), encryption (at various levels). The problem domain is complex, so the tools are complex. The tool you're very likely using right now to read this would be another case in point: the Web browser. Not just for CERN any more.
- lobster_johnson 10y ago
  That's one interpretation, but it's not really what people mean when they refer to "one tool". The progenitor of the expression is Doug McIlroy: This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface. Interoperability is part of DOTADIW idea, but it flows from the "one thing": If each tool does one thing, you can compose them using pipes. Of course, interoperability also implies tools can be swapped out, though as with DOTADIW, it's more of a myth than reality, because reality isn't that simple.
  - dredmorbius 10y ago
    I didn't say that interoperability was the goal, but that it's the effect. See Steve McConnell, Code Complete, for a treatment of modular design in software and the flexibility it provides. (The same logic applies well outside software.)
geezerjay 10y ago
> That's why we have both "ls" and "find" I don't get what you're trying to say. "ls" lists files in a specific path while "find" searches for objects in a file system that match a specific query. Are you sure you're not confused and somehow assuming that "does just one thing well" should mean "another tool must not accomplish anything that might be remotely achievable by misusing another tool"?
- lobster_johnson 10y ago
  Both "ls" and "find" "list files in a specific path" and "search for objects that match a specific query". There's total overlap. The difference is that "ls" is a presentation-oriented tool, whereas "find" is geared towards pipelines.
  - geezerjay 10y ago
    > Both "ls" and "find" "list files in a specific path" Only if you use "find" with a query that means "get me all files matching any and all criteria as long as they are in this specific path". That's a corner case, and a very silly one. > There's total overlap. A very silly and far-fetched one.

int_19h 10y ago

Powershell did one thing wrong, in that its tools emit objects, not data. In other words, they emit data with behavior (methods). This, in turn, imposes the CLR object and memory model on the whole thing, and makes the pipeline impossible to use between distinct processes. The right way to do this is to pick some reasonable text-based structured interchange format - s-exprs, JSON, whatever. Actually, it wouldn't be a bad thing to have a binary protocol as well, so long as all that complexity is negotiable, and is implemented by the standard library.

lobster_johnson 10y ago
Agreed. One huge benefit to a text format is that you can generate small "ad hoc" snippets (e.g. echo) easily. Otherwise I'd love something like Protocol Buffers here.
EdiX 10y ago
The problem here is that most unix tools (ls being the exception) do double duty, they both emit output for other programs and output for the user. Which one you get and how the user output is presented being controlled by flags. If you want them to only emit data in a standardized format you are implicitly cutting out the user. Since direct use by a human is arguably the main use case, output formatting needs to be reindtroduced somehow. You could make some adapter that you always pipe into that handles most formatting concerns but it's unlikely that it will be truly general. The only general way to do that is for the tool to output, along with the data, also some code that can be used to properly format the data. So there are only three ways to solve this: * Unix solution, with tools doing double duty and each tool having its own incompatible set of flags * Actual programming language solution, tools emit pure data. Great for programming, bad for interactive use. * Powershell solution, programs emit data+code: you are locked into some VM.
- skissane 10y ago
  Another option is to check what stdout is hooked up to – if it is hooked up to a terminal output something human readable, if it is hooked up to a pipe output something machine readable like JSON. (Actually, a number of utilities do this terminal-vs-nonterminal check already – e.g. GNU ls with --color=auto). One problem I've seen with this approach in practice, is sometimes you want the human readable output to go to a pipe. For example, when you use jq, it does syntax highlighting of JSON when talking to a terminal, and omits it when talking to a pipe or a file, but then you want to use a pager which understands ANSI escape sequences (e.g. less -R), so now you have to pass an option to tell the command to output ANSI escape sequences even though it is talking to a pipe not a terminal (e.g. jq -C). I wish there was a standard mechanism in Unix for the two ends of a pipe to negotiate about what data format goes down the pipe – MIME type, character set, etc. Then if I was piping jq to less, jq could ask less "do you understand ANSI color escapes?" And less could reply "yes I do please send them". Then I wouldn't need to remember to pass -C to jq and -R to less. (Maybe this could be implemented as ioctls at each end, by which the sender transmits a list of MIME types it supports, and the receiver replies by choosing one of them...)
  - fragmede 10y ago
    Less supports setting options in the LESS environment variable, so if you set LESS='-R' in your .bashrc/whatever, you don't have to pass -R to less. (If you do `jq foo | LESS='' less` for the rare times you need less to not behave that way.) Sadly, it doesn't seem like jq will read an environment setting to always specific -C
- TeMPOraL 10y ago
  > most unix tools (ls being the exception) do double duty, they both emit output for other programs and output for the user. Which one you get and how the user output is presented being controlled by flags. Which is where they fundamentally break the principle of "doing one thing, and doing it well". Formatting output for user should be a separate step (even if implicitly handled by your shell).
  - 10y ago
tonyedgecombe 10y ago
You would have to be prepared for a flood of data then, if I type get-printer I get a handful of useful properties, if I add ConvertTo-Json then I get 385 lines.

dredmorbius 10y ago

"Each tool does just one thing well" doesn't mean "just one tool does one thing". In fact, ETDJOTW means that interoperable alternative tools -- loose coupling -- is more readily supported. So that, say, you can transparently migrate from unsecured rsh to secured ssh. Actually: you can even gain capabilities, since rsh is (misfeature) limited to the 512 low-numbered ports, and hence, a maximum of 512 outbound connections. SSH has no such limitation, and can scale far higher as a management tool. (That bug, and option, was what finally convinced one old-time shop to swap in SSH for RSH for such tasks, after I identified the problem and the solution.) A better example would be "one tools" that do many things. Rob Pike suggested Perl, though IMO that's kind of foolish: Perl is Yet Another Programming Language, and programming languages, rather by design, tend to do multiple things (not always, or not always easily: see sed as an example of both caveats -- text replacement or branching code). Other examples are generally nexus points of complex interactions. Editors (most especially Emacs). Email management (communications protocols, format protocols, ties to DNS, anti-spam, filtering, remote-access (POP and IMAP), extensions, authentication (at various levels), encryption (at various levels). The problem domain is complex, so the tools are complex. The tool you're very likely using right now to read this would be another case in point: the Web browser. Not just for CERN any more.

lobster_johnson 10y ago
That's one interpretation, but it's not really what people mean when they refer to "one tool". The progenitor of the expression is Doug McIlroy: This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface. Interoperability is part of DOTADIW idea, but it flows from the "one thing": If each tool does one thing, you can compose them using pipes. Of course, interoperability also implies tools can be swapped out, though as with DOTADIW, it's more of a myth than reality, because reality isn't that simple.
- dredmorbius 10y ago
  I didn't say that interoperability was the goal, but that it's the effect. See Steve McConnell, Code Complete, for a treatment of modular design in software and the flexibility it provides. (The same logic applies well outside software.)

geezerjay 10y ago

> That's why we have both "ls" and "find" I don't get what you're trying to say. "ls" lists files in a specific path while "find" searches for objects in a file system that match a specific query. Are you sure you're not confused and somehow assuming that "does just one thing well" should mean "another tool must not accomplish anything that might be remotely achievable by misusing another tool"?

lobster_johnson 10y ago
Both "ls" and "find" "list files in a specific path" and "search for objects that match a specific query". There's total overlap. The difference is that "ls" is a presentation-oriented tool, whereas "find" is geared towards pipelines.
- geezerjay 10y ago
  > Both "ls" and "find" "list files in a specific path" Only if you use "find" with a query that means "get me all files matching any and all criteria as long as they are in this specific path". That's a corner case, and a very silly one. > There's total overlap. A very silly and far-fetched one.