8 ms·

Let's just be careful what we call Unix though. In SVR4, I don't think "ls" had that many options. It didn't even have colour (not that I would know; my Wyse 60

Let's just be careful what we call Unix though. In SVR4, I don't think "ls" had that many options. It didn't even have colour (not that I would know; my Wyse 60 could only have Green, Orange or White text... and the colour was set in the factory). It's the same for many programs that started small but grew loads of knobs with age. You can't claim that this was the Unix vision, because what we call Unix today isn't even technically Unix. I'm not saying Linux is bad or anything; I'm just pointing out that it's much more organic than Unix ever was, and that the resulting system is not necessarily carefully orchestrated.

noelwelsh 3y ago
IMO the problem with the Unix model is not that it's been corrupted over time, but that it was fundamentally flawed from the start. I'm hardly the first to point this out, but sending around unstructured text makes doing some things inordinately hard (imagine writing a separate program to color ls's output, instead of having that as a flag) + going beyond pipes leads to the awful experience of programming in shell.
- benj111 3y ago
  Ls seems like a bad example. Anything that you'd want to colour, the data should already be there. Further I'm not sure the Unix way inherently requires completely unstructured text
  - rcxdude 3y ago
    Unstructured text is the unix way. You can have pipelines sending other things between your programs (see powershell), but that's definitely not Unix.
    - rakoo 3y ago
      I disagree, the unix way has structure: lines of text each consisting of space-separated fields. The real issue is that sometimes your data has spaces, or even newlines, so the structure quickly isn't enough. And that "sometimes" happens xay more often than it has to, and the escape hatch is a mess.
      - masfuerte 3y ago
        > lines of text each consisting of space-separated fields I wish! Back in the day, tools like iostat and vmstat printed tabular data using fixed width fields with no spaces between. This worked fine until the computers got bigger and faster, then the larger numbers filled the fields so they ran together making the output unreadable. A surprising number of vendors had this problem.
      - BirAdam 3y ago
        But, you can always override IFS, so this isn’t as much of an issue.
      - bobbylarrybobby 3y ago
        It's bizarre that this is the case given that ascii contains characters for this very purpose. There's record separator, group separator... Now if your data contains those, that's bad. But that's much rarer than containing ordinary whitespace.
    - Kinrany 3y ago
      I disagree: text is the minimum structure. It's often enough sufficient too, but using something higher level like JSON is also perfectly UNIX.
      - rcxdude 3y ago
        I don't think so, this idea of JSON is a more modern interpretation and I'd say is counter to the UNIX philosophy, not an extension to it (it just so happens to squeeze into existing primitives a little better than e.g. a binary serialisation protocol).
    - bregma 3y ago
      The Unix way is streams of bytes. How those bytes are interpreted is up to the endpoints of the streams. Sure, a lot of endpoints interpret byte streams as text. That's not Unix, that's a particular endpoint. You will notice that Unix does not even offer 'text' files (like Windows does). Or ISAM files, or random access files. or Fortran carriage control. Just binary files. Because the Unix way is streams of bytes, and nothing more.
    - benj111 3y ago
      I disagree, unstructured text is itself fuzzy. CSV, line delimited, is that unstructured? Further if you wanted to go with JSON or something you can, there's nothing preventing it. Pipes are based on unstructured text so you have that flexibility. If you want to add JSON ls and filter that through JSON sed to JSON less you can. Finally 'the Unix way' is a cultural thing. It isn't set in stone. It can, and probably has evolved. So while you could argue that it meant that to some people at some time, that doesn't make it true today or in the future. The down side is that we have these disagreements over what 'the Unix way' is. But any philosophy that's been around for 40 years, in tech no less, has to be flexible, that's why it's stay relevant.
- AnimalMuppet 3y ago
  The Unix command line tools were designed to be usable in a lot of ad hoc situations. In an ad hoc situation, unstructured text is often what you have. So, to be usable for that, the command line tools had to be able to operate on unstructured text as input. That left two options - either also accept something structured as input, or accept only unstructured text as input. "Usable in an ad hoc situation" gets us back to composability. If you can compose existing parts to meet a novel situation, that's better than having to carefully architect a solution. And the key to being able to do that is common interfaces. Unix did exactly that. And, for all the criticism, it works pretty well. Can a well-designed interface do better? Sure. Can a well-designed interface deal with something outside the design parameters as well? Probably not.
  - grumblingdev 3y ago
    I mean people still use `ls` everyday so it in fact did win. We have super rich GUIs, REPLs, the start menu, live programming environments and notebooks, web browser file listing capability, the ability to build your own web app to list files visually anyway you want. And yet everyone still uses `ls`.
    - tmtvl 3y ago
      I think that most people who need ls's functionality (listing files in a directory) use a GUI application.
      - AnimalMuppet 3y ago
        Depends on how you define "functionality". You could say that ls's functionality is "providing a text listing of files in a directory. You can't get that from a GUI, certainly not without extra steps. Why would anyone care about that difference? If I'm reading the listing, I don't care. My eyeballs don't care if the photons came from a GUI or text in a shell. But if I'm trying to pipe the results to another program, then I care. GUIs don't pipe well. And people like me pipe the results of ls all the time.
tsimionescu 3y ago
It's still a failing of the Unix philosophy that people found it easier to add all these flags to ls than to string together a few "simple&composable" commands.
- jcelerier 3y ago
  It's for the very simple reason that flags are more easily discoverable (with simply a couple of tab key presses I can iterate and see the list of most flags of most of the cli apps I use) than "whichever program amongst the 7362 executables with obscure names I apparently have in /usr/bin will do the thing I want"
  - grumblingdev 3y ago
    I guess they could have implemented some of the flags as aliases or rather bash functions to a preset chain of invocations. The merging of flags would be difficult though. There are many flags that change the behavior of the file system calls that would be made...such as getting additional information. To support all the formatting commands, the piped output of `ls` would need to be very verbose and structured too...like json or something. They had to draw boundaries of functionality at some point. So the unix idea of "one thing" was probably defined a bit more vaguely or more end-user use case related. Convenience seemed to be king. It would have be nicer if they could have added a shell-like pipe syntax to C (and designed the code in a more modular and stream-like way). In the `ls` [source code][1] you can see simple blocks of steps like: if (!file_ignored(file)) { ... sort_files (); print_current_files (); clear_files (); } For example the `file_ignored` function in `ls` would have made a nice reusable library (with standardized configuration params) and a cli tool (with standardized flags). Missed opportunity to unify the shell and C, but I guess SmallTalk was kicking around at that time which went a whole lot further. It's funny that we are still very much lacking this unification... We don't have a high-level interpreted language that can also perform well as a system's language. New systems languages are popping up all the time, but they are more and more hardcore (looking at you Rust). They don't embrace any aspect of scriptability. The best effort I can think of is https://www.modular.com/ https://www.modular.com/. Surely though, a native TypeScript-based language is the way. It by far has the nicest syntax out of the popular scripting languages today. We have come so close...like with Dart...but still everyone seems to be avoiding the inevitability. [1]: https://github.com/wertarbyte/coreutils/blob/master/src/ls.c https://github.com/wertarbyte/coreutils/blob/master/src/ls.c
eesmith 3y ago
FWIW, ls in Research-V6 back in 1975 had 10 options. https://github.com/dspinellis/unix-history-repo/blob/Research-V6-Snapshot-Development/usr/source/s1/ls.c https://github.com/dspinellis/unix-history-repo/blob/Researc... By BSD 3 in 1980 it had 11 options. https://github.com/dspinellis/unix-history-repo/blob/BSD-3-Snapshot-Development/usr/src/cmd/ls/ls.c https://github.com/dspinellis/unix-history-repo/blob/BSD-3-S... The thing is, we can see even from the 1970s 'ls' how the Unix model doesn't meet the goal "to chain these simple programs together to create complex behaviors". There is no option to escape or NUL terminate a filename, making it possible to construct a filename containing a newline which makes the output look like two file entries. The option for that was added later. There's also the issue that embedded terminal codes will be interpreted by the terminal.

noelwelsh 3y ago

IMO the problem with the Unix model is not that it's been corrupted over time, but that it was fundamentally flawed from the start. I'm hardly the first to point this out, but sending around unstructured text makes doing some things inordinately hard (imagine writing a separate program to color ls's output, instead of having that as a flag) + going beyond pipes leads to the awful experience of programming in shell.

benj111 3y ago
Ls seems like a bad example. Anything that you'd want to colour, the data should already be there. Further I'm not sure the Unix way inherently requires completely unstructured text
- rcxdude 3y ago
  Unstructured text is the unix way. You can have pipelines sending other things between your programs (see powershell), but that's definitely not Unix.
  - rakoo 3y ago
    I disagree, the unix way has structure: lines of text each consisting of space-separated fields. The real issue is that sometimes your data has spaces, or even newlines, so the structure quickly isn't enough. And that "sometimes" happens xay more often than it has to, and the escape hatch is a mess.
    - masfuerte 3y ago
      > lines of text each consisting of space-separated fields I wish! Back in the day, tools like iostat and vmstat printed tabular data using fixed width fields with no spaces between. This worked fine until the computers got bigger and faster, then the larger numbers filled the fields so they ran together making the output unreadable. A surprising number of vendors had this problem.
    - BirAdam 3y ago
      But, you can always override IFS, so this isn’t as much of an issue.
    - bobbylarrybobby 3y ago
      It's bizarre that this is the case given that ascii contains characters for this very purpose. There's record separator, group separator... Now if your data contains those, that's bad. But that's much rarer than containing ordinary whitespace.
  - Kinrany 3y ago
    I disagree: text is the minimum structure. It's often enough sufficient too, but using something higher level like JSON is also perfectly UNIX.
    - rcxdude 3y ago
      I don't think so, this idea of JSON is a more modern interpretation and I'd say is counter to the UNIX philosophy, not an extension to it (it just so happens to squeeze into existing primitives a little better than e.g. a binary serialisation protocol).
  - bregma 3y ago
    The Unix way is streams of bytes. How those bytes are interpreted is up to the endpoints of the streams. Sure, a lot of endpoints interpret byte streams as text. That's not Unix, that's a particular endpoint. You will notice that Unix does not even offer 'text' files (like Windows does). Or ISAM files, or random access files. or Fortran carriage control. Just binary files. Because the Unix way is streams of bytes, and nothing more.
  - benj111 3y ago
    I disagree, unstructured text is itself fuzzy. CSV, line delimited, is that unstructured? Further if you wanted to go with JSON or something you can, there's nothing preventing it. Pipes are based on unstructured text so you have that flexibility. If you want to add JSON ls and filter that through JSON sed to JSON less you can. Finally 'the Unix way' is a cultural thing. It isn't set in stone. It can, and probably has evolved. So while you could argue that it meant that to some people at some time, that doesn't make it true today or in the future. The down side is that we have these disagreements over what 'the Unix way' is. But any philosophy that's been around for 40 years, in tech no less, has to be flexible, that's why it's stay relevant.
AnimalMuppet 3y ago
The Unix command line tools were designed to be usable in a lot of ad hoc situations. In an ad hoc situation, unstructured text is often what you have. So, to be usable for that, the command line tools had to be able to operate on unstructured text as input. That left two options - either also accept something structured as input, or accept only unstructured text as input. "Usable in an ad hoc situation" gets us back to composability. If you can compose existing parts to meet a novel situation, that's better than having to carefully architect a solution. And the key to being able to do that is common interfaces. Unix did exactly that. And, for all the criticism, it works pretty well. Can a well-designed interface do better? Sure. Can a well-designed interface deal with something outside the design parameters as well? Probably not.
- grumblingdev 3y ago
  I mean people still use `ls` everyday so it in fact did win. We have super rich GUIs, REPLs, the start menu, live programming environments and notebooks, web browser file listing capability, the ability to build your own web app to list files visually anyway you want. And yet everyone still uses `ls`.
  - tmtvl 3y ago
    I think that most people who need ls's functionality (listing files in a directory) use a GUI application.
    - AnimalMuppet 3y ago
      Depends on how you define "functionality". You could say that ls's functionality is "providing a text listing of files in a directory. You can't get that from a GUI, certainly not without extra steps. Why would anyone care about that difference? If I'm reading the listing, I don't care. My eyeballs don't care if the photons came from a GUI or text in a shell. But if I'm trying to pipe the results to another program, then I care. GUIs don't pipe well. And people like me pipe the results of ls all the time.

tsimionescu 3y ago

It's still a failing of the Unix philosophy that people found it easier to add all these flags to ls than to string together a few "simple&composable" commands.

jcelerier 3y ago
It's for the very simple reason that flags are more easily discoverable (with simply a couple of tab key presses I can iterate and see the list of most flags of most of the cli apps I use) than "whichever program amongst the 7362 executables with obscure names I apparently have in /usr/bin will do the thing I want"
- grumblingdev 3y ago
  I guess they could have implemented some of the flags as aliases or rather bash functions to a preset chain of invocations. The merging of flags would be difficult though. There are many flags that change the behavior of the file system calls that would be made...such as getting additional information. To support all the formatting commands, the piped output of `ls` would need to be very verbose and structured too...like json or something. They had to draw boundaries of functionality at some point. So the unix idea of "one thing" was probably defined a bit more vaguely or more end-user use case related. Convenience seemed to be king. It would have be nicer if they could have added a shell-like pipe syntax to C (and designed the code in a more modular and stream-like way). In the `ls` [source code][1] you can see simple blocks of steps like: if (!file_ignored(file)) { ... sort_files (); print_current_files (); clear_files (); } For example the `file_ignored` function in `ls` would have made a nice reusable library (with standardized configuration params) and a cli tool (with standardized flags). Missed opportunity to unify the shell and C, but I guess SmallTalk was kicking around at that time which went a whole lot further. It's funny that we are still very much lacking this unification... We don't have a high-level interpreted language that can also perform well as a system's language. New systems languages are popping up all the time, but they are more and more hardcore (looking at you Rust). They don't embrace any aspect of scriptability. The best effort I can think of is https://www.modular.com/ https://www.modular.com/. Surely though, a native TypeScript-based language is the way. It by far has the nicest syntax out of the popular scripting languages today. We have come so close...like with Dart...but still everyone seems to be avoiding the inevitability. [1]: https://github.com/wertarbyte/coreutils/blob/master/src/ls.c https://github.com/wertarbyte/coreutils/blob/master/src/ls.c

eesmith 3y ago

FWIW, ls in Research-V6 back in 1975 had 10 options. https://github.com/dspinellis/unix-history-repo/blob/Research-V6-Snapshot-Development/usr/source/s1/ls.c https://github.com/dspinellis/unix-history-repo/blob/Researc... By BSD 3 in 1980 it had 11 options. https://github.com/dspinellis/unix-history-repo/blob/BSD-3-Snapshot-Development/usr/src/cmd/ls/ls.c https://github.com/dspinellis/unix-history-repo/blob/BSD-3-S... The thing is, we can see even from the 1970s 'ls' how the Unix model doesn't meet the goal "to chain these simple programs together to create complex behaviors". There is no option to escape or NUL terminate a filename, making it possible to construct a filename containing a newline which makes the output look like two file entries. The option for that was added later. There's also the issue that embedded terminal codes will be interpreted by the terminal.