13 ms·

Unix is not an acceptable Unix

• useerup 11y ago
  Interesting examples, especially in light of the recent debates about Unix shells and PowerShell. In PowerShell, the `ls` command is a simple command that produces output without flags to control output format, ordering or filtering. Like with higher order functional programming, those tasks are delegated to cmdlets like Format-List, Format-Table, Select-Object, Where-Object. Some PowerShell examples: To list the files and directories (children) of a directory: ls To get just the names or the full path: ls | select Name ls | select FullName To discover which properties are available from `ls`: ls | gm To select only files larger than 200MB: ls | ? Length -gt 200MB (ls is alias for Get-ChildItem, gm is alias for Get-Member, ? is alias for Where-Object) It is somewhat ironic that PowerShell is more true to the principle of "do one thing and do it well" than the Unix shells.
  • MrBuddyCasino 11y ago
    Exactly. I think the Unix philosophy of "everything is a file" has an impedance mismatch with "streams of text". Files and directories are hierarchical lists of objects with certain properties, similar to JSON. The "directories of files" abstraction is versatile and useful, simply because it is a versatile and simple data structure. Streams of text are too limited.
  • vbezhenar 11y ago
    Another UNIX principle is that programs deal with text. PowerShell is huge violation of that principle. You can't easily write PowerShell command with C or Rust or Java, as far as I understand. Yet you can write command-line tool with any language and use it with combination of hundreds existing tools.
    • wz1000 11y ago
      A solution to this might be a standardized object format, like JSON.
      • vbezhenar 11y ago
        JSON doesn't support behaviour (methods). They are very useful for composing programs IMO.
        • wz1000 11y ago
          Behaviour could be implemented by shell commands. That way, if you wanted to implement a method for a particular object, you could just write a binary/shell script/whatever that reads the object from stdin and writes its result to stdout
        • masklinn 11y ago
          Methods are useful to organise and abstract programs, they're really quite bad at composition. Many functions operating on the same primitives is much more composable than small sets of functions operating on their own custom primitives.
        • sudioStudio64 11y ago
          if the deserialization mapped to a system wide type system then the methods could be mapped in at that point.
        • current_call 11y ago
          Then you'd need a JavaScript interpreter. Which would be terrible.
          • icebraining 11y ago
            Well, it doesn't have to be JS specifically. In theory, it could even be a piece of machine code.
      • chaz72 11y ago
        And a small number of tools for composing JSON.
      • Slackwise 11y ago
        YAML seems more preferable, since it would be far more readable when STDOUT is the shell. Oh, and JSON is a functional subset of YAML, so you'd still be able to output as JSON and a YAML parser will read it.
    • sudioStudio64 11y ago
      PS will take the text output of any command line program and turn it into .Net strings that can then be manipulated like any other object. You can write PS cmdlets in C/C++, there is an SDK. It will also marshal COM objects into and out of the appropriate .Net types. When dealing with output from command line apps I usually take the time to parse out the data I want from the strings and turn them into strong types if I'm writing a script...If I'm just trying to get something done with a command prompt I just do whatever gets it done the fastest.
      • cremno 11y ago
        >You can write PS cmdlets in C/C++, there is an SDK. Are you sure? AFAIK you have to use C++/CLI which isn't C++ and the official examples are either C# or VB.NET: https://code.msdn.microsoft.com/site/search?f[0].Type=Topic&f[0].Value=Windows%20PowerShell&f[0].Text=Windows%20PowerShell https://code.msdn.microsoft.com/site/search?f[0].Type=Topic&...
        • sudioStudio64 11y ago
          In the 2012 R2 time frame Jeff Snover said that they opened up cmdlet authoring to subsystem teams to use C++ to build cmdlets. Maybe they haven't released it yet? That may mean that you have to implement at least some part of it as a .Net class. That may mean that they are doing COM components that inherit certain interfaces...it may mean that they are doing PInvoke...to be honest I haven't looked into it. It may be that it's still internal. Huh. I should look that up.
    • nailer 11y ago
      Do your own apps communicate via text? Eg, rather than use JSON for the REST APIs you write, do you use only strings? Text made sense because text was the universal format. Now we have JSON. In the Windows world .net objects are universal (though I'd prefer JSON).
      • espadrine 11y ago
        I don't understand your point, since JSON is a textual format. Sure, the fact that Unix chose text meant they created a lot of different formats, some non-standard, but I don't see this being an issue, except for configuration purposes.
        • nailer 11y ago
          Do you deal with the serialized JSON text directly, or run JSON.parse() and JSON.stringify() to turn your JSON into objects? Do you ever use regular expressions to parse unstructured text when using a Unix shell? Do you think 'grep' and thinking of a regex is more or less efficient that using 'where'?
          • camperman 11y ago
            "or run JSON.parse() and JSON.stringify() to turn your JSON into objects?" By that time, the communication has already happened.
            • nailer 11y ago
              Well the only point of serialising was communication, so I'd argue unwrapping a presentation layer format is included as part of communication.
              • camperman 11y ago
                Fair enough. I was coming at this from my current experience where a couple of different programs I have grab JSON from a server: the Python one puts it into an object, the bash script doesn't because I couldn't be bothered :)
    • wumbernang 11y ago
      You can. Half way through a powershell script you can switch to C#, VB, JavaScript if you want and implement a pipeline or shell out to a C++ program, talk to something over the network or even Cygwin if you really want.
  • hurin 11y ago
    I'm not familiar with PowerShell, but wouldn't following through with this principle mean that the default program has to give maximally verbose output to the piped formatter, since the formatter can only filter rather than extend the original command. I imagine to achieve the default functionality your command must be significantly more verbose, if this philosophy is followed through 100%, or you have a lot of per-configured default formatters (and you have to remember what to pipe to which). Maybe I'm misunderstanding this, it's very neat in principle though.
    • wz1000 11y ago
      AFAIK PowerShell pipes stream objects, not text. I've never used it though.
    • AndrewDucker 11y ago
      PowerShell returns objects, not strings. Which means they can have as much information as you like without being overly verbose :-)
      • hurin 11y ago
        > PowerShell returns objects, not strings. Which means they can have as much information as you like without being overly verbose :-) How does the user know exactly which output he is going to get? The formatting program cannot know anything about default expected output - so it either must be specified explicitly, or objects must distinguish between default_for_formatter_to_output fields and more_verbose_hidden_fields - I'm really not a fan of the amount of man <command_name> using Unix involves, - but is the alternative really much better?
        • sudioStudio64 11y ago
          There are some sane defaults in the PS case along with heuristics that try to map arguments between pipe-out and pipe-in cmdlets. The parsing/serialization and formatting are done by the environment, not by an individual program. It works pretty well. In the end there is also some convention that helps it work more consistently. In practice, it is kind of cool to be able to construct new objects as the pipeline progresses so that can customize that behavior. I've found that with some aliases for default commands it can be reasonably succinct as well.
  • ekidd 11y ago
    I recently discovered that I can do this on Linux, too! After over two decades of using Unix and Linux, I ran into jq, a tool for querying and transforming structured JSON pipelines: http://stedolan.github.io/jq/ http://stedolan.github.io/jq/ This can be used to do many of things you demonstrate with PowerShell. Here's an example from the docs: curl 'https://api.github.com/repos/stedolan/jq/commits?per_page=5' | \ jq '.[] | {message: .commit.message, name: .commit.committer.name}' This outputs: { "name": "Nicolas Williams", "message": "Add --tab and -indent n options" } ... more JSON blobs... You can output a series of individual JSON documents, or convert the output to a regular JSON array. You can also output raw lines of text to interoperate with existing tools. And this works especially well with commands like 'docker inspect' that produce JSON output. I think that PowerShell and jq get this right: More command-line tools should produce explicitly-structured output, and we should have powerful query and transformation tools for working with that output. For day-to-day use, I do prefer jq's use of JSON over a binary format.
    • Canada 11y ago
      That's cool, though powershell deals in actual objects which are more powerful than better structured input and output. For example, the objects returned by ps have methods for interacting with them.
      • myhnaccount108 11y ago
        I would argue the object approach limits the universality of the PowerShell way as a general purpose computer interface because it binds it to the necessity of a particular flavor of object system (e.g. .Net) and mutable data which is venom to general purpose pipe and filter systems. jq looks very interesting though note that it builds upon an underlying notion of text to serialize JSON.
        • Canada 11y ago
          You can parse streams of text in PS too, so it's not like cmdlets are making the pipeline any less powerful. As for binding to .NET, I don't think that's very limiting. A surprising amount of stuff is already easily hosted in .NET. I would argue that all PS needs to be more competitive is: Ports to Linux/*BSD/OSX and a terminal implementation for Windows that doesn't suck. Cmd.exe is a piece of shit that needs to die: - Command editing. Is support for at least ^A, ^E, ^W, and friends too much to ask? - Completion. Who wants to cycle through 4253 possibilities one at a time? - Copy/paste. Programs actually STOP executing when you select text, like as if you ^Z in Unix. Even with quick edit enabled so you don't have to ALT-SPACE-E-ENTER<select stuff>ENTER, the fastest way to paste is 4 keys: ALT-SPACE-E-P. - No SSH. Microsoft is addressing this. It's borderline criminal that Windows doesn't just ship with OpenSSH server already. - No screen/tmux. I can't even talk about how it deals with resizing without using a lot of profanity. - Lack of echo by default is seriously annoying. In short, make the terminal feel like Putty and the editing/completion features feel like bash and I think PS could give all existing shells a run for their money.
  • agumonkey 11y ago
    I am really curious about how much parsing and formatting occupy *nix source code. (IIRC 30% of ls), and if it would be a good idea to decouple the way you mention it. my json-infused distro is still on my mind. ps: this lisp machine talk mention how 'programs' (I guess functions) exchanged plain old data rather than serialized ascii streams making things very fast. There are caveats of adress space sharing though but it's another nice hint we could investigate other ways. http://www.youtube.com/watch?v=o4-YnLpLgtk http://www.youtube.com/watch?v=o4-YnLpLgtk
    • sudioStudio64 11y ago
      That's an interesting point. abstracting out parsing/serialization from the data piped between commands would lead to more consistent argument handling for all commands.
      • agumonkey 11y ago
        somebody wrote docopt (started as python lib) as a generic POSIX usage string parsing (part of the standard). Maybe it could lead to simpler argument parsing and 'user interface' generation, whether static documentation or shell completion scripts. Also structured output may lead to more relation tools, less regexful ad-hoc parsing, maybe some kind of typecheck so callers can be notified when they need to be rewritten.
  • nailer 11y ago
    This, a thousand times. As a Linux user / developer, it's surprising to hear colleagues talk about how important it is to separate content from presentation regarding, say, TeX, but ignore the benefits of Powershell doing the same. Unix users actually expel cycles trying to express things like times and file size as regular expressions to operate on strings, rather than dealing with them using their real data types. A big part of this is that remoting has sucked - most webb app developers don't use Windows - so a lot of people who should have been able to find out for themselves hasn't been able to. Microsoft supporting SSH (as they announcedd recently) should fix that.
    • chongli 11y ago
      it's surprising to hear colleagues talk about how important it is to separate content from presentation regarding, say, TeX This debate never ends because, like the static vs dynamic types debate, it ignores the main tradeoff in favour of a one-size-fits-all solution. In both debates (separation and types) the tradeoff can be expressed most generally as an up-front vs deferred time investment. Static types require more up-front time investment with the advantage that they save time later. The same goes for having separated content from presentation. Now, in light of the tradeoffs above what is the appropriate choice to make? That depends on how much time you're going to spend writing the script/document/whatever and how much time you're going to spend running/maintaining it later. For scripts, it makes no sense to have a heavyweight type system if you're only going to run the thing once and throw it away. Likewise, for documents it makes no sense to put in the extra time planning all the styles if you're only going to make a document (say, a shopping list) and then throw it away. For a long-term document such as a book or a thesis it makes a ton of sense to use something like TeX.
      • alkonaut 11y ago
        > Static types require more up-front time investment with the advantage that they save time later. What exactly is it that saves time? Is it to not have to type as much? In my experience if you use a lang with good type inference you type a lot less with strict typing than without, since you don't have to have unit tests to guard for things like typos/param order/param types.
  • andmarios 11y ago
    So, what would happen if I ran: ssh myrouter "ls /var/log" | ? Length -gt 50ΚΒ Unix shells aren't confined on one's computer. It may be on your server, your android phone or your ten year old adsl router.
    • chris_wot 11y ago
      That won't work, because where-object needs a PowerShell object, and ssh is returning just text. I'm curious how Microsoft will deal with this.
    • useerup 11y ago
      irm myrouter {ls c:\logs | ? length -gt 50kb} This will emit "deserialized" objects from myrouter. PowerShell remoting works across machine boundaries by defining an xml based format for serializing and deserializing objects. During serialization, all properties are serialized, recursively to a configured max depth. The local client will see the deserialized objects - ie they are not "live" any more. Note that the PS team recently announced that they would support SSH as a remoting mechanism. I suspect that they will still send CliXML objects when using powershell to powershell communication. Powershell uses an industry standard for http based remote commands. It should be adaptable to SSH.
      • geofft 11y ago
        Or `irm myrouter {ls c:\logs} | ? length -gt 50kb` to match the original example, right? Serializing structured data is a pretty solved problem, and I wonder if decades of UNIX going out of its way to screw this up is confusing the parent poster. XML is certainly a way to do it that works just fine, but there are a million others. None of them require awk or counting spaces.
        • useerup 11y ago
          You're right. And I also screwed up "irm". It should have been "icm". Apologies.
    • alkonaut 11y ago
  • crdoconnor 11y ago
    My one wish for UNIX coreutils would be that they all had a switch to output JSON. ls --json ps aux --json If they did that, all that powershell stuff would become trivial. Wouldn't even be hard.
    • bishop_mandible 11y ago
      Why not TOML?
      • hk__2 11y ago
        JSON is a standardized format, TOML is not.
    • Comaleaf 11y ago
      How would that work for late-evaluated data? A PowerShell object doesn't have to provide all of the data up-front, and can give updated data when checked at a later date. JSON is still just text, it still needs to provide all of the data you might need up front.
      • jsjohnst 11y ago
        Great point that I think all the JSON advocates are missing.
        • rlguarino 11y ago
          Text streams also have that problem.
          • jsjohnst 11y ago
            Not necessarily. Text streams don't have to by default provide all possible data for the next process in the pipe. Sure, you could keep all the command line arguments you had before to make JSON output manageable, but then you have two problems rather than one.
    • 11y ago
  • CrLf 11y ago
    However, after a few years of using both, my conclusion is that PowerShell isn't a better shell. It is programatically superior (for scripting) but inferior as a CLI language. And your examples are one of the reasons why. "Do one thing and do it well" is about the goal of the tool, not about its features. With all its options, `ls` is still just about listing directory contents. And that motto is still just a guideline, not an absolute rule. What makes the *nix shell great is that it follows that motto enough to be sensible and structured, but not enough that it becomes a burden. Just like a human language does.
    • wumbernang 11y ago
      ls != list files in powershell. ls is an alias for Get-ChildItem which is simply "get me a list of objects attached to the path". That's about as orthogonal as you can possibly muster. It's actually more like plan 9 than unix.
    • Canada 11y ago
      If only it wasn't hosted in the awful cmd style terminal with it's frustrating style of edit functionality and lousy completion. If it had the feel of a Unix terminal I think I could get used to it.
  • nickysielicki 11y ago
    The reason that a program should strive to be unixlike is portability. The nice side effect is little bloat. Being unixlike is about the scope of a program rather than trying to make the implementation as basic as possible to the extent of not taking flags. If a program relies on another program to be useful, that's clearly not portable. It's no different if it's a library or another executable. These cmdlets are excessively small in scope and that makes them intrinsically tied to one another. They're useless without their sibling cmdlets. In effect they're one giant program. So I completely disagree that these are more unixlike.
  • mercurial 11y ago
    > It is somewhat ironic that PowerShell is more true to the principle of "do one thing and do it well" than the Unix shells. It's not particularly surprising that a system designed a long time after the original is more consistent (I wonder how the Plan 9 shell fares in this regard?).
  • hobarrera 11y ago
    This actually looks pretty nice and usable. I do wonder why they use TitleCase for somethings, and lowercase for others ("ls", "gt"). I think it kind of makes this non-obvious and non-predicable. (reminds me of PHP a bit).
• sigsergv 11y ago
  Linux is not Unix, that's just fine, I'm happy it is not. I've tried “re-evaluated” shell (Powershell), no, thanks, it's not intended for using by humans. Currently shell is still simple and powerful. Powershell is powerful and NOT simple.
  • Svip 11y ago
    Indeed. In PowerShell, you have to write 'programs' to get things done, in Bash (as an example), you write commands to get things done. I'd like to think that commands are allowed to contain shorthands. I don't see a problem with `ls`'s flags, even if they are numerous, because they all help reduce the interaction from programs to commands.
    • useerup 11y ago
      > In PowerShell, you have to write 'programs' to get things done, in Bash (as an example), you write commands to get things done Huh? I use PowerShell (almost) daily. I do not write 'programs'. I use small steps, modify and repeat if the result was not as desired, just as with any REPL. If you are referring to how one creates commands for the shell, PowerShell cmdlets can be written as PowerShell functions or as .NET classes. Not "programs". A very nice feature of PowerShell in that regard, is how the shell performs the parameter parsing. The cmdlet declares the parameters with name, type and optionally position, and the shell does any type coercion from e.g. string to the parameter type. This means that PS cmdlets - unlike sh commands - contain no parameter parsing logic whatsoever - only the logic and declarative parameters. This also means that the information is readily available to the shell (or extensions) to be used for auto suggestions ("intellisense"), tab completion, early error checking etc.
  • UK-AL 11y ago
    What does intended for humans even mean? At least powershell has command names that describe what they do.
    • dijit 11y ago
      Powershell is readable but barely writeable. Bash is the opposite, it's very easy to do complex things if you have some basic knowledge. for instance a one liner nested loop to replace a string in a bunch of text files which are a few directories deep. Easy to write but it takes a while to read.
• guard-of-terra 11y ago
  "A system composed of a zillion tiny modules is itself a pile of mud" Having so many args to `ls' might be an overkill, but I certainly don't want to combine four programs to get full coloured sorted listing of current directory.
  • icebraining 11y ago
    Why not, if you can combine them permanently using a shell script?
    • guard-of-terra 11y ago
      I won't be doing that. And if someone else does that for me, I've suddently got 400 (n^2) aliases that I can't reason about.
      • icebraining 11y ago
        I wouldn't say you need 400 aliases. You'd only need aliases for the commonly typed pattern, not for any possible combination you might ever use.
  • wz1000 11y ago
    Many shells already come with a few default aliases that call ls with different arguments. Your "four programs" can be combined in a simple alias.
    • nly 11y ago
      And what would you call an alias that gave you a coloured, sorted listing of the current directory?
      • mseepgood 11y ago
        lsc
  • Dewie3 11y ago
• icebraining 11y ago
  I found this a weird post. The shell does know what files are (how else could you do "echo * "?) and programs do take arguments and have return values. In any case, I think the author is kind of tilting at windmills; as far as I know, that Unix and Linux don't really follow the Unix philosophy is reasonably accepted today, even if it wasn't back in '83, when Rob Pike and Brian Kernighan made the presentation called "UNIX Style, or cat -v Considered Harmful", which pointed out exactly those problems, which were already in development: Bell Laboratories Murray Hill, NJ (dec!ucb)wav!research!rob It seems that UNIX has become the victim of cancerous growth at the hands of organizations such as UCB. 4.2BSD is an order of magnitude larger than Version 5, but, Pike claims, not ten times better. The talk reviews reasons for UNIX's popularity and shows, using UCB cat as a primary example, how UNIX has grown fat. cat isn't for printing files with line numbers, it isn't for compressing multiple blank lines, it's not for looking at non-printing ASCII characters, it's for concatenating files. We are reminded that ls isn't the place for code to break a single column into multiple ones, and that mailnews shouldn't have its own more processing or joke encryption code. Rob carried the standard well for the "spirit of UNIX," and you can look forward to a deeper look at the philosophy of UNIX in his forthcoming book. Yet, Rob Pike himself uses Linux nowadays - the problem today isn't so much acceptance, but backward compatibility.
• pjmlp 11y ago
  Yes, there is this strange idea in new generations that GNU/Linux is UNIX, without having tried DG/UX, Tru64, Solaris, HP-UX, Aix, Irix and many others. Just drop a GNU/Linux user into a default HP-UX installation and watch them getting around the system. Hint, those nice GNU flags and utilities aren't there. As for what UNIX turned out to be, I think Rob Pike as one of its creators is a good quote: <quote> I didn't use Unix at all, really, from about 1990 until 2002, when I joined Google. (I worked entirely on Plan 9, which I still believe does a pretty good job of solving those fundamental problems.) I was surprised when I came back to Unix how many of even the little things that were annoying in 1990 continue to annoy today. In 1975, when the argument vector had to live in a 512-byte-block, the 6th Edition system would often complain, 'arg list too long'. But today, when machines have gigabytes of memory, I still see that silly message far too often. The argument list is now limited somewhere north of 100K on the Linux machines I use at work, but come on people, dynamic memory allocation is a done deal! I started keeping a list of these annoyances but it got too long and depressing so I just learned to live with them again. We really are using a 1970s era operating system well past its sell-by date. We get a lot done, and we have fun, but let's face it, the fundamental design of Unix is older than many of the readers of Slashdot, while lots of different, great ideas about computing and networks have been developed in the last 30 years. Using Unix is the computing equivalent of listening only to music by David Cassidy. </quote> Taken from http://interviews.slashdot.org/story/04/10/18/1153211/rob-pike-responds http://interviews.slashdot.org/story/04/10/18/1153211/rob-pi...
  • FooBarWidget 11y ago
    > Yes, there is this strange idea in new generations that GNU/Linux is UNIX, without having tried DG/UX, Tru64, Solaris, HP-UX, Aix, Irix and many others. I think that it's not so strange, seeing that you have to fork over lots of $$$ to get your hands on hardware that would run Solaris/HP-UX/AIX/etc, plus licensing fees. You can't easily rent a cloud server for an hour to play with it (with some exceptions, like SmartOS on Joyent). The high barrier to tinkering reduced proprietary Unices to expensive niche environments that are willing to pay lots of cash.
    • justincormack 11y ago
• mangecoeur 11y ago
  I think there's a bit of a disconnect between how people talk about/defend the unix shell and the reason it's actually still there. People often defend the simplicity or philosophy - despite, as the article points out very well, much of it being clearly poorly designed. But really the reason most of it exists unchanged is simply path-dependancy - we used it then so we use it now. It works well enough that there's not enough incentive to drive a large change. There are dozens of different shell-like environments and scripting languages, but they never really took over out of plain intertia. That, and the conservative attitudes of people defending their Unix traditions (related in no small part to how much effort they put into learning the thing in the first place). For new people arriving on the scene, its still less effort to learn things, even when they get truely arcane, than to use something which relatively few other people use or even roll your own alternative...
• nspattak 11y ago
  Do one thing and do one thing well doesn't equal do one simple thing. Furthermore, ls can actually do One Thing Only if this is enough for you and that is plain ls. What would be the better alternative? Have multiple ls, ls-color, ls-long-color, etc? The discussion about Unix philosophy, standards, etc is a different story but I did not see that from the articles examples.
  • Dewie3 11y ago
    "Doing one not-simple thing" is sometimes "doing multiple simple things which they couldn't decompose" in disguise.
• Garlef 11y ago
  In other words: The shell should be a lisp.
  • GlennS 11y ago
    So, http://scsh.net/ http://scsh.net/?
• golergka 11y ago
  The `ls` example is especially interesting: it made me think about perfomance of pipelines. Imagine a folder with 10^6 files. Listing them would take some time; if you only need 10^3 of files selected by the simplest filter (LIMIT 1000, for example), piping the whole list, in whatever format, to the filter program would be very far from optimal. What would be a perfect solution is to use iterators instead of lists (the best example that I know of is Linq, but it seems that this approach is common enough). However, now we don't even exchange data between different programs; now we couple them, running at the same time, by a complicated interface! It seems that if we take one more logical step in this direction, we'll get generics. Don't you start to feel that this kind of ideal shell, piping different functions together in the most correct way possible already exists, and it's already installed on your computer? I'm talking about interpreted languages like Python, Perl, Javascript, Ruby or whatever else you fancy. Of course, I don't seriously think that we should abandon shells for language interpreters. It's just that balance between complexity and universality is a very delicate thing, and convention, however bad, is good just because it's a standard. In the meanwhile, I'm quite happy with the Fish shell.
  • radiospiel 11y ago
    Well, Unix is prepared to handle that nice enough: when the pipe reader terminates (because it received 1000 lines) the writer would receive SIGPIPE on the next write to the pipe and consequently be killed (as the default, at least.) Case closed.
    • golergka 11y ago
      Well, `LIMIT 1000` would of course be the simplest case, so closing it isn't really an achievement. What if I want to list all information about files with some mode? `ls` will still send 10^6 lines, carefully formatted, and almost all of them will be discarded with a simplest filter it could perform before reading all additional information from disc.
      • anewhnaccount 11y ago
        To execute this automatically in an efficient way we need a query planner which can look at the whole pipeline and decide to use a different set of primitives if the naive/implied ones aren't sufficient. What you're talking about is implemented in relation database management systems but it requires the query planner to know about the whole system - that is it's the opposite of Unix - there is a piece of the system that needs to know about the whole system. As far as Unix is concerned, the whole list is not stored in memory. Pipes are buffered streams (they are iterators, just with a buffer attached, which makes them more efficient not less).
• sudioStudio64 11y ago
  these arguments about unix philosophy, to me, seem to miss the point about what Unix actually is...which, again...to me, is a bunch of tools to get things done. being able to get things done with Unix means that you just put up with the weird bits and move on. there doesn't need to be an overriding philosophy. there are places where there seems to be consistent concepts, but their implementation isn't and doesn't really need to be consistent. the way that new ideas get so intensely opposed needs to be really looked at though. there's no need to be so insulting if something isn't a good idea...it just won't get used.
  • iopq 11y ago
    Maybe for you, but for me it's just a bunch of tools I don't really know that well with lots of arcane options. I just google how to do what I want because it's not possible to memorize through light usage.
    • sudioStudio64 11y ago
      I agree. No one ever memorizes all of it...or even a common core of it. It's weird like that.
• mtrn 11y ago
  > Granted, we’ve developed graphical user interfaces that keep “ordinary users” away from the command line, but we still expect “serious developers” to drop down into a demonstrably inhumane environment to get anything meaningful done. Maybe I am too familiar with it, but I have yet to encounter a graphical environment that matches the effectiveness of the terminal and command line programs. I saw systems, that tried to replace shell one-liners with a hundred thousand line GUI application, that no one cared to use.
• andmarios 11y ago
  As I see it ls does one job (list directory contents) and does it well (by supporting 37 flags). If ls was a human, we would call him an expert. Since unix tools expect input as plain text, the flags that ls supports are absolutely useful and I believe this is obvious to anyone who has written more than 50 lines of bash. When your input doesn't support a standard complex structure (e.g json with some standard set fields), claiming that “sort by creation time” is a simple filter is funny. First you would need a ls flag so that ls will output creation times in an easy to process time format (seconds since epoch), then you would need to ask sort to work on a certain (1 flag) numerical (1 more flag) field. Last you would have to pass sort's output to a stream processor to get rid of unwanted fields. Current “list files by creation time”: ls --sort=time --time=ctime Replace sort flag with filters: ls -l --time=ctime --time-style="+%s" | sort -n -k 6 -r | awk '{ $1=$2=$3=$4=$5=$6=""; print substr($0,7) }'
  • zymhan 11y ago
    I totally agree that trying to filter the output of ls is madness when compared to using the built-in flags. But doesn't this also indicate that command output should be more unified in it's output format? "cut", for example, is a useful tool, but it seems ham-fisted as a way of formatting output data for anything other than a one-time job.
  • joepvd 11y ago
    Nitpick, sorry. `ctime` is not Creation Time, but Change Time. The traditional *NIX file systems do not have a meta data field for creation time.
    • andmarios 11y ago
      Thank you; I naively assumed since we have modification time (mtime), ctime would stand for creation time. :)
  • useerup 11y ago
    The same in PowerShell - using ls without the abundance of parameters - would look like: ls | sort -des LastWriteTime | select name 1: Get the "child" items of the current directory 2: sort them descending according to the last write time 3: strip out all other properties but name No need for ls to support a plethora of formatting options. And the typing is actually pretty short due to tab completion: ls | sort -des *time<tab><tab><tab><tab><tab> | select n<tab> I could not remember what the "last change time" was called, so I just entered *time and used tab completion. PowerShell is smart enough to be aware that (one of) the output types from "ls" has a property called "LastWriteTime" - it came up after 5 tabs. Similarly when pressing n<tab> in the select it completed directly to Name because it knew (still from the pipeline) that output of "ls" would have a "Name" property - even if sorted first.
• adambatkin 11y ago
  This article gets it all wrong: The ls command does one thing well (LiSting files) and it isn't optimized for terminal size, it's optimized for PROGRAMMER productivity. The ls command gives you a default view that is exactly what you want most of the time and is easily copy/pastable for other uses (it's a single column) or more importantly, piped to some other program for further processing. And then there are easy-access options for common tasks (extra details, hidden files, sorting by attributes, etc...) Powershell largely does exactly what the author is talking about. And yet it's completely unusable for interactive use (though it's great for writing longer-lived automations).
• peter303 11y ago
  Ed is still one of favorite editors out of UNIX-like terseness.
• chris_wot 11y ago
  Someone needs to port PowerShell to Unix :-) I'm confused about the response to his tweet though - ls has a lot of switches, but I can't really see any that don't do something with listing files. And then the response is "Unfortunately, the lineage is _not_ Unix -> Linux. It is Unix -> Plan 9. Linux doesn't follow Unix philosophy :-("... Except Unix's ls eventually had 19 options, and the ls used by Linux is a port Gnu's coreutils. Nevertheless, I tend to agree. ls does a lot of formatting that could be moved to ancillary utilities. I guess what I like about PowerShell is the way it allows for redirecting into objects - that really is an innovation that Unix shells don't have (at least as far as I'm aware...).
• knowledgesale 11y ago
  Incidentally, there is plenty of ls alternatives like lsp: https://github.com/dborzov/lsp https://github.com/dborzov/lsp
• taylodl 11y ago
  I dunno, the shell combined with sed and awk is pretty powerful and so far has taken care of everything I need to do
• mwcampbell 11y ago
  The essay "Free Your Technical Aesthetic from the 1970s" [1] by James Hague is appropriate here. See also The UNIX-HATERS Handbook; much of it is outdated, but I think some of it is still relevant, particularly the part on the shortcomings of pipelines. I think it's unfortunate that even if a good open-source PowerShell clone is developed (or PowerShell itself is open-sourced), it will probably never gain widespread acceptance outside of Microsoft shops. Tribalism so often trumps actual merit. But if someone did a PowerShell-like thing on top of Node.js, I could see that taking off, particularly if it were integrated with the Atom editor and its package manager. [1]: http://prog21.dadgum.com/74.html http://prog21.dadgum.com/74.html
  • bishop_mandible 11y ago
    > But if someone did a PowerShell-like thing on top of Node.js, I could see that taking off I think it should also use React.
  • ska 11y ago
    > it will probably never gain widespread acceptance outside of Microsoft shops. This seems fairly natural for something so deeply coupled with .NET. Perhaps mono and Microsofts current efforts will make this less of a constraint, but still -- .NET has made some deep architectural tradeoffs that you will mostly have to live with if you want to play in this particular sandbox, and I can't see that helping with wider spread acceptance.
  • RubyPinch 11y ago
    "open-source PowerShell" PaSh http://pash.sourceforge.net/ http://pash.sourceforge.net/ , a bit unloved, but definitely exists "node.js" termkit, which is what partially inspired the current article. I think the problem is there is no authoritative backing for anything on the linux side, with windows, if you are going to make a object-aware-pipes tool, its going to be powershell without a doubt. In linuxland, the decision is not so clear (does one implement their own? go with termkit? go with pash? go with one of the other more deviant shells?)
• soyiuz 11y ago
  I think the author misses the point. As far as I see it, Unix tools are optimized for one thing: the human user. If `ls` is bloated, it is only bloated from the system point of view. As a user, I only see a small command that is quick and consistent (to a reasonable extent) with other tools. I don't need to know about 39 flags--that complexity is revealed with time, as I need to accomplish more complex tasks. For now, I only use and "see" two or three. And if more is needed, the program is small enough to be reasonably well documented. `man ls` perhaps with `grep` is all most people need. This brings me to the second point. "Streams of text." Just like `ls`, streams of text are an optimal format/convention for humans. Many other things are better at being more compact or more efficient etc. But as formats and conventions proliferate, streams of plain text remain: readable and universal. Humans will ALWAYS be able to work with text. It is something that all humans kind of agreed upon--which cannot be said for any other formats or standards, which can offer various technical benefits at the expense of longevity, universality, and readability. These two features (`ls` being relatively light and text streamy) leads to the "bootstrapping" effects sought after by the first generation of Unix developers. Learning about pipes and filters in one part of the system is applicable to all others. These tools scale with your level of expertise. They grow with you because despite the small quirks, there's a remarkable consistency of interface: text! Consequently `ls` (along with many other core tools) is implemented in the same way across a staggering variety of platforms. It has survived decades of alternatives touted as "better," "faster," "more usable," etc. etc. etc. That is the remarkable achievement of *nix / GNU etc. approach to creating human-centric software. As we architect ever more complex systems, we would do well to understand why and how Unix has endured, warts and all.
  • iopq 11y ago
    No, I'd rather be working with FILES, not paths to files. This is how you get `rm -rf $STEAMROOT/*` problems
• Myk267 11y ago
  I think it's easy to misinterpret any of the "Unix philosophies". It's Ye Olde "MIT vs New Jersey" again, and Unix is relentlessly straight out of NJ. Simplicity in the "NJ" sense isn't what you'd expect if you're of the "MIT" mindset.
  • jstimpfle 11y ago
    That's a good point. Makes me think again of "slow devices" and interrupts leading to EINTR or partial reads/writes. In my eyes a beautifully pragmatic solution to a real world problem, and yet at some point SA_RESTART was devised.
• Dewie3 11y ago
  Meta: I think someone made a comment that forced the whole page to be wider than it normally is? like when you post an image on a forum that forces the whole page to widen to fit it horizontally.
  • avn2109 11y ago
    Yeah this is super annoying. I am Chrome/OSX and scrolling is awful :(
• vanilla 11y ago
  I can't follow the comparison. Linux is just a Kernel, most distros use the coreutils package which is developed/maintained by GNU [1]. So any comparison is not between Linux and Unix, rather the GNU part in GNU/Linux [1]: https://www.gnu.org/software/coreutils/coreutils.html https://www.gnu.org/software/coreutils/coreutils.html
  • justizin 11y ago
    Right, and OS X is based on FreeBSD.
• markhahn 11y ago
  ls's plethora of flags doesn't stop "echo *" from working.
• ised 11y ago
  I have never thought that UNIX approach is the "best" in a universal sense. But when compared to how large, slow, complicated and opaque the "alternatives" are, UNIX is the clear choice for me. I can modify and recompile UNIX to meet my own ends. That is all but impossible if I chose an alternative such as Windows. For example, if I do not want ls to have 30+ options, I can trim it down to just a few options and recompile. There are other utilities besides ls for viewing file information, e.g., BSD stat(1) or mtree -cp. The later displays mode information in octal which is something ls, despite its 30+ options, does not do. Or I can write my own simple utility. I am given the full UNIX source code. Where is the source code for Windows? Personally I keep my filenames short and never use spaces, so I sometimes use the shell's echo builtin and tr(1) to get a quick list of files. echo * |tr '\040' '\012' If there were non-UNIX alteratives that were small, simple and transparent, perhaps I might not be using UNIX. Because I have become very comfortable with UNIX, any alternatives that others suggest have to be comparable with UNIX on size and simplicity before I will take them seriously. Currently, I use a kernel source tree that compresses to under 40MB; I can compile kernels with about 200MB of RAM and fully loaded kernels are about 17MB. Userland utilities are usually around 5MB as I prefer to put them in the kernel's embedded filesystem. I do not like to rely on disks. My "IDE" is the same system I am compiling. There is no GUI overhead, everything can be done in textmode. The importance of the preceding two sentences cannot be understated. I am always willing to consider non-UNIX alternatives that can offer the same or better flexibility, size constraints and simplicity. But after decades of being open to alternatives, I am still not aware of any.
  • throwawaykf05 11y ago
    Why do you need the source code to write a utility when all you need is an API? I've written software for Linux and Solaris but never had to look at the source code, so I imagine it's the same for Windows.
    • ised 11y ago
      "... you need the source code to write a utility..." Where in the comment is this statement? Personally the primary reasons I would want the source code for the kernel and utilities would be 1. to assess its quality and, assuming the quality meets my standards, 2. to modify it to meet my own ends. In my case, the less I have to write things from scratch the better. Let me know if you still have questions.
• ggchappell 11y ago
  > You can’t write a function that returns a list of files because the shell doesn’t know what a “list” is, doesn’t know what “files” are, and couldn’t tell you the difference between a “function” and a “program” if its life depended on it. Could it be that what this article is really groping towards is Perl? I mean Perl not as it has become, but as it was -- Perl 4 perhaps: basically the capabilities of the shell, augmented with a couple of data structures. So there are strings, lists, hashes, and functions, and that's pretty much it.
• dbbolton 11y ago
  Seems like the article, and several comments here, equivocate Linux with GNU. While they are most commonly used in conjunction with one another, I still don't believe that "this GNU program does too many things; therefore Linux doesn't adhere to the Unix philosophy" is a valid argument.
• gumby 11y ago
  I think you have it backwards: there are two legitimate counterarguments. One is that 'ls' is the common program for querying the directory data structure (which in original Unix was just a kind of file). So it makes sense for it to output the data in different kinds of ways. The second is that its a user mode command. It's easier than having a lot of little commands (ls for all files, basically ls -1; lsF for showing file types, etc). In fact I'll often to 'ls' then realise I want more info so will type 'ls -l' (say) which is really my brain saying "oh yea, just like what I just asked for but more info". Better to pick something more deranged like tar or cpio!
• solidsnack9000 11y ago
  UNIX emerged in an era of more structured systems, though. Many OSes of the time did not have files in a naive sense -- they had record-based formats. "Everything is a file" was a revolutionary idea, and made UNIX simple and lightweight and portable. It is interesting that the author doesn't highlight "everything is a file" which I have always understood to be the fundamental UNIX philosophy. ("Do one thing well" applies to so much more than UNIX.) Our understanding of how to use types and how to structure data on disk and for streaming has greatly matured; but that doesn't mean we can ever afford to pay more to get nothing. Say for a moment that `ls` provided data in tab-separated value format (a format which is, unlike CSV, line filterable). Wouldn't that go a long way toward allowing for the HOFs the author mentions? And it wouldn't involve a shell type system (or "Shell Object Model").
• ezy 11y ago
  "Many of the usability issues raised by Don Norman in his 1981 criticism of Unix have gone largely unaddressed" That's because Don's complaints were not about intrinsic issues, but complaints from someone who had a frustrating time developing a mental model of the components of a system and who tried to use the just-so story of "cognitive design" to justify his inability. The non-point about prompting y/n to delete is particularly misguided. Similar to this article. "ls" is called "ls", not because of slow 80-char terminals, but because humans are, on average, not fast typists. One wants to list the contents of a directory a lot, and "ls" is quicker than "catalog". Similarly, "ps" vs "tasklist" and "cc" vs "compile-a-c-program" :-). Notably, in powershell, most people have aliases that do this for powershell's necessarily verbose query functions. It's primarily a command shell for controlling the machine, not a programming language. And the UI choices made over the last 30 years reflect that. That fact that ls takes arguments is a UI improvement that keeps you from having to write tiny programs all the time and pipe it's output to sort and awk all day long. It is also why the "one job well" crew is only half right, and why Powershell kind of feels awkward even though it seems to get more things "correct" in that regard. This is just yet another just-so story by someone who wants ideological purity in UI, which is completely misguided when UI, by definition is oriented towards a universe which is not ideologically pure.
• darkr 11y ago
  One of the reasons for ls having additional functionality, is that it is used in restricted shells - such as those used on rssh (SCP,SFTP, rsync and the like). There are no filtering utilities available in these shells: -rbash-3.2$ ls $PATH chmod cp groups ls mv rm scp