10 ms·

Interesting examples, especially in light of the recent debates about Unix shells and PowerShell. In PowerShell, the `ls` command is a simple command that prod

• MrBuddyCasino 11y ago
  Exactly. I think the Unix philosophy of "everything is a file" has an impedance mismatch with "streams of text". Files and directories are hierarchical lists of objects with certain properties, similar to JSON. The "directories of files" abstraction is versatile and useful, simply because it is a versatile and simple data structure. Streams of text are too limited.
• vbezhenar 11y ago
  Another UNIX principle is that programs deal with text. PowerShell is huge violation of that principle. You can't easily write PowerShell command with C or Rust or Java, as far as I understand. Yet you can write command-line tool with any language and use it with combination of hundreds existing tools.
  • wz1000 11y ago
    A solution to this might be a standardized object format, like JSON.
    • vbezhenar 11y ago
      JSON doesn't support behaviour (methods). They are very useful for composing programs IMO.
      • wz1000 11y ago
        Behaviour could be implemented by shell commands. That way, if you wanted to implement a method for a particular object, you could just write a binary/shell script/whatever that reads the object from stdin and writes its result to stdout
      • masklinn 11y ago
        Methods are useful to organise and abstract programs, they're really quite bad at composition. Many functions operating on the same primitives is much more composable than small sets of functions operating on their own custom primitives.
      • sudioStudio64 11y ago
        if the deserialization mapped to a system wide type system then the methods could be mapped in at that point.
      • current_call 11y ago
        Then you'd need a JavaScript interpreter. Which would be terrible.
        • icebraining 11y ago
          Well, it doesn't have to be JS specifically. In theory, it could even be a piece of machine code.
    • chaz72 11y ago
      And a small number of tools for composing JSON.
    • Slackwise 11y ago
      YAML seems more preferable, since it would be far more readable when STDOUT is the shell. Oh, and JSON is a functional subset of YAML, so you'd still be able to output as JSON and a YAML parser will read it.
  • sudioStudio64 11y ago
    PS will take the text output of any command line program and turn it into .Net strings that can then be manipulated like any other object. You can write PS cmdlets in C/C++, there is an SDK. It will also marshal COM objects into and out of the appropriate .Net types. When dealing with output from command line apps I usually take the time to parse out the data I want from the strings and turn them into strong types if I'm writing a script...If I'm just trying to get something done with a command prompt I just do whatever gets it done the fastest.
    • cremno 11y ago
      >You can write PS cmdlets in C/C++, there is an SDK. Are you sure? AFAIK you have to use C++/CLI which isn't C++ and the official examples are either C# or VB.NET: https://code.msdn.microsoft.com/site/search?f[0].Type=Topic&f[0].Value=Windows%20PowerShell&f[0].Text=Windows%20PowerShell https://code.msdn.microsoft.com/site/search?f[0].Type=Topic&...
      • sudioStudio64 11y ago
        In the 2012 R2 time frame Jeff Snover said that they opened up cmdlet authoring to subsystem teams to use C++ to build cmdlets. Maybe they haven't released it yet? That may mean that you have to implement at least some part of it as a .Net class. That may mean that they are doing COM components that inherit certain interfaces...it may mean that they are doing PInvoke...to be honest I haven't looked into it. It may be that it's still internal. Huh. I should look that up.
  • nailer 11y ago
    Do your own apps communicate via text? Eg, rather than use JSON for the REST APIs you write, do you use only strings? Text made sense because text was the universal format. Now we have JSON. In the Windows world .net objects are universal (though I'd prefer JSON).
    • espadrine 11y ago
      I don't understand your point, since JSON is a textual format. Sure, the fact that Unix chose text meant they created a lot of different formats, some non-standard, but I don't see this being an issue, except for configuration purposes.
      • nailer 11y ago
        Do you deal with the serialized JSON text directly, or run JSON.parse() and JSON.stringify() to turn your JSON into objects? Do you ever use regular expressions to parse unstructured text when using a Unix shell? Do you think 'grep' and thinking of a regex is more or less efficient that using 'where'?
        • camperman 11y ago
          "or run JSON.parse() and JSON.stringify() to turn your JSON into objects?" By that time, the communication has already happened.
          • nailer 11y ago
            Well the only point of serialising was communication, so I'd argue unwrapping a presentation layer format is included as part of communication.
            • camperman 11y ago
              Fair enough. I was coming at this from my current experience where a couple of different programs I have grab JSON from a server: the Python one puts it into an object, the bash script doesn't because I couldn't be bothered :)
  • wumbernang 11y ago
    You can. Half way through a powershell script you can switch to C#, VB, JavaScript if you want and implement a pipeline or shell out to a C++ program, talk to something over the network or even Cygwin if you really want.
• hurin 11y ago
  I'm not familiar with PowerShell, but wouldn't following through with this principle mean that the default program has to give maximally verbose output to the piped formatter, since the formatter can only filter rather than extend the original command. I imagine to achieve the default functionality your command must be significantly more verbose, if this philosophy is followed through 100%, or you have a lot of per-configured default formatters (and you have to remember what to pipe to which). Maybe I'm misunderstanding this, it's very neat in principle though.
  • wz1000 11y ago
    AFAIK PowerShell pipes stream objects, not text. I've never used it though.
  • AndrewDucker 11y ago
    PowerShell returns objects, not strings. Which means they can have as much information as you like without being overly verbose :-)
    • hurin 11y ago
      > PowerShell returns objects, not strings. Which means they can have as much information as you like without being overly verbose :-) How does the user know exactly which output he is going to get? The formatting program cannot know anything about default expected output - so it either must be specified explicitly, or objects must distinguish between default_for_formatter_to_output fields and more_verbose_hidden_fields - I'm really not a fan of the amount of man <command_name> using Unix involves, - but is the alternative really much better?
      • sudioStudio64 11y ago
        There are some sane defaults in the PS case along with heuristics that try to map arguments between pipe-out and pipe-in cmdlets. The parsing/serialization and formatting are done by the environment, not by an individual program. It works pretty well. In the end there is also some convention that helps it work more consistently. In practice, it is kind of cool to be able to construct new objects as the pipeline progresses so that can customize that behavior. I've found that with some aliases for default commands it can be reasonably succinct as well.
• ekidd 11y ago
  I recently discovered that I can do this on Linux, too! After over two decades of using Unix and Linux, I ran into jq, a tool for querying and transforming structured JSON pipelines: http://stedolan.github.io/jq/ http://stedolan.github.io/jq/ This can be used to do many of things you demonstrate with PowerShell. Here's an example from the docs: curl 'https://api.github.com/repos/stedolan/jq/commits?per_page=5' | \ jq '.[] | {message: .commit.message, name: .commit.committer.name}' This outputs: { "name": "Nicolas Williams", "message": "Add --tab and -indent n options" } ... more JSON blobs... You can output a series of individual JSON documents, or convert the output to a regular JSON array. You can also output raw lines of text to interoperate with existing tools. And this works especially well with commands like 'docker inspect' that produce JSON output. I think that PowerShell and jq get this right: More command-line tools should produce explicitly-structured output, and we should have powerful query and transformation tools for working with that output. For day-to-day use, I do prefer jq's use of JSON over a binary format.
  • Canada 11y ago
    That's cool, though powershell deals in actual objects which are more powerful than better structured input and output. For example, the objects returned by ps have methods for interacting with them.
    • myhnaccount108 11y ago
      I would argue the object approach limits the universality of the PowerShell way as a general purpose computer interface because it binds it to the necessity of a particular flavor of object system (e.g. .Net) and mutable data which is venom to general purpose pipe and filter systems. jq looks very interesting though note that it builds upon an underlying notion of text to serialize JSON.
      • Canada 11y ago
        You can parse streams of text in PS too, so it's not like cmdlets are making the pipeline any less powerful. As for binding to .NET, I don't think that's very limiting. A surprising amount of stuff is already easily hosted in .NET. I would argue that all PS needs to be more competitive is: Ports to Linux/*BSD/OSX and a terminal implementation for Windows that doesn't suck. Cmd.exe is a piece of shit that needs to die: - Command editing. Is support for at least ^A, ^E, ^W, and friends too much to ask? - Completion. Who wants to cycle through 4253 possibilities one at a time? - Copy/paste. Programs actually STOP executing when you select text, like as if you ^Z in Unix. Even with quick edit enabled so you don't have to ALT-SPACE-E-ENTER<select stuff>ENTER, the fastest way to paste is 4 keys: ALT-SPACE-E-P. - No SSH. Microsoft is addressing this. It's borderline criminal that Windows doesn't just ship with OpenSSH server already. - No screen/tmux. I can't even talk about how it deals with resizing without using a lot of profanity. - Lack of echo by default is seriously annoying. In short, make the terminal feel like Putty and the editing/completion features feel like bash and I think PS could give all existing shells a run for their money.
• agumonkey 11y ago
  I am really curious about how much parsing and formatting occupy *nix source code. (IIRC 30% of ls), and if it would be a good idea to decouple the way you mention it. my json-infused distro is still on my mind. ps: this lisp machine talk mention how 'programs' (I guess functions) exchanged plain old data rather than serialized ascii streams making things very fast. There are caveats of adress space sharing though but it's another nice hint we could investigate other ways. http://www.youtube.com/watch?v=o4-YnLpLgtk http://www.youtube.com/watch?v=o4-YnLpLgtk
  • sudioStudio64 11y ago
    That's an interesting point. abstracting out parsing/serialization from the data piped between commands would lead to more consistent argument handling for all commands.
    • agumonkey 11y ago
      somebody wrote docopt (started as python lib) as a generic POSIX usage string parsing (part of the standard). Maybe it could lead to simpler argument parsing and 'user interface' generation, whether static documentation or shell completion scripts. Also structured output may lead to more relation tools, less regexful ad-hoc parsing, maybe some kind of typecheck so callers can be notified when they need to be rewritten.
• nailer 11y ago
  This, a thousand times. As a Linux user / developer, it's surprising to hear colleagues talk about how important it is to separate content from presentation regarding, say, TeX, but ignore the benefits of Powershell doing the same. Unix users actually expel cycles trying to express things like times and file size as regular expressions to operate on strings, rather than dealing with them using their real data types. A big part of this is that remoting has sucked - most webb app developers don't use Windows - so a lot of people who should have been able to find out for themselves hasn't been able to. Microsoft supporting SSH (as they announcedd recently) should fix that.
  • chongli 11y ago
    it's surprising to hear colleagues talk about how important it is to separate content from presentation regarding, say, TeX This debate never ends because, like the static vs dynamic types debate, it ignores the main tradeoff in favour of a one-size-fits-all solution. In both debates (separation and types) the tradeoff can be expressed most generally as an up-front vs deferred time investment. Static types require more up-front time investment with the advantage that they save time later. The same goes for having separated content from presentation. Now, in light of the tradeoffs above what is the appropriate choice to make? That depends on how much time you're going to spend writing the script/document/whatever and how much time you're going to spend running/maintaining it later. For scripts, it makes no sense to have a heavyweight type system if you're only going to run the thing once and throw it away. Likewise, for documents it makes no sense to put in the extra time planning all the styles if you're only going to make a document (say, a shopping list) and then throw it away. For a long-term document such as a book or a thesis it makes a ton of sense to use something like TeX.
    • alkonaut 11y ago
      > Static types require more up-front time investment with the advantage that they save time later. What exactly is it that saves time? Is it to not have to type as much? In my experience if you use a lang with good type inference you type a lot less with strict typing than without, since you don't have to have unit tests to guard for things like typos/param order/param types.
• andmarios 11y ago
  So, what would happen if I ran: ssh myrouter "ls /var/log" | ? Length -gt 50ΚΒ Unix shells aren't confined on one's computer. It may be on your server, your android phone or your ten year old adsl router.
  • chris_wot 11y ago
    That won't work, because where-object needs a PowerShell object, and ssh is returning just text. I'm curious how Microsoft will deal with this.
  • useerup 11y ago
    irm myrouter {ls c:\logs | ? length -gt 50kb} This will emit "deserialized" objects from myrouter. PowerShell remoting works across machine boundaries by defining an xml based format for serializing and deserializing objects. During serialization, all properties are serialized, recursively to a configured max depth. The local client will see the deserialized objects - ie they are not "live" any more. Note that the PS team recently announced that they would support SSH as a remoting mechanism. I suspect that they will still send CliXML objects when using powershell to powershell communication. Powershell uses an industry standard for http based remote commands. It should be adaptable to SSH.
    • geofft 11y ago
      Or `irm myrouter {ls c:\logs} | ? length -gt 50kb` to match the original example, right? Serializing structured data is a pretty solved problem, and I wonder if decades of UNIX going out of its way to screw this up is confusing the parent poster. XML is certainly a way to do it that works just fine, but there are a million others. None of them require awk or counting spaces.
      • useerup 11y ago
        You're right. And I also screwed up "irm". It should have been "icm". Apologies.
  • alkonaut 11y ago
• crdoconnor 11y ago
  My one wish for UNIX coreutils would be that they all had a switch to output JSON. ls --json ps aux --json If they did that, all that powershell stuff would become trivial. Wouldn't even be hard.
  • bishop_mandible 11y ago
    Why not TOML?
    • hk__2 11y ago
      JSON is a standardized format, TOML is not.
  • Comaleaf 11y ago
    How would that work for late-evaluated data? A PowerShell object doesn't have to provide all of the data up-front, and can give updated data when checked at a later date. JSON is still just text, it still needs to provide all of the data you might need up front.
    • jsjohnst 11y ago
      Great point that I think all the JSON advocates are missing.
      • rlguarino 11y ago
        Text streams also have that problem.
        • jsjohnst 11y ago
          Not necessarily. Text streams don't have to by default provide all possible data for the next process in the pipe. Sure, you could keep all the command line arguments you had before to make JSON output manageable, but then you have two problems rather than one.
  • 11y ago
• CrLf 11y ago
  However, after a few years of using both, my conclusion is that PowerShell isn't a better shell. It is programatically superior (for scripting) but inferior as a CLI language. And your examples are one of the reasons why. "Do one thing and do it well" is about the goal of the tool, not about its features. With all its options, `ls` is still just about listing directory contents. And that motto is still just a guideline, not an absolute rule. What makes the *nix shell great is that it follows that motto enough to be sensible and structured, but not enough that it becomes a burden. Just like a human language does.
  • wumbernang 11y ago
    ls != list files in powershell. ls is an alias for Get-ChildItem which is simply "get me a list of objects attached to the path". That's about as orthogonal as you can possibly muster. It's actually more like plan 9 than unix.
  • Canada 11y ago
    If only it wasn't hosted in the awful cmd style terminal with it's frustrating style of edit functionality and lousy completion. If it had the feel of a Unix terminal I think I could get used to it.
• nickysielicki 11y ago
  The reason that a program should strive to be unixlike is portability. The nice side effect is little bloat. Being unixlike is about the scope of a program rather than trying to make the implementation as basic as possible to the extent of not taking flags. If a program relies on another program to be useful, that's clearly not portable. It's no different if it's a library or another executable. These cmdlets are excessively small in scope and that makes them intrinsically tied to one another. They're useless without their sibling cmdlets. In effect they're one giant program. So I completely disagree that these are more unixlike.
• mercurial 11y ago
  > It is somewhat ironic that PowerShell is more true to the principle of "do one thing and do it well" than the Unix shells. It's not particularly surprising that a system designed a long time after the original is more consistent (I wonder how the Plan 9 shell fares in this regard?).
• hobarrera 11y ago
  This actually looks pretty nice and usable. I do wonder why they use TitleCase for somethings, and lowercase for others ("ls", "gt"). I think it kind of makes this non-obvious and non-predicable. (reminds me of PHP a bit).