6 ms·

> a structured data format like S-expressions could be passed around instead. Why would a structured format be a better lowest common denominator than an unstr

> a structured data format like S-expressions could be passed around instead. Why would a structured format be a better lowest common denominator than an unstructured one like text files? Or is it files that are the issue?

snazz 6y ago
Unix pipes are great, but having to parse plain-text streams is inefficient and error-prone. Imagine how much more powerful it would be if commands returned data in structures that could be used by other applications without parsing an ad-hoc text format. The filesystem can be used to approximate this, like it is in Plan 9, but I think that using text as the lowest common denominator leaves a lot of potential power on the table.
- zozbot234 6y ago
  Unix-like pipes are not restricted to sending text, they may send general bytestreams. Modern data serialization formats (flatbuffers, capnproto, msgpack, etc.) are highly efficient, and using serialization makes it easier to deploy applications in a distributed setting.
  - snazz 6y ago
    Sure, but the standard GNU or BSD userland doesn't do that, does it? If you're going to the trouble of implementing your own userspace, then you might as well use a consistent, proper data serialization format (either text or binary, but text is more accessible).
  - throwaway894345 6y ago
    The value is in having everything speak a common standard. In unix, that 'standard' is lines of text, which is inherently linear and lacks the ability to idiomatically represent hierarchy. Conceivably, one could use modern serialization formats or we could do something a bit more interesting, like supporting some notion of "objects" directly in the operating system or runtime (in the case of powershell, the "runtime" is the .net vm; in unix it's the OS).
    - brutt 6y ago
      Format: * Field type: S - 16 bytes, D - 32, L - 64, Q - 128, B - 256, P - 512, T - 1024. * Field ID - 3 bytes ASCII Letters, numbers, '_'; * Field-value separator - 2 bytes: ':' and field value terminator, e.g. ': ' or ':"' or ':[' or ':{' or ':(' ; * Field value 9 bytes ASCII for S, 27 for D, and so on, padded with ' '; * Field separator: 1 byte '\n'; SOBJ: X23456789\n SFN_:"Length" \n SFV_: 23 \n SEND: \n Such text can be parsed by both human and computer, because computer can just jump from field to field without parsing everything in between, like humans do.
- d0mine 6y ago
  You don't need to send text (but you should by default for 99%+ cases when the performance is good enough and use the advantages of a human-readable format) The simplicity of a pipe is a feature.
  - zozbot234 6y ago
    Ultimately, these "text" formats are simply a matter of convention that makes some binary-formatted streams easier to document for humans' benefit. There's nothing wrong with a binary format if it's properly documented in some way, and modern serialization formats can be augmented with schemas and other features that help provide this.
    - laumars 6y ago
      Being able to easily read and edit them matters too (pipes are just files so one might choose that file to be a persistent file on disk). Having a none-text format would mean all your file editing tools would have to then understand this new binary format. So you’d have to rewrite every tool to manipulate that data as if it’s text. By which point you’ve effectively just reinvented ASCII.
- travbrack 6y ago
  Powershell has been growing on me lately. It uses objects to pass data around and it's quite nice.
  - AnIdiotOnTheNet 6y ago
    Using powershell for a month or so and then going back to doing things with text-parsing UNIX commands really highlights how much better things could be if only we moved past doing things the same way they were done in the 70s.
    - chacha2 6y ago
      Have you checked out NuShell?
  - strbean 6y ago
    I'm really sad I missed the boat so hard on Powershell. I grabbed it at the first public release (beta? alpha?), and the thing was absurdly slow. Trying to `ls` took around 10 seconds. I nope'd out and didn't touch it again. Now I have a coworker who uses it religiously (on linux, who would have thought!) and when I'm watching over his shoulder I'm astounded by how great it is.
    - throwaway894345 6y ago
      How do people use environment variables with powershell? I often want to do `FOO=bar ./program`, but this doesn't seem to be supported. I googled briefly, but there weren't many satisfactory answers.
      - majkinetor 6y ago
        $Env:FOO='bar'; ./program
        brutt 6y ago
        It's equivalent of `export FOO='bar'; ./program` , which is not same as `FOO='bar' ./program`, which, in turn, is equivalent to `OLD_FOO="$FOO"; export FOO='bar'; ./program; export FOO="$OLD_FOO";` .
- riazrizvi 6y ago
  Interesting idea, might be important to brush up on the implementation rationale of Unix. To that end there is The Design of The Unix Operating System, Bach [1]. Or A Fast File System for Unix [2]. Or The Design and Implementation of a Log-Structured File System [3]. It seems to me that files solve for the problem of how to deliver an economic memory management system, and they do it by paging cheap disk space so that it's an effective proxy for fast-but-expensive memory. Also files, with their file pointers, can certainly be used as data structures, but this 'understanding' of what each section means is clearly variable, so is best left to specific applications and it's not good to build into the system as a default. Different applications will want different structures for optimal performance... [1] https://www.amazon.com/Design-UNIX-Operating-System/dp/0132017997 https://www.amazon.com/Design-UNIX-Operating-System/dp/01320... [2] https://people.eecs.berkeley.edu/~brewer/cs262/FFS.pdf https://people.eecs.berkeley.edu/~brewer/cs262/FFS.pdf [3] https://people.eecs.berkeley.edu/~brewer/cs262/LFS.pdf https://people.eecs.berkeley.edu/~brewer/cs262/LFS.pdf

snazz 6y ago

Unix pipes are great, but having to parse plain-text streams is inefficient and error-prone. Imagine how much more powerful it would be if commands returned data in structures that could be used by other applications without parsing an ad-hoc text format. The filesystem can be used to approximate this, like it is in Plan 9, but I think that using text as the lowest common denominator leaves a lot of potential power on the table.

zozbot234 6y ago
Unix-like pipes are not restricted to sending text, they may send general bytestreams. Modern data serialization formats (flatbuffers, capnproto, msgpack, etc.) are highly efficient, and using serialization makes it easier to deploy applications in a distributed setting.
- snazz 6y ago
  Sure, but the standard GNU or BSD userland doesn't do that, does it? If you're going to the trouble of implementing your own userspace, then you might as well use a consistent, proper data serialization format (either text or binary, but text is more accessible).
- throwaway894345 6y ago
  The value is in having everything speak a common standard. In unix, that 'standard' is lines of text, which is inherently linear and lacks the ability to idiomatically represent hierarchy. Conceivably, one could use modern serialization formats or we could do something a bit more interesting, like supporting some notion of "objects" directly in the operating system or runtime (in the case of powershell, the "runtime" is the .net vm; in unix it's the OS).
  - brutt 6y ago
    Format: * Field type: S - 16 bytes, D - 32, L - 64, Q - 128, B - 256, P - 512, T - 1024. * Field ID - 3 bytes ASCII Letters, numbers, '_'; * Field-value separator - 2 bytes: ':' and field value terminator, e.g. ': ' or ':"' or ':[' or ':{' or ':(' ; * Field value 9 bytes ASCII for S, 27 for D, and so on, padded with ' '; * Field separator: 1 byte '\n'; SOBJ: X23456789\n SFN_:"Length" \n SFV_: 23 \n SEND: \n Such text can be parsed by both human and computer, because computer can just jump from field to field without parsing everything in between, like humans do.
d0mine 6y ago
You don't need to send text (but you should by default for 99%+ cases when the performance is good enough and use the advantages of a human-readable format) The simplicity of a pipe is a feature.
- zozbot234 6y ago
  Ultimately, these "text" formats are simply a matter of convention that makes some binary-formatted streams easier to document for humans' benefit. There's nothing wrong with a binary format if it's properly documented in some way, and modern serialization formats can be augmented with schemas and other features that help provide this.
  - laumars 6y ago
    Being able to easily read and edit them matters too (pipes are just files so one might choose that file to be a persistent file on disk). Having a none-text format would mean all your file editing tools would have to then understand this new binary format. So you’d have to rewrite every tool to manipulate that data as if it’s text. By which point you’ve effectively just reinvented ASCII.
travbrack 6y ago
Powershell has been growing on me lately. It uses objects to pass data around and it's quite nice.
- AnIdiotOnTheNet 6y ago
  Using powershell for a month or so and then going back to doing things with text-parsing UNIX commands really highlights how much better things could be if only we moved past doing things the same way they were done in the 70s.
  - chacha2 6y ago
    Have you checked out NuShell?
- strbean 6y ago
  I'm really sad I missed the boat so hard on Powershell. I grabbed it at the first public release (beta? alpha?), and the thing was absurdly slow. Trying to `ls` took around 10 seconds. I nope'd out and didn't touch it again. Now I have a coworker who uses it religiously (on linux, who would have thought!) and when I'm watching over his shoulder I'm astounded by how great it is.
  - throwaway894345 6y ago
    How do people use environment variables with powershell? I often want to do `FOO=bar ./program`, but this doesn't seem to be supported. I googled briefly, but there weren't many satisfactory answers.
    - majkinetor 6y ago
      $Env:FOO='bar'; ./program
      - brutt 6y ago
        It's equivalent of `export FOO='bar'; ./program` , which is not same as `FOO='bar' ./program`, which, in turn, is equivalent to `OLD_FOO="$FOO"; export FOO='bar'; ./program; export FOO="$OLD_FOO";` .
riazrizvi 6y ago
Interesting idea, might be important to brush up on the implementation rationale of Unix. To that end there is The Design of The Unix Operating System, Bach [1]. Or A Fast File System for Unix [2]. Or The Design and Implementation of a Log-Structured File System [3]. It seems to me that files solve for the problem of how to deliver an economic memory management system, and they do it by paging cheap disk space so that it's an effective proxy for fast-but-expensive memory. Also files, with their file pointers, can certainly be used as data structures, but this 'understanding' of what each section means is clearly variable, so is best left to specific applications and it's not good to build into the system as a default. Different applications will want different structures for optimal performance... [1] https://www.amazon.com/Design-UNIX-Operating-System/dp/0132017997 https://www.amazon.com/Design-UNIX-Operating-System/dp/01320... [2] https://people.eecs.berkeley.edu/~brewer/cs262/FFS.pdf https://people.eecs.berkeley.edu/~brewer/cs262/FFS.pdf [3] https://people.eecs.berkeley.edu/~brewer/cs262/LFS.pdf https://people.eecs.berkeley.edu/~brewer/cs262/LFS.pdf