4 ms·

> Null separators between fields is structured data. It might just be the simplest that can possibly work (because the null character is outside the normal data

> Null separators between fields is structured data. It might just be the simplest that can possibly work (because the null character is outside the normal data range for text, unlike the newline, which demonstrably doesn't work in that sense) but it is structure. A list structure, specifically. Sometimes I daydream about a parallel universe in which the designers of Unix decided on record-oriented IO instead of stream-oriented IO. If pipes were defined in terms of records as opposed to an unstructured byte stream, there'd be no need for a special character (whether newline or null) to separate records. How is in-band signalling in pipes any better than in-band signalling in telecommunications?

b112 3y ago
Sometimes I daydream about a parallel universe in which the designers of Unix decided on record-oriented IO instead of stream-oriented IO. I would attack you with tea bags and waxed paper, if you tried to alter the timeline, and make this so. On the other hand, I would happily sing your praises, if you invented another kind of pipe, to your specs, it sounds like a great additional. Imagine combining the two, in one series of commands!
- skissane 3y ago
  > On the other hand, I would happily sing your praises, if you invented another kind of pipe, to your specs, it sounds like a great additional Already exists - Unix domain sockets. Some shells (in particular some versions of ksh) use them to implement pipes. And, on some platforms (Linux yes, but I think maybe not macOS???) Unix domain sockets support record-oriented operation (SOCK_SEQPACKET). The problem is that for it to work you don’t just need the kernel to support it and the shell to use it, you also need all the utilities to support it too-that’s a big ask. The idea has been implemented on IBM mainframes (CMS pipelines aka Hartmann pipelines). But that’s a radically different platform, and IBM has never tried porting that to a non-mainframe platform, and nobody has ever sought to directly clone it (although stuff like PowerShell and NuShell share some of its ideas, albeit none of the details)
  - b112 3y ago
    Hmm. Shame it never caught on in the OSS/GNU world. Thanks for the FYI.
euroderf 3y ago
What if most standard Unix commands had a new option to insert into their output those nifty separators ? (FS, GS, RS, US) That ought to make life simpler for downstream commands that try to go beyond processing a text stream.
- skissane 3y ago
  > What if most standard Unix commands had a new option to insert into their output those nifty separators ? (FS, GS, RS, US) That's not what I mean by record-oriented IO though. That's signalling record boundaries in-band. I'm talking about signalling them out-of-band. So you have record lengths kept separately from the data, and passed around by APIs separately from the data bytes.
  - euroderf 3y ago
    I've had the same idea. It would (I guess) rewriting the kernel from the ground up to turn every stream into two streams ? Telecoms uses both concepts (in-band signaling and out-of-band signaling) but I guess the Unix developers didn't get the memo ?
    - skissane 3y ago
      > I've had the same idea. It would (I guess) rewriting the kernel from the ground up to turn every stream into two streams ? Many (but not all) Unix implementations already have support for “record-oriented pipes” in the kernel, it is just that support has rarely been used. Linux has record-oriented Unix domain sockets (SOCK_SEQPACKET), but few use them. > Telecoms uses both concepts (in-band signaling and out-of-band signaling) but I guess the Unix developers didn't get the memo ? The original Bell Labs Unix team created STREAMS, which does support true record-oriented IO. Most commercial Unix implementations (such as Solaris and AIX) support it (or at least did at one point). But Berkeley Sockets won the mindshare competition, and open source Unix-likes (such as Linux and BSDs) never added streams support. There was a project to add it to Linux, but Linus was opposed to the idea, so it never got merged, and I think it has since been abandoned. Even before STREAMS, Unix supported record-oriented IO in the terminal subsystem (cooked mode). But it wasn’t general, it was very specific to the needs of interactive use. STREAMS was intended as a generalisation but it never caught on. So even today all Unix-likes have a purpose-specific (rather than general) record-IO implementation in their tty subsystems, pseudoterminals, etc

b112 3y ago

Sometimes I daydream about a parallel universe in which the designers of Unix decided on record-oriented IO instead of stream-oriented IO. I would attack you with tea bags and waxed paper, if you tried to alter the timeline, and make this so. On the other hand, I would happily sing your praises, if you invented another kind of pipe, to your specs, it sounds like a great additional. Imagine combining the two, in one series of commands!

skissane 3y ago
> On the other hand, I would happily sing your praises, if you invented another kind of pipe, to your specs, it sounds like a great additional Already exists - Unix domain sockets. Some shells (in particular some versions of ksh) use them to implement pipes. And, on some platforms (Linux yes, but I think maybe not macOS???) Unix domain sockets support record-oriented operation (SOCK_SEQPACKET). The problem is that for it to work you don’t just need the kernel to support it and the shell to use it, you also need all the utilities to support it too-that’s a big ask. The idea has been implemented on IBM mainframes (CMS pipelines aka Hartmann pipelines). But that’s a radically different platform, and IBM has never tried porting that to a non-mainframe platform, and nobody has ever sought to directly clone it (although stuff like PowerShell and NuShell share some of its ideas, albeit none of the details)
- b112 3y ago
  Hmm. Shame it never caught on in the OSS/GNU world. Thanks for the FYI.

euroderf 3y ago

What if most standard Unix commands had a new option to insert into their output those nifty separators ? (FS, GS, RS, US) That ought to make life simpler for downstream commands that try to go beyond processing a text stream.

skissane 3y ago
> What if most standard Unix commands had a new option to insert into their output those nifty separators ? (FS, GS, RS, US) That's not what I mean by record-oriented IO though. That's signalling record boundaries in-band. I'm talking about signalling them out-of-band. So you have record lengths kept separately from the data, and passed around by APIs separately from the data bytes.
- euroderf 3y ago
  I've had the same idea. It would (I guess) rewriting the kernel from the ground up to turn every stream into two streams ? Telecoms uses both concepts (in-band signaling and out-of-band signaling) but I guess the Unix developers didn't get the memo ?
  - skissane 3y ago
    > I've had the same idea. It would (I guess) rewriting the kernel from the ground up to turn every stream into two streams ? Many (but not all) Unix implementations already have support for “record-oriented pipes” in the kernel, it is just that support has rarely been used. Linux has record-oriented Unix domain sockets (SOCK_SEQPACKET), but few use them. > Telecoms uses both concepts (in-band signaling and out-of-band signaling) but I guess the Unix developers didn't get the memo ? The original Bell Labs Unix team created STREAMS, which does support true record-oriented IO. Most commercial Unix implementations (such as Solaris and AIX) support it (or at least did at one point). But Berkeley Sockets won the mindshare competition, and open source Unix-likes (such as Linux and BSDs) never added streams support. There was a project to add it to Linux, but Linus was opposed to the idea, so it never got merged, and I think it has since been abandoned. Even before STREAMS, Unix supported record-oriented IO in the terminal subsystem (cooked mode). But it wasn’t general, it was very specific to the needs of interactive use. STREAMS was intended as a generalisation but it never caught on. So even today all Unix-likes have a purpose-specific (rather than general) record-IO implementation in their tty subsystems, pseudoterminals, etc