7 ms·

One thing I've been thinking of: how about making typed file descriptors? Besides stdio and stdout, one of the best-accepted conventions on modern operating sy

One thing I've been thinking of: how about making typed file descriptors? Besides stdio and stdout, one of the best-accepted conventions on modern operating systems running a graphical shell seems to be the clipboard. Programs can offer different mime types trough a clipboard, so why not do so trough a pipe or file descriptor? That would bring us a bit closer to powershell, while remaining backwards-compatible. cat pipe.dot <- text/plain | dot <- text/plain, text/dot, image/svg+xml, application/pdf, application/postscript, etc | xclip Instead of xclip in the above example, one could use an image viewer, a multiplexer a bit like tee (1) but to redirect to standard input, etc. One could force a pipe to filter out some mimetypes, or use a program dedicated to that. One could imagine bidirectional communications over a channel. xeyes | ssh somehost record-audio.sh | mail someone@somewhere nmtui | guitoolkit rsync | guitoolkit ssh somehost rsync | guitoolkit Basically, building on the strengths of UNIX.

TeMPOraL 6y ago
Come to think of it, you could abuse NTFS Alternate Data Streams for this. Imagine: > # See alternate data streams > Get-Item diagram.dot -Stream * Stream Length ------ ------ :$DATA 123 text/plain 4444 image/svg+xml 65432 image/png 456134 > # Send the whole bulk > Get-Item diagram.dot -Stream * | ... > #... or pick just one: > Get-Content diagram.dot -Stream "image/svg+xml" | Send-Email .... Would have to ensure everything is aware of the streams and the convention, though, so that you wouldn't accidentally truncate them when copying files around.
- MayeulC 6y ago
  Ah, that's interesting, thanks for mentioning it. That's pretty much what I mean, but on arbitrary file descriptors instead of files. I agree that both ends need to be aware of these extra formats to make use of them, but just like with clipboards, programs could always fall back to pretending only text/plain exists :)
  - HelloNurse 6y ago
    Typing, in the programming language sense, is a major can of worms that a general purpose shell usually avoids with a deliberately bare bones type system (e.g. text, binary, directory and symlink). How can a shell do anything serious with asserted file types that can be missing, wrong, malformed, unknown or mendacious? It's hard to go beyond offering a best-effort "open with the associated program" application as an alternative to vi, cat etc.
    - MayeulC 6y ago
      > How can a shell do anything serious with asserted file types that can be missing, wrong, malformed, unknown or mendacious? It's hard to go beyond offering a best-effort "open with the associated program" application as an alternative to vi, cat etc. How would that be any worse than the status quo, though? Programs already output, depending on their invocation and the utility at hand, binary data in arbitrary formats, shell escape sequences, text, ASCII art, etc. The only issue I can think of is for portability of the scripts, if a program gets updated to support a new type as input or output, and that causes issues compared to the testing environment. But that's arguably already an issue too, and it would be rather simple to annotate expected types.
      - HelloNurse 6y ago
        The cost of specifying MIME types or the like can easily exceed its value, which is very low because we are already specifying file types (to applications that can deal with more than one) well enough with file extensions (undependable, but vastly more convenient than new infrastructure for new metadata), with self-describing or recognizable file formats, and often explicitly.
- monocasa 6y ago
  Not just NTFS, but pre OSX Macs would do stuff like this. Like a formatted document would have a text/plain fork, and the formatting information in another fork. Or an executable could keep it's symbol table or relocation information in other forks.
  - TeMPOraL 6y ago
    Thanks! I just checked, and it seems that ADS were introduced in NTFS in order to support Mac resource forks. https://en.wikipedia.org/wiki/NTFS#Alternate_data_stream_(ADS) https://en.wikipedia.org/wiki/NTFS#Alternate_data_stream_(AD...
  - watersb 6y ago
    You can muck around alternate data streams like a classic resource fork of a file on macOS by appending /..namedfork/rsrc to the file name. ls ./groovyFontFile/..namedfork/rsrc https://stackoverflow.com/a/42216808/8878480 https://stackoverflow.com/a/42216808/8878480 I've tried it with ancient font suitcases -- the ones that pre-date dfont (data fork only) fonts. It's somewhat ironic that all desktop file systems these days support myriad data streams on a file. I've implemented some on NTFS, played with extended attributes across tar archives between Linux and Mac and Windows. It's all there, but no longer used.
hnlmorg 6y ago
This is something I've been working on with my shell, https://github.com/lmorg/murex https://github.com/lmorg/murex The problem isn't so much finding a way of passing metadata but how do you retain compatibility with existing GNU / UNIX tools? It would be easy to create an entirely new shell (like Powershell) that throws out backwards compatibility but that wouldn't be very practical on Linux/UNIX when so much of it is powered by old POSIX standards. I addressed this by having a wrapper around pipes. Anything that connects out to a classic executable is a traditional POSIX byte stream with no type data sent. Anything sent to a command that is written to support my shell can make use of said meta data. As for handling graphical components, that's a little easier. Many command line programs already alter their behaviour depending on whether STDOUT is a TTY or not. I take things a little further and have a media handler function, called `open`, that inlines stuff in a shell that can be (it will render images in the terminal) but if STDOUT is a TTY then it will pipe the image data instead.
- MayeulC 6y ago
  Wow, your shell seems to be pretty far along. > The problem isn't so much finding a way of passing metadata but how do you retain compatibility with existing GNU / UNIX tools Honestly? I'd have suggested a new syscall that provides an interface similar to isatty(). It seems a matter of adding a few constants to the ioctl syscall, prototype it and prove it is useful, then upstream it. Better make a crude prototype, see if that's any useful, and ask upstream what they think of it. If the new ioctl isn't available, nothing changes, programs continue exchanging whatever they expect. If the new interface is available, most programs won't know about it, and would continue their business as usual. However, programs that know about it could get more metadata trough it, and communicate in a smarter way. Looks like a standard way of introducing new functionality, can be deprecated if needed, and keeps backwards-compatibility, at the expense of a tiny bit of complexity (dwarfed by having to support multiple formats, which some tools already do). The way you do it seems quite fine too, not sure which would be better. Staying with traditional file descriptors also has the issues that you are limited to unidirectional communication, and only one file descriptor, so maybe you need a protocol like html, as a sibling joked. > As for handling graphical components, that's a little easier. Many command line programs already alter their behaviour depending on whether STDOUT is a TTY or not. Well, that's kind of my point: you could output more than curses interface if you wanted, something like an xml ui file, or a list of commands on a side channel. But I make no judgement on this: it was just to illustrate what kind use-cases this would enable.
sime2009 6y ago
A possible way of doing what you describe might be to add an extra file descriptor or two when invoking processes. Along side the normal stdin, stdout, and stderr, you might have a couple more where a protocol can be used to negotiate and share typed data. It might even be possible to do it a backwards compatible way. (If in an app the extra file descriptors are missing, then fall back to old school stdin/stdout.) This blog from years ago describes something similar: http://acko.net/blog/on-termkit/ http://acko.net/blog/on-termkit/ i.e. expanding the process model
- MayeulC 6y ago
  Thank you for the blog post, it resonates quite well with me. I thought a bit about this, but that becomes a bit complicated without additional metadata on a file descriptor (is it really that additional interface, or just a random FD we inherited?). It might be possible to achieve in a robust way with mount namespaces and named pipes, but that's another can of worms... That said, it would likely work 90% of the time that way.
  - sime2009 6y ago
    I pay attention to things in this area as I've been working a terminal which aims to do new and interesting things with text and also non-text datatypes. See "Shell integration" and below at https://extraterm.org/features.html https://extraterm.org/features.html Integrating some new shell projects like murex and nushell with Extraterm is on my TODO list.
  - paulfurtado 6y ago
    I'd imagine that in a shell context, most of your FDs would be closed on exec unless explicitly passed. That said, why not use env vars which tell the child process what to expect on which file descriptor numbers? Depending how crazy you want to get, you could also use unix sockets to pass around FDs instead of inheritance. In that scenario, a process would be passed a unix socket either by a socket pair file descriptor or the shell would have a named socket which it tells child processes about using an env var. You can then use the sendfd functionality of unix sockets to pass around arbitrary file descriptors with arbitrary metadata. You could also use the unix socket to instead have the child process optionally ask the shell what the descriptor's metadata is. That way you'd keep backwards compatibility with unix tools, but the child process could sendfd the file descriptor back to its parent to ask for data about it. The parent could look it up by calling fstat on it and comparing the fstat data to the other file descriptors it sent to the child. But regardless, when you can't add a new syscall and need to do file descriptor stuff on behalf of another process, unix sockets + sendfd can really start to look like a userspace syscall interface which you can do a lot with.
Someone 6y ago
The clipboard nowadays is populated lazily (you do not want that dot call to generate Postscript, svg, pdf, png, gif, and whatever else it can generate while, in the end, you’ll need only one of them), so you would have to keep those processes running until you know nobody needs the data anymore. That would be quite a change from the traditional pipe model.
- MayeulC 6y ago
  Oh, but the traditional pipe model can also produce data lazily (wait until a given FD is open before producing output), so that wouldn't be that much of an issue. That wouldn't change much from the pipe model. At least on wayland, wl-copy forks to the background. If you wanted the producer to finish producing data and exit, surely you could make a flag for it in the receiving program, or pipe it trough one that does.
  - Someone 6y ago
    This is different. You need communication back through the pipeline to inform the data producers what data you want. Also, in your example ending in | xclip, xclip cannot know what format data it has to ask from dot until it gets pasted somewhere. Alternatively, it would either have to guess what format(s) the user would like to paste later or it would have to cover all bases and ask dot for output in every format it supports.
jodrellblank 6y ago
Why not simply make the pipe explicitly HTTP? first | second | third "first" starts up a HTTP server, "second" is a HTTP client which makes a standard HTTP request to "first" declaring in the headers what content types it can support, and "first" returns a standard HTTP response declaring in the headers what kind of content it is sending. "Second" is also a HTTP server for "third" in the same way, and so on down the pipeline. All the OS has to do is coordinate a free TCP port for each pipe. The HTTP client can simply be a lightweight headless Chrome Electron bundle, and the whole pipeline gets all the benefits of HTTP (standard UTF-8 encoding, compression), with only a small overhead of cookie policies, cross pipe scripting, modern auth tokens, 2-factor authentication between client and server, SAML, and a tiny surface area increase for security concerns but only a few dozen MB or so. Plus the pipeline then has ongoing two-way communication up and down it.
- karatinversion 6y ago
  This is awful, I love it.
- devenblake 6y ago
  This is a dangerous joke to make / idea to perpetuate given the audience of HN.
  - TeMPOraL 6y ago
    Agreed. I'm tempted to simultaneously upvote and flag this idea :).
- MayeulC 6y ago
  The idea isn't that bad. Obviously the following can't be taken at face value: > lightweight headless Chrome Electron [...] with only a small overhead of cookie policies, cross pipe scripting, modern auth tokens, 2-factor authentication between client and server, SAML, and a tiny surface area increase for security concerns but only a few dozen MB or so. However, there's probably much to be learned from the history of the web when it comes to security and compatibility between different implementations and successive evolutions of a protocol. http certainly has merits, given its long history. It isn't flawless either, of course. Even features such as compression could make sense at first glance, though they would be a better fit for a dedicated tool like piping to zstd, since you control both ends of a pipe. Range requests, multiplexing, server-sent events, etc, could have some use. And it isn't like making a http server or client is so hard with a small library, given you already pull in libc or worse. http probably wouldn't be suitable as-is, certainly not over TCP (though the fopen/fclose dance is quite close to it), but making it a fully fledged bidirectional protocol could be interesting, especially if you can somehow build on top of the system IPC on a microkernel. It could be a way to forward hardware access and such.

TeMPOraL 6y ago

Come to think of it, you could abuse NTFS Alternate Data Streams for this. Imagine: > # See alternate data streams > Get-Item diagram.dot -Stream * Stream Length ------ ------ :$DATA 123 text/plain 4444 image/svg+xml 65432 image/png 456134 > # Send the whole bulk > Get-Item diagram.dot -Stream * | ... > #... or pick just one: > Get-Content diagram.dot -Stream "image/svg+xml" | Send-Email .... Would have to ensure everything is aware of the streams and the convention, though, so that you wouldn't accidentally truncate them when copying files around.

MayeulC 6y ago
Ah, that's interesting, thanks for mentioning it. That's pretty much what I mean, but on arbitrary file descriptors instead of files. I agree that both ends need to be aware of these extra formats to make use of them, but just like with clipboards, programs could always fall back to pretending only text/plain exists :)
- HelloNurse 6y ago
  Typing, in the programming language sense, is a major can of worms that a general purpose shell usually avoids with a deliberately bare bones type system (e.g. text, binary, directory and symlink). How can a shell do anything serious with asserted file types that can be missing, wrong, malformed, unknown or mendacious? It's hard to go beyond offering a best-effort "open with the associated program" application as an alternative to vi, cat etc.
  - MayeulC 6y ago
    > How can a shell do anything serious with asserted file types that can be missing, wrong, malformed, unknown or mendacious? It's hard to go beyond offering a best-effort "open with the associated program" application as an alternative to vi, cat etc. How would that be any worse than the status quo, though? Programs already output, depending on their invocation and the utility at hand, binary data in arbitrary formats, shell escape sequences, text, ASCII art, etc. The only issue I can think of is for portability of the scripts, if a program gets updated to support a new type as input or output, and that causes issues compared to the testing environment. But that's arguably already an issue too, and it would be rather simple to annotate expected types.
    - HelloNurse 6y ago
      The cost of specifying MIME types or the like can easily exceed its value, which is very low because we are already specifying file types (to applications that can deal with more than one) well enough with file extensions (undependable, but vastly more convenient than new infrastructure for new metadata), with self-describing or recognizable file formats, and often explicitly.
monocasa 6y ago
Not just NTFS, but pre OSX Macs would do stuff like this. Like a formatted document would have a text/plain fork, and the formatting information in another fork. Or an executable could keep it's symbol table or relocation information in other forks.
- TeMPOraL 6y ago
  Thanks! I just checked, and it seems that ADS were introduced in NTFS in order to support Mac resource forks. https://en.wikipedia.org/wiki/NTFS#Alternate_data_stream_(ADS) https://en.wikipedia.org/wiki/NTFS#Alternate_data_stream_(AD...
- watersb 6y ago
  You can muck around alternate data streams like a classic resource fork of a file on macOS by appending /..namedfork/rsrc to the file name. ls ./groovyFontFile/..namedfork/rsrc https://stackoverflow.com/a/42216808/8878480 https://stackoverflow.com/a/42216808/8878480 I've tried it with ancient font suitcases -- the ones that pre-date dfont (data fork only) fonts. It's somewhat ironic that all desktop file systems these days support myriad data streams on a file. I've implemented some on NTFS, played with extended attributes across tar archives between Linux and Mac and Windows. It's all there, but no longer used.

hnlmorg 6y ago

This is something I've been working on with my shell, https://github.com/lmorg/murex https://github.com/lmorg/murex The problem isn't so much finding a way of passing metadata but how do you retain compatibility with existing GNU / UNIX tools? It would be easy to create an entirely new shell (like Powershell) that throws out backwards compatibility but that wouldn't be very practical on Linux/UNIX when so much of it is powered by old POSIX standards. I addressed this by having a wrapper around pipes. Anything that connects out to a classic executable is a traditional POSIX byte stream with no type data sent. Anything sent to a command that is written to support my shell can make use of said meta data. As for handling graphical components, that's a little easier. Many command line programs already alter their behaviour depending on whether STDOUT is a TTY or not. I take things a little further and have a media handler function, called `open`, that inlines stuff in a shell that can be (it will render images in the terminal) but if STDOUT is a TTY then it will pipe the image data instead.

MayeulC 6y ago
Wow, your shell seems to be pretty far along. > The problem isn't so much finding a way of passing metadata but how do you retain compatibility with existing GNU / UNIX tools Honestly? I'd have suggested a new syscall that provides an interface similar to isatty(). It seems a matter of adding a few constants to the ioctl syscall, prototype it and prove it is useful, then upstream it. Better make a crude prototype, see if that's any useful, and ask upstream what they think of it. If the new ioctl isn't available, nothing changes, programs continue exchanging whatever they expect. If the new interface is available, most programs won't know about it, and would continue their business as usual. However, programs that know about it could get more metadata trough it, and communicate in a smarter way. Looks like a standard way of introducing new functionality, can be deprecated if needed, and keeps backwards-compatibility, at the expense of a tiny bit of complexity (dwarfed by having to support multiple formats, which some tools already do). The way you do it seems quite fine too, not sure which would be better. Staying with traditional file descriptors also has the issues that you are limited to unidirectional communication, and only one file descriptor, so maybe you need a protocol like html, as a sibling joked. > As for handling graphical components, that's a little easier. Many command line programs already alter their behaviour depending on whether STDOUT is a TTY or not. Well, that's kind of my point: you could output more than curses interface if you wanted, something like an xml ui file, or a list of commands on a side channel. But I make no judgement on this: it was just to illustrate what kind use-cases this would enable.

sime2009 6y ago

A possible way of doing what you describe might be to add an extra file descriptor or two when invoking processes. Along side the normal stdin, stdout, and stderr, you might have a couple more where a protocol can be used to negotiate and share typed data. It might even be possible to do it a backwards compatible way. (If in an app the extra file descriptors are missing, then fall back to old school stdin/stdout.) This blog from years ago describes something similar: http://acko.net/blog/on-termkit/ http://acko.net/blog/on-termkit/ i.e. expanding the process model

MayeulC 6y ago
Thank you for the blog post, it resonates quite well with me. I thought a bit about this, but that becomes a bit complicated without additional metadata on a file descriptor (is it really that additional interface, or just a random FD we inherited?). It might be possible to achieve in a robust way with mount namespaces and named pipes, but that's another can of worms... That said, it would likely work 90% of the time that way.
- sime2009 6y ago
  I pay attention to things in this area as I've been working a terminal which aims to do new and interesting things with text and also non-text datatypes. See "Shell integration" and below at https://extraterm.org/features.html https://extraterm.org/features.html Integrating some new shell projects like murex and nushell with Extraterm is on my TODO list.
- paulfurtado 6y ago
  I'd imagine that in a shell context, most of your FDs would be closed on exec unless explicitly passed. That said, why not use env vars which tell the child process what to expect on which file descriptor numbers? Depending how crazy you want to get, you could also use unix sockets to pass around FDs instead of inheritance. In that scenario, a process would be passed a unix socket either by a socket pair file descriptor or the shell would have a named socket which it tells child processes about using an env var. You can then use the sendfd functionality of unix sockets to pass around arbitrary file descriptors with arbitrary metadata. You could also use the unix socket to instead have the child process optionally ask the shell what the descriptor's metadata is. That way you'd keep backwards compatibility with unix tools, but the child process could sendfd the file descriptor back to its parent to ask for data about it. The parent could look it up by calling fstat on it and comparing the fstat data to the other file descriptors it sent to the child. But regardless, when you can't add a new syscall and need to do file descriptor stuff on behalf of another process, unix sockets + sendfd can really start to look like a userspace syscall interface which you can do a lot with.

Someone 6y ago

The clipboard nowadays is populated lazily (you do not want that dot call to generate Postscript, svg, pdf, png, gif, and whatever else it can generate while, in the end, you’ll need only one of them), so you would have to keep those processes running until you know nobody needs the data anymore. That would be quite a change from the traditional pipe model.

MayeulC 6y ago
Oh, but the traditional pipe model can also produce data lazily (wait until a given FD is open before producing output), so that wouldn't be that much of an issue. That wouldn't change much from the pipe model. At least on wayland, wl-copy forks to the background. If you wanted the producer to finish producing data and exit, surely you could make a flag for it in the receiving program, or pipe it trough one that does.
- Someone 6y ago
  This is different. You need communication back through the pipeline to inform the data producers what data you want. Also, in your example ending in | xclip, xclip cannot know what format data it has to ask from dot until it gets pasted somewhere. Alternatively, it would either have to guess what format(s) the user would like to paste later or it would have to cover all bases and ask dot for output in every format it supports.

jodrellblank 6y ago

Why not simply make the pipe explicitly HTTP? first | second | third "first" starts up a HTTP server, "second" is a HTTP client which makes a standard HTTP request to "first" declaring in the headers what content types it can support, and "first" returns a standard HTTP response declaring in the headers what kind of content it is sending. "Second" is also a HTTP server for "third" in the same way, and so on down the pipeline. All the OS has to do is coordinate a free TCP port for each pipe. The HTTP client can simply be a lightweight headless Chrome Electron bundle, and the whole pipeline gets all the benefits of HTTP (standard UTF-8 encoding, compression), with only a small overhead of cookie policies, cross pipe scripting, modern auth tokens, 2-factor authentication between client and server, SAML, and a tiny surface area increase for security concerns but only a few dozen MB or so. Plus the pipeline then has ongoing two-way communication up and down it.

karatinversion 6y ago
This is awful, I love it.
devenblake 6y ago
This is a dangerous joke to make / idea to perpetuate given the audience of HN.
- TeMPOraL 6y ago
  Agreed. I'm tempted to simultaneously upvote and flag this idea :).
MayeulC 6y ago
The idea isn't that bad. Obviously the following can't be taken at face value: > lightweight headless Chrome Electron [...] with only a small overhead of cookie policies, cross pipe scripting, modern auth tokens, 2-factor authentication between client and server, SAML, and a tiny surface area increase for security concerns but only a few dozen MB or so. However, there's probably much to be learned from the history of the web when it comes to security and compatibility between different implementations and successive evolutions of a protocol. http certainly has merits, given its long history. It isn't flawless either, of course. Even features such as compression could make sense at first glance, though they would be a better fit for a dedicated tool like piping to zstd, since you control both ends of a pipe. Range requests, multiplexing, server-sent events, etc, could have some use. And it isn't like making a http server or client is so hard with a small library, given you already pull in libc or worse. http probably wouldn't be suitable as-is, certainly not over TCP (though the fopen/fclose dance is quite close to it), but making it a fully fledged bidirectional protocol could be interesting, especially if you can somehow build on top of the system IPC on a microkernel. It could be a way to forward hardware access and such.