7 ms·

I didn't realise I needed this until I saw this post. Now I'm wondering why this isn't the standard already for every command line out there. Seems like a no br

I didn't realise I needed this until I saw this post. Now I'm wondering why this isn't the standard already for every command line out there. Seems like a no brainer improvment with non to few downsides.

hnlmorg 6y ago
I've been working on a way to implement this into my own shell (murex). The shell already has a builtin support for known "safe" commands, that is commands that are generally safe for background execution. The reason murex has this feature is because it supports executing the pipeline to generate more relevant autocompletion suggestions (eg when picking out nodes from a JSON or YAML file). Thus implementing an `up` like feature should be a pretty easy addition....however I ran into a bug that stalled the development of said feature and it's been in that stalled state for the last ~2 years. The reason development stalled for so long is because personally I'm not sure this is something that should be widely supported. Even with murex, which has an internal catalogue of known safe(ish) commands, the dangers of tools like this is pretty high. Running this against a regular shell which doesn't even do that seems like potential suicide.
- rocqua 6y ago
  How about spinning up a container with the entire host file-system mounted in a Copy-on-Write configuration. This way you can trash the container as much as you want without affecting anything on the outside. Combine it with syscall logging and you might actually have a great tool for figuring out exactly how a command is affecting your system.
  - hnlmorg 6y ago
    That'll certainly help but you'd still have issues with any connected devices (USB, network API alls, etc). I don't think it is a solvable problem though. There is no way to know ahead of time which usage is safe and which is destructive because of the variety of UNIX-like configurations out there, variety of coreutils and variety of optional flags. Not to mention all the 3rd party tools out there. And some seemingly destructive usages are vital for safe usage (eg tmp files). Worse still, by the time you build a safe container you've have spent more time spinning up a sandbox than you've had spent using the CLI preview tools anyways (yeah ZFS could help here but few people run ZFS on their dev machines and you still haven't solved mocking APIs for connected devices). I just can't see how this can be achieved safely without completely redesigning how SYSCALLs work from the ground up to follow a method more akin to iOS or FireFox's on-demand permissions. At which point you've gone well beyond the realm of "yak shaving"
    - rocqua 6y ago
      Network API calls seems like a scary one. But for things like USB and sys-calls, I'd suggest a "block and log" method. Like "running this command shows this output (with some extra error messages" and tried to delete these files and do these system calls.
      - hnlmorg 6y ago
        But my point is even the safest of coreutils will make dozens of SYSCALLs. The moment you start pipelining commands there will be several writes since that's how pipelining works and if you break even one of those writes the entire pipeline fails and the usefulness of this pipeline preview utility falls flat. There isn't a way you can make this utility safe and useful. You can make it safe, but then you're drifting into a whole new field of computing with regards to analysing suspicious binaries. Unfortunately you then don't have reliable, accurate, real time pipeline previews. Or you can have the real time previews but you then have to accept there is some risk involved. But you can't have it both safe and risk free. This is why murex took the approach of having a safe list of trusted executables. It doesn't remove the risk but it at least reduces the risk to a subset of commands that are typically read only. However even that is far from a perfect solution.
        rocqua 6y ago
        Thinking about it more, the kind of guarantees you want are the kind given by a VM for running untrusted binaries. The only difference is that you don't mind read-access to the host system. Yes, there will be certain things that involve system-calls or network activity that your previewer will not accurately show. But those cases fall far outside the goal of "live preview of text output of data processing pipelines".
        hnlmorg 6y ago
        No they don’t. Believe it or not a lot of people make network API calls from the command line. I know this because I’m one of them. I’ve even published guides I’m on how to make said API calls and parse them all as one command pipeline.

hnlmorg 6y ago

I've been working on a way to implement this into my own shell (murex). The shell already has a builtin support for known "safe" commands, that is commands that are generally safe for background execution. The reason murex has this feature is because it supports executing the pipeline to generate more relevant autocompletion suggestions (eg when picking out nodes from a JSON or YAML file). Thus implementing an `up` like feature should be a pretty easy addition....however I ran into a bug that stalled the development of said feature and it's been in that stalled state for the last ~2 years. The reason development stalled for so long is because personally I'm not sure this is something that should be widely supported. Even with murex, which has an internal catalogue of known safe(ish) commands, the dangers of tools like this is pretty high. Running this against a regular shell which doesn't even do that seems like potential suicide.

rocqua 6y ago
How about spinning up a container with the entire host file-system mounted in a Copy-on-Write configuration. This way you can trash the container as much as you want without affecting anything on the outside. Combine it with syscall logging and you might actually have a great tool for figuring out exactly how a command is affecting your system.
- hnlmorg 6y ago
  That'll certainly help but you'd still have issues with any connected devices (USB, network API alls, etc). I don't think it is a solvable problem though. There is no way to know ahead of time which usage is safe and which is destructive because of the variety of UNIX-like configurations out there, variety of coreutils and variety of optional flags. Not to mention all the 3rd party tools out there. And some seemingly destructive usages are vital for safe usage (eg tmp files). Worse still, by the time you build a safe container you've have spent more time spinning up a sandbox than you've had spent using the CLI preview tools anyways (yeah ZFS could help here but few people run ZFS on their dev machines and you still haven't solved mocking APIs for connected devices). I just can't see how this can be achieved safely without completely redesigning how SYSCALLs work from the ground up to follow a method more akin to iOS or FireFox's on-demand permissions. At which point you've gone well beyond the realm of "yak shaving"
  - rocqua 6y ago
    Network API calls seems like a scary one. But for things like USB and sys-calls, I'd suggest a "block and log" method. Like "running this command shows this output (with some extra error messages" and tried to delete these files and do these system calls.
    - hnlmorg 6y ago
      But my point is even the safest of coreutils will make dozens of SYSCALLs. The moment you start pipelining commands there will be several writes since that's how pipelining works and if you break even one of those writes the entire pipeline fails and the usefulness of this pipeline preview utility falls flat. There isn't a way you can make this utility safe and useful. You can make it safe, but then you're drifting into a whole new field of computing with regards to analysing suspicious binaries. Unfortunately you then don't have reliable, accurate, real time pipeline previews. Or you can have the real time previews but you then have to accept there is some risk involved. But you can't have it both safe and risk free. This is why murex took the approach of having a safe list of trusted executables. It doesn't remove the risk but it at least reduces the risk to a subset of commands that are typically read only. However even that is far from a perfect solution.
      - rocqua 6y ago
        Thinking about it more, the kind of guarantees you want are the kind given by a VM for running untrusted binaries. The only difference is that you don't mind read-access to the host system. Yes, there will be certain things that involve system-calls or network activity that your previewer will not accurately show. But those cases fall far outside the goal of "live preview of text output of data processing pipelines".
        hnlmorg 6y ago
        No they don’t. Believe it or not a lot of people make network API calls from the command line. I know this because I’m one of them. I’ve even published guides I’m on how to make said API calls and parse them all as one command pipeline.