6 ms·

I have a pretty good idea for a shell but I don't know when I'll ever find time to implement it. The idea is that the shell should be able to optimize pipeline

I have a pretty good idea for a shell but I don't know when I'll ever find time to implement it. The idea is that the shell should be able to optimize pipelines. Pipeline here meaning a chain of commands piped into one-another. So if you have a pipeline like grep ^2017-05-29 /var/log/somefile | grep -v 'INFO|WARN' | tail -n5 | cut -f1 -f3 Then the idea is that instead of piping the whole result from command to command, your shell would compile a set of commands into a single program and run that. Now you might say that that sounds like it goes against the Unix philosophy, but actually it doesn't need to. If all of the core utilities were implemented in such a way that their logic could be extracted without duplication of code then the shell can still be doing "one thing and one thing well". --- Another idea I have is to make these core utilities pipe objects instead of text, like on Windows. I am very fond of bash but I think one thing that Microsoft seems to have done right is to have the idea of being able to work on objects in PowerShell. But I don't want PowerShell. I want a mostly Unix shell, except, as I said, with objects. I just wish I had more time :(

geocar 9y ago
You might find running across four cores is actually faster than "compiling into a single program"
- glangdale 9y ago
  Hmmn. A 'smart enough' compiler would combine the two grep patterns into a single pattern and run them in reverse, stopping at the first 5 lines. :-) Even if we don't assume we have such astounding transformations, I think you're vastly overestimating the power of multithreading in this example. There's very little for each core to do, especially the final step (which is getting a whopping 5 lines to process). All that being said, I'm not sure that, if I wanted highly parallel text processing OR deeply clever compiler-style optimizations, I'd start from "existing Unix utilities and their associated cruft".
  - eriknstr 9y ago
    Not everyone understood what I meant to say but you did, thanks :) Starting from the Unix utilites is simply because I'm so used to them. I can assure you, however, that anyone who was under the impression that my system was actually Unix would be in for a nasty surprise :-> My system would be, shall we say, not POSIX compliant. My system would take the things I like about Unix, such as being command-line centric and having utilities focused on being good at small individual tasks and using pipes to glue them together seemingly (but actually as mentioned each pipeline is compiled to an optimized program when run), and the filesystem would remain essential and hierarchical. I would separate Operating System binary data, Operating System config files, Application binary data, Application config files, User config files, User downloaded data and User created data STRICTLY! Also like I mentioned, to have structured binary data instead of text. This would lead to commands which are named like in POSIX but which operate a bit differently with regards to the arguments they take. Probably a lot of other things as well that I've been thinking about but which escapes me at the moment. But yeah, all of that remains a dream because it would involve more work than I have time for :(
  - geocar 9y ago
    > Hmmn. A 'smart enough' compiler would combine the two grep patterns into a single pattern and run them in reverse, stopping at the first 5 lines. :-) Only if `/var/log/somefile` is a regular file and isn't opened for writing by any other process. Or your "smart enough compiler" doesn't care about doing the same thing.
iUsedToCode 9y ago
Yeah, but why? Pipes are NEVER a performance issue for me. Maybe you have some very tricky use cases, but for 99.95% of users it doesn't matter. Almost always you'll loose more time on I/O than on pipe processing.
laumars 9y ago
UNIX shells already optimise pipelines by running the commands along it concurrently. In your example, the first grep will be outputing it's results while it's still sucking in /var/log/somefile; and the second grep will be parsing the output of the first grep as and when the first grep outputs. tail is a little different because it needs to find the end of the file but the greps certainly don't wait for each other to finish. As for your "objects" point. The shell I'm working on (discussed elsewhere in this thread) uses JSON to pass objects around functions written for that shell while still falling back to standard flat text files when piping out to traditional UNIX/GNU/whatever tools. Though I still need to make the whole thing more intuitive.
- eriknstr 9y ago
  >UNIX shells already optimise pipelines by running the commands along it concurrently. I am aware of that. Still, performing multiple operations on the data within a single loop and with less copying would be an optimization.
  - laumars 9y ago
    I get what you're saying but technically those tools are running inside the same single loop (as that file is only iterated the once most of that pipeline) but distributed across multiple OS threads. Shell scripts are surprisingly efficient at pipelined work where you're just dealing with streaming data (as your example mostly does). It's the more complex logic they fall significantly short on. The weakest area and why I kept saying "most" of that pipeline would be the use of tail part way through the chain (as others have mentioned). But that could be optimised within Bash to something like: tail -r somefile | grep 'expression' | egrep 'expression' | head -n number | cut ... (I think -r flag on tail is GNU rather than POSIX though) You're object point is definitely relevant though. For example how many times have we seen shell scripts break because they don't consider spaces in filenames? Talking about annoyances in Bash I'd also throw in exception handling as a major problem for traditional shells. Having Bash fail in expected ways and handle them cleanly is a huge pain in the arse. Those above two points (and wanting cleaner syntax for iteration) are what drove me to write my own $SHELL so it does sound like I'm addressing some of your annoyances - albeit not all of them. However it's a long way from being something usable in production environments but I'm happy to access pull requests if you do ever feel you want to contribute to something that's trying to work towards at least some of your goals :)
vram22 9y ago
> cut -f1 -f3 Minor point: Can't you change that to cut -f1,3 ?
- eriknstr 9y ago
  Yes. I just forgot to :)
10165 9y ago
"grep ^2017-05-29 /var/log/somefile | grep -v 'INFO|WARN' | tail -n5 | cut -f1 -f3" Did you mean cut -d' ' -f1,3? Assuming delimiter is a space the above example can be reduced to: sed ' /^2017-05-29/!d; /INFO/d; /WARN/d; s/ /INFO/; s/ /INFO/; s/ .*//; s/INFO.*INFO/ /; ' /var/log/somefile \ |tail -n5 "... compile a set of commands into a single program and run that." Linux: see busybox ("multicall binary") BSD: see binaries on install media ("crunched binary") sed and tail are both compiled into busybox re: bsd compiling in tail with crunchgen is easy As for "objects", the k language can mmap the output of UNIX commands and run parallel computations on those results as "data" not "text". It can be faster than I/O.
- eriknstr 9y ago
  I intended for the delimiter to be tab in this example. Using sed or awk is an option, yes, but I am so used to the standard utilities that I would rather keep using them. Also I'm not sure if your sed script does what I intended for it to do. 1. Take ever line that starts with 2017-05-29. 2. Out of the lines we have, remove any that contain INFO or WARN. 3. Take the five last of all of those lines. 4. Take the first and third fields of all of those lines. Let's create a sample file cat > /tmp/somefile <<EOF 2017-05-28T08:30+0200 nobody ERR: Foobar failed to xyzzy. 2017-05-29T13:01+0200 nobody INFO: Garply initiated by grault scheduler. 2017-05-29T13:37+0200 nobody DEBUG: Garply exited with 0. 2017-05-29T14:12+0200 nobody WARN: Plugh quux corge. 2017-05-29T14:55+0200 nobody ERR: PLUGH QUUX CORGE! 2017-05-30T00:17+0200 nobody ERR: Failed to retrieve baz needed for thud. EOF (Note that due to how HN formatting works you won't be able to copy-paste this because HN requires leading space to format as code but then it includes the spaces in the output also and we don't want leading space in the file.) grep ^2017-05-29 /tmp/somefile | egrep -v 'INFO|WARN' | tail -n5 | cut -f1 -f3 Note: I had a typo in my original example where I said "grep -v" where I meant "egrep -v". Result: 2017-05-29T13:37+0200 DEBUG: Garply exited with 0. 2017-05-29T14:55+0200 ERR: PLUGH QUUX CORGE! Running it with your pipeline sed ' /^2017-05-29/!d; /INFO/d; /WARN/d; s/ /INFO/; s/ /INFO/; s/ .*//; s/INFO.*INFO/ /; ' /tmp/somefile \ |tail -n5 results instead in: 2017-05-29T13:37+0200 nobody DEBUG: exited 2017-05-29T14:55+0200 nobody ERR: QUUX But that's just because you assumed that space was the delimiter when it was not. Your use of the word "INFO" as a placeholder for the first two occurrences of the space character threw me off quite a bit when reading your script, since the word "INFO" occurs in the file we are working with itself, but it makes sense to use a word that we know is no longer possible to be present since we've already removed all lines that contain it. However, while a neat trick, those kinds of strange hacks are the kinds of things that has brought me to believe that having objects (like they have in Microsoft PowerShell) instead of pure text would be beneficial in Unix also. --- As for your comments on compiling into a single program and run that, I don't think you understood what I meant, or I don't think that busybox and those chrunched binaries you mentioned perform the optimization I am talking about, do they. --- Using the k language you mention in that fashion seems more like a hack and will require a lot of work each time. I would rather rewrite all core commands of my system so that they produced true objects, or actually, rather than objects just structured binary data. I don't need the output to have methods you can call. One of the main things I want from structured binary data is to be able to select the columns of data by name instead of by index and without having a mess of some commands using tab for delimiter and others space and so on. So instead of zfs list -H | cut -f1,3,4 I would like to zfs list | cut name used avail for example, or something like that. Also all commands that output tabular data must have a "header" command that will show the column headers. So to see the headers that the 'list' subcommand of the zfs command will output, I would say zfs header list And it would tell me NAME USED AVAIL REFER MOUNTPOINT and furthermore since I am authoring the shell, when you tab to complete a command it will call the binary with the header subcommand and output the headers so that you can have them easily accessible while you're working on a pipeline. And furthermore this shall work with pipes already present. So if I type zfs list | cut and then tab to complete, it will call zfs header list and pipe that to cut header which will return the input it saw Naturally, "header" will be a reserved word. All commands will understand how to work with tabular data. When you use grep you will either specify which column to use, or you can tell it to look across all columns with * The shell will only expand * to file names when the * is positionally last in the argument list of a command since all commands will take the list of files as the last ones in their list of arguments. --- All of this being said, I appreciate all replies, including yours.
chubot 9y ago
This is related to loop fusion, e.g.: https://julialang.org/blog/2017/01/moredots https://julialang.org/blog/2017/01/moredots I don't have the citation but I believe they mentioned that the Chapel language also does this. Rather than materializing intermediate arrays, it just does everything element-wise in one pass. I think this optimization may make sense in some cases that arise in practice, but probably not very many. There are a lot of other things that need to be fixed with the shell first.
- eriknstr 9y ago
  Loop fusion is a good name for what I want, thanks.

geocar 9y ago

You might find running across four cores is actually faster than "compiling into a single program"

glangdale 9y ago
Hmmn. A 'smart enough' compiler would combine the two grep patterns into a single pattern and run them in reverse, stopping at the first 5 lines. :-) Even if we don't assume we have such astounding transformations, I think you're vastly overestimating the power of multithreading in this example. There's very little for each core to do, especially the final step (which is getting a whopping 5 lines to process). All that being said, I'm not sure that, if I wanted highly parallel text processing OR deeply clever compiler-style optimizations, I'd start from "existing Unix utilities and their associated cruft".
- eriknstr 9y ago
  Not everyone understood what I meant to say but you did, thanks :) Starting from the Unix utilites is simply because I'm so used to them. I can assure you, however, that anyone who was under the impression that my system was actually Unix would be in for a nasty surprise :-> My system would be, shall we say, not POSIX compliant. My system would take the things I like about Unix, such as being command-line centric and having utilities focused on being good at small individual tasks and using pipes to glue them together seemingly (but actually as mentioned each pipeline is compiled to an optimized program when run), and the filesystem would remain essential and hierarchical. I would separate Operating System binary data, Operating System config files, Application binary data, Application config files, User config files, User downloaded data and User created data STRICTLY! Also like I mentioned, to have structured binary data instead of text. This would lead to commands which are named like in POSIX but which operate a bit differently with regards to the arguments they take. Probably a lot of other things as well that I've been thinking about but which escapes me at the moment. But yeah, all of that remains a dream because it would involve more work than I have time for :(
- geocar 9y ago
  > Hmmn. A 'smart enough' compiler would combine the two grep patterns into a single pattern and run them in reverse, stopping at the first 5 lines. :-) Only if `/var/log/somefile` is a regular file and isn't opened for writing by any other process. Or your "smart enough compiler" doesn't care about doing the same thing.

iUsedToCode 9y ago

Yeah, but why? Pipes are NEVER a performance issue for me. Maybe you have some very tricky use cases, but for 99.95% of users it doesn't matter. Almost always you'll loose more time on I/O than on pipe processing.

laumars 9y ago

UNIX shells already optimise pipelines by running the commands along it concurrently. In your example, the first grep will be outputing it's results while it's still sucking in /var/log/somefile; and the second grep will be parsing the output of the first grep as and when the first grep outputs. tail is a little different because it needs to find the end of the file but the greps certainly don't wait for each other to finish. As for your "objects" point. The shell I'm working on (discussed elsewhere in this thread) uses JSON to pass objects around functions written for that shell while still falling back to standard flat text files when piping out to traditional UNIX/GNU/whatever tools. Though I still need to make the whole thing more intuitive.

eriknstr 9y ago
>UNIX shells already optimise pipelines by running the commands along it concurrently. I am aware of that. Still, performing multiple operations on the data within a single loop and with less copying would be an optimization.
- laumars 9y ago
  I get what you're saying but technically those tools are running inside the same single loop (as that file is only iterated the once most of that pipeline) but distributed across multiple OS threads. Shell scripts are surprisingly efficient at pipelined work where you're just dealing with streaming data (as your example mostly does). It's the more complex logic they fall significantly short on. The weakest area and why I kept saying "most" of that pipeline would be the use of tail part way through the chain (as others have mentioned). But that could be optimised within Bash to something like: tail -r somefile | grep 'expression' | egrep 'expression' | head -n number | cut ... (I think -r flag on tail is GNU rather than POSIX though) You're object point is definitely relevant though. For example how many times have we seen shell scripts break because they don't consider spaces in filenames? Talking about annoyances in Bash I'd also throw in exception handling as a major problem for traditional shells. Having Bash fail in expected ways and handle them cleanly is a huge pain in the arse. Those above two points (and wanting cleaner syntax for iteration) are what drove me to write my own $SHELL so it does sound like I'm addressing some of your annoyances - albeit not all of them. However it's a long way from being something usable in production environments but I'm happy to access pull requests if you do ever feel you want to contribute to something that's trying to work towards at least some of your goals :)

vram22 9y ago

> cut -f1 -f3 Minor point: Can't you change that to cut -f1,3 ?

eriknstr 9y ago
Yes. I just forgot to :)

10165 9y ago

"grep ^2017-05-29 /var/log/somefile | grep -v 'INFO|WARN' | tail -n5 | cut -f1 -f3" Did you mean cut -d' ' -f1,3? Assuming delimiter is a space the above example can be reduced to: sed ' /^2017-05-29/!d; /INFO/d; /WARN/d; s/ /INFO/; s/ /INFO/; s/ .*//; s/INFO.*INFO/ /; ' /var/log/somefile \ |tail -n5 "... compile a set of commands into a single program and run that." Linux: see busybox ("multicall binary") BSD: see binaries on install media ("crunched binary") sed and tail are both compiled into busybox re: bsd compiling in tail with crunchgen is easy As for "objects", the k language can mmap the output of UNIX commands and run parallel computations on those results as "data" not "text". It can be faster than I/O.

eriknstr 9y ago
I intended for the delimiter to be tab in this example. Using sed or awk is an option, yes, but I am so used to the standard utilities that I would rather keep using them. Also I'm not sure if your sed script does what I intended for it to do. 1. Take ever line that starts with 2017-05-29. 2. Out of the lines we have, remove any that contain INFO or WARN. 3. Take the five last of all of those lines. 4. Take the first and third fields of all of those lines. Let's create a sample file cat > /tmp/somefile <<EOF 2017-05-28T08:30+0200 nobody ERR: Foobar failed to xyzzy. 2017-05-29T13:01+0200 nobody INFO: Garply initiated by grault scheduler. 2017-05-29T13:37+0200 nobody DEBUG: Garply exited with 0. 2017-05-29T14:12+0200 nobody WARN: Plugh quux corge. 2017-05-29T14:55+0200 nobody ERR: PLUGH QUUX CORGE! 2017-05-30T00:17+0200 nobody ERR: Failed to retrieve baz needed for thud. EOF (Note that due to how HN formatting works you won't be able to copy-paste this because HN requires leading space to format as code but then it includes the spaces in the output also and we don't want leading space in the file.) grep ^2017-05-29 /tmp/somefile | egrep -v 'INFO|WARN' | tail -n5 | cut -f1 -f3 Note: I had a typo in my original example where I said "grep -v" where I meant "egrep -v". Result: 2017-05-29T13:37+0200 DEBUG: Garply exited with 0. 2017-05-29T14:55+0200 ERR: PLUGH QUUX CORGE! Running it with your pipeline sed ' /^2017-05-29/!d; /INFO/d; /WARN/d; s/ /INFO/; s/ /INFO/; s/ .*//; s/INFO.*INFO/ /; ' /tmp/somefile \ |tail -n5 results instead in: 2017-05-29T13:37+0200 nobody DEBUG: exited 2017-05-29T14:55+0200 nobody ERR: QUUX But that's just because you assumed that space was the delimiter when it was not. Your use of the word "INFO" as a placeholder for the first two occurrences of the space character threw me off quite a bit when reading your script, since the word "INFO" occurs in the file we are working with itself, but it makes sense to use a word that we know is no longer possible to be present since we've already removed all lines that contain it. However, while a neat trick, those kinds of strange hacks are the kinds of things that has brought me to believe that having objects (like they have in Microsoft PowerShell) instead of pure text would be beneficial in Unix also. --- As for your comments on compiling into a single program and run that, I don't think you understood what I meant, or I don't think that busybox and those chrunched binaries you mentioned perform the optimization I am talking about, do they. --- Using the k language you mention in that fashion seems more like a hack and will require a lot of work each time. I would rather rewrite all core commands of my system so that they produced true objects, or actually, rather than objects just structured binary data. I don't need the output to have methods you can call. One of the main things I want from structured binary data is to be able to select the columns of data by name instead of by index and without having a mess of some commands using tab for delimiter and others space and so on. So instead of zfs list -H | cut -f1,3,4 I would like to zfs list | cut name used avail for example, or something like that. Also all commands that output tabular data must have a "header" command that will show the column headers. So to see the headers that the 'list' subcommand of the zfs command will output, I would say zfs header list And it would tell me NAME USED AVAIL REFER MOUNTPOINT and furthermore since I am authoring the shell, when you tab to complete a command it will call the binary with the header subcommand and output the headers so that you can have them easily accessible while you're working on a pipeline. And furthermore this shall work with pipes already present. So if I type zfs list | cut and then tab to complete, it will call zfs header list and pipe that to cut header which will return the input it saw Naturally, "header" will be a reserved word. All commands will understand how to work with tabular data. When you use grep you will either specify which column to use, or you can tell it to look across all columns with * The shell will only expand * to file names when the * is positionally last in the argument list of a command since all commands will take the list of files as the last ones in their list of arguments. --- All of this being said, I appreciate all replies, including yours.

chubot 9y ago

This is related to loop fusion, e.g.: https://julialang.org/blog/2017/01/moredots https://julialang.org/blog/2017/01/moredots I don't have the citation but I believe they mentioned that the Chapel language also does this. Rather than materializing intermediate arrays, it just does everything element-wise in one pass. I think this optimization may make sense in some cases that arise in practice, but probably not very many. There are a lot of other things that need to be fixed with the shell first.

eriknstr 9y ago
Loop fusion is a good name for what I want, thanks.