3 ms·

Yes being able to store the FDs for pipes in variables is something that would make it more explicit and flexible. (It could also introduce deadlocks though)

by chubot 6y ago

Yes being able to store the FDs for pipes in variables is something that would make it more explicit and flexible. (It could also introduce deadlocks though) Also, dgsh is along these lines: https://www.spinellis.gr/sw/dgsh/#compress-compare https://www.spinellis.gr/sw/dgsh/#compress-compare It appears to use Unix domain sockets, not pipes. I think I might want to use the Ruby-like blocks for syntax, something like grep FOO *.py | wc -l | sort -n pipeline { pipe :p1 # variables that are pipes pipe :p2 # this syntax isn't great but shows the idea grep FOO *.py > &p1 < &p1 wc -l > &p2 < &p2 sort -n } ----- Although I think richer pipelines make sense, structured data will probably come first: https://github.com/oilshell/oil/wiki/Structured-Data-in-Oil https://github.com/oilshell/oil/wiki/Structured-Data-in-Oil

peter_d_sherman 6y ago
That's another excellent point, by the way... All Unix (and derivative OS utilities) typically pass unstructured raw data to one another via pipes... That's sort of a good thing and a bad thing at the same time... It's great for such things as binary files, binary streams, anything binary in nature... But equal-and-oppositely, it's terrible for say, if I wanted to put the output of a 'ls -al', and I wanted to put all of those fields into something like a structured XML or JSON or BSON or what-have-you. I also like your point about FD's for pipes... that's a good idea too. Yes, I could see deadlocks as possible, but then, maybe yet another extra step of engineering/forethought is necessary -- how do we manage locking in those scenarios? Maybe we could create timers that automatically break a deadlock after a certain number of seconds, and/or put the lock explicitly under program control (like the FD's for pipes), etc., etc. Anyway, some fascinating stuff! >"Although I think richer pipelines make sense, structured data will probably come first: https://github.com/oilshell/oil/wiki/Structured-Data-in-Oil https://github.com/oilshell/oil/wiki/Structured-Data-in-Oil" Looks like you've been doing your homework! <g>

by chubot 6y ago

peter_d_sherman 6y ago

That's another excellent point, by the way... All Unix (and derivative OS utilities) typically pass unstructured raw data to one another via pipes... That's sort of a good thing and a bad thing at the same time... It's great for such things as binary files, binary streams, anything binary in nature... But equal-and-oppositely, it's terrible for say, if I wanted to put the output of a 'ls -al', and I wanted to put all of those fields into something like a structured XML or JSON or BSON or what-have-you. I also like your point about FD's for pipes... that's a good idea too. Yes, I could see deadlocks as possible, but then, maybe yet another extra step of engineering/forethought is necessary -- how do we manage locking in those scenarios? Maybe we could create timers that automatically break a deadlock after a certain number of seconds, and/or put the lock explicitly under program control (like the FD's for pipes), etc., etc. Anyway, some fascinating stuff! >"Although I think richer pipelines make sense, structured data will probably come first: https://github.com/oilshell/oil/wiki/Structured-Data-in-Oil https://github.com/oilshell/oil/wiki/Structured-Data-in-Oil" Looks like you've been doing your homework! <g>