3 ms·

(author here) Thanks for the comment! I do think you hit on something interesting, and it partly explains why the project is so big :) To me, shell and Python

by chubot 6y ago

(author here) Thanks for the comment! I do think you hit on something interesting, and it partly explains why the project is so big :) To me, shell and Python/JS feel kind of similar, and that was sort of the thesis at the start of the project. But if you just incrementally add features to Bourne shell, you basically get Korn shell, which is where bash got all the "bashisms" that people don't like. This paper was surprising to me; the proprietary AT&T ksh was used for GUIs and so forth, AND it was embeddable like Lua: http://www.oilshell.org/archive/ksh-usenix.pdf http://www.oilshell.org/archive/ksh-usenix.pdf [ksh] has the capabilities of perl and tcl, yet it is backward compatible with the Bourne shell. Applications of up to 25,000 lines have been written in ksh and are in production use. Much of the impetus for ksh–93 was wksh, which allows graphical user interfaces to be written in ksh. It's a lot more work to make something like Python or JS! Garbage collection is one issue; shells don't have it because they don't have recursive compound data structures or true functions. So I would say that ksh "failed" because it took a lot of shortcuts. Perl, Python, Ruby, JS, and PHP won. So with Oil I also am upgrading shell, but it was a lot more work since I wanted to make it like the latter languages. It wasn't obvious at the outset how different these things are! So in a sense Oil is upgrading shell to be more like the languages that won, which I still think is a good idea. It's weird that every computer boots to into a programming language REPL, but you're discouraged from learning it because it sucks. There's no reason that language shouldn't be a good one. --- Yes pipelines are a big deal and the runtime is solid now so they can be enhanced. Ideas here, feel free to chime in: https://github.com/oilshell/oil/issues/843 https://github.com/oilshell/oil/issues/843 :)

peter_d_sherman 6y ago
>"It's a lot more work to make something like Python or JS! Garbage collection is one issue; shells don't have it because they don't have recursive compound data structures or true functions." An excellent point! >"Yes pipelines are a big deal and the runtime is solid now so they can be enhanced." Maybe synchronous (aka, guaranteed linear) and asynchronous (no guarantees) are better ways to look at it... For example, a simple shell command like 'echo' could be run either synchronously or asynchronously, but if we're assigning the result of it to some variable, it makes sense to run it synchronously, to get that result into the variable before proceeding to the next line of the program... But, if we're running a disk defragmenting tool, or other process that we don't know when it's going to complete, then it makes sense to run it asynchronously. Shell commands that are chained together via pipelines may run asynchronously, but the shell may wait for everything to complete to scoop up the output; so you've sort of have the entire line that was sent to the shell running synchronously (if you're assigning the result to a variable), even though parts of it are running asynchronously (with respect to one another)... And of course, via ampersand, you could have the entire line run asynchronous to your program... See, I'd almost say that one of the problems is that shell scripting languages almost have no knowledge (and correct me if I am wrong, I might be!) about the pipes they create inside of commands -- an ideal shell language would be able to know about these and hook these for various purposes, and specify what would be ran synchronously and asynchronously... Or, another weird corner case -- what if you have a shell script that launches other shell scripts asynchronously (in subshells) and they launch other shell scripts also asynchronously -- in deeper level subshells? How would your language/shell track/handle that case? How do you communicate/track/determine when/where to assign results to things? It's sort of like the language/shell -- must be able to : A) Explicitly say what is asynchronous and what is synchronous... B) For asynchronous things, be able to communicate with them, track their status, get results from them when they complete, etc., etc. Now, of all of the programming languages in existence, I think Go is a good candidate for understanding synchronous vs. non-synchronous -- but you don't exactly get an "easily scriptable" beginner-friendly shell-scripting language with Go... Python seems to be the right balance between "ease of use" and "gives you control". So I'm agreed with all that you've said about Python, and your use of Python in code...
- chubot 6y ago
  Yes being able to store the FDs for pipes in variables is something that would make it more explicit and flexible. (It could also introduce deadlocks though) Also, dgsh is along these lines: https://www.spinellis.gr/sw/dgsh/#compress-compare https://www.spinellis.gr/sw/dgsh/#compress-compare It appears to use Unix domain sockets, not pipes. I think I might want to use the Ruby-like blocks for syntax, something like grep FOO *.py | wc -l | sort -n pipeline { pipe :p1 # variables that are pipes pipe :p2 # this syntax isn't great but shows the idea grep FOO *.py > &p1 < &p1 wc -l > &p2 < &p2 sort -n } ----- Although I think richer pipelines make sense, structured data will probably come first: https://github.com/oilshell/oil/wiki/Structured-Data-in-Oil https://github.com/oilshell/oil/wiki/Structured-Data-in-Oil
  - peter_d_sherman 6y ago
    That's another excellent point, by the way... All Unix (and derivative OS utilities) typically pass unstructured raw data to one another via pipes... That's sort of a good thing and a bad thing at the same time... It's great for such things as binary files, binary streams, anything binary in nature... But equal-and-oppositely, it's terrible for say, if I wanted to put the output of a 'ls -al', and I wanted to put all of those fields into something like a structured XML or JSON or BSON or what-have-you. I also like your point about FD's for pipes... that's a good idea too. Yes, I could see deadlocks as possible, but then, maybe yet another extra step of engineering/forethought is necessary -- how do we manage locking in those scenarios? Maybe we could create timers that automatically break a deadlock after a certain number of seconds, and/or put the lock explicitly under program control (like the FD's for pipes), etc., etc. Anyway, some fascinating stuff! >"Although I think richer pipelines make sense, structured data will probably come first: https://github.com/oilshell/oil/wiki/Structured-Data-in-Oil https://github.com/oilshell/oil/wiki/Structured-Data-in-Oil" Looks like you've been doing your homework! <g>

by chubot 6y ago

peter_d_sherman 6y ago

>"It's a lot more work to make something like Python or JS! Garbage collection is one issue; shells don't have it because they don't have recursive compound data structures or true functions." An excellent point! >"Yes pipelines are a big deal and the runtime is solid now so they can be enhanced." Maybe synchronous (aka, guaranteed linear) and asynchronous (no guarantees) are better ways to look at it... For example, a simple shell command like 'echo' could be run either synchronously or asynchronously, but if we're assigning the result of it to some variable, it makes sense to run it synchronously, to get that result into the variable before proceeding to the next line of the program... But, if we're running a disk defragmenting tool, or other process that we don't know when it's going to complete, then it makes sense to run it asynchronously. Shell commands that are chained together via pipelines may run asynchronously, but the shell may wait for everything to complete to scoop up the output; so you've sort of have the entire line that was sent to the shell running synchronously (if you're assigning the result to a variable), even though parts of it are running asynchronously (with respect to one another)... And of course, via ampersand, you could have the entire line run asynchronous to your program... See, I'd almost say that one of the problems is that shell scripting languages almost have no knowledge (and correct me if I am wrong, I might be!) about the pipes they create inside of commands -- an ideal shell language would be able to know about these and hook these for various purposes, and specify what would be ran synchronously and asynchronously... Or, another weird corner case -- what if you have a shell script that launches other shell scripts asynchronously (in subshells) and they launch other shell scripts also asynchronously -- in deeper level subshells? How would your language/shell track/handle that case? How do you communicate/track/determine when/where to assign results to things? It's sort of like the language/shell -- must be able to : A) Explicitly say what is asynchronous and what is synchronous... B) For asynchronous things, be able to communicate with them, track their status, get results from them when they complete, etc., etc. Now, of all of the programming languages in existence, I think Go is a good candidate for understanding synchronous vs. non-synchronous -- but you don't exactly get an "easily scriptable" beginner-friendly shell-scripting language with Go... Python seems to be the right balance between "ease of use" and "gives you control". So I'm agreed with all that you've said about Python, and your use of Python in code...

chubot 6y ago
Yes being able to store the FDs for pipes in variables is something that would make it more explicit and flexible. (It could also introduce deadlocks though) Also, dgsh is along these lines: https://www.spinellis.gr/sw/dgsh/#compress-compare https://www.spinellis.gr/sw/dgsh/#compress-compare It appears to use Unix domain sockets, not pipes. I think I might want to use the Ruby-like blocks for syntax, something like grep FOO *.py | wc -l | sort -n pipeline { pipe :p1 # variables that are pipes pipe :p2 # this syntax isn't great but shows the idea grep FOO *.py > &p1 < &p1 wc -l > &p2 < &p2 sort -n } ----- Although I think richer pipelines make sense, structured data will probably come first: https://github.com/oilshell/oil/wiki/Structured-Data-in-Oil https://github.com/oilshell/oil/wiki/Structured-Data-in-Oil
- peter_d_sherman 6y ago
  That's another excellent point, by the way... All Unix (and derivative OS utilities) typically pass unstructured raw data to one another via pipes... That's sort of a good thing and a bad thing at the same time... It's great for such things as binary files, binary streams, anything binary in nature... But equal-and-oppositely, it's terrible for say, if I wanted to put the output of a 'ls -al', and I wanted to put all of those fields into something like a structured XML or JSON or BSON or what-have-you. I also like your point about FD's for pipes... that's a good idea too. Yes, I could see deadlocks as possible, but then, maybe yet another extra step of engineering/forethought is necessary -- how do we manage locking in those scenarios? Maybe we could create timers that automatically break a deadlock after a certain number of seconds, and/or put the lock explicitly under program control (like the FD's for pipes), etc., etc. Anyway, some fascinating stuff! >"Although I think richer pipelines make sense, structured data will probably come first: https://github.com/oilshell/oil/wiki/Structured-Data-in-Oil https://github.com/oilshell/oil/wiki/Structured-Data-in-Oil" Looks like you've been doing your homework! <g>