5 ms·

If you want to see a pretty complete shell in a higher level language (Python, 11K lines of code), check out: https://github.com/oilshell/oil/ https://github.c

by chubot 10y ago

If you want to see a pretty complete shell in a higher level language (Python, 11K lines of code), check out: https://github.com/oilshell/oil/ https://github.com/oilshell/oil/ And a blog here: http://www.oilshell.org/ http://www.oilshell.org/ (index: http://www.oilshell.org/blog/2016/11/20.html http://www.oilshell.org/blog/2016/11/20.html) This article is good in that it shows that starting processes boils down to the system calls fork(), exec(), wait(), etc. My shell has function calls, loops, conditionals, pipelines, subshells, command substitution, redirects, here docs, etc. (The code has some messiness, but the architecture is clean IMO.) It turns out that these features require just a few more system calls -- dup2() and fcntl() to manipulate file descriptors, pipe(), and open() / read() / write() / close(). It's a good reminder of how simple Unix is. The complexity of the shell is basically all in string processing, and not in interfacing with the kernel (although the interactive parts of the shell do complicate things). These system calls are all available in Python, so the code ends up being comparable to what you would write in C (and I'm in the process of porting to C++ now that the architecture has settled.)

avinassh 10y ago
There is also xonsh, pure python and works well. [0] - http://xon.sh/ http://xon.sh/ [1] - https://github.com/xonsh/xonsh https://github.com/xonsh/xonsh
pjmlp 10y ago
And in the process also safer.
- ars 10y ago
  How would it be safer? You should never execute untrusted code in the shell. And bugs that would cause problems in the shell are unlikely to be of the unterminated string type, but more likely to be logic errors (redirect to the wrong file, mess up globbing, etc), and Python would not help you there. In fact thinking that Python makes you safer is a very dangerous attitude. Someone programming with that mindset is likely to have less safe code than a C programmer who knows he has to do it right.
  - pjmlp 10y ago
    Free from memory corruption errors. Of course there is still the possibility of logical errors.
  - fulafel 10y ago
    The security bugs in shells are not of the "executing untrusted shell scripts" variety, but more complex. Shell scripts are often handling untrusted input and dealing with a untrusted multiuser filesystem, there are lots of ways these can go wrong. Yes, a lot of these are not memory safety bugs. But some are. Have a look here: http://www.cvedetails.com/product/21050/GNU-Bash.html?vendor_id=72 http://www.cvedetails.com/product/21050/GNU-Bash.html?vendor...
  - adrianN 10y ago
    You have logic errors no matter what programming language you choose. But only in some programming languages you have the additional error class of memory corruption. Hence, choosing a memory-safe language is probably safer (assuming it's not so complicated to use that you introduce more logic errors).
jodrellblank 10y ago
Microsoft PowerShell is pretty complete in a higher level langage (C#, mostly) and is open source on Github here: https://github.com/PowerShell/PowerShell https://github.com/PowerShell/PowerShell
- userbinator 10y ago
  ...and absurdly complex. One of the quick tests I like to do to ascertain the complexity of a codebase is "how long does it take to find the entrypoint" and "how long does it take to find the core functionality"? In this case, I have been clicking around through nearly empty directories for 15+ minutes and haven't seen anything that looks like an entrypoint yet, nevermind the main loop. In contrast, I downloaded the Bash source and within 5 minutes found the main loop (eval.c) and entrypoint (shell.c). Now I'm curious to know what the cmd.exe source is like, but MS seems to have no intention of releasing that.
  - waf 10y ago
    After clicking that github link, it appears to be in `src/powershell/Program.cs`, along with a readme explaining how the entrypoint works. I'm not making any statement on the complexity of the general Powershell codebase, but finding the entrypoint at least seems straightforward.
  - i336_ 10y ago
    Some hand-wavily relevant comments: - I had the exact same problem - "how do I get at the entrypoint and follow this" - only earlier today, with Chromium. I was trying to figure out why sync was broken and thought I'd try and follow the (C++) breadcrumbs. It didn't go very well. https://bugs.chromium.org/p/chromium/issues/detail?id=671498 https://bugs.chromium.org/p/chromium/issues/detail?id=671498 - AFAIK, cmd.exe is basically a horrible, horrible hack. I'm not 100% sure, but I think an instance of cmd.exe is the only way you can run things with a stdout, as in Windows has no concept of TTYs - the cmd.exe process uses a bunch of undocumented kernel interface APIs to "make it work™". Related: https://github.com/Microsoft/BashOnWindows/issues/111#issuecomment-238302654 https://github.com/Microsoft/BashOnWindows/issues/111#issuec...
fiatjaf 10y ago
I've skimmed oilshell, but couldn't find what it does differently. It just says everywhere that it is a shell that can do shell work. If it is just a normal shell, why would I want to use it instead of bash or others?
- dijit 10y ago
  My understanding is that it's easier to understand a simple source tree, you can understand how the more complex shells work underneath. The author probably started it as a learning excercise.
  - chubot 10y ago
    See my reply -- that's why I posted it here, so people could see an example of the algorithms and data structures for shell, but that's not the purpose of the project.
- chubot 10y ago
  The main thing is that it will be an entirely new programming language that you can convert bash scripts to. That's why I went to the trouble of writing a pretty complete bash parser as well as an executor. I mention that on the blog, but I haven't written about it a lot since I haven't implemented the language yet. It's designed with a big doc but not implemented yet. Have you ever written a completion function in bash? IMO it's kind of a nightmare. That should be one of the initial use cases for a better language. It would be closer to writing it in Python or ES6 rather than bash. If you just wanted to use sh syntax, then the reason you would use it is because it will have better parse errors and better runtime errors, which I talk about on my blog. But right now I don't actually expect anyone to use it, because it's in the early stages, like I say.

by chubot 10y ago

avinassh 10y ago

There is also xonsh, pure python and works well. [0] - http://xon.sh/ http://xon.sh/ [1] - https://github.com/xonsh/xonsh https://github.com/xonsh/xonsh

pjmlp 10y ago

And in the process also safer.

ars 10y ago
How would it be safer? You should never execute untrusted code in the shell. And bugs that would cause problems in the shell are unlikely to be of the unterminated string type, but more likely to be logic errors (redirect to the wrong file, mess up globbing, etc), and Python would not help you there. In fact thinking that Python makes you safer is a very dangerous attitude. Someone programming with that mindset is likely to have less safe code than a C programmer who knows he has to do it right.
- pjmlp 10y ago
  Free from memory corruption errors. Of course there is still the possibility of logical errors.
- fulafel 10y ago
  The security bugs in shells are not of the "executing untrusted shell scripts" variety, but more complex. Shell scripts are often handling untrusted input and dealing with a untrusted multiuser filesystem, there are lots of ways these can go wrong. Yes, a lot of these are not memory safety bugs. But some are. Have a look here: http://www.cvedetails.com/product/21050/GNU-Bash.html?vendor_id=72 http://www.cvedetails.com/product/21050/GNU-Bash.html?vendor...
- adrianN 10y ago
  You have logic errors no matter what programming language you choose. But only in some programming languages you have the additional error class of memory corruption. Hence, choosing a memory-safe language is probably safer (assuming it's not so complicated to use that you introduce more logic errors).

jodrellblank 10y ago

Microsoft PowerShell is pretty complete in a higher level langage (C#, mostly) and is open source on Github here: https://github.com/PowerShell/PowerShell https://github.com/PowerShell/PowerShell

userbinator 10y ago
...and absurdly complex. One of the quick tests I like to do to ascertain the complexity of a codebase is "how long does it take to find the entrypoint" and "how long does it take to find the core functionality"? In this case, I have been clicking around through nearly empty directories for 15+ minutes and haven't seen anything that looks like an entrypoint yet, nevermind the main loop. In contrast, I downloaded the Bash source and within 5 minutes found the main loop (eval.c) and entrypoint (shell.c). Now I'm curious to know what the cmd.exe source is like, but MS seems to have no intention of releasing that.
- waf 10y ago
  After clicking that github link, it appears to be in `src/powershell/Program.cs`, along with a readme explaining how the entrypoint works. I'm not making any statement on the complexity of the general Powershell codebase, but finding the entrypoint at least seems straightforward.
- i336_ 10y ago
  Some hand-wavily relevant comments: - I had the exact same problem - "how do I get at the entrypoint and follow this" - only earlier today, with Chromium. I was trying to figure out why sync was broken and thought I'd try and follow the (C++) breadcrumbs. It didn't go very well. https://bugs.chromium.org/p/chromium/issues/detail?id=671498 https://bugs.chromium.org/p/chromium/issues/detail?id=671498 - AFAIK, cmd.exe is basically a horrible, horrible hack. I'm not 100% sure, but I think an instance of cmd.exe is the only way you can run things with a stdout, as in Windows has no concept of TTYs - the cmd.exe process uses a bunch of undocumented kernel interface APIs to "make it work™". Related: https://github.com/Microsoft/BashOnWindows/issues/111#issuecomment-238302654 https://github.com/Microsoft/BashOnWindows/issues/111#issuec...

fiatjaf 10y ago

I've skimmed oilshell, but couldn't find what it does differently. It just says everywhere that it is a shell that can do shell work. If it is just a normal shell, why would I want to use it instead of bash or others?

dijit 10y ago
My understanding is that it's easier to understand a simple source tree, you can understand how the more complex shells work underneath. The author probably started it as a learning excercise.
- chubot 10y ago
  See my reply -- that's why I posted it here, so people could see an example of the algorithms and data structures for shell, but that's not the purpose of the project.
chubot 10y ago
The main thing is that it will be an entirely new programming language that you can convert bash scripts to. That's why I went to the trouble of writing a pretty complete bash parser as well as an executor. I mention that on the blog, but I haven't written about it a lot since I haven't implemented the language yet. It's designed with a big doc but not implemented yet. Have you ever written a completion function in bash? IMO it's kind of a nightmare. That should be one of the initial use cases for a better language. It would be closer to writing it in Python or ES6 rather than bash. If you just wanted to use sh syntax, then the reason you would use it is because it will have better parse errors and better runtime errors, which I talk about on my blog. But right now I don't actually expect anyone to use it, because it's in the early stages, like I say.