7 ms·

Why did we keep the language of the shell and the OS separate? It seems like a needless abstraction which creates more harm than good (read a shell script vs an

by wlib 8y ago

Why did we keep the language of the shell and the OS separate? It seems like a needless abstraction which creates more harm than good (read a shell script vs any other language). While I'm at it, why is the filesystem and syscall api not just part of a standard userland language? For example, the filesystem could be exposed like an object tree rather than some syscall ritual. The syscalls could just be invisible, where the language compiler deals with it instead of the programmer. I think that the old LISP machines got this right while we are stuck in a usless POSIX compatibility trap. The only reason I think they didn't design unix this way was because C was too low level, but we could write the OS in a "higher level" functional language.

techntoke 8y ago
> The only reason I think they didn't design unix this way was because C was too low level, but we could write the OS in a "higher level" functional language. You've already lost my interest. Bash/Shell is incredibly powerful and doesn't need a higher-level abstraction. That is what programming languages and CLI tools are for.
- wlib 8y ago
  That misses my poorly written point. Do you use a separate, awkward tool to call functions you wrote in python? Or do you call the function in python itself? The shell is a useless abstraction that was only necessary in unix because the alternative was c. Now that we have better languages, why not use them to write the OS bottom-up?
  - minitech 8y ago
    Because writing new OSes is hard and writing new OSes where existing software that people want to use keeps working is even harder.
    - AnIdiotOnTheNet 8y ago
      And thus the tech industry piled abstraction upon abstraction, decade after decade, until finally all software collapsed into a singularity and destroyed the earth. Which, frankly, came as somewhat of a relief to all the people forced to use it.
  - dfc 8y ago
    It's strange to me to imagine that there would be one language that was the best choice for OS implementation and day to day user interaction/automation. I'm curious what language do you think would be good for both use cases?
    - wlib 8y ago
      A more syntactically friendly Haskell-like language
      - int_19h 8y ago
        How would you deal with mutable state like, say, the current directory?
        jrs95 8y ago
        When all you have is a Haskell everything starts to look like a monad
        Risord 8y ago
        I think that if we use purish FP as our system language it doesn't make sense to just emulate imperative/mutable system. But mm... why you would want to mutable state like current directory in first place? Or even mutable filesystem? Why aren't those just immutable parameters of your program?
        adwn 8y ago
        > But mm... why you would want to mutable state like current directory in first place? Or even mutable filesystem? Why aren't those just immutable parameters of your program? Because that's how just about any OS, application, library, filesystem, device driver, database, nearly everything related to computing at all, works. Reinventing the universe has never, ever lead to success. If you want even the slightest hope for non-negligible adoption of your operating system, you need to be able to interface with the rest of the world. And that's why you don't want the filesystem to be an "immuatble parameter of your program".
  - algesten 8y ago
    The choice of python as your shell would surely be as arbitrary as bash? There is a long history of alternative shells csh, tcsh, zsh, fish, etc. All with various level of abstraction and various amounts of "programming language type constructs" (for lack of a better term). At the end of the day, long arguments have been had over which is the better shell. It's all just personal preference. Hence it can be set in /etc/passwd per user. You prefer python? chsh is there to change it.
    - wlib 8y ago
      My point wasn't about python, just an example of how a shell around python, made only for python function calls, would be a useless abstraction.
      - otabdeveloper2 8y ago
        Something like IPython, amirite? XD
  - nurettin 8y ago
    I have a different theory; perhaps line printers (which terminals try to emulate) are to blame. Space on a line is limited and one directional, so thr tools naturally evolved towards strange single line incantations instead of full fledged programs. Commodore Pet did it right by introducing a navigable terminal where you could move between lines and don't need to open an editor to write multiline programs.
    - pmontra 8y ago
      I don't have time to Google now but I think that mainframe terminals worked like that. 3270 should be the model to search for.
      - TheOtherHobbes 8y ago
        This was already a solved problem in the 70s. IBM's 3270 series offered full buffered access to a character matrix display, and DEC's VT52 and later terminals added bi-directional scrolling. Modern CLIs can usually emulate at least a few of the old scrolling character matrix displays, but only a few bash commands (e.g. top) use the extra features.
  - dragonwriter 8y ago
    > Do you use a separate, awkward tool to call functions you wrote in python? I use a separate tool to call a lot of functions written in C. That tool is called “Python”. Or “Ruby”. Sometimes I use a separate tool to call functions written in Python (or JavaScript, or...) and that tool is called PostgreSQL. Sometimes I use Ruby to call C to call Postgres to call JS. > Now that we have better languages, why not use them to write the OS bottom-up? What better languages for implementing an OS? Lisp? We had that before C. Rust? Either way—and I like both languages—I don't want to use either for a shell. Red? Maybe, but I don't think we're to the point of a Red OS yet, and while it's probably eventually usable for that purpose, I don't see it as necessarily ideal for OS implementation though it might be tolerable as a shell language.
charlesdaniels 8y ago
If you're doing it right, you are solving very different problems with shell vs any other language. Shell is best used as a tool for orchestrating other programs, you should not be implementing your programs in shell. Syscalls, in general, are used in lieu of objects or other abstractions because they more accurate mirror what the underlying hardware is doing. This isn't always the case, some syscalls are maintained for POSIX-compatibility and add a lot of complexity to emulate behavior that is no longer reflective of the hardware. At the end of the day, you'll find that it's very difficult to maintain the highest levels of performance while also presenting an API that has a high level of abstraction. Things like dynamically-resizable lists, using hash tables for everything, runtime metaprogramming, and other such niceties of modern HLLs aren't free from a performance perspective. If you really want to know more, I would suggest reading one of McKusick books on operating system design (the most recent being The Design and Implementation of the FreeBSD Operating System 2/e, but even the older ones are still largely relevant). Maintaining this "useless POSIX compatibility trap" has a certain amount of utility; I for one like not having to re-write all of my programs every few years. I imagine others feel the same. In closing, some projects that are pushing the boundaries of OS design which you may want to check out include: * Redox OS (https://www.redox-os.org/ https://www.redox-os.org/) - a UNIX-like OS done from scratch in Rust * OpenBSD (https://www.openbsd.org/ https://www.openbsd.org/) - one of the old-school Unices, written in plain old C, but with some modern security tricks up it's sleeves * Helen OS (http://www.helenos.org/ http://www.helenos.org/) - a new microkernel OS written from scratch in C++, Helen OS is not UNIX-like * DragonFlyBSD (https://www.dragonflybsd.org/ https://www.dragonflybsd.org/) - a FreeBSD fork focused on file systems research * Haiku (https://www.haiku-os.org/ https://www.haiku-os.org/) - binary and source compatible with BeOS, mostly written in C++, but also has a POSIX compatibility layer
- jxy 8y ago
  It's odd not including Plan 9 here. But I guess that is the fate of Plan 9.
  - dralley 8y ago
    Redox reuses a lot of concepts from Plan 9
darkpuma 8y ago
I think you might like TempleOS.
_emacsomancer_ 8y ago
It's not quite the same thing, but I've been enjoying using eshell (https://www.gnu.org/software/emacs/manual/html_mono/eshell.html https://www.gnu.org/software/emacs/manual/html_mono/eshell.h...). It has the usual shell interaction, except you can also interleave it with arbitrary elisp. It's not quite the OS, but it allows interaction with any parts of Emacs, which is at least OS-like (and can itself interact with the OS in various ways).
jxy 8y ago
> stuck in a usless POSIX compatibility trap Exactly. Unfortunately. Unless we have a solution that is significantly better than POSIX/UNIX, switching to anything else incurs a significant cost that no one is willing to pay. Short of that, we probably need a technology breakthrough that brings in a complete architectural change.
- majewsky 8y ago
  > Unless we have a solution that is significantly better than POSIX/UNIX, switching to anything else incurs a significant cost that no one is willing to pay. The problem is that it doesn't even suffice for the new thing to be "significantly better". Because of the huge sunk cost, the new thing needs to be able to do desirable things that a POSIX-compatible OS strictly cannot do. Otherwise, it will always be easier and faster to just glue another subsystem onto the Linux kernel and continue using that.
LukeShu 8y ago
That was kind of the idea of Unix. C was just a nicer from Assembly. You might write a performance-sensitive or low-level routine in C, much like you might drop to C when writing a Python library. But the high-level language of the system was the shell. The `dc` executable wasn't just meant to be a user-facing calculator program, it was also meant to be the system's "bignum" library. That was big-picture Unix. The syscalls and C were little-picture Unix. This 2013 thread shaped a lot of the way I think about Unix: https://news.ycombinator.com/item?id=6530180 https://news.ycombinator.com/item?id=6530180
- wlib 8y ago
  Good insight, thank you
  - tmcb 8y ago
    On the same line, let me recommend you a book: "The Design of the Unix Operating System," by Maurice J. Bach. If I am not mistaken, Mr. Bach was a member at Bell Labs. My personal take on the book is that the kernel was meant to be a portable virtual machine, extensible through processes, and that those would be the building blocks of user applications for which the shell would act as glue. In other words, the shell and the OS are separate because most of what we commonly call "the OS" may be interpreted as mere encapsulation of less versatile hardware architectures.
- krylon 8y ago
  This makes me think of PowerShell. Individual Cmdlets can be written in PowerShell itself or in any language that compiles to .Net IL, plus one has dire t access to the entire .Net framework.
  - useerup 8y ago
    PowerShell takes it a bit further: There is no "magic" built-in commands needed for e.g. manipulating current directory, environment variables etc. In PowerShell the cmdlets execute in-process and may have access to the session and it's state. Indeed all of the built-in cmdlets are just cmdlets defined not by the shell, but by the core modules
fabricexpert 8y ago
If you think of the browser as an OS, which it almost is these days, JavaScript is that language.
nothrabannosir 8y ago
(re: Bash-vs-OS integration) bash is a programming language like any other, and you could use anything with a REPL as your shell. Python should do. In fact, I'll try it right now.. Yes, it works. Just sudo chsh -s /usr/bin/python <username> and off you go. Once you start doing this for a bit, you'll notice that the Python REPL is an incredibly poor UI for repeated execution of subprocesses. It is very elaborate. Having to constantly wrap your strings in quotes, calling subprocess modules, exec, painstaking handing of streams, etc. Then you start looking for a language that has better syntax for calling external programs.. hmm... Bash. Or zsh, or ksh, etc. These languages excel at it. But that's all they are: programming languages that happen to be super easy to use when it comes to starting external programs. This is why it makes little sense to bind them to the OS. As far as the OS is concerned: there is no Bash. Just like there is no Python. There is just syscalls.
- meditate 8y ago
  You may be interested in xonsh, a shell that supports both python and bash-like expressions: https://xon.sh/ https://xon.sh/
  - nurettin 8y ago
    is this like fish?
    - leadingthenet 8y ago
      It's pretty much like fish, but with Python goodness.
  - wlib 8y ago
    Thank you for this
- klibertp 8y ago
  > Python REPL is an incredibly poor UI for repeated execution of subprocesses Python REPL, even with recent additions of TAB completion, is a poor REPL, period. IPython, on the other hand, offers a much better programming environment than shell while still allowing easy access to most of the things you mention. Example: In [1]: from pathlib import Path In [2]: file = Path("~/.sbclrc").expanduser().read_text() In [3]: !echo "$file" | awk '!/^;;/' # EDIT: to clarify, this shows passing # Python variable to a shell pipeline. #-quicklisp (let ((quicklisp-init (merge-pathnames quicklisp/setup.lisp (user-homedir-pathname)))) (when (probe-file quicklisp-init) (load quicklisp-init))) All things considered, between !, %, %% and /, IPython is a decent shell. I was using it for a few years as my main shell, actually - its QT Console implementation specifically. I was working on Windows back then, PowerShell didn't exist yet, and `cmd.exe` was... well, exactly the same as today. TLDR: a shell is just a REPL with a few conveniences for specific tasks. Re-implement these and suddenly any REPL becomes a workable shell.
m3at 8y ago
You might want to look at the oilshell project [1]. It's an attempt to bring some sanity to the shell while keeping the ability to run existing scripts. [1] http://www.oilshell.org/blog/2018/01/28.html http://www.oilshell.org/blog/2018/01/28.html

by wlib 8y ago

techntoke 8y ago

> The only reason I think they didn't design unix this way was because C was too low level, but we could write the OS in a "higher level" functional language. You've already lost my interest. Bash/Shell is incredibly powerful and doesn't need a higher-level abstraction. That is what programming languages and CLI tools are for.

wlib 8y ago
That misses my poorly written point. Do you use a separate, awkward tool to call functions you wrote in python? Or do you call the function in python itself? The shell is a useless abstraction that was only necessary in unix because the alternative was c. Now that we have better languages, why not use them to write the OS bottom-up?
- minitech 8y ago
  Because writing new OSes is hard and writing new OSes where existing software that people want to use keeps working is even harder.
  - AnIdiotOnTheNet 8y ago
    And thus the tech industry piled abstraction upon abstraction, decade after decade, until finally all software collapsed into a singularity and destroyed the earth. Which, frankly, came as somewhat of a relief to all the people forced to use it.
- dfc 8y ago
  It's strange to me to imagine that there would be one language that was the best choice for OS implementation and day to day user interaction/automation. I'm curious what language do you think would be good for both use cases?
  - wlib 8y ago
    A more syntactically friendly Haskell-like language
    - int_19h 8y ago
      How would you deal with mutable state like, say, the current directory?
      - jrs95 8y ago
        When all you have is a Haskell everything starts to look like a monad
      - Risord 8y ago
        I think that if we use purish FP as our system language it doesn't make sense to just emulate imperative/mutable system. But mm... why you would want to mutable state like current directory in first place? Or even mutable filesystem? Why aren't those just immutable parameters of your program?
        adwn 8y ago
        > But mm... why you would want to mutable state like current directory in first place? Or even mutable filesystem? Why aren't those just immutable parameters of your program? Because that's how just about any OS, application, library, filesystem, device driver, database, nearly everything related to computing at all, works. Reinventing the universe has never, ever lead to success. If you want even the slightest hope for non-negligible adoption of your operating system, you need to be able to interface with the rest of the world. And that's why you don't want the filesystem to be an "immuatble parameter of your program".
- algesten 8y ago
  The choice of python as your shell would surely be as arbitrary as bash? There is a long history of alternative shells csh, tcsh, zsh, fish, etc. All with various level of abstraction and various amounts of "programming language type constructs" (for lack of a better term). At the end of the day, long arguments have been had over which is the better shell. It's all just personal preference. Hence it can be set in /etc/passwd per user. You prefer python? chsh is there to change it.
  - wlib 8y ago
    My point wasn't about python, just an example of how a shell around python, made only for python function calls, would be a useless abstraction.
    - otabdeveloper2 8y ago
      Something like IPython, amirite? XD
- nurettin 8y ago
  I have a different theory; perhaps line printers (which terminals try to emulate) are to blame. Space on a line is limited and one directional, so thr tools naturally evolved towards strange single line incantations instead of full fledged programs. Commodore Pet did it right by introducing a navigable terminal where you could move between lines and don't need to open an editor to write multiline programs.
  - pmontra 8y ago
    I don't have time to Google now but I think that mainframe terminals worked like that. 3270 should be the model to search for.
    - TheOtherHobbes 8y ago
      This was already a solved problem in the 70s. IBM's 3270 series offered full buffered access to a character matrix display, and DEC's VT52 and later terminals added bi-directional scrolling. Modern CLIs can usually emulate at least a few of the old scrolling character matrix displays, but only a few bash commands (e.g. top) use the extra features.
- dragonwriter 8y ago
  > Do you use a separate, awkward tool to call functions you wrote in python? I use a separate tool to call a lot of functions written in C. That tool is called “Python”. Or “Ruby”. Sometimes I use a separate tool to call functions written in Python (or JavaScript, or...) and that tool is called PostgreSQL. Sometimes I use Ruby to call C to call Postgres to call JS. > Now that we have better languages, why not use them to write the OS bottom-up? What better languages for implementing an OS? Lisp? We had that before C. Rust? Either way—and I like both languages—I don't want to use either for a shell. Red? Maybe, but I don't think we're to the point of a Red OS yet, and while it's probably eventually usable for that purpose, I don't see it as necessarily ideal for OS implementation though it might be tolerable as a shell language.

charlesdaniels 8y ago

If you're doing it right, you are solving very different problems with shell vs any other language. Shell is best used as a tool for orchestrating other programs, you should not be implementing your programs in shell. Syscalls, in general, are used in lieu of objects or other abstractions because they more accurate mirror what the underlying hardware is doing. This isn't always the case, some syscalls are maintained for POSIX-compatibility and add a lot of complexity to emulate behavior that is no longer reflective of the hardware. At the end of the day, you'll find that it's very difficult to maintain the highest levels of performance while also presenting an API that has a high level of abstraction. Things like dynamically-resizable lists, using hash tables for everything, runtime metaprogramming, and other such niceties of modern HLLs aren't free from a performance perspective. If you really want to know more, I would suggest reading one of McKusick books on operating system design (the most recent being The Design and Implementation of the FreeBSD Operating System 2/e, but even the older ones are still largely relevant). Maintaining this "useless POSIX compatibility trap" has a certain amount of utility; I for one like not having to re-write all of my programs every few years. I imagine others feel the same. In closing, some projects that are pushing the boundaries of OS design which you may want to check out include: * Redox OS (https://www.redox-os.org/ https://www.redox-os.org/) - a UNIX-like OS done from scratch in Rust * OpenBSD (https://www.openbsd.org/ https://www.openbsd.org/) - one of the old-school Unices, written in plain old C, but with some modern security tricks up it's sleeves * Helen OS (http://www.helenos.org/ http://www.helenos.org/) - a new microkernel OS written from scratch in C++, Helen OS is not UNIX-like * DragonFlyBSD (https://www.dragonflybsd.org/ https://www.dragonflybsd.org/) - a FreeBSD fork focused on file systems research * Haiku (https://www.haiku-os.org/ https://www.haiku-os.org/) - binary and source compatible with BeOS, mostly written in C++, but also has a POSIX compatibility layer

jxy 8y ago
It's odd not including Plan 9 here. But I guess that is the fate of Plan 9.
- dralley 8y ago
  Redox reuses a lot of concepts from Plan 9

darkpuma 8y ago

I think you might like TempleOS.

_emacsomancer_ 8y ago

It's not quite the same thing, but I've been enjoying using eshell (https://www.gnu.org/software/emacs/manual/html_mono/eshell.html https://www.gnu.org/software/emacs/manual/html_mono/eshell.h...). It has the usual shell interaction, except you can also interleave it with arbitrary elisp. It's not quite the OS, but it allows interaction with any parts of Emacs, which is at least OS-like (and can itself interact with the OS in various ways).

jxy 8y ago

> stuck in a usless POSIX compatibility trap Exactly. Unfortunately. Unless we have a solution that is significantly better than POSIX/UNIX, switching to anything else incurs a significant cost that no one is willing to pay. Short of that, we probably need a technology breakthrough that brings in a complete architectural change.

majewsky 8y ago
> Unless we have a solution that is significantly better than POSIX/UNIX, switching to anything else incurs a significant cost that no one is willing to pay. The problem is that it doesn't even suffice for the new thing to be "significantly better". Because of the huge sunk cost, the new thing needs to be able to do desirable things that a POSIX-compatible OS strictly cannot do. Otherwise, it will always be easier and faster to just glue another subsystem onto the Linux kernel and continue using that.

LukeShu 8y ago

That was kind of the idea of Unix. C was just a nicer from Assembly. You might write a performance-sensitive or low-level routine in C, much like you might drop to C when writing a Python library. But the high-level language of the system was the shell. The `dc` executable wasn't just meant to be a user-facing calculator program, it was also meant to be the system's "bignum" library. That was big-picture Unix. The syscalls and C were little-picture Unix. This 2013 thread shaped a lot of the way I think about Unix: https://news.ycombinator.com/item?id=6530180 https://news.ycombinator.com/item?id=6530180

wlib 8y ago
Good insight, thank you
- tmcb 8y ago
  On the same line, let me recommend you a book: "The Design of the Unix Operating System," by Maurice J. Bach. If I am not mistaken, Mr. Bach was a member at Bell Labs. My personal take on the book is that the kernel was meant to be a portable virtual machine, extensible through processes, and that those would be the building blocks of user applications for which the shell would act as glue. In other words, the shell and the OS are separate because most of what we commonly call "the OS" may be interpreted as mere encapsulation of less versatile hardware architectures.
krylon 8y ago
This makes me think of PowerShell. Individual Cmdlets can be written in PowerShell itself or in any language that compiles to .Net IL, plus one has dire t access to the entire .Net framework.
- useerup 8y ago
  PowerShell takes it a bit further: There is no "magic" built-in commands needed for e.g. manipulating current directory, environment variables etc. In PowerShell the cmdlets execute in-process and may have access to the session and it's state. Indeed all of the built-in cmdlets are just cmdlets defined not by the shell, but by the core modules

fabricexpert 8y ago

If you think of the browser as an OS, which it almost is these days, JavaScript is that language.

nothrabannosir 8y ago

(re: Bash-vs-OS integration) bash is a programming language like any other, and you could use anything with a REPL as your shell. Python should do. In fact, I'll try it right now.. Yes, it works. Just sudo chsh -s /usr/bin/python <username> and off you go. Once you start doing this for a bit, you'll notice that the Python REPL is an incredibly poor UI for repeated execution of subprocesses. It is very elaborate. Having to constantly wrap your strings in quotes, calling subprocess modules, exec, painstaking handing of streams, etc. Then you start looking for a language that has better syntax for calling external programs.. hmm... Bash. Or zsh, or ksh, etc. These languages excel at it. But that's all they are: programming languages that happen to be super easy to use when it comes to starting external programs. This is why it makes little sense to bind them to the OS. As far as the OS is concerned: there is no Bash. Just like there is no Python. There is just syscalls.

meditate 8y ago
You may be interested in xonsh, a shell that supports both python and bash-like expressions: https://xon.sh/ https://xon.sh/
- nurettin 8y ago
  is this like fish?
  - leadingthenet 8y ago
    It's pretty much like fish, but with Python goodness.
- wlib 8y ago
  Thank you for this
klibertp 8y ago
> Python REPL is an incredibly poor UI for repeated execution of subprocesses Python REPL, even with recent additions of TAB completion, is a poor REPL, period. IPython, on the other hand, offers a much better programming environment than shell while still allowing easy access to most of the things you mention. Example: In [1]: from pathlib import Path In [2]: file = Path("~/.sbclrc").expanduser().read_text() In [3]: !echo "$file" | awk '!/^;;/' # EDIT: to clarify, this shows passing # Python variable to a shell pipeline. #-quicklisp (let ((quicklisp-init (merge-pathnames quicklisp/setup.lisp (user-homedir-pathname)))) (when (probe-file quicklisp-init) (load quicklisp-init))) All things considered, between !, %, %% and /, IPython is a decent shell. I was using it for a few years as my main shell, actually - its QT Console implementation specifically. I was working on Windows back then, PowerShell didn't exist yet, and `cmd.exe` was... well, exactly the same as today. TLDR: a shell is just a REPL with a few conveniences for specific tasks. Re-implement these and suddenly any REPL becomes a workable shell.

m3at 8y ago

You might want to look at the oilshell project [1]. It's an attempt to bring some sanity to the shell while keeping the ability to run existing scripts. [1] http://www.oilshell.org/blog/2018/01/28.html http://www.oilshell.org/blog/2018/01/28.html