4 ms·

What programming language should the OS be? That's the problem. What he's saying will never happen, because nobody can design a programming language that will

by chubot 10y ago

What programming language should the OS be? That's the problem. What he's saying will never happen, because nobody can design a programming language that will fit all use cases. Unix is polyglot by design; that's a feature and not a bug. Everybody wants the OS to be easier for THEIR use case, underestimating the diversity of computing. "Why can't I just have a whole OS in node.js, e.g. https://node-os.com/ https://node-os.com/ ? That's all I need." Well some people need to run linear algebra on computers and don't have any need for that stuff. Likewise you don't have any need for Fortran. The "language as OS" thing has been tried with Lisp and Smalltalk, and failed to gain adoption for good reason. They are both just languages now. Microsoft already tried and abandoned the second idea (WinFS). File systems are complex, but databases are an order of magnitude more complex. OSes evolve on a slower time scale; databases and languages on a relatively faster time scale. Multiple databases are also a feature. Both sqlite and Postgres/MySQL work on top of the file system. The file system supports the minimum API you need to write a database. (It's somewhat bad, but the way to fix the API isn't to replace it with a database) If you had bundled a a relational database with the OS, then you wouldn't have had the right abstractions to make distributed databases like Dynamo and BigTable and whatnot. A database just has more design parameters than a file system and so you can't write one that will fit all applications. Again, the problem is underestimating the diversity of use cases. So yeah I think both of these ideas are badly mistaken. EDIT: I also don't think idea about shared memory is good idea. You can can have IPC without the network stack using Unix domain sockets, and that's what people actually use to connect to databases on the same machine. It has the benefit that you can move the database to another machine with minimal changes. A shared memory interface between and application and a database sounds like a bad idea.

markdoubleyou 10y ago
Clearly TempleOS/HolyC is the answer we've been looking for. http://www.templeos.org http://www.templeos.org
gavanwoolery 10y ago
I agree with all of your points, I just wonder (hypothetically) if things could be improved, or are we at the peak already? I think certain things, like the web browser, could be made much better if we rethought it and simplified it. Of course, we are fighting a huge battle against legacy, but big things have been killed in the past (see Flash)
- chubot 10y ago
  I agree about the web browser, but unfortunately there's a tradeoff between features and stability. A browser from 2016 has to do way more than a browser from 2006 or 1996. It's supporting much more of the world's interaction now, in a very literal sense. So we have gained a lot, but it comes at a cost. Both Firefox and Chrome have had severe stability issues, and I hope that when this period of rapid web evolution subsides, there will be a rethinking and consolidation. I believe Douglas Crockford said that the period of stasis after IE6 and before Firefox was the one of best things for JavaScript, and I think there is something to that argument. Although I would ask you -- improved along what dimension? Everything is a tradeoff. I agree that software is much too buggy and slow. As far as being "bloated" and complex, I used to be part of the church of simplicity, but I've come to realize that the use cases for computing are just very diverse. Though I would like to get all these features without the associated instability and performance drop (i.e. paying for what you don't use). Bjarne Stroustrop expresses this well; he says he hears this all the time: "C++ should be a much smaller and simpler language. Please add this tiny feature I'm missing". I think there's a lot of truth to that. I'm working on a shell [1] because it's a language that connects other languages. It's the glue that makes some of the complexity manageable. And it's also the language closest to the OS other than C -- you might call it the "second language" that boots up on any system. Shell is of a horrible legacy mess too. It's an instance of failure by success, much like PHP. I think the reason for the mess is that piling new crap on top is easier than fixing and replacing old stuff. It's much easier to write a new language than to replace an old language, and probably more fun. A lot of my blog is basically software archaeology. If people want things to be better, they have to roll up their sleeves and dig into the systems that people ACTUALLY USE and fix them, rather than just proposing "revolutionary stuff" that only adds to the pile. [1]: http://www.oilshell.org/blog/ http://www.oilshell.org/blog/
david927 10y ago
Which leads us to ask, "Where are the revolutionaries?" It used to be the younger generation, around college-age, would rethink their parents' systems: political, business, etc. These kids would look at the buildings around them and imagine what could be accomplished in razing those buildings and creating a new foundation. I don't see that anymore. I see college-aged kids simply wanting to add a new floor on top of a very rickety shack that should have been discarded ages ago. Ironically, the people writing these types of manifestos for Computer Science are almost exclusively older. They often wrote that first foundational layer and know that it needs to be overhauled. By the way, I would phrase it more that "any righteous programming language is also an operating system," and that "no righteous operating system operates on files, so no righteous programming language is coded in files." Data that is not stored or accessed as data eventually will be. It's true we've tried many variations on these themes in the past, but if we do the logical syllogism in our heads, we come invariably to the conclusion this is how the future should be. We know these things. So why give up? What does failure mean other than that we need to continue trying? It infuriates me that we don't try -- when people say "we've tried everything and there's no way forward." No, we're just lazy. Charles Duell never actually said in 1899 that "everything has already been invented," yet, oh my God, we can't stop saying it now. So I'm going to ask again, "Where are the revolutionaries?"
- chubot 10y ago
  It absolutely does need to be overhauled, but the problem is that people don't agree on the direction. I think most of your ideas are profoundly mistaken, and have been disproven by history. I'm sure you think my ideas are wrong too. I think that people tend to dismiss the success or failure of certain systems as luck or even marketing, rather than deeply reflecting on the intrinsic properties of those systems that led them to be adopted or not adopted. Unix and the web have lots of flaws, but they're not an accident of history. They have fundamental architectural properties that made them succeed. (In particular, REST is the same idea architecturally as Unix/Plan 9 style file systems. Long story, because it's basically avoiding O(m*n) problems in the ecosystem.) You also have to accept evolution as a fundamental force in computing (as Linus Torvalds does). To ignore it is to set yourself up for failure and frustration. After all, humans are by no means the optimal living being either, but they're the best that evolution came up with. And as I said in my other post, "revolutionary" ideas tend to just add to the pile. They fail to meet their expectations and then Unix subsumes them. It's almost an economic inevitability. And Linux WAS revolutionary, if you compare to to Microsoft. The first couple decades of my life were DOS and Windows based, so from that perspective the revolution happened. Microsoft is porting all their stuff to Linux now. http://www.revolution-os.com/ http://www.revolution-os.com/ If you want a single language OS, check out Mirage: https://mirage.io/ https://mirage.io/ . I know somebody is going to say "I don't like OCaml". Now you understand why Unix dominates: it supports heterogeneity. Even Mirage can't really get rid of Unix, because it runs on Xen, which typically runs on Linux.
nickpsecurity 10y ago
The Smalltalk and LISP methods were successful in the commercial space. Still are at software level with LISP machines doing it at hardware level with incredible benefits for a while. Reasons for failure seem social or economic instead of technical so far. I'm not dismissing them for customizable OS's yet as we might see a comeback of such things in new form.
- lispm 10y ago
  Some of the technical problems of Lisp Machines: * relatively complex code * grown system with lots of legacy code * depended on certain hardware or its emulation -> not portable to other architectures * almost no security story * optimized for single-user GUI workstation use, other types (headless servers, ...) not so well supported * not multi-user capable * weak support for terminals * would have needed more bug fixing * more difficult to use with standard keyboards etc etc
  - tlb 10y ago
    Was the lisp machine OS fundamentally non-portable to other hardware? It's my impression that it could have been, with similar amount of effort to making the first port of any OS. Unix is intended to be portable, but creating and maintaining a port (including compilers and drivers) is a large effort, and somehow a community never materialized around supporting Lisp Machine on PCs, the way Unix on PCs did.
    - nickpsecurity 10y ago
      That they ended up emulating Genera instead of porting it should tell you something. Maybe. Also, look at the trouble people go through to do that: http://fare.tunes.org/LispM.html http://fare.tunes.org/LispM.html Then there's this that makes it look easy but also shows the keyboard problem LispM was probably referring to: http://www.loomcom.com/genera/genera-install.html http://www.loomcom.com/genera/genera-install.html What I find on getting them running without LISP hardware is somewhere between tough and "wow, I feel for that person." Maybe things have improved. There just seems to be a huge mismatch between many aspects of LISP machines and modern machines that means it's probably easier to do a clean-slate LISP machine that builds on modern primitives & interfaces better.
sedachv 10y ago
> Unix is polyglot by design; that's a feature and not a bug. No it is not. Unix was always built around C with other languages as an afterthought, even late in its development. Here are some quotes from my copy of the UNIX time-sharing system: UNIX Programmer's Manual revised edition from 1983: "System calls are entries into the UNIX supervisor. Every system call has one or more C language interfaces..." "An assortment of subroutines is available... The functions are described in terms of C, but most will work with Fortran as well." "The three principal languages in UNIX are provided by the C compiler cc(1), the Fortran compiler f77(1), and the assembler as(1)." Saying that Unix is a "polyglot" operating system makes as little sense as saying that Symbolics machines were "polyglot." Symbolics had better C and Fortran development environments than Unix but it does not change the fact that it was a Lisp machine. Unix is a time-sharing system with a C standard library, C memory management conventions, C calling conventions, C stack layout, C process memory layout, C linking and loading. Working against any of these conventions is possible but very awkward.
- chubot 10y ago
  Yes, you're right that C is special in Unix. There's a bias toward writing applications in C because the system interface is provided as C headers, and the ABI is architecture-dependent. But C turned out to be a great language for writing programming languages as well as kernels. The JVM is written in C, Python/Perl/Ruby/v8 etc. are written in C or C++. So in practice you do get an ecosystem with multiple languages. It's certainly nicer to write languages in C than in assembly. Treating everything as byte streams is another big reason that it is polyglot. This has obvious downsides, but if you stored everything as s-expressions, or C structs, then it would privilege one language over another. Traditional Unix utilities don't use architecture-dependent binary formats, and this is one of the main reasons why. But honestly, the only other solution would have been to define an RPC-like architecture-independent interface -- what operating systems do that? I don't know of any.
  - sedachv 10y ago
    > The JVM is written in C, Python/Perl/Ruby/v8 etc. are written in C or C++. This is disingenuous - that's exactly like saying that ZetaC on the Lisp Machine was written in Lisp and not in microcode, so Symbolics is a polyglot machine. The reason that a lot of dynamic programming language runtimes (like the JVM, SBCL, etc) use C is exactly because the only alternative on Unix is to hand-code system calls in assembly. The code generators are almost always self-hosting in these languages. > It's certainly nicer to write languages in C than in assembly. C is a bad language to target compilers and transpilers to if you need non-standard control flow because it forces you into a stack discipline. Doing things like continuations or restartable exceptions requires Rube Goldberg machine-level workarounds. > Treating everything as byte streams is another big reason that it is polyglot. This has obvious downsides, but if you stored everything as s-expressions, or C structs, then it would privilege one language over another. Traditional Unix utilities don't use architecture-dependent binary formats, and this is one of the main reasons why. Even the idea of byte streams is a C-ism. Bytes on the PDP-10 would "naturally" come out to 6 bits (out of a 36-bit word) - the instruction set was based around flexible bit fields. > But honestly, the only other solution would have been to define an RPC-like architecture-independent interface -- what operating systems do that? I don't know of any. That sounds like a microkernel. But I think the real takeaway here is that any operating system is going to come with baggage for language implementors, and it is important to recognize and make explicit what these assumptions are and why they were made (like for example I don't think many people ever think about the non-8-bit-bytes thing, but it is important if you want to provide a nice interface for bit-banging: http://clhs.lisp.se/Body/f_ldb.htm#ldb http://clhs.lisp.se/Body/f_ldb.htm#ldb)

by chubot 10y ago

markdoubleyou 10y ago

Clearly TempleOS/HolyC is the answer we've been looking for. http://www.templeos.org http://www.templeos.org

gavanwoolery 10y ago

I agree with all of your points, I just wonder (hypothetically) if things could be improved, or are we at the peak already? I think certain things, like the web browser, could be made much better if we rethought it and simplified it. Of course, we are fighting a huge battle against legacy, but big things have been killed in the past (see Flash)

chubot 10y ago
I agree about the web browser, but unfortunately there's a tradeoff between features and stability. A browser from 2016 has to do way more than a browser from 2006 or 1996. It's supporting much more of the world's interaction now, in a very literal sense. So we have gained a lot, but it comes at a cost. Both Firefox and Chrome have had severe stability issues, and I hope that when this period of rapid web evolution subsides, there will be a rethinking and consolidation. I believe Douglas Crockford said that the period of stasis after IE6 and before Firefox was the one of best things for JavaScript, and I think there is something to that argument. Although I would ask you -- improved along what dimension? Everything is a tradeoff. I agree that software is much too buggy and slow. As far as being "bloated" and complex, I used to be part of the church of simplicity, but I've come to realize that the use cases for computing are just very diverse. Though I would like to get all these features without the associated instability and performance drop (i.e. paying for what you don't use). Bjarne Stroustrop expresses this well; he says he hears this all the time: "C++ should be a much smaller and simpler language. Please add this tiny feature I'm missing". I think there's a lot of truth to that. I'm working on a shell [1] because it's a language that connects other languages. It's the glue that makes some of the complexity manageable. And it's also the language closest to the OS other than C -- you might call it the "second language" that boots up on any system. Shell is of a horrible legacy mess too. It's an instance of failure by success, much like PHP. I think the reason for the mess is that piling new crap on top is easier than fixing and replacing old stuff. It's much easier to write a new language than to replace an old language, and probably more fun. A lot of my blog is basically software archaeology. If people want things to be better, they have to roll up their sleeves and dig into the systems that people ACTUALLY USE and fix them, rather than just proposing "revolutionary stuff" that only adds to the pile. [1]: http://www.oilshell.org/blog/ http://www.oilshell.org/blog/

david927 10y ago

Which leads us to ask, "Where are the revolutionaries?" It used to be the younger generation, around college-age, would rethink their parents' systems: political, business, etc. These kids would look at the buildings around them and imagine what could be accomplished in razing those buildings and creating a new foundation. I don't see that anymore. I see college-aged kids simply wanting to add a new floor on top of a very rickety shack that should have been discarded ages ago. Ironically, the people writing these types of manifestos for Computer Science are almost exclusively older. They often wrote that first foundational layer and know that it needs to be overhauled. By the way, I would phrase it more that "any righteous programming language is also an operating system," and that "no righteous operating system operates on files, so no righteous programming language is coded in files." Data that is not stored or accessed as data eventually will be. It's true we've tried many variations on these themes in the past, but if we do the logical syllogism in our heads, we come invariably to the conclusion this is how the future should be. We know these things. So why give up? What does failure mean other than that we need to continue trying? It infuriates me that we don't try -- when people say "we've tried everything and there's no way forward." No, we're just lazy. Charles Duell never actually said in 1899 that "everything has already been invented," yet, oh my God, we can't stop saying it now. So I'm going to ask again, "Where are the revolutionaries?"

chubot 10y ago
It absolutely does need to be overhauled, but the problem is that people don't agree on the direction. I think most of your ideas are profoundly mistaken, and have been disproven by history. I'm sure you think my ideas are wrong too. I think that people tend to dismiss the success or failure of certain systems as luck or even marketing, rather than deeply reflecting on the intrinsic properties of those systems that led them to be adopted or not adopted. Unix and the web have lots of flaws, but they're not an accident of history. They have fundamental architectural properties that made them succeed. (In particular, REST is the same idea architecturally as Unix/Plan 9 style file systems. Long story, because it's basically avoiding O(m*n) problems in the ecosystem.) You also have to accept evolution as a fundamental force in computing (as Linus Torvalds does). To ignore it is to set yourself up for failure and frustration. After all, humans are by no means the optimal living being either, but they're the best that evolution came up with. And as I said in my other post, "revolutionary" ideas tend to just add to the pile. They fail to meet their expectations and then Unix subsumes them. It's almost an economic inevitability. And Linux WAS revolutionary, if you compare to to Microsoft. The first couple decades of my life were DOS and Windows based, so from that perspective the revolution happened. Microsoft is porting all their stuff to Linux now. http://www.revolution-os.com/ http://www.revolution-os.com/ If you want a single language OS, check out Mirage: https://mirage.io/ https://mirage.io/ . I know somebody is going to say "I don't like OCaml". Now you understand why Unix dominates: it supports heterogeneity. Even Mirage can't really get rid of Unix, because it runs on Xen, which typically runs on Linux.

nickpsecurity 10y ago

The Smalltalk and LISP methods were successful in the commercial space. Still are at software level with LISP machines doing it at hardware level with incredible benefits for a while. Reasons for failure seem social or economic instead of technical so far. I'm not dismissing them for customizable OS's yet as we might see a comeback of such things in new form.

lispm 10y ago
Some of the technical problems of Lisp Machines: * relatively complex code * grown system with lots of legacy code * depended on certain hardware or its emulation -> not portable to other architectures * almost no security story * optimized for single-user GUI workstation use, other types (headless servers, ...) not so well supported * not multi-user capable * weak support for terminals * would have needed more bug fixing * more difficult to use with standard keyboards etc etc
- tlb 10y ago
  Was the lisp machine OS fundamentally non-portable to other hardware? It's my impression that it could have been, with similar amount of effort to making the first port of any OS. Unix is intended to be portable, but creating and maintaining a port (including compilers and drivers) is a large effort, and somehow a community never materialized around supporting Lisp Machine on PCs, the way Unix on PCs did.
  - nickpsecurity 10y ago
    That they ended up emulating Genera instead of porting it should tell you something. Maybe. Also, look at the trouble people go through to do that: http://fare.tunes.org/LispM.html http://fare.tunes.org/LispM.html Then there's this that makes it look easy but also shows the keyboard problem LispM was probably referring to: http://www.loomcom.com/genera/genera-install.html http://www.loomcom.com/genera/genera-install.html What I find on getting them running without LISP hardware is somewhere between tough and "wow, I feel for that person." Maybe things have improved. There just seems to be a huge mismatch between many aspects of LISP machines and modern machines that means it's probably easier to do a clean-slate LISP machine that builds on modern primitives & interfaces better.

sedachv 10y ago

> Unix is polyglot by design; that's a feature and not a bug. No it is not. Unix was always built around C with other languages as an afterthought, even late in its development. Here are some quotes from my copy of the UNIX time-sharing system: UNIX Programmer's Manual revised edition from 1983: "System calls are entries into the UNIX supervisor. Every system call has one or more C language interfaces..." "An assortment of subroutines is available... The functions are described in terms of C, but most will work with Fortran as well." "The three principal languages in UNIX are provided by the C compiler cc(1), the Fortran compiler f77(1), and the assembler as(1)." Saying that Unix is a "polyglot" operating system makes as little sense as saying that Symbolics machines were "polyglot." Symbolics had better C and Fortran development environments than Unix but it does not change the fact that it was a Lisp machine. Unix is a time-sharing system with a C standard library, C memory management conventions, C calling conventions, C stack layout, C process memory layout, C linking and loading. Working against any of these conventions is possible but very awkward.

chubot 10y ago
Yes, you're right that C is special in Unix. There's a bias toward writing applications in C because the system interface is provided as C headers, and the ABI is architecture-dependent. But C turned out to be a great language for writing programming languages as well as kernels. The JVM is written in C, Python/Perl/Ruby/v8 etc. are written in C or C++. So in practice you do get an ecosystem with multiple languages. It's certainly nicer to write languages in C than in assembly. Treating everything as byte streams is another big reason that it is polyglot. This has obvious downsides, but if you stored everything as s-expressions, or C structs, then it would privilege one language over another. Traditional Unix utilities don't use architecture-dependent binary formats, and this is one of the main reasons why. But honestly, the only other solution would have been to define an RPC-like architecture-independent interface -- what operating systems do that? I don't know of any.
- sedachv 10y ago
  > The JVM is written in C, Python/Perl/Ruby/v8 etc. are written in C or C++. This is disingenuous - that's exactly like saying that ZetaC on the Lisp Machine was written in Lisp and not in microcode, so Symbolics is a polyglot machine. The reason that a lot of dynamic programming language runtimes (like the JVM, SBCL, etc) use C is exactly because the only alternative on Unix is to hand-code system calls in assembly. The code generators are almost always self-hosting in these languages. > It's certainly nicer to write languages in C than in assembly. C is a bad language to target compilers and transpilers to if you need non-standard control flow because it forces you into a stack discipline. Doing things like continuations or restartable exceptions requires Rube Goldberg machine-level workarounds. > Treating everything as byte streams is another big reason that it is polyglot. This has obvious downsides, but if you stored everything as s-expressions, or C structs, then it would privilege one language over another. Traditional Unix utilities don't use architecture-dependent binary formats, and this is one of the main reasons why. Even the idea of byte streams is a C-ism. Bytes on the PDP-10 would "naturally" come out to 6 bits (out of a 36-bit word) - the instruction set was based around flexible bit fields. > But honestly, the only other solution would have been to define an RPC-like architecture-independent interface -- what operating systems do that? I don't know of any. That sounds like a microkernel. But I think the real takeaway here is that any operating system is going to come with baggage for language implementors, and it is important to recognize and make explicit what these assumptions are and why they were made (like for example I don't think many people ever think about the non-8-bit-bytes thing, but it is important if you want to provide a nice interface for bit-banging: http://clhs.lisp.se/Body/f_ldb.htm#ldb http://clhs.lisp.se/Body/f_ldb.htm#ldb)