7 ms·

Read the section "Conclusion" at the end, if nothing else. It is extremely well written. The author throws out some potentially controversial ideas, but he (lik

Read the section "Conclusion" at the end, if nothing else. It is extremely well written. The author throws out some potentially controversial ideas, but he (like many people I have seen) are all hoping for a common outcome - dramatically "rethinking" the concept of the OS and programming language, but really embracing ideas that are as old as time. I'll highlight one section: "Most of all, let’s rethink the received wisdom that you should teach your computer to do things in a programming language and run the resulting program on an operating system. A righteous operating system should be a programming language. And for goodness’ sake, let’s not use the entire network stack just to talk to another process on the same machine which is responsible for managing a database using the filesystem stack. At least let’s use shared memory (with transactional semantics, naturally – which Intel’s latest CPUs support in hardware). But if we believe in the future – if we believe in ourselves – let’s dare to ask why, anyway, does the operating system give you this “filesystem” thing that’s no good as a database and expect you to just accept that “stuff on computers goes in folders, lah”? Any decent software environment ought to have a fully featured database, built in, and no need for a 'filesystem'."

chubot 10y ago
What programming language should the OS be? That's the problem. What he's saying will never happen, because nobody can design a programming language that will fit all use cases. Unix is polyglot by design; that's a feature and not a bug. Everybody wants the OS to be easier for THEIR use case, underestimating the diversity of computing. "Why can't I just have a whole OS in node.js, e.g. https://node-os.com/ https://node-os.com/ ? That's all I need." Well some people need to run linear algebra on computers and don't have any need for that stuff. Likewise you don't have any need for Fortran. The "language as OS" thing has been tried with Lisp and Smalltalk, and failed to gain adoption for good reason. They are both just languages now. Microsoft already tried and abandoned the second idea (WinFS). File systems are complex, but databases are an order of magnitude more complex. OSes evolve on a slower time scale; databases and languages on a relatively faster time scale. Multiple databases are also a feature. Both sqlite and Postgres/MySQL work on top of the file system. The file system supports the minimum API you need to write a database. (It's somewhat bad, but the way to fix the API isn't to replace it with a database) If you had bundled a a relational database with the OS, then you wouldn't have had the right abstractions to make distributed databases like Dynamo and BigTable and whatnot. A database just has more design parameters than a file system and so you can't write one that will fit all applications. Again, the problem is underestimating the diversity of use cases. So yeah I think both of these ideas are badly mistaken. EDIT: I also don't think idea about shared memory is good idea. You can can have IPC without the network stack using Unix domain sockets, and that's what people actually use to connect to databases on the same machine. It has the benefit that you can move the database to another machine with minimal changes. A shared memory interface between and application and a database sounds like a bad idea.
- markdoubleyou 10y ago
  Clearly TempleOS/HolyC is the answer we've been looking for. http://www.templeos.org http://www.templeos.org
- gavanwoolery 10y ago
  I agree with all of your points, I just wonder (hypothetically) if things could be improved, or are we at the peak already? I think certain things, like the web browser, could be made much better if we rethought it and simplified it. Of course, we are fighting a huge battle against legacy, but big things have been killed in the past (see Flash)
  - chubot 10y ago
    I agree about the web browser, but unfortunately there's a tradeoff between features and stability. A browser from 2016 has to do way more than a browser from 2006 or 1996. It's supporting much more of the world's interaction now, in a very literal sense. So we have gained a lot, but it comes at a cost. Both Firefox and Chrome have had severe stability issues, and I hope that when this period of rapid web evolution subsides, there will be a rethinking and consolidation. I believe Douglas Crockford said that the period of stasis after IE6 and before Firefox was the one of best things for JavaScript, and I think there is something to that argument. Although I would ask you -- improved along what dimension? Everything is a tradeoff. I agree that software is much too buggy and slow. As far as being "bloated" and complex, I used to be part of the church of simplicity, but I've come to realize that the use cases for computing are just very diverse. Though I would like to get all these features without the associated instability and performance drop (i.e. paying for what you don't use). Bjarne Stroustrop expresses this well; he says he hears this all the time: "C++ should be a much smaller and simpler language. Please add this tiny feature I'm missing". I think there's a lot of truth to that. I'm working on a shell [1] because it's a language that connects other languages. It's the glue that makes some of the complexity manageable. And it's also the language closest to the OS other than C -- you might call it the "second language" that boots up on any system. Shell is of a horrible legacy mess too. It's an instance of failure by success, much like PHP. I think the reason for the mess is that piling new crap on top is easier than fixing and replacing old stuff. It's much easier to write a new language than to replace an old language, and probably more fun. A lot of my blog is basically software archaeology. If people want things to be better, they have to roll up their sleeves and dig into the systems that people ACTUALLY USE and fix them, rather than just proposing "revolutionary stuff" that only adds to the pile. [1]: http://www.oilshell.org/blog/ http://www.oilshell.org/blog/
marcosdumay 10y ago
I'm all for making system calls and IPC cheaper. I'm also all for making system calls indistinguishable from IPC for all practical purposes. I'm in support of those ideas even if one does not take them to their logical destination at a microkernel. But I just don't think filesystems should be as powerful as databases. At a minimum, no computer exists alone, and sharing database structures is a much more complex action than sharing opinionated byte streams.
andrewl 10y ago
I like the built-in database functionality he describes. Wasn't that a feature of the BeOS file system?
- gavanwoolery 10y ago
  Not sure, but it does sound interesting. :)
- kpil 10y ago
  No idea, but VMS have something like that, and probably also other mini computers of the era. I'm only 40 so I'm to young to know :-) I think that Unix' "everything is a stream" concept won because it's simple and powerful, but it's a good example of the failure of our educational systems that only old geezers have a clue that there are alternatives. I feel like 99 % of the things I do at work, some guy at IBM should have made a generic solution for in 1979. Actually someone probably did.
  - tyingq 10y ago
    Probably not the earliest implementation, but IBM mainframes did indeed have record oriented files in the 1960's. https://en.wikipedia.org/wiki/Data_set_(IBM_mainframe) https://en.wikipedia.org/wiki/Data_set_(IBM_mainframe)
  - dom0 10y ago
    This is actually one of the main reasons why I'm interested in non-PC computers and non-Unixish/non-NTish operating systems :)
  - deleted 10y ago
    [deleted]
- fnord123 10y ago
  Maybe. You can query on BFS to match meta data entries. But for me it's more important that file systems be transactional. So instead of doing a stupid write-to-tmp-file and then move-into-place-on-same-physical-volume-hoping-it's-atomic dance you could just say open('filename').then_write(buf).then_commit() or whatever and it would do the work as a transaction; or fail. But you wouldn't end up with corrupt files with half written chunks.
Clubber 10y ago
These are good comments. >A righteous operating system should be a programming language. Like the Commodore 64, VIC 20, TRS 80 and many early home computers. >Any decent software environment ought to have a fully featured database, built in, and no need for a 'filesystem'. Microsoft tried to have MSSQL as a file system in Windows called WinFS. https://en.wikipedia.org/wiki/WinFS https://en.wikipedia.org/wiki/WinFS If he's referring to non-relational, then most FS as we know it are key/value databases. You put in a key: c:\files\myfile.txt (/usr/root/myfile) and you get a byte array. It's more sophisticated than simple numeric tokens because it adds a 3d structure in the key. Part of the reason for these seemingly odd things is we stand on the shoulders of giants that came before us. Because of this, iterative technologies must be backwards compatible to survive.
- gavanwoolery 10y ago
  With regards to standing on the shoulders of giants, the rational part of me wholeheartedly agrees, but the kid in me just wants to burn it all to the ground and rebuild it from scratch. :) Of course, it is nearly impossible to rewrite things better because inevitably you hit one of these traps: not enough time/money, the bureaucracy that is attached to money, the "too many chefs" problem, or the inadequate supply of experienced/battle-hardened minds to put on such a project - i.e. people that have written compilers, operating systems, GUIs, etc - enough times over to know what they are doing.
- nickpsecurity 10y ago
  Whereas the strategy worked for AS/400's. Helped them be more consistent, reliable, and easily managed. I'm more for pluggable storage with filesystems, object systems, and RDBMS's being options to choose from. Microsoft's failure doesn't invalidate the concept, though, so much as their use-case and solution didn't work out.
- tlb 10y ago
  A database as a file system is fundamentally a good idea. But so much software expects a regular file system, and it's surprisingly hard to implement an efficient regular file system on top of a relational DB.
buzzybee 10y ago
Everything I've seen about revolutionizing software points to the likelihood of it succeeding being zero. There's a lifecycle to it, and what we preserve are data and protocols, but not the incidental complexity of the systems themselves. At every point of the continuum, computer technology is more complex than it has to be in an academic sense("why have a computer at all if you can do equations in your head?"), but it solves enough problems that it stays alive. Or to see it from a different light: once it's programmable, you've doomed it to die, the more so the more programmable it is. And this is borne out by how fast we burn through hardware. Our code survives the best where it's more driven towards a known destination format - e.g. an old TeX source document is more likely to be rebuildable into a rendered artifact than equivalent C code into usable software. The Lisp or Smalltalk attitude to this - which is to remove code/data boundaries altogether - mostly seems to add further uncertainty and less room for curated archival. Either the whole thing runs or it doesn't.
- gavanwoolery 10y ago
  Good points - see my comment below ("With regards to standing on the shoulders of giants..."). I wonder if there is some sort of enforcement of rules you could apply that would result in a programmable thing maintaining a relatively clean / non-bloated state. Of course, once you introduce more rules, you reduce freedom...

chubot 10y ago

What programming language should the OS be? That's the problem. What he's saying will never happen, because nobody can design a programming language that will fit all use cases. Unix is polyglot by design; that's a feature and not a bug. Everybody wants the OS to be easier for THEIR use case, underestimating the diversity of computing. "Why can't I just have a whole OS in node.js, e.g. https://node-os.com/ https://node-os.com/ ? That's all I need." Well some people need to run linear algebra on computers and don't have any need for that stuff. Likewise you don't have any need for Fortran. The "language as OS" thing has been tried with Lisp and Smalltalk, and failed to gain adoption for good reason. They are both just languages now. Microsoft already tried and abandoned the second idea (WinFS). File systems are complex, but databases are an order of magnitude more complex. OSes evolve on a slower time scale; databases and languages on a relatively faster time scale. Multiple databases are also a feature. Both sqlite and Postgres/MySQL work on top of the file system. The file system supports the minimum API you need to write a database. (It's somewhat bad, but the way to fix the API isn't to replace it with a database) If you had bundled a a relational database with the OS, then you wouldn't have had the right abstractions to make distributed databases like Dynamo and BigTable and whatnot. A database just has more design parameters than a file system and so you can't write one that will fit all applications. Again, the problem is underestimating the diversity of use cases. So yeah I think both of these ideas are badly mistaken. EDIT: I also don't think idea about shared memory is good idea. You can can have IPC without the network stack using Unix domain sockets, and that's what people actually use to connect to databases on the same machine. It has the benefit that you can move the database to another machine with minimal changes. A shared memory interface between and application and a database sounds like a bad idea.

markdoubleyou 10y ago
Clearly TempleOS/HolyC is the answer we've been looking for. http://www.templeos.org http://www.templeos.org
gavanwoolery 10y ago
I agree with all of your points, I just wonder (hypothetically) if things could be improved, or are we at the peak already? I think certain things, like the web browser, could be made much better if we rethought it and simplified it. Of course, we are fighting a huge battle against legacy, but big things have been killed in the past (see Flash)
- chubot 10y ago
  I agree about the web browser, but unfortunately there's a tradeoff between features and stability. A browser from 2016 has to do way more than a browser from 2006 or 1996. It's supporting much more of the world's interaction now, in a very literal sense. So we have gained a lot, but it comes at a cost. Both Firefox and Chrome have had severe stability issues, and I hope that when this period of rapid web evolution subsides, there will be a rethinking and consolidation. I believe Douglas Crockford said that the period of stasis after IE6 and before Firefox was the one of best things for JavaScript, and I think there is something to that argument. Although I would ask you -- improved along what dimension? Everything is a tradeoff. I agree that software is much too buggy and slow. As far as being "bloated" and complex, I used to be part of the church of simplicity, but I've come to realize that the use cases for computing are just very diverse. Though I would like to get all these features without the associated instability and performance drop (i.e. paying for what you don't use). Bjarne Stroustrop expresses this well; he says he hears this all the time: "C++ should be a much smaller and simpler language. Please add this tiny feature I'm missing". I think there's a lot of truth to that. I'm working on a shell [1] because it's a language that connects other languages. It's the glue that makes some of the complexity manageable. And it's also the language closest to the OS other than C -- you might call it the "second language" that boots up on any system. Shell is of a horrible legacy mess too. It's an instance of failure by success, much like PHP. I think the reason for the mess is that piling new crap on top is easier than fixing and replacing old stuff. It's much easier to write a new language than to replace an old language, and probably more fun. A lot of my blog is basically software archaeology. If people want things to be better, they have to roll up their sleeves and dig into the systems that people ACTUALLY USE and fix them, rather than just proposing "revolutionary stuff" that only adds to the pile. [1]: http://www.oilshell.org/blog/ http://www.oilshell.org/blog/

marcosdumay 10y ago

I'm all for making system calls and IPC cheaper. I'm also all for making system calls indistinguishable from IPC for all practical purposes. I'm in support of those ideas even if one does not take them to their logical destination at a microkernel. But I just don't think filesystems should be as powerful as databases. At a minimum, no computer exists alone, and sharing database structures is a much more complex action than sharing opinionated byte streams.

andrewl 10y ago

I like the built-in database functionality he describes. Wasn't that a feature of the BeOS file system?

gavanwoolery 10y ago
Not sure, but it does sound interesting. :)
kpil 10y ago
No idea, but VMS have something like that, and probably also other mini computers of the era. I'm only 40 so I'm to young to know :-) I think that Unix' "everything is a stream" concept won because it's simple and powerful, but it's a good example of the failure of our educational systems that only old geezers have a clue that there are alternatives. I feel like 99 % of the things I do at work, some guy at IBM should have made a generic solution for in 1979. Actually someone probably did.
- tyingq 10y ago
  Probably not the earliest implementation, but IBM mainframes did indeed have record oriented files in the 1960's. https://en.wikipedia.org/wiki/Data_set_(IBM_mainframe) https://en.wikipedia.org/wiki/Data_set_(IBM_mainframe)
- dom0 10y ago
  This is actually one of the main reasons why I'm interested in non-PC computers and non-Unixish/non-NTish operating systems :)
- deleted 10y ago
  [deleted]
fnord123 10y ago
Maybe. You can query on BFS to match meta data entries. But for me it's more important that file systems be transactional. So instead of doing a stupid write-to-tmp-file and then move-into-place-on-same-physical-volume-hoping-it's-atomic dance you could just say open('filename').then_write(buf).then_commit() or whatever and it would do the work as a transaction; or fail. But you wouldn't end up with corrupt files with half written chunks.

Clubber 10y ago

These are good comments. >A righteous operating system should be a programming language. Like the Commodore 64, VIC 20, TRS 80 and many early home computers. >Any decent software environment ought to have a fully featured database, built in, and no need for a 'filesystem'. Microsoft tried to have MSSQL as a file system in Windows called WinFS. https://en.wikipedia.org/wiki/WinFS https://en.wikipedia.org/wiki/WinFS If he's referring to non-relational, then most FS as we know it are key/value databases. You put in a key: c:\files\myfile.txt (/usr/root/myfile) and you get a byte array. It's more sophisticated than simple numeric tokens because it adds a 3d structure in the key. Part of the reason for these seemingly odd things is we stand on the shoulders of giants that came before us. Because of this, iterative technologies must be backwards compatible to survive.

gavanwoolery 10y ago
With regards to standing on the shoulders of giants, the rational part of me wholeheartedly agrees, but the kid in me just wants to burn it all to the ground and rebuild it from scratch. :) Of course, it is nearly impossible to rewrite things better because inevitably you hit one of these traps: not enough time/money, the bureaucracy that is attached to money, the "too many chefs" problem, or the inadequate supply of experienced/battle-hardened minds to put on such a project - i.e. people that have written compilers, operating systems, GUIs, etc - enough times over to know what they are doing.
nickpsecurity 10y ago
Whereas the strategy worked for AS/400's. Helped them be more consistent, reliable, and easily managed. I'm more for pluggable storage with filesystems, object systems, and RDBMS's being options to choose from. Microsoft's failure doesn't invalidate the concept, though, so much as their use-case and solution didn't work out.
tlb 10y ago
A database as a file system is fundamentally a good idea. But so much software expects a regular file system, and it's surprisingly hard to implement an efficient regular file system on top of a relational DB.

buzzybee 10y ago

Everything I've seen about revolutionizing software points to the likelihood of it succeeding being zero. There's a lifecycle to it, and what we preserve are data and protocols, but not the incidental complexity of the systems themselves. At every point of the continuum, computer technology is more complex than it has to be in an academic sense("why have a computer at all if you can do equations in your head?"), but it solves enough problems that it stays alive. Or to see it from a different light: once it's programmable, you've doomed it to die, the more so the more programmable it is. And this is borne out by how fast we burn through hardware. Our code survives the best where it's more driven towards a known destination format - e.g. an old TeX source document is more likely to be rebuildable into a rendered artifact than equivalent C code into usable software. The Lisp or Smalltalk attitude to this - which is to remove code/data boundaries altogether - mostly seems to add further uncertainty and less room for curated archival. Either the whole thing runs or it doesn't.

gavanwoolery 10y ago
Good points - see my comment below ("With regards to standing on the shoulders of giants..."). I wonder if there is some sort of enforcement of rules you could apply that would result in a programmable thing maintaining a relatively clean / non-bloated state. Of course, once you introduce more rules, you reduce freedom...