5 ms·

DBOS: A DBMS-oriented Operating System [pdf]

80 points by matt_d 5y ago

SigmundA 5y ago
Maybe one day we will finally get orthogonally persistent os that is not niche: http://tunes.org/wiki/orthogonal_20persistence.html http://tunes.org/wiki/orthogonal_20persistence.html
- Findecanor 5y ago
  I used to be a fan of the idea, but over time I formed the opinion that "ordinary" software is just too error-prone to be worthy of full orthogonal persistence and life-long up-times. Tunes was supposed to be based on a high-level type-safe, memory-safe language runtime — but that would just be the least requirement. IMHO, formal methods and proof-carrying code (or some other novel idea altogether) would have to be applied throughout to avoid errors that would persist. Orthogonal persistence has got an up-swing in academia in recent years after the introduction of new non-volatile memory tech, such as 3D CrossPoint. But I think that some of these researchers have applied it a bit naïvely. The best NVRAM is still much less reliable than DRAM, so it still needs e.g. wear-levelling and error-correction (and even write-throttling) — like what modern file-system stacks do.
marcodiego 5y ago
These kinds of ideas always remember me of WinFS.
melony 5y ago
Maybe Google can add a NFS frontend to Spanner, I wonder what that would look like.
- taywrobel 5y ago
  Probably would be better to try this as a layer on top of FoundationDB. It was kind of designed to be a distributed systems building block that you can develop interfaces on top of without worrying much about implementing the durability, consistency, or distribution of data.
tracyhenry 5y ago
DBOS was presented at the CIDR conference a couple weeks ago: https://youtu.be/L4LsYda9hKU https://youtu.be/L4LsYda9hKU
CalChris 5y ago
I remember back in the day, overly Unix people had a mantra, a filesystem is not a database which is kind of a non-sentence like a Macintosh is not a typewriter, also common at the time. The dominant storage technology then, disks, had structure and performance was determined largely by that structure. Ignoring that structure did not solve the performance problem. I thought that maybe a filesystem actually needed a schema and wrote a semester paper for that for CS262. Well that went over like a lead balloon. It's less necessary now with SSDs, but I often think that Unix/Linux oversimplifies and then pays technical debt for decades for having adopted a simpler approach than VMS/...
- GauntletWizard 5y ago
  And the success of VMS in the modern era speaks to that.
  - Aloha 5y ago
    Some of the structures of Files-11 lives on in NTFS in part. Some of the structure of VMS lives on in NT.
- pjmlp 5y ago
  It is just another example of all the things UNIX people decided to ignore, yet thanks to Worse is Better, now it is everywhere with plenty of technical debt to fix.
  - lproven 5y ago
    Whenever I read a HN comment and recoil because it's something that I could well have written myself, inevitably, it was @pjmlp. :-D
- SigmundA 5y ago
  Seems obvious to me that a file system is a type of hierarchal k/v database.
dang 5y ago
There was a small thread recently (we invited matt_d to repost): DBOS: A DBMS-oriented Operating System [pdf] - https://news.ycombinator.com/item?id=29803799 https://news.ycombinator.com/item?id=29803799 - Jan 2022 (6 comments)
mmcgaha 5y ago
I am sure there are a lot of details that I missed, but this sounds a lot like an IBM i series. If this is free software I can see a market for it. If this is commercial software then . . . well I don't know good luck.
indymike 5y ago
This idea has been around for a while... BeOS allowed for indexing and metadata which allowed you to use the file system a lot like a relational database. It was a really cool feature and made it easy to write software that stored tabular data. MS tried back in the Windows Longhorn era... and gave up on it. I think there's more than one 70s and 80s era big iron OS that did this, too. The file system has been really hard to get rid of - it hides a surprising lot of details on how bits get read and written from hardware, and that hardware has changed a lot in the past decades. SQLite is a great codebase to look at to see what really goes on there.
- zokier 5y ago
  Filesystem is small aspect here, the big deal here is putting stuff like task scheduling and IPC on top of DBMS. And even for files, the point here is more on having distributed filesystem as first-class citizen and less on being able to do sql-like queries.
  - pjmlp 5y ago
    Although not the same as what the paper talks about, an approximation would be Oracle running with raw volumes managed by the DB, on a cluster, using Web based UIs done on Apex.
- gjvnq 5y ago
  I'm toying with the same idea but also with the idea of treating files and folders as the same thing: nodes in the filesystem. It's an interesting idea but coding it is somewhat hard as I also want decentralized synchronization.
zokier 5y ago
Feels like someone took hadoop and osquery and smashed them together. And like hadoop it feels like overkill if you don't have a large cluster of computers to distribute work on.
AtlasLion 5y ago
reminds me a lot of IBM OS400.
boznz 5y ago
briefly brought back memories of the PICK OS https://en.wikipedia.org/wiki/Pick_operating_system https://en.wikipedia.org/wiki/Pick_operating_system which ran our council tax system back in the 90s when I was an IT Manager <shiver>

80 points by matt_d 5y ago

Maybe one day we will finally get orthogonally persistent os that is not niche: http://tunes.org/wiki/orthogonal_20persistence.html http://tunes.org/wiki/orthogonal_20persistence.html

Findecanor 5y ago
I used to be a fan of the idea, but over time I formed the opinion that "ordinary" software is just too error-prone to be worthy of full orthogonal persistence and life-long up-times. Tunes was supposed to be based on a high-level type-safe, memory-safe language runtime — but that would just be the least requirement. IMHO, formal methods and proof-carrying code (or some other novel idea altogether) would have to be applied throughout to avoid errors that would persist. Orthogonal persistence has got an up-swing in academia in recent years after the introduction of new non-volatile memory tech, such as 3D CrossPoint. But I think that some of these researchers have applied it a bit naïvely. The best NVRAM is still much less reliable than DRAM, so it still needs e.g. wear-levelling and error-correction (and even write-throttling) — like what modern file-system stacks do.

marcodiego 5y ago

These kinds of ideas always remember me of WinFS.

melony 5y ago

Maybe Google can add a NFS frontend to Spanner, I wonder what that would look like.

taywrobel 5y ago
Probably would be better to try this as a layer on top of FoundationDB. It was kind of designed to be a distributed systems building block that you can develop interfaces on top of without worrying much about implementing the durability, consistency, or distribution of data.

tracyhenry 5y ago

DBOS was presented at the CIDR conference a couple weeks ago: https://youtu.be/L4LsYda9hKU https://youtu.be/L4LsYda9hKU

CalChris 5y ago

I remember back in the day, overly Unix people had a mantra, a filesystem is not a database which is kind of a non-sentence like a Macintosh is not a typewriter, also common at the time. The dominant storage technology then, disks, had structure and performance was determined largely by that structure. Ignoring that structure did not solve the performance problem. I thought that maybe a filesystem actually needed a schema and wrote a semester paper for that for CS262. Well that went over like a lead balloon. It's less necessary now with SSDs, but I often think that Unix/Linux oversimplifies and then pays technical debt for decades for having adopted a simpler approach than VMS/...

GauntletWizard 5y ago
And the success of VMS in the modern era speaks to that.
- Aloha 5y ago
  Some of the structures of Files-11 lives on in NTFS in part. Some of the structure of VMS lives on in NT.
pjmlp 5y ago
It is just another example of all the things UNIX people decided to ignore, yet thanks to Worse is Better, now it is everywhere with plenty of technical debt to fix.
- lproven 5y ago
  Whenever I read a HN comment and recoil because it's something that I could well have written myself, inevitably, it was @pjmlp. :-D
SigmundA 5y ago
Seems obvious to me that a file system is a type of hierarchal k/v database.

dang 5y ago

There was a small thread recently (we invited matt_d to repost): DBOS: A DBMS-oriented Operating System [pdf] - https://news.ycombinator.com/item?id=29803799 https://news.ycombinator.com/item?id=29803799 - Jan 2022 (6 comments)

mmcgaha 5y ago

I am sure there are a lot of details that I missed, but this sounds a lot like an IBM i series. If this is free software I can see a market for it. If this is commercial software then . . . well I don't know good luck.

indymike 5y ago

This idea has been around for a while... BeOS allowed for indexing and metadata which allowed you to use the file system a lot like a relational database. It was a really cool feature and made it easy to write software that stored tabular data. MS tried back in the Windows Longhorn era... and gave up on it. I think there's more than one 70s and 80s era big iron OS that did this, too. The file system has been really hard to get rid of - it hides a surprising lot of details on how bits get read and written from hardware, and that hardware has changed a lot in the past decades. SQLite is a great codebase to look at to see what really goes on there.

zokier 5y ago
Filesystem is small aspect here, the big deal here is putting stuff like task scheduling and IPC on top of DBMS. And even for files, the point here is more on having distributed filesystem as first-class citizen and less on being able to do sql-like queries.
- pjmlp 5y ago
  Although not the same as what the paper talks about, an approximation would be Oracle running with raw volumes managed by the DB, on a cluster, using Web based UIs done on Apex.
gjvnq 5y ago
I'm toying with the same idea but also with the idea of treating files and folders as the same thing: nodes in the filesystem. It's an interesting idea but coding it is somewhat hard as I also want decentralized synchronization.

zokier 5y ago

Feels like someone took hadoop and osquery and smashed them together. And like hadoop it feels like overkill if you don't have a large cluster of computers to distribute work on.

AtlasLion 5y ago

reminds me a lot of IBM OS400.

boznz 5y ago

briefly brought back memories of the PICK OS https://en.wikipedia.org/wiki/Pick_operating_system https://en.wikipedia.org/wiki/Pick_operating_system which ran our council tax system back in the 90s when I was an IT Manager <shiver>