6 ms·

Stonebraker Seeks to Invert the Computing Paradigm with DBOS

89 points by signa11 2y ago

js8 2y ago
I have been thinking about that idea (putting database in front of operating system, i.e. applications interact with the environment through database) for a while now, I think it's really neat. Another good reason is testing and verification. Your application becomes decoupled from the OS resources and you can verify what it does to them by comparing the database. Because the OS state is managed by database, it is formally described and you can verify what your application does to it. In some sense, it's an implementation of functional core/imperative shell idea. Also, because DBs are transactional, transactions will become a first-class concept in your operating system. Which is something that's long overdue for applications (although there are systems such as CICS that have been doing this for some time now).
- digitalsankhara 2y ago
  Have you heard of PickOS [0]? Maybe similar to what you are describing. I admit I just read the first couple of paragraphs of the article and thought, yeah - PickOS vibes. [0] https://en.wikipedia.org/wiki/Pick_operating_system https://en.wikipedia.org/wiki/Pick_operating_system
  - kcartlidge 2y ago
    I was a Pick user back in the late 1980s and that was, indeed, my first thought too.
thisislife2 2y ago
Interesting - we've been hearing about a "database file system" for some time now, and it's exciting to see it popping up slowly. But DBOS still has a long way to go - runs only on AWS currently and you are limited to Typescript for development (which is a curious choice - any idea why?). > DBOS will resist being on-prem and will resist being POSIX compliant, Stonebraker said ... They may also want things like support for Python and Java programming environments, and support for running in Azure and GCP, which will be determined in the future.
- phlakaton 2y ago
  BeOS had a database-based filesystem way back in the early 1990s. It was slow and difficult to keep up to date. They swapped it out with BFS and the rest was history. It's possible this will do better than Be's old database system, but this is not a novel idea.
  - jimhefferon 2y ago
    In the article he says that DBs are now much faster.
    - snovv_crash 2y ago
      People are also much happier these days to introduce a factor of 10 slowdown relative to what the hardware can natively support.
  - pjmlp 2y ago
    IBM big iron systems still use databases as file system.
deleted 2y ago
[deleted]
deepfriedchokes 2y ago
Sounds kind of like what we live in. The universe is the database and life the operating system(s).
- hestefisk 2y ago
  And the network is the computer :)
random3 2y ago
This isn't a bad idea. However, assuming heterogenous hardware(CPUs, GPUs, RAM, Network, capabilities) the OS problem is primarily a driver problem. Resource management seem like a candidate for a microkernel and this can expose a relational interface to a distributed OS. The distributed OS is primarily a resource manager and well fit for a similar relational interface. Provided that, besides drivers, the state management (tasks) is a relatively big issue, this makes the whole setup not far from a DB architecture and most managers/schedulers are arguably easily generalizable to a relational interface. In fact most distributed systems run on some form of distributed state management / DB (e.g. Raft, Zookeeper) This said, I think the more granular approach for the kernel is maybe a cleaner way to think about it. Also, any pragmatic approach would need to take drivers into account and I wonder if it's realistic to assume anything else but major kernels.
- rwmj 2y ago
  As they seem to be targeting only cloud, the set of drivers you actually need to implement is relatively small. eg. Plenty of "toy" OSes get away with implementing just a few virtio drivers and hence can run on qemu, KVM, and AWS, which is enough for them.
  - random3 2y ago
    fair enough, but wouldn't a virtualization layer in between defeat the purpose?
    - rwmj 2y ago
      I'm not quite sure what you mean, but if you target the cloud as the platform (as is stated in the article), then you always have a virtualization layer, even if you provision on something like AWS Bare Metal Instances. (The reason bare metal instances use virtualization is not obvious: It's so you don't try to reflash the firmware on devices for a persistent attack.)
      - random3 2y ago
somat 2y ago
I worked at one place, they had heavily invested into pick style databases, when pick databases still made sense, and for some of the crusty old pick programmers still around, that was the os, it was their development environment and shell. https://en.wikipedia.org/wiki/Pick_operating_system https://en.wikipedia.org/wiki/Pick_operating_system Note: I don't think they had any actual pick systems left, by the time I got there they were mostly using unidata on linux, but there were still a lot unidata on sco boxes around.
- tonnydourado 2y ago
  > It is named after one of its developers, Dick Pick. You gotta be fucking kidding me
  - kcartlidge 2y ago
    And: > Pick was originally implemented as the Generalized Information Retrieval Language System (GIRLS) As an aside, though, I really enjoyed working on a Pick system a few decades back.
xixixao 2y ago
This boils down to a serverless platform with storage and scheduling - which is a great idea! Add reactivity and you get Convex: https://www.convex.dev/ https://www.convex.dev/
- sgt101 2y ago
  I think that they are more radical - it seems like they have implemented a unikernel that supports the database service and their claim is that that's all you need for app dev. I think it looks rather similar (as a component) to something like snowflake - lots of data optimised services, but lower level with more app capability. The problem they will have is that a lot of the benefit that they are pushing is created by the big cloud data warehouses already, and the pain of a total re-implementation rather than re-architect or lift-and-shift completely terrifies most CIO's
yarg 2y ago
Reminds me of what Microsoft was trying to do for Longhorn. https://en.wikipedia.org/wiki/WinFS https://en.wikipedia.org/wiki/WinFS
- pjmlp 2y ago
  Nowadays part of it survives as ReFS. https://learn.microsoft.com/en-us/windows-server/storage/refs/refs-overview https://learn.microsoft.com/en-us/windows-server/storage/ref...
Guthur 2y ago
[flagged]
- clcaev 2y ago
  I wonder how much will be open source, and how much is special sauce. https://github.com/dbos-project https://github.com/dbos-project > This repository is research code and no longer maintained. The students working on it have graduated This is from "apiary". Lotus also seems experimental and limited to GCP, so I guess not?
- deleted 2y ago
  [deleted]
fmajid 2y ago
There is still an OS. Calling it a “minimal kernel” in the diagram is just semantic word games.
Validark 2y ago
Looks cool, but if you're building a stack for everyone else to build atop, why would you start with non-bare metal languages? I see C++ is in there, which is fine depending on who is wielding it, but Java and TypeScript necessarily add overhead that to me just doesn't make sense to add to all systems that want to run on your platform?
- unnouinceput 2y ago
  Java was in 1st iteration and was replaced by Typescript
  - slowmovintarget 2y ago
    So they replaced an entire language with a linter?
nialse 2y ago
Sounds a lot like OS/400 in my ears. A machine interface that virtualizes the hardware and a database on top of that which applications are built upon. So, maybe not that new of an idea?
- gregw2 2y ago
  For a thread on this from an earlier discussion, see: https://news.ycombinator.com/item?id=32705776 https://news.ycombinator.com/item?id=32705776
- cootercharger 2y ago
  its called IBM i these days. DB2 originally didnt have a name because it was part of OS/400... Later DB2 name was applied and standalone versions released. "IBM i" is basically DB2OS if we want to talk about flipping the paradigm
  - pmcjones 2y ago
    DB2 was not part of OS/400 -- it was for MVS. Its early history is covered in this: https://mcjones.org/System_R/SQL_Reunion_95/index.html https://mcjones.org/System_R/SQL_Reunion_95/index.html
- natmaka 2y ago
  Wasn't the old "38" ( https://en.wikipedia.org/wiki/IBM_System/38 https://en.wikipedia.org/wiki/IBM_System/38 ) already built upon this concept?
gregw2 2y ago
Prior threads on DBOS: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=Dbos&sort=byPopularity&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
spaintech 2y ago
Been following this development here for some time… many threads already provide for some great comments on the idea. I find it both interesting and exciting to see a new os for the could era. I do feel like the DBOS is still not singular, PICK ( already mentioned) but also mentioned in other thread where Taurus OS and the Plant 9 project, which also leverages a database on a file system as a core for system resource management. In my own thinking of a cloud OS, and particularly considering the focus on server-less applications as mentioned in the article, I have been inspired by Forth’s Dictionary data structure as a lightweight data structure concept where the core state and application contexts are managed through modular dictionaries, each with isolated and secure contexts. OS has a Core Dictiory, and each application is contained with in it’s own namespace dictionary… ect. Message passing between applications could be efficiently handled using a tagging/queue system, potentially extending to RDMA for distributed environments. This could offer a lightweight, performance-optimized alternative current OS options, particularly in resource-constrained or highly distributed systems for server-less infrastructures.
ChicagoDave 2y ago
Not sure if relational was the optimal choice. A graph db like Neo4j might be more effective.
bregma 2y ago
By 'invert the computing paradigm' they seem to mean 'reinvent IBM MVS and CICS from the 1970s'. Of course, it's still around and dominates some industries, but hey, where's the marketing spin in that?
- 082349872349872 2y ago
  If current OS's are large enough that it makes sense to put an entire DBMS under them, that sounds like one of the best arguments I've heard for microkernels since the 1980s (1970s?). (one might imagine that a microkernel schedules cores [time] and provides block access/mapping to storage layers [space]; while the DB layers would build first locking and tables, and then transactions, on top of that; after which come applications...)
  - bregma 2y ago
    That's exactly how QNX works today, except all drivers (eg. block device access, storage space) are just userspace processes, as is the database layer. But in that case, the OS is not layered on top of the database, they're all peers.
- cootercharger 2y ago
  The OS currently known as "IBM i" (fka os400) is in essence "DB2 OS". DB2 is available as a "standalone" RDBMS also, and they rename it every decade or so for marketing
- karmakaze 2y ago
  The way it could be different this time is that we have arrived at "The Network is the Computer" age. A datacenter running Facebook can look like one giant computer, so it could make sense to have a distributed operating system to run that on with a distributed database under it. Of course this effort may be unrelated to this view.
netmonk 2y ago
Foxpro on mdsos was already doing that.
deleted 2y ago
[deleted]
adamgordonbell 2y ago
Github: https://github.com/dbos-inc/ https://github.com/dbos-inc/ Commercial site: https://www.dbos.dev/ https://www.dbos.dev/
- yencabulator 2y ago
  Oh, weird. Their old abandoned code is at https://dbos-project.github.io/ https://dbos-project.github.io/ https://github.com/DBOS-project https://github.com/DBOS-project with no indication of this move.
pjmlp 2y ago
Why should this be surprising at all? Big iron RDMS have had the enterprise option to run as full OSes for quite some time now.
cootercharger 2y ago
this makes me think of the IBM i (fka OS400) Operating System, where the DB integration is so tight in with the OS. Originally DB2 did not have a name because it was so tightly part of an OS, it didnt need one until it came time to compare it to ther dbs and make standalone offerings of it. So therefore "IBM i" is basically DB2OS then?
- jasomill 2y ago
  The original (ca. 1988) name of the OS/400 SQL facility was SQL/400. DB2 was originally an unrelated RDBMS that ran on IBM's MVS mainframe OS, announced in 1983. Several years later, IBM expanded the use of the DB2 name to refer to a variety of separately-developed RDBMS products (the original DB2 for MVS, an older mainframe database that ran on DOS/VSE and VM/CMS, the AS/400 database, and the AIX RDBMS that eventually became DB2 for Linux, UNIX, and Windows).
deleted 2y ago
[deleted]
yencabulator 2y ago
Where's the code? I see a Typescript framework for writing database-using programs, but I see nothing resembling a kernel or an OS. It seems the SaaS is proprietary. This is weird since even the article points out they got early feedback that proprietary systems weren't interesting to people: > The first version of DBOS was written in Java and used VoltDB, the fast relational database created by Stonebraker over a decade ago. But early feedback from interested parties said a proprietary system was a no-go, so the commercial version was rewritten to use FoundationDB [...]

89 points by signa11 2y ago

js8 2y ago

I have been thinking about that idea (putting database in front of operating system, i.e. applications interact with the environment through database) for a while now, I think it's really neat. Another good reason is testing and verification. Your application becomes decoupled from the OS resources and you can verify what it does to them by comparing the database. Because the OS state is managed by database, it is formally described and you can verify what your application does to it. In some sense, it's an implementation of functional core/imperative shell idea. Also, because DBs are transactional, transactions will become a first-class concept in your operating system. Which is something that's long overdue for applications (although there are systems such as CICS that have been doing this for some time now).

digitalsankhara 2y ago
Have you heard of PickOS [0]? Maybe similar to what you are describing. I admit I just read the first couple of paragraphs of the article and thought, yeah - PickOS vibes. [0] https://en.wikipedia.org/wiki/Pick_operating_system https://en.wikipedia.org/wiki/Pick_operating_system
- kcartlidge 2y ago
  I was a Pick user back in the late 1980s and that was, indeed, my first thought too.

thisislife2 2y ago

Interesting - we've been hearing about a "database file system" for some time now, and it's exciting to see it popping up slowly. But DBOS still has a long way to go - runs only on AWS currently and you are limited to Typescript for development (which is a curious choice - any idea why?). > DBOS will resist being on-prem and will resist being POSIX compliant, Stonebraker said ... They may also want things like support for Python and Java programming environments, and support for running in Azure and GCP, which will be determined in the future.

phlakaton 2y ago
BeOS had a database-based filesystem way back in the early 1990s. It was slow and difficult to keep up to date. They swapped it out with BFS and the rest was history. It's possible this will do better than Be's old database system, but this is not a novel idea.
- jimhefferon 2y ago
  In the article he says that DBs are now much faster.
  - snovv_crash 2y ago
    People are also much happier these days to introduce a factor of 10 slowdown relative to what the hardware can natively support.
- pjmlp 2y ago
  IBM big iron systems still use databases as file system.

deleted 2y ago

[deleted]

deepfriedchokes 2y ago

Sounds kind of like what we live in. The universe is the database and life the operating system(s).

hestefisk 2y ago
And the network is the computer :)

random3 2y ago

This isn't a bad idea. However, assuming heterogenous hardware(CPUs, GPUs, RAM, Network, capabilities) the OS problem is primarily a driver problem. Resource management seem like a candidate for a microkernel and this can expose a relational interface to a distributed OS. The distributed OS is primarily a resource manager and well fit for a similar relational interface. Provided that, besides drivers, the state management (tasks) is a relatively big issue, this makes the whole setup not far from a DB architecture and most managers/schedulers are arguably easily generalizable to a relational interface. In fact most distributed systems run on some form of distributed state management / DB (e.g. Raft, Zookeeper) This said, I think the more granular approach for the kernel is maybe a cleaner way to think about it. Also, any pragmatic approach would need to take drivers into account and I wonder if it's realistic to assume anything else but major kernels.

rwmj 2y ago
As they seem to be targeting only cloud, the set of drivers you actually need to implement is relatively small. eg. Plenty of "toy" OSes get away with implementing just a few virtio drivers and hence can run on qemu, KVM, and AWS, which is enough for them.
- random3 2y ago
  fair enough, but wouldn't a virtualization layer in between defeat the purpose?
  - rwmj 2y ago
    I'm not quite sure what you mean, but if you target the cloud as the platform (as is stated in the article), then you always have a virtualization layer, even if you provision on something like AWS Bare Metal Instances. (The reason bare metal instances use virtualization is not obvious: It's so you don't try to reflash the firmware on devices for a persistent attack.)
    - random3 2y ago

somat 2y ago

I worked at one place, they had heavily invested into pick style databases, when pick databases still made sense, and for some of the crusty old pick programmers still around, that was the os, it was their development environment and shell. https://en.wikipedia.org/wiki/Pick_operating_system https://en.wikipedia.org/wiki/Pick_operating_system Note: I don't think they had any actual pick systems left, by the time I got there they were mostly using unidata on linux, but there were still a lot unidata on sco boxes around.

tonnydourado 2y ago
> It is named after one of its developers, Dick Pick. You gotta be fucking kidding me
- kcartlidge 2y ago
  And: > Pick was originally implemented as the Generalized Information Retrieval Language System (GIRLS) As an aside, though, I really enjoyed working on a Pick system a few decades back.

xixixao 2y ago

This boils down to a serverless platform with storage and scheduling - which is a great idea! Add reactivity and you get Convex: https://www.convex.dev/ https://www.convex.dev/

sgt101 2y ago
I think that they are more radical - it seems like they have implemented a unikernel that supports the database service and their claim is that that's all you need for app dev. I think it looks rather similar (as a component) to something like snowflake - lots of data optimised services, but lower level with more app capability. The problem they will have is that a lot of the benefit that they are pushing is created by the big cloud data warehouses already, and the pain of a total re-implementation rather than re-architect or lift-and-shift completely terrifies most CIO's

yarg 2y ago

Reminds me of what Microsoft was trying to do for Longhorn. https://en.wikipedia.org/wiki/WinFS https://en.wikipedia.org/wiki/WinFS

pjmlp 2y ago
Nowadays part of it survives as ReFS. https://learn.microsoft.com/en-us/windows-server/storage/refs/refs-overview https://learn.microsoft.com/en-us/windows-server/storage/ref...

Guthur 2y ago

[flagged]

clcaev 2y ago
I wonder how much will be open source, and how much is special sauce. https://github.com/dbos-project https://github.com/dbos-project > This repository is research code and no longer maintained. The students working on it have graduated This is from "apiary". Lotus also seems experimental and limited to GCP, so I guess not?
deleted 2y ago
[deleted]

fmajid 2y ago

There is still an OS. Calling it a “minimal kernel” in the diagram is just semantic word games.

Validark 2y ago

Looks cool, but if you're building a stack for everyone else to build atop, why would you start with non-bare metal languages? I see C++ is in there, which is fine depending on who is wielding it, but Java and TypeScript necessarily add overhead that to me just doesn't make sense to add to all systems that want to run on your platform?

unnouinceput 2y ago
Java was in 1st iteration and was replaced by Typescript
- slowmovintarget 2y ago
  So they replaced an entire language with a linter?

nialse 2y ago

Sounds a lot like OS/400 in my ears. A machine interface that virtualizes the hardware and a database on top of that which applications are built upon. So, maybe not that new of an idea?

gregw2 2y ago
For a thread on this from an earlier discussion, see: https://news.ycombinator.com/item?id=32705776 https://news.ycombinator.com/item?id=32705776
cootercharger 2y ago
its called IBM i these days. DB2 originally didnt have a name because it was part of OS/400... Later DB2 name was applied and standalone versions released. "IBM i" is basically DB2OS if we want to talk about flipping the paradigm
- pmcjones 2y ago
  DB2 was not part of OS/400 -- it was for MVS. Its early history is covered in this: https://mcjones.org/System_R/SQL_Reunion_95/index.html https://mcjones.org/System_R/SQL_Reunion_95/index.html
natmaka 2y ago
Wasn't the old "38" ( https://en.wikipedia.org/wiki/IBM_System/38 https://en.wikipedia.org/wiki/IBM_System/38 ) already built upon this concept?

gregw2 2y ago

Prior threads on DBOS: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=Dbos&sort=byPopularity&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...

spaintech 2y ago

Been following this development here for some time… many threads already provide for some great comments on the idea. I find it both interesting and exciting to see a new os for the could era. I do feel like the DBOS is still not singular, PICK ( already mentioned) but also mentioned in other thread where Taurus OS and the Plant 9 project, which also leverages a database on a file system as a core for system resource management. In my own thinking of a cloud OS, and particularly considering the focus on server-less applications as mentioned in the article, I have been inspired by Forth’s Dictionary data structure as a lightweight data structure concept where the core state and application contexts are managed through modular dictionaries, each with isolated and secure contexts. OS has a Core Dictiory, and each application is contained with in it’s own namespace dictionary… ect. Message passing between applications could be efficiently handled using a tagging/queue system, potentially extending to RDMA for distributed environments. This could offer a lightweight, performance-optimized alternative current OS options, particularly in resource-constrained or highly distributed systems for server-less infrastructures.

ChicagoDave 2y ago

Not sure if relational was the optimal choice. A graph db like Neo4j might be more effective.

bregma 2y ago

By 'invert the computing paradigm' they seem to mean 'reinvent IBM MVS and CICS from the 1970s'. Of course, it's still around and dominates some industries, but hey, where's the marketing spin in that?

082349872349872 2y ago
If current OS's are large enough that it makes sense to put an entire DBMS under them, that sounds like one of the best arguments I've heard for microkernels since the 1980s (1970s?). (one might imagine that a microkernel schedules cores [time] and provides block access/mapping to storage layers [space]; while the DB layers would build first locking and tables, and then transactions, on top of that; after which come applications...)
- bregma 2y ago
  That's exactly how QNX works today, except all drivers (eg. block device access, storage space) are just userspace processes, as is the database layer. But in that case, the OS is not layered on top of the database, they're all peers.
cootercharger 2y ago
The OS currently known as "IBM i" (fka os400) is in essence "DB2 OS". DB2 is available as a "standalone" RDBMS also, and they rename it every decade or so for marketing
karmakaze 2y ago
The way it could be different this time is that we have arrived at "The Network is the Computer" age. A datacenter running Facebook can look like one giant computer, so it could make sense to have a distributed operating system to run that on with a distributed database under it. Of course this effort may be unrelated to this view.

netmonk 2y ago

Foxpro on mdsos was already doing that.

deleted 2y ago

[deleted]

adamgordonbell 2y ago

Github: https://github.com/dbos-inc/ https://github.com/dbos-inc/ Commercial site: https://www.dbos.dev/ https://www.dbos.dev/

yencabulator 2y ago
Oh, weird. Their old abandoned code is at https://dbos-project.github.io/ https://dbos-project.github.io/ https://github.com/DBOS-project https://github.com/DBOS-project with no indication of this move.

pjmlp 2y ago

Why should this be surprising at all? Big iron RDMS have had the enterprise option to run as full OSes for quite some time now.

cootercharger 2y ago

this makes me think of the IBM i (fka OS400) Operating System, where the DB integration is so tight in with the OS. Originally DB2 did not have a name because it was so tightly part of an OS, it didnt need one until it came time to compare it to ther dbs and make standalone offerings of it. So therefore "IBM i" is basically DB2OS then?

jasomill 2y ago
The original (ca. 1988) name of the OS/400 SQL facility was SQL/400. DB2 was originally an unrelated RDBMS that ran on IBM's MVS mainframe OS, announced in 1983. Several years later, IBM expanded the use of the DB2 name to refer to a variety of separately-developed RDBMS products (the original DB2 for MVS, an older mainframe database that ran on DOS/VSE and VM/CMS, the AS/400 database, and the AIX RDBMS that eventually became DB2 for Linux, UNIX, and Windows).

deleted 2y ago

[deleted]

yencabulator 2y ago

Where's the code? I see a Typescript framework for writing database-using programs, but I see nothing resembling a kernel or an OS. It seems the SaaS is proprietary. This is weird since even the article points out they got early feedback that proprietary systems weren't interesting to people: > The first version of DBOS was written in Java and used VoltDB, the fast relational database created by Stonebraker over a decade ago. But early feedback from interested parties said a proprietary system was a no-go, so the commercial version was rewritten to use FoundationDB [...]