12 ms·
DBOS: A database-oriented operating system
- amelius 4y agoSounds like something you could implement on top of an existing OS like Linux (even in userspace) and get mostly the same advantages.
- laweijfmvo 4y agoThis was my first thought, and probably should be the first pass IMO. If they have to implement an entire novel OS to support this it will never be anything beyond a lab experiment.
- Jtsummers 4y agoSounds like that's actually their plan for the first pass (reading their papers now): > While the DBMS engine will need some basic resource management functionality to bootstrap its execution, this could be done over a cluster of servers running current OSs, and eventually bootstrapped over the new DBOS. https://arxiv.org/abs/2007.11112 https://arxiv.org/abs/2007.11112
- esjeon 4y agoYeah, and TBH this is not a good reason to forge up a completely new OS. Even though the idea itself will surely improve performance, the immatureness alone will likely offset the benefits. It'll need years of improvements and tuning, which directly translates to $$$. Worse, normal people will not test the new OS for free, which is how things worked for Linux.
- melony 4y agoWindows Registry doesn't count?
- karmakaze 4y agoPerhaps not, but WinFS (Windows Future Storage)[0] comes closer. > In 2013 Bill Gates cited WinFS as his greatest disappointment at Microsoft and that the idea of WinFS was ahead of its time, which will re-emerge. WinFS was supposed to be released with the codename Longhorn version of Windows, but Vista was released without it. WinFS was disbanded and parts incorporated into other products and features. [0] https://en.wikipedia.org/wiki/WinFS https://en.wikipedia.org/wiki/WinFS
- rwmj 4y agoIs there any work or implementation? Edit yes, this would be a better link: https://dbos-project.github.io/ https://dbos-project.github.io/
- wyan 4y agoAS/400 follows a similar idea if I recall correctly, on top of DB2
- skissane 4y agoDubious. IBM's marketing wants to convince you it does, but as far as I can work out, the integration of DB2 into the system is nowhere near as deep as the marketing makes it sound.
- nrr 4y agoIt isn't actually Db2. They call it "Db2 for i," but it's a completely different codebase from what you'll find on z/OS and elsewhere. That said, the object-relational database goes down pretty deep in the system. IBM i (as OS/400 is now called) relies on it for storing things like source code and bound programs in addition to, well, data. Aside from IFS, which seems seldom used except for Java, it's basically the way to store things. If you have a library (IBM i-ese for a directory, kinda, but it's actually more like a schema in something like PostgreSQL), say, called YCBR1 that contains a physical file named QRPGLESRC, you can run `SELECT srcseq, srcdta FROM ycbr1/qrpglesrc` against it even though the members in that physical file are RPG IV source code, and you'll get the usual kind of resultset that you'd expect from a relational database. Likewise, you'll get something similar for binaries.
- skissane 4y ago> It isn't actually Db2. They call it "Db2 for i," but it's a completely different codebase from what you'll find on z/OS and elsewhere. From what I understand, the DB2/400 code base was forked from SQL/DS – so its closest relative is Db2 for VM/VSE – and Db2 for i is as much Db2 as Db2 for VM/VSE or Db2 for LUW is. (Db2 for z/OS is itself descended from SQL/DS – it started out as a port of SQL/DS from VM/CMS to MVS, although I believe most or all of the SQL/DS code was rewritten early in its development history and there may be little or none left by now.) > IBM i (as OS/400 is now called) relies on it for storing things like source code and bound programs in addition to, well, data Not really. You are talking about storing objects (including files) in the classic OS/400 filesystem – which hasn't fundamentally changed from that of S/38 – and S/38 didn't have SQL. During the development of OS/400, they took the higher layers of SQL/DS and ported them on top of the S/38 file system. But if we are talking about how RPG code is stored, that's all about those lower layers which predate SQL support > YCBR1 that contains a physical file named QRPGLESRC, you can run `SELECT srcseq, srcdta FROM ycbr1/qrpglesrc` against it even though the members in that physical file are RPG IV source code Nowadays, many other RDBMS systems support querying OS files as if they were DB tables. I won't deny that IBM i makes it somewhat more seamless than most other systems – but I think that's mainly because it is seen as somewhat of a niche requirement, other systems could easily make it more seamless if it was seen as a priority. No "deep" integration with the OS is necessary to implement it, either
- crazygringo 4y agoI understand conceptually how you could architect a desktop/server OS based on a local database instead of a local filesystem, and it's deeply intriguing to me in terms of how everything could share a common data language that is far more flexible than files but far more structured than text. Presumably things like terminal output would be formatted as query results instead of text. The database wouldn't reside in files on disk, it would reside directly in blocks on disk and in-memory as required. But this seems to be proposing a distributed database that runs across computers, which confuses me. Anything that runs across computers seems to me to be application-level, not OS-level, by definition. What does it even mean to be running a distributed database on "microkernel services" instead of on fully-fledged OS's? And then... where is the OS-level CPU coming from? If the database is distributed among computers then fine, but which computer is "running" the OS? Both the nomenclature and distributed aspect are really throwing me off here.
- pjmlp 4y agoHave a look at IBM i (nee AS/400), it uses database instead of files, aptly named catalogs.
- skissane 4y agoDisagree, IBM i uses files: how do you create a database table in it? CRTPF command ("Create Physical File"). You create a file. And I don't know what you mean by "catalogs", in the context of IBM i. Are you talking about DB2 catalog views? (Which exist in DB2 on every other platform, and most other RDBMS have something equivalent, such as the ANSI standard INFORMATION_SCHEMA) Or are you confusing IBM i with MVS (in which the OS contains databases called "catalogs", in which you lookup a file name or file name prefix to find out which disk volume a file is stored on?)
- pjmlp 4y agoI might have mixed it up with object libraries.
- 4y ago
- drewcoo 4y agoNeat idea, but so many questions . . . how are process boundaries on data preserved? If data can be shared between processes via the db, how do they enforce clean, clear, testable interfaces (like monoliths lack)? And given all that, how do they manage data schema changes that we'd handle now with API versioning?
- marginalia_nu 4y ago> clean, clear, testable interfaces (like monoliths lack) What prevents a monolith from having these things?
- quickthrower2 4y agoOther than Conway’s law; nothing
- balentio 4y agoThe cloud is all ready a security nightmare. I don't want to put more stuff in it.
- pjmlp 4y agoWell, Oracle APEX applications written in PL/SQL are a way to approach this.
- PaulDavisThe1st 4y agoOK, so I've got 50GB of audio samples. Does anyone actually believe that these 50GB (400 giga-bits) are more efficiently stored in a database designed for "stuff" than they are in a database designed for files ("a filesystem") ?
- wswope 4y agoCan be; it’s all contextual. https://www.sqlite.org/fasterthanfs.html https://www.sqlite.org/fasterthanfs.html
- PaulDavisThe1st 4y agoFair enough. The problem is that at some point, the data has to hit some sort of storage hardware. Presumably between the DB and the hardware, there's some layer that somewhat abstracts the storage hardware. Isn't that ... a filesystem?
- tremon 4y agoThat's a storage volume (partition, raid volume, zfs pool, etc), not a filesystem. A filesystem is the abstraction layer on top of the storage volume translating the user-assigned data identifiers (aka file names) to byte ranges. Talking specifically about databases: they often implement their own data organization. Oracle and Sybase famously performed better when working on raw partitions than with files.
- samus 4y agoA filesystem offers a hierachical interface. Meanwhile, a DBMS needs nothing more from the OS than access to blocks and preferably information about HDD layout. That's a level below.
- hcta 4y agoCan you clarify what point you're making? If you're trying to argue that adding an extra layer can only reduce performance, any cache is an obvious exception to that. Are you saying it's extraneous to use a database as a storage abstraction because they have to sit on top on filesystems, and filesystems already exist?
- didgetmaster 4y agoI have been working for years on a system designed to do much of what is described in their 'Level 2' layer. It is a single system that can effectively manage unstructured (i.e. file) data, semi-structured data (NoSql), and highly structured data (RDBMS). I don't have the resources available to me like this team, so there are still a lot of features on my TODO list that would enable some of the things they are looking for, but it can do many things now. It can manage 200M+ files and find subsets of them in sub-second speed. It can build relational tables with hundreds of millions of rows and thousands of columns and perform queries faster than many conventional DBs. www.Didgets.com is where you can download the beta software. Demo videos at https://www.youtube.com/channel/UC-L1oTcH0ocMXShifCt4JQQ https://www.youtube.com/channel/UC-L1oTcH0ocMXShifCt4JQQ
- goldenkey 4y agoHow do you deal with the aspects of CAP or ACID?
- didgetmaster 4y agoAs many people know, CAP stands for Consistency, Availability, and Partitioning which applies to distributed systems. You get to pick two. I chose the AP part with eventual consistency. 'Eventual' is the term with variability. Some systems are faster than others in bringing the distributed nodes up-to-date with data changes across the system. I believe I have some algorithms that complement the structure of the data to make the window of inconsistency as small as possible. The system is also designed for full ACID compliance, but there is still some work to be done in this area.
- AlbertCory 4y ago"A database-oriented operating system" -- where have I heard this before? Oh, right: https://en.wikipedia.org/wiki/Pick_operating_system https://en.wikipedia.org/wiki/Pick_operating_system (No, I never used this.) Incidentally, the name of its creator would be pretty unfortunate these days: Dick Pick.
- deleted 4y ago[deleted]
- fernly 4y agoI expect they will acknowledge the Pick OS? "a demand-paged, multiuser, virtual memory, time-sharing computer operating system based around a MultiValue database." [0] https://en.wikipedia.org/wiki/Pick_operating_system https://en.wikipedia.org/wiki/Pick_operating_system
- jacknews 4y ago"Pick was originally implemented as the Generalized Information Retrieval Language System (GIRLS) on an IBM System/360 in 1965 by Don Nelson and Richard (Dick) Pick" I'm sot sure the acronym would stand today, nor indeed Richard Pick's nickname.
- skissane 4y agoPick's "multi-value database" is essentially just a flat-file database. I don't see how – if we put aside the marketing – it was really any more "database-oriented" than your average mainframe/minicomputer operating system with a record-oriented filesystem, such as MVS (especially VSAM), VM/CMS, OpenVMS (in particular its Files-11 RMS component), etc.
- Cupertino95014 4y agoFrom my (very dim) memory, Pick was popular because it did everything some users needed and was easy to use (relatively speaking). I never met anyone who used it, though.
- skissane 4y agoIt was built on BASIC – and it really took off at a time (late 1970s / early 1980s) when BASIC was still a very major programming language. I think its database features have been somewhat overhyped – but no doubt an area of strength when compared to other lower-end systems of the same time period. The fact that the OS, programming language and database were all a single software package, developed together and sold as a unit, attracted people. Much of its initial success was also due to its marketing strategy – target ISVs who would resell it as the basis of turnkey solutions. It was based on a virtual machine, which made it easy to port to disparate hardware platforms – everything from PCs, to multiple incompatible minicomputer lines, to IBM mainframes – a key selling point in its day – although not a unique approach (see UCSD Pascal). Other approaches to portability – such as the C programming language – hadn't really taken off yet.
- airocker 4y ago""" At Level 1, a kernel provides low-level OS services such as device drivers and memory management. At Level 2, a distributed DBMS runs on those services. At Level 3, we build high-level OS services such as a distributed file system, cluster scheduler, and distributed inter-process communication (IPC) subsystem on top of the DBMS. At Level 4, users write applications. """ How would it be different from installing today's kernel and then installing postgres, Gluster, docker and etcd on top of it?
- airocker 4y agowould you force level 4 applications to only access level 3 services and model everything that OS does (device management, process management, memory management etc) as a layer on top of it? So essentially all devices are a table? I think just modeling memory access as a table would be a big win. Not sure how atomicity and consistency would help applications exactly though with every memory access. Would love to know.
- Jtsummers 4y agoProbably shortens the path between the applications (built on their distributed database manager) and the hardware/network compared to running applications on top of Postgres or other DBs today (which still end up calling out to the kernel and other subsystems). If all your applications (as seems to be their intent) don't need the Linux kernel, but only the DB, then push the DB service into the operating system. Building on top of too many layers increases the overall complexity and reduces overall performance. Periodically chopping out part of the system and creating what you actually need, fresh, is sometimes necessary. Even if it doesn't produce a new product or final system on its own, gives you a direction for moving other systems if the theory pans out in practice.
- deleted 4y ago[deleted]
- airocker 4y agoI guess the changes will be mostly within the kernel and not outside(possibly some new APIs). I thought applications would have to change which possibly is too hard to do. So anyone who misunderstood it the same way I did, nothing will change for an application developer.
- shrubble 4y agoYou could start doing an open-source version of this by basically installing a barebones Linux kernel, installing PostgreSQL on it, then, writing all services to run within PG. If you want multi-server, distributed, there are already ways to do that with PostgreSQL. It would get you 90% of the way there...
- pstuart 4y agoThe second and third 90% can be challenging.
- shrubble 4y agoIt's funny, but it's true!
- ta988 4y agoPostgresql multi-server for reads yes. For writes it is a bit harder. There are commercial offerings for that last I checked.
- esjeon 4y agoSo it's time to create fuse-postgres?
- shrubble 4y agoI doubt this would run well on a current version of PostgreSQL, but might have some ideas... https://www.linuxjournal.com/article/1383 https://www.linuxjournal.com/article/1383 (from 1997!)
- chunsj 4y agoOS/400?
- tapirl 4y agoor Mesos?
- wikitopian 4y agoThis is the future, but they're doing it backwards. Start at the top with sqlite compiled to webassembly as a virtual machine and prove that the tabular paradigm is superior to the scalar, vector, and object paradigms. It'll be more fun that way. The way that webassembly and wasi are designed to not fall back on unix tradition creates the opening to escape from the traditional files, streams, and objects universe.
- quasarj 4y agoHonest question, what are some (or at least one) distributed databases that actually work? Asking for.. a friend
- alexnewman 4y agoremember aix? big claim to fame was it was built around a db. nothing new under the sun i guess
- stonogo 4y agoWhen I read the title I immediately thought of PalmOS, and indeed it seems to be similar in architecture, with the major difference being that the database is hosted elsewhere. I'm interested to know if the advantages that came from PalmOS' database-focused APIs carry over into the distributed systems world!
- claytongulick 4y agoI'm just going to be the first to propose it: SQLiteOS
- euroderf 4y agoEvery "subcollection" is just another SQLite DB embedded as a (searchable) BLOB.
- deleted 4y ago[deleted]
- Scootwilli90 4y ago[flagged]
- zubairq 4y agoThere are already some Dbos type systems out there. I built one which stores program state in SQLite databases and process state and programs are also stored in SQLite. In the past I believe things like silver stream did the same too. The project I made is open source too: https://github.com/yazz/yazz https://github.com/yazz/yazz
- throwaway398035 4y agoI understand an OS to be something like the "base" software that runs on a single computer to unify the various pieces of hardware that are in it. I was surprised that this is a distributed system, but on second thought, it makes sense: "multiple CPU's" is just really "more hardware". The network is just another bus, in a sense. In my opinion it's an extra "innovation token" though, since it's not standard practice to build distributed OS's. With the addition of building a database-oriented system is going to be a hell of a challenge.
- bob1029 4y agoI feel like we are getting close to something approximating this with an experimental branch of our B2B product. We designed a schema and range of application-defined functions that allow for us to use SQLite as a domain-specific language. In addition to domain data, the schema also stores the authoritative view state as well as any intermediate function arguments. The really nice thing about this is all of the support around the edges. You can use SQLite’s EXPLAIN keyword to get an output of how the virtual machine would execute your statements, including any custom functions. You can also trace the result of every command in separate tables for troubleshooting - Each command starts with SELECT so we might as well do something useful with this. It’s a “reflective” approach since the scripts for handing the various events reside in the same database upon when they execute and can be inspected or altered by script execution. There’s also the magic of the WAL. If you are doing everything in the database (presumably because it’s your operating system) then things like incremental replication of state to another computer become trivial.
- samsquire 4y agoI journal computer ideas and the ideas from database engineering are yet to percolate everywhere, especially to the desktop environment. Why is every company building frontends and backends when the CRUD problem could be solved properly once and for all and reused everywhere? We did the same for communication and kernels with Linux, Windows and BSD, and BSD sockets which is shared by practically everybody. Your React frontend is legacy and shall be rewritten in 5 years. But BSD sockets or the Linux kernel doesn't get rewritten everyday. Rather than writing hand rolled code for querying data structures and manipulating them as Linux does, we can define queries that retrieve data structures in in the shape we're looking for. To put this simply, this is extremely high level, and the idea that data layout, data structure and algorithm can be unaggregated for cache locality and performance and developer experience. We can form materialized views on top of other materialized views and calculate the most efficient retrieval and storage format based on the structure of the data. I suspect a materialized view, as in the data structures of the Linux kernel is more efficient than materializing a join at runtime. One of my ideas is "ideas4 9. Query for data structure", https://github.com/samsquire/ideas4#9-query-for-data-structure https://github.com/samsquire/ideas4#9-query-for-data-structu... which is the idea we should be capable of querying to retrieve data structure in the shape we want. The shape of the data lends itself to solving certain kinds of problems. An ideas3 is "Query database" https://github.com/samsquire/ideas3#17-query-database https://github.com/samsquire/ideas3#17-query-database, we persist queries as we persist data and use them to optimise data storage format. Rather than using a cost based optimiser to optimise how to retrieve data in the same uniform structure, we optimise how to store data itself. For example, for write heavy loads, we use LSM trees, we persist columnular when a query uses every field of every row and so on. I also had the idea # 10. in ideas4 to persist data access patterns directly and optimise that. https://github.com/samsquire/ideas4#10-access-pattern-serialization https://github.com/samsquire/ideas4#10-access-pattern-serial...
- jonahbenton 4y agoI read this overview several times and I remain confused. It seems like a rehash of plan 9/network-is-computer paradigms. I get all the reasons that is interesting now- networking is REALLY fast and lots of cache hierarchy and other assumptions in kernels just don't apply any more- but to place the emphasis on the areas of innovation on the use of tables and transactions as core distributed primitives for applications...is strange. While maybe beneficial for system programmers, it definitely isn't a more understandable paradigm for application programmers- very few of whom who write code that saves data understand the varieties of transaction semantics. And in terms of applications, what decades of failed "database under the filesystem with typed object storage" work has shown also that the data types and semantics applications need are always more imaginative than those exposed at lower levels. Someone used to working with graphs, when given tables, is going to walk away. Where is the ML layer that optimizes under a "save this thing" surface that actually presents a new stable abstraction? But in true armchair architecture sniper fashion, I'm going to leave my confusion as is and move on. Best wishes to them.