16 ms·

Database as Filesystem [video]

175 points by enobrev 7y ago

kragen 7y ago
This was the idea behind murdererfs, right? It worked pretty well to power mp3.com for a few years. The great https://philip.greenspun.com/panda/databases-choosing https://philip.greenspun.com/panda/databases-choosing talks a bit about the issue: The great thing about the file system is its invisibility. You probably didn't purchase it separately, you might not be aware of its existence, you won't have to run an ad in the newspaper for a file system administrator with 5+ years of experience, and it will pretty much work as advertised. All you need to do with a file system is back it up to tape every day or two. He goes on to say that the main thing RDBMSes add to your web-serving life is ACID transactions, which I think is pretty much true. Sadly modern filesystems still don't support ACID transactions, though often they do manage to export sufficient functionality for user-level code like Postgres and SQLite to do so.
- madhadron 7y ago
  > you won't have to run an ad in the newspaper for a file system administrator with 5+ years of experience Until you reach an extreme scale, at which point you hire the author of the filesystem.
- brandmeyer 7y ago
  Its going to be hard to retrofit transactions into the POSIX model with anything other than a very coarse granularity. In the SQL model, statements are assigned to a transaction based on which connection they came through. How should POSIX filesystem transactions be scoped? Without rewriting software, its hard to use anything finer-grained than an entire process tree.
  - kragen 7y ago
    An entire process tree costs 100 us of CPU time to create and destroy on modern Linux, so that mechanism could deliver tens of thousands of transactions per second on a modern cellphone. That's probably more than the underlying Flash can make durable, but with nested transactions you might be able to take advantage of more than that. Still, it might be adequate. Unfortunately we are saddled with three generations of software that uses threads like they're going out of style (which they are) due to an unfortunate confluence of Win16 refugees with a misguided attempt to run our transaction processing software on a virtual machine designed for a set-top box with no MMU. You can fork and exit on Linux in 100 us, but not if you have sixteen threads and 512 megabytes of virtual address space spread across 128 mappings.
- the8472 7y ago
  > Sadly modern filesystems still don't support ACID transactions They offer range locking, optimistic locking, snapshots, atomic swaps, durable writes and various other interesting features as primitives. If you're willing to build some data storage around a specific filesystem you can do a lot. The correct incantations to get those features can be a bit gnarly, but so is coding database engines. NTFS even offers actual transactions, although their use is discouraged.
  - kragen 7y ago
    Interesting, why is their use discouraged? Because Jim Gray’s sailboat sank?
    - the8472 7y ago
      https://docs.microsoft.com/en-us/windows/win32/fileio/deprecation-of-txf https://docs.microsoft.com/en-us/windows/win32/fileio/deprec...
      - kragen 7y ago
        This just says that since nobody uses it they might delete it, but it doesn't explain why nobody uses it. Presumably because it doesn't work well or is hard to use, but which, and how?
- deleted 7y ago
  [deleted]
- Someone 7y ago
  https://en.m.wikipedia.org/wiki/Transactional_NTFS https://en.m.wikipedia.org/wiki/Transactional_NTFS provides ACID transactions, even, I think, across machines.
jamesblonde 7y ago
HopsFS is based around using an in-memory database as the metadata store for a filesystem - https://www.usenix.org/conference/fast17/technical-sessions/presentation/niazi https://www.usenix.org/conference/fast17/technical-sessions/... HopsFS is a drop-in replacement for HDFS, and the above paper showed 16X throughput improvement for Spotify's Hadoop workload over Apache HDFS. Since then, HopsFS has added small files in the metadata layer, with over 60X throughput improvement and an order of magnitude lower latency for writing small files (ACM Middleware'18): https://www.logicalclocks.com/millions-and-millions-of-files-deep-learning-at-scale-with-hopsfs/ https://www.logicalclocks.com/millions-and-millions-of-files...
lioeters 7y ago
Building a POSIX-compatible file system on top of a database engine (Postgres, MariaDB) on raw partitions.. I could see advantages like being able to have indexes, relationships, run queries on files' metadata, import/export the whole tree or branches (folders), maybe easier managing of distributed file systems.. Aside from the performance overhead for the sake of experimentation, from my not-very-knowledgeable perspective, the idea seems to have enough interesting implications that I'm curious to see where it could go. I also like other variations on this, like "file system as database", "Git repo as file system", and so on.
- dfee 7y ago
  Or alternatively, I'd like to see variations like: "Git repo as a database"; i.e. sqlite backed git. There are a whole class of problems here that I have little or no experience with, but seemingly could be incredibly impactful. For instance, further research and investment into versioning, event-logging, etc. could bring about a very rich revolution in OS / user experience. "What if... data (e.g. file) manipulations where semantic – rather than CRUD, you had application-layer higher level blocks that presented different abstractions to data below." Or, as I have more experience with SQL than FS, I could just be looking at the world through the lens of having a hammer and seeing everything as a nail.
  - notduncansmith 7y ago
    Not exactly “SQLite-backed Git” but have you looked at Fossil? https://www.fossil-scm.org/home/doc/trunk/www/index.wiki https://www.fossil-scm.org/home/doc/trunk/www/index.wiki
  - syn0byte 7y ago
    Everything is a nail. Whenever developer sorts get all starry eyed over "DB as an FS", they never consider data exchange between applications vendors and platforms. Their applications always exist alone in a digital vacuum. Example: Imagine trying to attach an MS Word document stored in this DBFS to an email from a Thunderbird client. What does that look like to the user? What does it look like to the Thunderbird developer? Abstract and encapsulate so that each application can use their own container format that includes schema and encoding details? Other applications can read this metadata directly from each containerized db object? Cool, we just reinvented the filesystem again.
RocketSyntax 7y ago
graph database could handle the infinite directory nesting with ease
amelius 7y ago
What happens to performance if you store a database inside this filesystem?
- matharmin 7y ago
  Just speculating, but the database is mostly used for the metadata. If implemented well, there will be very little overhead for reading and writing to a file, especially if the size stays constant. Some databases rely on specific low level file system operations, which may not be optimised in a niche filesystem. But in theory there shouldn't be anything preventing it from being optimized, and the database from running at close to maximum performance.
  - amelius 7y ago
    Ok, but if you store only metadata then you lose the advantage of running all the file operations inside a transaction. Then again, posix has no notion of transactions ...
  - Someone 7y ago
    ”there will be very little overhead for reading and writing to a file, especially if the size stays constant” Not if you add primary keys, indexes, ‘on update’ clauses, etc.
- rhacker 7y ago
  Docker -> Mysql Image -> This DB Filesystem -> Separate docker images that mount the DBFS over other Mysql image. Should be quite fun for a lot of people.
deepspace 7y ago
Altria did something similar almost 30 years ago with MVFS - https://www-01.ibm.com/support/docview.wss?uid=swg21230196 https://www-01.ibm.com/support/docview.wss?uid=swg21230196 - on top of RDM. The SCM tool (IBM Rational Clearcase) that makes use of MVFS to map a database of modifications to a file system is still around. I always admired the concept behind Clearcase / MVFS, though in practice it was a nightmare to administer.
- pjmlp 7y ago
  Clearcase might be a monster to deal with, but it has several advance features that I miss in FOSS alternatives. One being the ability to share build binaries across users.
bullen 7y ago
I do it the other way around, filesystem as database: http://root.rupy.se http://root.rupy.se Just learned to partition the SD card for my RPi 2 cluster properly with ext4 type "small" so I don't run out of inodes. Also purchased 6x 256GB Toshiba for 40$ each, I have been waiting for large and cheap microSD cards since 2015!
- flukus 7y ago
  TMSU (https://tmsu.org/ https://tmsu.org/) is another one like this. It's simple and inter-operates with everything that currently exists and it offers most features you'd want out of a database file system.
  - bullen 7y ago
    Ok, do you know how tmsu stores it's metadata? I found references to sqlite3 in the source. ROOT actually uses the filesystem AS database, it has no separate index files except the "link" ones and those only store a binary list of long id's to other files. ROOT has some features that are missing from tmsu though; like distributed real-time replication over async. HTTP (scales across continents). HTTP API with JSON support. User management and security. All of that within ~4000 lines of code. I built it to have a MMO database that doesn't need manual moving of data for region switching and backup while avoiding the sync. bottleneck of all other databases. Here are two sites that use it: http://talk.binarytask.com http://talk.binarytask.com and http://fuse.rupy.se http://fuse.rupy.se
andrewl 7y ago
The original Be File System (BFS) for BeOS had some attributes of a database. I'm happy to see more research being done on the concept.
cyphar 7y ago
This reminds me a little bit of the ideas behind Reiser4[1] -- the idea being that we could unify different namespaces of data (files, metadata of files, databases, anything) into a unified namespace which you could then use Unix tools on. It is a bit of a shame that the ideas behind Reiser4 have been lost with Hans Reiser's murder conviction, and nobody has really picked them back up since then. One of the neat ideas more directly related to this talk is that Reiser4 had the ability for users to define their own data layout (I think they were called "modules") which would allow you to optimise the filesystem layout for databases while still allowing non-database software to read the contents. [1]: https://www.youtube.com/watch?v=mIrMVPnxa04 https://www.youtube.com/watch?v=mIrMVPnxa04
vkaku 7y ago
There are different portions to a FS that actually need to be treated indivudually: - Bricks / Blocks / Blob data. - Metadata / Tags / Xactional data. The deal is that any DB <-> FS has to support both of these separately. The Bricks/Blocks are optimized for raw performance/seek/access pattern with very low overhead; The Metadata are optimized for fancy querying / fast lookup semantics. Any DB/Key Value store for large data could be modeled on top of a volume layer, like LVM / ZVOL; This could absolutely become the foundation of a FS as well. The deal is that extensible / flexible storage space used to be something filesystems used to provide, but those functions have been moved to a volume manager in regular systems. The reliability and metadata part can be moved to the database layer, so that leaves the filesystem doing a No-Op.
stackzero 7y ago
In a similar vein, I ran a little experimental project using redis as a distributed filesystem using this: https://steve.fi/Software/redisfs/ https://steve.fi/Software/redisfs/
MikeTaylor 7y ago
This kind of thing was one of the foundational notions of Plan 9, back in the late 1980s and early 1990s. I often wonder how different our world would be if they'd only had the foresight to release the code freely instead of keeping it proprietary.
oscargrouch 7y ago
Thats one of the reasons why need to rely more on micro-kernel design OS first, so it's easiar to advance and experiment with different designs for the storage layer. The "everything is a file" mindset was great to move us from the chaos they were in back in the day to where we are right now. But with the modern ecosystem is holding us all back, because is too rigid, and a abstraction that its a poor fit to a lot of modern scenarios. With micro-kernels, we could have the kernel dealing only with persistent storage blocks and leave the storage abstraction to the userspace where it belongs. If im not mistaken, The Zircon kernel go even further by abstracting through pages.. so you could just map a collection of blocks to those pages and deal only with the pages in memory, no matter they source (if disk, network or only on memory). Leaving the abstractions to userspace and by doing so, enabling us to download, install, and mix those abstractions in a way that is a good fit for a giving scenario. As we have seen, the monolithic kernel have trouble with the bazaar approach as those experiments have to be shipped in kernel-space, with all the trouble this entails.
burmecia 7y ago
Database and filesystem share a lot similarity, but still have their own characteristics. To use database as a back-end for filesystem has some benefits, such as reliable persistence, ACID transaction and etc. It reduces complexity to achieve durable storage which filesystem should deal with. Many filesystems are able to use DB as back-end, I also made a filesystem ZboxFS (https://github.com/zboxfs/zbox https://github.com/zboxfs/zbox) which can also use sqlite as underlying storage.
deleted 7y ago
[deleted]
gfody 7y ago
I think the real interesting value in putting an RDBMS underneath the filesystem would come from relationally modeling the objects in the filesystem, ie files and types and their contents, and then being able to query those things with SQL. If their relational schema is just inodes and blocks I don’t really see the value - after all the filesystem is just a miniature RDBMS purpose-built and optimized for that fixed schema and so it’s not surprising that their system is so much slower.
techntoke 7y ago
Would be awesome to query a DB, like `ls /db/america/ny/cities` using tags or something in files to automatically and predictably sort data into folders/files.
hartzell 7y ago
See also Michael Olson's "The Design and Implementation of the Inversion File System (1993)", from the 1993 USENIX conference (https://www.usenix.org/legacy/publications/library/proceedings/sd93/ https://www.usenix.org/legacy/publications/library/proceedin...), PDF here: http://db.cs.berkeley.edu/papers/S2K-93-28.pdf http://db.cs.berkeley.edu/papers/S2K-93-28.pdf

175 points by enobrev 7y ago

This was the idea behind murdererfs, right? It worked pretty well to power mp3.com for a few years. The great https://philip.greenspun.com/panda/databases-choosing https://philip.greenspun.com/panda/databases-choosing talks a bit about the issue: The great thing about the file system is its invisibility. You probably didn't purchase it separately, you might not be aware of its existence, you won't have to run an ad in the newspaper for a file system administrator with 5+ years of experience, and it will pretty much work as advertised. All you need to do with a file system is back it up to tape every day or two. He goes on to say that the main thing RDBMSes add to your web-serving life is ACID transactions, which I think is pretty much true. Sadly modern filesystems still don't support ACID transactions, though often they do manage to export sufficient functionality for user-level code like Postgres and SQLite to do so.

madhadron 7y ago
> you won't have to run an ad in the newspaper for a file system administrator with 5+ years of experience Until you reach an extreme scale, at which point you hire the author of the filesystem.
brandmeyer 7y ago
Its going to be hard to retrofit transactions into the POSIX model with anything other than a very coarse granularity. In the SQL model, statements are assigned to a transaction based on which connection they came through. How should POSIX filesystem transactions be scoped? Without rewriting software, its hard to use anything finer-grained than an entire process tree.
- kragen 7y ago
  An entire process tree costs 100 us of CPU time to create and destroy on modern Linux, so that mechanism could deliver tens of thousands of transactions per second on a modern cellphone. That's probably more than the underlying Flash can make durable, but with nested transactions you might be able to take advantage of more than that. Still, it might be adequate. Unfortunately we are saddled with three generations of software that uses threads like they're going out of style (which they are) due to an unfortunate confluence of Win16 refugees with a misguided attempt to run our transaction processing software on a virtual machine designed for a set-top box with no MMU. You can fork and exit on Linux in 100 us, but not if you have sixteen threads and 512 megabytes of virtual address space spread across 128 mappings.
the8472 7y ago
> Sadly modern filesystems still don't support ACID transactions They offer range locking, optimistic locking, snapshots, atomic swaps, durable writes and various other interesting features as primitives. If you're willing to build some data storage around a specific filesystem you can do a lot. The correct incantations to get those features can be a bit gnarly, but so is coding database engines. NTFS even offers actual transactions, although their use is discouraged.
- kragen 7y ago
  Interesting, why is their use discouraged? Because Jim Gray’s sailboat sank?
  - the8472 7y ago
    https://docs.microsoft.com/en-us/windows/win32/fileio/deprecation-of-txf https://docs.microsoft.com/en-us/windows/win32/fileio/deprec...
    - kragen 7y ago
      This just says that since nobody uses it they might delete it, but it doesn't explain why nobody uses it. Presumably because it doesn't work well or is hard to use, but which, and how?
deleted 7y ago
[deleted]
Someone 7y ago
https://en.m.wikipedia.org/wiki/Transactional_NTFS https://en.m.wikipedia.org/wiki/Transactional_NTFS provides ACID transactions, even, I think, across machines.

jamesblonde 7y ago

HopsFS is based around using an in-memory database as the metadata store for a filesystem - https://www.usenix.org/conference/fast17/technical-sessions/presentation/niazi https://www.usenix.org/conference/fast17/technical-sessions/... HopsFS is a drop-in replacement for HDFS, and the above paper showed 16X throughput improvement for Spotify's Hadoop workload over Apache HDFS. Since then, HopsFS has added small files in the metadata layer, with over 60X throughput improvement and an order of magnitude lower latency for writing small files (ACM Middleware'18): https://www.logicalclocks.com/millions-and-millions-of-files-deep-learning-at-scale-with-hopsfs/ https://www.logicalclocks.com/millions-and-millions-of-files...

lioeters 7y ago

Building a POSIX-compatible file system on top of a database engine (Postgres, MariaDB) on raw partitions.. I could see advantages like being able to have indexes, relationships, run queries on files' metadata, import/export the whole tree or branches (folders), maybe easier managing of distributed file systems.. Aside from the performance overhead for the sake of experimentation, from my not-very-knowledgeable perspective, the idea seems to have enough interesting implications that I'm curious to see where it could go. I also like other variations on this, like "file system as database", "Git repo as file system", and so on.

dfee 7y ago
Or alternatively, I'd like to see variations like: "Git repo as a database"; i.e. sqlite backed git. There are a whole class of problems here that I have little or no experience with, but seemingly could be incredibly impactful. For instance, further research and investment into versioning, event-logging, etc. could bring about a very rich revolution in OS / user experience. "What if... data (e.g. file) manipulations where semantic – rather than CRUD, you had application-layer higher level blocks that presented different abstractions to data below." Or, as I have more experience with SQL than FS, I could just be looking at the world through the lens of having a hammer and seeing everything as a nail.
- notduncansmith 7y ago
  Not exactly “SQLite-backed Git” but have you looked at Fossil? https://www.fossil-scm.org/home/doc/trunk/www/index.wiki https://www.fossil-scm.org/home/doc/trunk/www/index.wiki
- syn0byte 7y ago
  Everything is a nail. Whenever developer sorts get all starry eyed over "DB as an FS", they never consider data exchange between applications vendors and platforms. Their applications always exist alone in a digital vacuum. Example: Imagine trying to attach an MS Word document stored in this DBFS to an email from a Thunderbird client. What does that look like to the user? What does it look like to the Thunderbird developer? Abstract and encapsulate so that each application can use their own container format that includes schema and encoding details? Other applications can read this metadata directly from each containerized db object? Cool, we just reinvented the filesystem again.

RocketSyntax 7y ago

graph database could handle the infinite directory nesting with ease

amelius 7y ago

What happens to performance if you store a database inside this filesystem?

matharmin 7y ago
Just speculating, but the database is mostly used for the metadata. If implemented well, there will be very little overhead for reading and writing to a file, especially if the size stays constant. Some databases rely on specific low level file system operations, which may not be optimised in a niche filesystem. But in theory there shouldn't be anything preventing it from being optimized, and the database from running at close to maximum performance.
- amelius 7y ago
  Ok, but if you store only metadata then you lose the advantage of running all the file operations inside a transaction. Then again, posix has no notion of transactions ...
- Someone 7y ago
  ”there will be very little overhead for reading and writing to a file, especially if the size stays constant” Not if you add primary keys, indexes, ‘on update’ clauses, etc.
rhacker 7y ago
Docker -> Mysql Image -> This DB Filesystem -> Separate docker images that mount the DBFS over other Mysql image. Should be quite fun for a lot of people.

deepspace 7y ago

Altria did something similar almost 30 years ago with MVFS - https://www-01.ibm.com/support/docview.wss?uid=swg21230196 https://www-01.ibm.com/support/docview.wss?uid=swg21230196 - on top of RDM. The SCM tool (IBM Rational Clearcase) that makes use of MVFS to map a database of modifications to a file system is still around. I always admired the concept behind Clearcase / MVFS, though in practice it was a nightmare to administer.

pjmlp 7y ago
Clearcase might be a monster to deal with, but it has several advance features that I miss in FOSS alternatives. One being the ability to share build binaries across users.

bullen 7y ago

I do it the other way around, filesystem as database: http://root.rupy.se http://root.rupy.se Just learned to partition the SD card for my RPi 2 cluster properly with ext4 type "small" so I don't run out of inodes. Also purchased 6x 256GB Toshiba for 40$ each, I have been waiting for large and cheap microSD cards since 2015!

flukus 7y ago
TMSU (https://tmsu.org/ https://tmsu.org/) is another one like this. It's simple and inter-operates with everything that currently exists and it offers most features you'd want out of a database file system.
- bullen 7y ago
  Ok, do you know how tmsu stores it's metadata? I found references to sqlite3 in the source. ROOT actually uses the filesystem AS database, it has no separate index files except the "link" ones and those only store a binary list of long id's to other files. ROOT has some features that are missing from tmsu though; like distributed real-time replication over async. HTTP (scales across continents). HTTP API with JSON support. User management and security. All of that within ~4000 lines of code. I built it to have a MMO database that doesn't need manual moving of data for region switching and backup while avoiding the sync. bottleneck of all other databases. Here are two sites that use it: http://talk.binarytask.com http://talk.binarytask.com and http://fuse.rupy.se http://fuse.rupy.se

andrewl 7y ago

The original Be File System (BFS) for BeOS had some attributes of a database. I'm happy to see more research being done on the concept.

cyphar 7y ago

This reminds me a little bit of the ideas behind Reiser4[1] -- the idea being that we could unify different namespaces of data (files, metadata of files, databases, anything) into a unified namespace which you could then use Unix tools on. It is a bit of a shame that the ideas behind Reiser4 have been lost with Hans Reiser's murder conviction, and nobody has really picked them back up since then. One of the neat ideas more directly related to this talk is that Reiser4 had the ability for users to define their own data layout (I think they were called "modules") which would allow you to optimise the filesystem layout for databases while still allowing non-database software to read the contents. [1]: https://www.youtube.com/watch?v=mIrMVPnxa04 https://www.youtube.com/watch?v=mIrMVPnxa04

vkaku 7y ago

There are different portions to a FS that actually need to be treated indivudually: - Bricks / Blocks / Blob data. - Metadata / Tags / Xactional data. The deal is that any DB <-> FS has to support both of these separately. The Bricks/Blocks are optimized for raw performance/seek/access pattern with very low overhead; The Metadata are optimized for fancy querying / fast lookup semantics. Any DB/Key Value store for large data could be modeled on top of a volume layer, like LVM / ZVOL; This could absolutely become the foundation of a FS as well. The deal is that extensible / flexible storage space used to be something filesystems used to provide, but those functions have been moved to a volume manager in regular systems. The reliability and metadata part can be moved to the database layer, so that leaves the filesystem doing a No-Op.

stackzero 7y ago

In a similar vein, I ran a little experimental project using redis as a distributed filesystem using this: https://steve.fi/Software/redisfs/ https://steve.fi/Software/redisfs/

MikeTaylor 7y ago

This kind of thing was one of the foundational notions of Plan 9, back in the late 1980s and early 1990s. I often wonder how different our world would be if they'd only had the foresight to release the code freely instead of keeping it proprietary.

oscargrouch 7y ago

Thats one of the reasons why need to rely more on micro-kernel design OS first, so it's easiar to advance and experiment with different designs for the storage layer. The "everything is a file" mindset was great to move us from the chaos they were in back in the day to where we are right now. But with the modern ecosystem is holding us all back, because is too rigid, and a abstraction that its a poor fit to a lot of modern scenarios. With micro-kernels, we could have the kernel dealing only with persistent storage blocks and leave the storage abstraction to the userspace where it belongs. If im not mistaken, The Zircon kernel go even further by abstracting through pages.. so you could just map a collection of blocks to those pages and deal only with the pages in memory, no matter they source (if disk, network or only on memory). Leaving the abstractions to userspace and by doing so, enabling us to download, install, and mix those abstractions in a way that is a good fit for a giving scenario. As we have seen, the monolithic kernel have trouble with the bazaar approach as those experiments have to be shipped in kernel-space, with all the trouble this entails.

burmecia 7y ago

Database and filesystem share a lot similarity, but still have their own characteristics. To use database as a back-end for filesystem has some benefits, such as reliable persistence, ACID transaction and etc. It reduces complexity to achieve durable storage which filesystem should deal with. Many filesystems are able to use DB as back-end, I also made a filesystem ZboxFS (https://github.com/zboxfs/zbox https://github.com/zboxfs/zbox) which can also use sqlite as underlying storage.

deleted 7y ago

[deleted]

gfody 7y ago

I think the real interesting value in putting an RDBMS underneath the filesystem would come from relationally modeling the objects in the filesystem, ie files and types and their contents, and then being able to query those things with SQL. If their relational schema is just inodes and blocks I don’t really see the value - after all the filesystem is just a miniature RDBMS purpose-built and optimized for that fixed schema and so it’s not surprising that their system is so much slower.

techntoke 7y ago

Would be awesome to query a DB, like `ls /db/america/ny/cities` using tags or something in files to automatically and predictably sort data into folders/files.

hartzell 7y ago

See also Michael Olson's "The Design and Implementation of the Inversion File System (1993)", from the 1993 USENIX conference (https://www.usenix.org/legacy/publications/library/proceedings/sd93/ https://www.usenix.org/legacy/publications/library/proceedin...), PDF here: http://db.cs.berkeley.edu/papers/S2K-93-28.pdf http://db.cs.berkeley.edu/papers/S2K-93-28.pdf