4 ms·

Building a POSIX-compatible file system on top of a database engine (Postgres, MariaDB) on raw partitions.. I could see advantages like being able to have inde

Building a POSIX-compatible file system on top of a database engine (Postgres, MariaDB) on raw partitions.. I could see advantages like being able to have indexes, relationships, run queries on files' metadata, import/export the whole tree or branches (folders), maybe easier managing of distributed file systems.. Aside from the performance overhead for the sake of experimentation, from my not-very-knowledgeable perspective, the idea seems to have enough interesting implications that I'm curious to see where it could go. I also like other variations on this, like "file system as database", "Git repo as file system", and so on.

dfee 7y ago
Or alternatively, I'd like to see variations like: "Git repo as a database"; i.e. sqlite backed git. There are a whole class of problems here that I have little or no experience with, but seemingly could be incredibly impactful. For instance, further research and investment into versioning, event-logging, etc. could bring about a very rich revolution in OS / user experience. "What if... data (e.g. file) manipulations where semantic – rather than CRUD, you had application-layer higher level blocks that presented different abstractions to data below." Or, as I have more experience with SQL than FS, I could just be looking at the world through the lens of having a hammer and seeing everything as a nail.
- notduncansmith 7y ago
  Not exactly “SQLite-backed Git” but have you looked at Fossil? https://www.fossil-scm.org/home/doc/trunk/www/index.wiki https://www.fossil-scm.org/home/doc/trunk/www/index.wiki
- syn0byte 7y ago
  Everything is a nail. Whenever developer sorts get all starry eyed over "DB as an FS", they never consider data exchange between applications vendors and platforms. Their applications always exist alone in a digital vacuum. Example: Imagine trying to attach an MS Word document stored in this DBFS to an email from a Thunderbird client. What does that look like to the user? What does it look like to the Thunderbird developer? Abstract and encapsulate so that each application can use their own container format that includes schema and encoding details? Other applications can read this metadata directly from each containerized db object? Cool, we just reinvented the filesystem again.
  - smilliken 7y ago
    You're right, any viable replacement for a filesystem must be compatible with existing filesystem APIs. That doesn't mean you can't implement the storage layer differently, or add additional APIs. Look at ZFS. It's a filesystem, but internally looks like a database. If you try to define the terms "database" and "filesystem" generally, you'll realize how difficult it is to separate the concepts.
- wodenokoto 7y ago
  > SQLite backed git SQLite development is actually version controlled by an open source SQLite backed version control system called fossil.
jamesblonde 7y ago
You can read on the background and challenges here in Salman Niazi's PhD - http://kth.diva-portal.org/smash/get/diva2:1260852/FULLTEXT02.pdf http://kth.diva-portal.org/smash/get/diva2:1260852/FULLTEXT0.... This is not a new topic - WinFS (part of the failed Longhorn project) was the best known example, which tried to use SQL Server as the metadata store for a new windows FS. Performance killed them. The reason why we managed to get better performance compared to all previous attempts it three-fold: 1. we used an in-memory high performance DB (NDB) and we did not de-normalize our schemas (no caching!) 2. with denormalized tables, we needed cross-partition transactions at high performance 3. we did a reasonable job at re-implementing HDFS' POSIX-like model using read-isolated transactions and row-level locking and new protocol for sub-tree operations.
- lioeters 7y ago
  Thank you for raising these relevant points. They led me down a fascinating rabbit hole on this topic of "database as file system". This is way over my head, but to pique others' curiosity, here's the abstract: Recent improvements in both the performance and scalability of shared-nothing, transactional, in-memory NewSQL databases have reopened the research question of whether distributed metadata for hierarchical file systems can be managed using commodity databases. In this paper, we introduce HopsFS, a next generation distribution of the Hadoop Distributed File System (HDFS) that replaces HDFS’ single node in-memory metadata service, with a distributed metadata service built on a NewSQL database. By removing the metadata bottleneck, HopsFS enables an order of magnitude larger and higher throughput clusters compared to HDFS. Metadata capacity has been increased to at least 37 times HDFS’ capacity, and in experiments based on a workload trace from Spotify, we show that HopsFS supports 16 to 37 times the throughput of Apache HDFS. HopsFS also has lower latency for many concurrent clients, and no downtime during failover. Finally, as metadata is now stored in a commodity database, it can be safely extended and easily exported to external systems for online analysis and free-text search.
  - daviesliu 7y ago
    Have you publish the workload tracing from Spotify? We are very interesting on the results you got, looking for a way to replicate that using our setup.
novaleaf 7y ago
I remember being excited that this would be (eventually) coming to windows in the pre-Vista. It was called WinFS: https://en.wikipedia.org/wiki/WinFS https://en.wikipedia.org/wiki/WinFS Unfortunately this got canceled, IIRC due to difficulties with back-compat.

dfee 7y ago

Or alternatively, I'd like to see variations like: "Git repo as a database"; i.e. sqlite backed git. There are a whole class of problems here that I have little or no experience with, but seemingly could be incredibly impactful. For instance, further research and investment into versioning, event-logging, etc. could bring about a very rich revolution in OS / user experience. "What if... data (e.g. file) manipulations where semantic – rather than CRUD, you had application-layer higher level blocks that presented different abstractions to data below." Or, as I have more experience with SQL than FS, I could just be looking at the world through the lens of having a hammer and seeing everything as a nail.

notduncansmith 7y ago
Not exactly “SQLite-backed Git” but have you looked at Fossil? https://www.fossil-scm.org/home/doc/trunk/www/index.wiki https://www.fossil-scm.org/home/doc/trunk/www/index.wiki
syn0byte 7y ago
Everything is a nail. Whenever developer sorts get all starry eyed over "DB as an FS", they never consider data exchange between applications vendors and platforms. Their applications always exist alone in a digital vacuum. Example: Imagine trying to attach an MS Word document stored in this DBFS to an email from a Thunderbird client. What does that look like to the user? What does it look like to the Thunderbird developer? Abstract and encapsulate so that each application can use their own container format that includes schema and encoding details? Other applications can read this metadata directly from each containerized db object? Cool, we just reinvented the filesystem again.
- smilliken 7y ago
  You're right, any viable replacement for a filesystem must be compatible with existing filesystem APIs. That doesn't mean you can't implement the storage layer differently, or add additional APIs. Look at ZFS. It's a filesystem, but internally looks like a database. If you try to define the terms "database" and "filesystem" generally, you'll realize how difficult it is to separate the concepts.
wodenokoto 7y ago
> SQLite backed git SQLite development is actually version controlled by an open source SQLite backed version control system called fossil.

jamesblonde 7y ago

You can read on the background and challenges here in Salman Niazi's PhD - http://kth.diva-portal.org/smash/get/diva2:1260852/FULLTEXT02.pdf http://kth.diva-portal.org/smash/get/diva2:1260852/FULLTEXT0.... This is not a new topic - WinFS (part of the failed Longhorn project) was the best known example, which tried to use SQL Server as the metadata store for a new windows FS. Performance killed them. The reason why we managed to get better performance compared to all previous attempts it three-fold: 1. we used an in-memory high performance DB (NDB) and we did not de-normalize our schemas (no caching!) 2. with denormalized tables, we needed cross-partition transactions at high performance 3. we did a reasonable job at re-implementing HDFS' POSIX-like model using read-isolated transactions and row-level locking and new protocol for sub-tree operations.

lioeters 7y ago
Thank you for raising these relevant points. They led me down a fascinating rabbit hole on this topic of "database as file system". This is way over my head, but to pique others' curiosity, here's the abstract: Recent improvements in both the performance and scalability of shared-nothing, transactional, in-memory NewSQL databases have reopened the research question of whether distributed metadata for hierarchical file systems can be managed using commodity databases. In this paper, we introduce HopsFS, a next generation distribution of the Hadoop Distributed File System (HDFS) that replaces HDFS’ single node in-memory metadata service, with a distributed metadata service built on a NewSQL database. By removing the metadata bottleneck, HopsFS enables an order of magnitude larger and higher throughput clusters compared to HDFS. Metadata capacity has been increased to at least 37 times HDFS’ capacity, and in experiments based on a workload trace from Spotify, we show that HopsFS supports 16 to 37 times the throughput of Apache HDFS. HopsFS also has lower latency for many concurrent clients, and no downtime during failover. Finally, as metadata is now stored in a commodity database, it can be safely extended and easily exported to external systems for online analysis and free-text search.
- daviesliu 7y ago
  Have you publish the workload tracing from Spotify? We are very interesting on the results you got, looking for a way to replicate that using our setup.

novaleaf 7y ago

I remember being excited that this would be (eventually) coming to windows in the pre-Vista. It was called WinFS: https://en.wikipedia.org/wiki/WinFS https://en.wikipedia.org/wiki/WinFS Unfortunately this got canceled, IIRC due to difficulties with back-compat.