5 ms·

In my opinion database as a filesystem was a noble goal and someone will repeat it successfully in the future. Just because MS failed at it, does not mean that

In my opinion database as a filesystem was a noble goal and someone will repeat it successfully in the future. Just because MS failed at it, does not mean that this idea is not possible. I think that we stuck in 80-s/90-s foundational tech for too long and there're many new ideas that will shape future computing.

jitl 4y ago
It seems like a solution looking for a problem. Indexers with file system hooks like APFS + Spotlight and the thing Haiku has seem like a more practical approach for similar results.
kvdveer 4y ago
In a way, this is already happening. Documents are moving to the cloud, where they exist mostly in a database-like structure. End-user computers are becoming more and more a just an interface for documents that are in an online database, or (if the app supports offline) local sqlite databases. The endgame is a machine with only an OS. The OS is where all the hard edge cases are, which prevented the db-as-fs idea from succeeding in the first place.
- gjvc 4y ago
  The endgame is a machine with only an OS. What does this mean?
  - 5e92cb50239222b 4y ago
    Something like Chrome OS, I would imagine.
  - mdp2021 4y ago
    In the context, it appears to intend "the strict set of what constitutes a terminal to access external storage". But of course, even in the context of the cloud a DB does not (necessarily?) replace a FS, and, more fundamentally, that which the poster presented looks more like a sociologistic presentation of trends (or, "big ideas" for the salesforce), oblivious of the absiological ground that "progress is not in decreased freedom", while the presented model is that of crippled machine (wanting to store locally is a basicmost demand).
  - thefounder 4y ago
    Perhaps a browser(i.e Chrome)
  - deleted 4y ago
    [deleted]
  - EVa5I7bHFq9mnYK 4y ago
    Terminal + mainframe. Everything new is well forgotten old)
  - tcmart14 4y ago
    Probably referring to a dumb client.
- usrbinbash 4y ago
  > The endgame is a machine with only an OS. This was tried multiple times already, and in each case, it turned out that the ability to easily store and organise data locally was both desired and required among the wider audience. Cloud storage is a nice feature to complement local storage, but will not replace it any time soon.
  - kasey_junk 4y ago
    Typically this has been because file sizes have either caused network issues or storage cost issues on the server side. We are approaching (or are already there) the place where the biggest consumer file formats (videos) don’t have this constraint. At my house already it is very nebulous what is stored locally and what is stored on a remote server. I’m sure there is some consumer technology coming around the bend that will make local storage again compelling but I think we are on the cusp of another server centric file storage era not the reverse.n I can easily see a db file system being the default in for remote resources in consumer devices while traditional local file systems are reserved for specialized use cases.
    - oblak 4y ago
      Why is local storage not compelling anymore for you? It's faster, cheaper and often more reliable than someone else's computer.
    - usrbinbash 4y ago
      > Typically this has been because file sizes have either caused network issues or storage cost issues on the server side And because of privacy concerns. And because people want to use their computers even when they cannot connect to the internet. And because people want the security of being sure they can access their files even if a company goes bankrupt/has an outage. And because, no matter how fast networks get; I somewhat doubt that it will be faster to load a 2GiB 4k video file or a 7GiB pytorch checkpoint from cloud storage, than via the SATA or PCIe bus.
- ryloric 4y ago
  The recent popularity of Obsidian and similar knowledge management software is part of the same trend. They're databases in their own way, localized but better organized like a graph you can run queries against than a bunch of text files.
- cmrdporcupine 4y ago
  Speed of light is a hard physical limit. Local storage will always have a major edge for this reason alone. That said I'm now in this weird mid-life confusion about the fact that my new WAN (3gbps fiber connection) is now faster than any reasonable home LAN I can set up. Needed to move some movies from one machine to another the other day and it was way faster to just bittorrent them again from upstream than actually copy, since even USB removable media was slower than my WAN. So I think we might be at least temporarily in an awkward situation where cloud storage outcompetes LAN storage. But truly local storage (nVME etc) has an extreme edge over anything networked, and always will.
  - thfuran 4y ago
    >But truly local storage (nVME etc) has an extreme edge over anything networked, and always will. Sure, for specific usecases that don't much resemble what most people spend their time doing.
  - AnIdiotOnTheNet 4y ago
    This doesn't make any sense to me, I must be missing something. If your WAN is 3Gb/s, and you LAN is less than that, how does pulling things to a local machine, presumably on the LAN and therefore limited to the LAN speed, end up being faster?
    - cmrdporcupine 4y ago
      The machine doing the torrenting is tied directly to the access point, everything else is over wifi.
      - wolrah 4y ago
        > The machine doing the torrenting is tied directly to the access point, everything else is over wifi. That still doesn't make it faster to just torrent them again. If the LAN is slower than the WAN then a large file where the transfer time is long enough that the few milliseconds of setup latency is irrelevant might be effectively equally fast to re-download from than to transfer over the LAN, but never faster, and in most cases the LAN is going to be more reliable. There are almost certainly some cases where BitTorrent can transfer a specific set of files faster than common LAN transfer protocols, especially FTP or older versions of SMB when faced with large collections of small files, but that's an entirely different matter. If the bottleneck in either case is the second PC's LAN interface how could the WAN possibly be a faster source than the LAN machine?
dietr1ch 4y ago
I'd love to use a db for a filesystem for many of my directories. Granted, it might just be sub-optimal in certain scenarios, but I already have to logically partition my data into big blobs that are rarely accessed and stuff that must go into the SSDs already. Maybe just giving that FS a hatch to deal with the stuff that must be special and odd/weird is ok. I don't really want the DB magic for my swap files anyway.
smcnally 4y ago
BeFS “includes support for extended file attributes, with indexing and querying characteristics to provide functionality similar to that of a relational database.” The BeOS file system, an OS geek retrospective | Ars Technica [https://arstechnica.com/information-technology/2018/07/the-beos-filesystem/ https://arstechnica.com/information-technology/2018/07/the-b...]
- microtonal 4y ago
  There is also a book at the BeFS filesystem: http://www.nobius.org/dbg/practical-file-system-design.pdf http://www.nobius.org/dbg/practical-file-system-design.pdf (Legal source, the website is the author's.) Dominic Giampaolo now works at Apple on APFS.
  - conradev 4y ago
    My APFS container corrupted itself while working at Apple and Dominic was able to manually repair it enough for me to be able to mount it He lives in Maine, though, so he did it over SSH!
- kalleboo 4y ago
  You mean like HFS+ and Spotlight?
pifm_guy 4y ago
If you look at the on-disk structure of NTFS, you see that it's design is rather similar to databases. Metadata and file data are treated in the same way. Every 'special' data structure, like for example the record of which disk blocks are free, is actually stored as a [hidden] file. If a file is short, the data doesn't need a whole disk block allocated to it. Likewise, if a file name is really long, that string doesn't need to be stored together with other metadata.
ryloric 4y ago
I'd like to see a different abstraction and UX for "files", something that feels like a database but with looser usage patterns from a user's perspective, more intuitive to use and reason about. I don't want to remember paths and folders and all that crap. I don't want to depend on my discipline to organize and I'm willing to trade some flexibility in return. There's some alternatives like tags, but none of them feel natural and require setup. I feel services like dropbox had a real chance to try something, but the best they came up with was showing me "recent documents" for a homepage, wasted opportunity if you ask me.
jasonwatkinspdx 4y ago
There's an interesting project that's going the other direction: they're starting with a distributed database, then building an OS of sorts atop it (not traditional posix API): https://dbos-project.github.io/ https://dbos-project.github.io/
jandrewrogers 4y ago
The practical problem is that filesystems and databases are highly optimized toward different objectives at an architectural level. You can treat one like the other but the performance will be comparatively terrible. This is made even more difficult by the practical necessity of backward compatibility with standards like POSIX, SQL, et al in the same implementation that were never intended to be interchangeable at the implementation level. In principle you could design a database to be used as an effective filesystem. In practice, the implementation wouldn't be very compatible with any other software you need to work with and the ability to integrate matters.
cmrdporcupine 4y ago
Unfortunately I don't agree. I say unfortunately because I personally believe the knowledge model underlying a relational database system is a beautiful and fundamentally superior method of information management. But I also have traveled enough roads in this industry to know that heterogeneity of approaches ends up defining actual practice -- for better and for worse -- and the "universality" of fopen/fread/fwrite/fclose of blobs of data is going to be hard to move away from. I'd say that the only way that the "database" takes over the filesystem is by basically dropping to the lowest common denominator and becoming more like a filesystem; basically a key-value store. Throwing away the benefits of what a good database offers. Mainstream databases (in the form of SQL) only do a subset of what's possible with the relational model that Date&Codd laid out decades ago. I fear a move into a lower level of the stack would only worsen this. In fact I think the trend is kind of going the other way. Filesystems are slowly adopting more and more of the storage techniques that came out of DB research, but using them for the fairly anemic storage Unix FS storage model. Because that's basically what the industry is asking for. Further, I think there remains a poverty of knowledge about databases generally in the industry. I don't have a CS degree myself, but almost everyone I know (outside of my current employer) who does have one has said some variant of this to me: "I didn't take any/many DB classes in school." Knowledge of the relational data model is weak/thin in our industry (the low point being the "NoSQL" wave a decade ago). And within CS academia, it seems that R&D in this field is mostly centred around Europe (with the notable exception in North America being the excellent programme @ CMU.) Disclaimer: Employed by a database company.
didgetmaster 4y ago
This is why I set out to build a completely new system from scratch that could perform both file system and database operations extremely well. It will let you store hundreds of millions of files in a single container (a pod); attach a multitude of contextual tags to each one; and then find subsets of those files using database-like queries (e.g. Find all JPEG photos where event = 'wedding' AND camera = 'Cannon EOS' AND size > 10 MB AND photographer = 'John'). It can find everything (even when there are millions of matches) in less than a second and even beats out Postgres and SQLite at most DB table queries. https://www.youtube.com/watch?v=dWIo6sia_hw https://www.youtube.com/watch?v=dWIo6sia_hw (file system functions) https://www.youtube.com/watch?v=Va5ZqfwQXWI https://www.youtube.com/watch?v=Va5ZqfwQXWI (benchmark vs SQLite) https://www.youtube.com/watch?v=OVICKCkWMZE https://www.youtube.com/watch?v=OVICKCkWMZE (benchmark vs Postgres)

jitl 4y ago

It seems like a solution looking for a problem. Indexers with file system hooks like APFS + Spotlight and the thing Haiku has seem like a more practical approach for similar results.

kvdveer 4y ago

In a way, this is already happening. Documents are moving to the cloud, where they exist mostly in a database-like structure. End-user computers are becoming more and more a just an interface for documents that are in an online database, or (if the app supports offline) local sqlite databases. The endgame is a machine with only an OS. The OS is where all the hard edge cases are, which prevented the db-as-fs idea from succeeding in the first place.

gjvc 4y ago
The endgame is a machine with only an OS. What does this mean?
- 5e92cb50239222b 4y ago
  Something like Chrome OS, I would imagine.
- mdp2021 4y ago
  In the context, it appears to intend "the strict set of what constitutes a terminal to access external storage". But of course, even in the context of the cloud a DB does not (necessarily?) replace a FS, and, more fundamentally, that which the poster presented looks more like a sociologistic presentation of trends (or, "big ideas" for the salesforce), oblivious of the absiological ground that "progress is not in decreased freedom", while the presented model is that of crippled machine (wanting to store locally is a basicmost demand).
- thefounder 4y ago
  Perhaps a browser(i.e Chrome)
- deleted 4y ago
  [deleted]
- EVa5I7bHFq9mnYK 4y ago
  Terminal + mainframe. Everything new is well forgotten old)
- tcmart14 4y ago
  Probably referring to a dumb client.
usrbinbash 4y ago
> The endgame is a machine with only an OS. This was tried multiple times already, and in each case, it turned out that the ability to easily store and organise data locally was both desired and required among the wider audience. Cloud storage is a nice feature to complement local storage, but will not replace it any time soon.
- kasey_junk 4y ago
  Typically this has been because file sizes have either caused network issues or storage cost issues on the server side. We are approaching (or are already there) the place where the biggest consumer file formats (videos) don’t have this constraint. At my house already it is very nebulous what is stored locally and what is stored on a remote server. I’m sure there is some consumer technology coming around the bend that will make local storage again compelling but I think we are on the cusp of another server centric file storage era not the reverse.n I can easily see a db file system being the default in for remote resources in consumer devices while traditional local file systems are reserved for specialized use cases.
  - oblak 4y ago
    Why is local storage not compelling anymore for you? It's faster, cheaper and often more reliable than someone else's computer.
  - usrbinbash 4y ago
    > Typically this has been because file sizes have either caused network issues or storage cost issues on the server side And because of privacy concerns. And because people want to use their computers even when they cannot connect to the internet. And because people want the security of being sure they can access their files even if a company goes bankrupt/has an outage. And because, no matter how fast networks get; I somewhat doubt that it will be faster to load a 2GiB 4k video file or a 7GiB pytorch checkpoint from cloud storage, than via the SATA or PCIe bus.
ryloric 4y ago
The recent popularity of Obsidian and similar knowledge management software is part of the same trend. They're databases in their own way, localized but better organized like a graph you can run queries against than a bunch of text files.
cmrdporcupine 4y ago
Speed of light is a hard physical limit. Local storage will always have a major edge for this reason alone. That said I'm now in this weird mid-life confusion about the fact that my new WAN (3gbps fiber connection) is now faster than any reasonable home LAN I can set up. Needed to move some movies from one machine to another the other day and it was way faster to just bittorrent them again from upstream than actually copy, since even USB removable media was slower than my WAN. So I think we might be at least temporarily in an awkward situation where cloud storage outcompetes LAN storage. But truly local storage (nVME etc) has an extreme edge over anything networked, and always will.
- thfuran 4y ago
  >But truly local storage (nVME etc) has an extreme edge over anything networked, and always will. Sure, for specific usecases that don't much resemble what most people spend their time doing.
- AnIdiotOnTheNet 4y ago
  This doesn't make any sense to me, I must be missing something. If your WAN is 3Gb/s, and you LAN is less than that, how does pulling things to a local machine, presumably on the LAN and therefore limited to the LAN speed, end up being faster?
  - cmrdporcupine 4y ago
    The machine doing the torrenting is tied directly to the access point, everything else is over wifi.
    - wolrah 4y ago
      > The machine doing the torrenting is tied directly to the access point, everything else is over wifi. That still doesn't make it faster to just torrent them again. If the LAN is slower than the WAN then a large file where the transfer time is long enough that the few milliseconds of setup latency is irrelevant might be effectively equally fast to re-download from than to transfer over the LAN, but never faster, and in most cases the LAN is going to be more reliable. There are almost certainly some cases where BitTorrent can transfer a specific set of files faster than common LAN transfer protocols, especially FTP or older versions of SMB when faced with large collections of small files, but that's an entirely different matter. If the bottleneck in either case is the second PC's LAN interface how could the WAN possibly be a faster source than the LAN machine?

dietr1ch 4y ago

I'd love to use a db for a filesystem for many of my directories. Granted, it might just be sub-optimal in certain scenarios, but I already have to logically partition my data into big blobs that are rarely accessed and stuff that must go into the SSDs already. Maybe just giving that FS a hatch to deal with the stuff that must be special and odd/weird is ok. I don't really want the DB magic for my swap files anyway.

smcnally 4y ago

BeFS “includes support for extended file attributes, with indexing and querying characteristics to provide functionality similar to that of a relational database.” The BeOS file system, an OS geek retrospective | Ars Technica [https://arstechnica.com/information-technology/2018/07/the-beos-filesystem/ https://arstechnica.com/information-technology/2018/07/the-b...]

microtonal 4y ago
There is also a book at the BeFS filesystem: http://www.nobius.org/dbg/practical-file-system-design.pdf http://www.nobius.org/dbg/practical-file-system-design.pdf (Legal source, the website is the author's.) Dominic Giampaolo now works at Apple on APFS.
- conradev 4y ago
  My APFS container corrupted itself while working at Apple and Dominic was able to manually repair it enough for me to be able to mount it He lives in Maine, though, so he did it over SSH!
kalleboo 4y ago
You mean like HFS+ and Spotlight?

pifm_guy 4y ago

If you look at the on-disk structure of NTFS, you see that it's design is rather similar to databases. Metadata and file data are treated in the same way. Every 'special' data structure, like for example the record of which disk blocks are free, is actually stored as a [hidden] file. If a file is short, the data doesn't need a whole disk block allocated to it. Likewise, if a file name is really long, that string doesn't need to be stored together with other metadata.

ryloric 4y ago

I'd like to see a different abstraction and UX for "files", something that feels like a database but with looser usage patterns from a user's perspective, more intuitive to use and reason about. I don't want to remember paths and folders and all that crap. I don't want to depend on my discipline to organize and I'm willing to trade some flexibility in return. There's some alternatives like tags, but none of them feel natural and require setup. I feel services like dropbox had a real chance to try something, but the best they came up with was showing me "recent documents" for a homepage, wasted opportunity if you ask me.

jasonwatkinspdx 4y ago

There's an interesting project that's going the other direction: they're starting with a distributed database, then building an OS of sorts atop it (not traditional posix API): https://dbos-project.github.io/ https://dbos-project.github.io/

jandrewrogers 4y ago

The practical problem is that filesystems and databases are highly optimized toward different objectives at an architectural level. You can treat one like the other but the performance will be comparatively terrible. This is made even more difficult by the practical necessity of backward compatibility with standards like POSIX, SQL, et al in the same implementation that were never intended to be interchangeable at the implementation level. In principle you could design a database to be used as an effective filesystem. In practice, the implementation wouldn't be very compatible with any other software you need to work with and the ability to integrate matters.

cmrdporcupine 4y ago

Unfortunately I don't agree. I say unfortunately because I personally believe the knowledge model underlying a relational database system is a beautiful and fundamentally superior method of information management. But I also have traveled enough roads in this industry to know that heterogeneity of approaches ends up defining actual practice -- for better and for worse -- and the "universality" of fopen/fread/fwrite/fclose of blobs of data is going to be hard to move away from. I'd say that the only way that the "database" takes over the filesystem is by basically dropping to the lowest common denominator and becoming more like a filesystem; basically a key-value store. Throwing away the benefits of what a good database offers. Mainstream databases (in the form of SQL) only do a subset of what's possible with the relational model that Date&Codd laid out decades ago. I fear a move into a lower level of the stack would only worsen this. In fact I think the trend is kind of going the other way. Filesystems are slowly adopting more and more of the storage techniques that came out of DB research, but using them for the fairly anemic storage Unix FS storage model. Because that's basically what the industry is asking for. Further, I think there remains a poverty of knowledge about databases generally in the industry. I don't have a CS degree myself, but almost everyone I know (outside of my current employer) who does have one has said some variant of this to me: "I didn't take any/many DB classes in school." Knowledge of the relational data model is weak/thin in our industry (the low point being the "NoSQL" wave a decade ago). And within CS academia, it seems that R&D in this field is mostly centred around Europe (with the notable exception in North America being the excellent programme @ CMU.) Disclaimer: Employed by a database company.

didgetmaster 4y ago

This is why I set out to build a completely new system from scratch that could perform both file system and database operations extremely well. It will let you store hundreds of millions of files in a single container (a pod); attach a multitude of contextual tags to each one; and then find subsets of those files using database-like queries (e.g. Find all JPEG photos where event = 'wedding' AND camera = 'Cannon EOS' AND size > 10 MB AND photographer = 'John'). It can find everything (even when there are millions of matches) in less than a second and even beats out Postgres and SQLite at most DB table queries. https://www.youtube.com/watch?v=dWIo6sia_hw https://www.youtube.com/watch?v=dWIo6sia_hw (file system functions) https://www.youtube.com/watch?v=Va5ZqfwQXWI https://www.youtube.com/watch?v=Va5ZqfwQXWI (benchmark vs SQLite) https://www.youtube.com/watch?v=OVICKCkWMZE https://www.youtube.com/watch?v=OVICKCkWMZE (benchmark vs Postgres)