5 ms·

Maybe the File System shouldn’t be hierarchical but rather relational as well.

by uvdn7 5y ago

seanhunter 5y ago
Microsoft wondered that too and for a while developed "CairoOFS" as a possible replacement for NTFS. It was intended as a relational "object filesystem" https://betawiki.net/wiki/Microsoft_Cairo https://betawiki.net/wiki/Microsoft_Cairo
taeric 5y ago
And then we could have fun looking at the execution plan of our file access. :D Meant in jest; though I think the idea of a "one true way" to access files is a pipe dream. The hierarchical works better on my computer than whatever scheme we've cooked up for our phones.
- Semaphor 5y ago
  > The hierarchical works better on my computer than whatever scheme we've cooked up for our phones. What do you mean? Hierarchical is how I access files on my phone. Is that an Apple thing?
  - taeric 5y ago
    I thought phones had moved to an "ownership" model where access to files goes through the applications. That said, trying right now, I see there is a files app that seems to be mainly types. I'm assuming that is actually by folder?
    - Semaphor 5y ago
      > I thought phones had moved to an "ownership" model where access to files goes through the applications. There are many file explorers for Android, and they all have a normal browsing interface. The only "ownership" thing I’ve encountered is with the Nextcloud client, where it makes sense as those directories and files are not necessarily on the phone. I’m only on Android 10, but I think I would have heard if Android 12 completely changed things.
      - taeric 5y ago
        Yeah. I can only think I was confused by the few times I've tried to mess with the data. I thought I read they were trying to not go folders based.
jasfi 5y ago
What is the equivalent of a directory, and of a file? From the relation point of view, what would your tables be? It doesn't map that well, the idea sounds intriguing, but in practice a filesystem seems like a better model for files.
- dagss 5y ago
  You would not have a "directory" in the sense you use the word (why would directory be required in a FILE system?) Instead a heap of files and you search for them using metadata (e.g. each file has an associated key value store that you search on). Music catalogues is a good example. In the 90s one vould wonder if the music sitting on HD should be organized as "Music/(genre)/(artist)" or "Music/(artist)/(genre)" as directories. Different choices was best for different persons. Eventually many music players for desktop (e.g. iTunes) just made a different metaphor. And Spotify, Netflix etc do not use a hierarchy but you search for items using metadata. Another example is e.g. /lib/libfoo-1.2.so Where one could instead have libraryname=foo scope=system version=1.2 sharedlibrary=true Or similar
  - mnsc 5y ago
    > Music/(artist)/(genre) Well that's only necessary when your music library contains Zappa.
  - jasfi 5y ago
    That's not relational, that's closer to a NoSQL store.
    - dagss 5y ago
      That is orthogonal, NoSQL can be used as a relational DB too. SQL is one way to do a relational DB. Also what I describe is close to SQL if e.g. you declare a set of properties and property types for each "filetype". Relational is one aspect of a DB. Schemaless or not is another aspect.
- zozbot234 5y ago
  A "file" is just a key-value entry where the "key" is some label in an arbitrary namespace, and the "value" is a blob of bytes. The "directories" of a relational FS would be dynamically generated: they would arise as query result sets based on user input, as opposed to being fixed and materialized on-disk. This is how the relational FS worked in BeOS, but the exact same "virtual directory" feature was also ported to some versions of Windows.
mftb 5y ago
Microsoft Sharepoint uses SQL Server for all user Item storage, so it's an example of an all relational file store. They did this around the time they were also experimenting with the Cairo a sibling comment mentions. Also as another commenter noted you query on the metadata. Folders/directories for instance are represented as metadata. Installations can also describe very detailed ontologies including ones provided by third parties. It's a lot of work. Edit: I think I was actually thinking of WinFS which came out of Cairo later, around 2000.
- kodemager 5y ago
  That means onedrive for business uses sql server as it uses sharepoint online, interesting.
  - mftb 5y ago
    That's an interesting point. As I recall (unfortunately it's been a while). There was tension at first who would be top dog. SharePoint or OneDrive, and OneDrive won, so SharePoint Online is actually mapped over OneDrive. MS could do that because ironically even though SharePoint On-prem had been entirely SQL Server underneath, SharePoint On-prem devs generally never wrote a line of SQL. They interacted with the SharePoint Object model and that was basically mapped over OneDrive. What OneDrive is underneath I don't think MS had to be as forthcoming about because of it's cloud-based nature. I was on my way out by that point though.
roenxi 5y ago
The only thing the relational models really gives you is the ability to join two relations, which lets data decompose for storage but re-compose for many different uses. I'm not immediately seeing how that would be useful in a file context, where generally you want to look up a specific blob of data (aka a file). A tree-based file-system is optimised for doing a search from the users perspective, finding a file takes log(files) steps and finding related files is trivially cheap. It is likely hard to outdo that with a relational model.
- klodolph 5y ago
  The filesystem is used by different people for different purposes. Any attempt to make sense about “what makes sense” for files is going to be colored very differently depending on what perspective you have. From an end-user perspective, files are documents that I create and I want to be able to find them in different ways. From the perspective of a typical app developer, the filesystem is a hierarchical key-value store. The perspective of a database developer, backup software developer, system administrator, etc. is going to be completely different yet.
  - roenxi 5y ago
    It is easy to do that though - set up a database. I've met one, maybe 2, people who don't use file systems the conventional way. They're rare and they generally just want a search index as opposed to relational data.
    - klodolph 5y ago
      > It is easy to do that though - set up a database. I think we have really, fundamentally, failed to communicate here. I’m 100% sure we are talking about different things.
- hvidgaard 5y ago
  I would see it more as a different and better way to store the same and additional information. In a RDBMS an index is already storing references in a tree format. If you map that to files and folders you now have basically the same thing as a file system. Doing it more as a DB enables the OS to use the knowledge from RDBMS for efficiency, which I'm sure rivals the best file systems and it's possible to create multiple indexes and views for other use cases. Our current view on file systems and the knowledge we have is heavily influenced by slow spinning disks, while RDBMS have leveraged RAM a lot more. With todays fast SSDs the file system operates in a reality that is more like RAM than a slow spinning disk.
dboreham 5y ago
> Maybe the File System shouldn’t be hierarchical but rather relational as well How do you know it isn't?

Maybe the File System shouldn’t be hierarchical but rather relational as well.

by uvdn7 5y ago

Maybe the File System shouldn’t be hierarchical but rather relational as well.

Microsoft wondered that too and for a while developed "CairoOFS" as a possible replacement for NTFS. It was intended as a relational "object filesystem" https://betawiki.net/wiki/Microsoft_Cairo https://betawiki.net/wiki/Microsoft_Cairo

taeric 5y ago

And then we could have fun looking at the execution plan of our file access. :D Meant in jest; though I think the idea of a "one true way" to access files is a pipe dream. The hierarchical works better on my computer than whatever scheme we've cooked up for our phones.

Semaphor 5y ago
> The hierarchical works better on my computer than whatever scheme we've cooked up for our phones. What do you mean? Hierarchical is how I access files on my phone. Is that an Apple thing?
- taeric 5y ago
  I thought phones had moved to an "ownership" model where access to files goes through the applications. That said, trying right now, I see there is a files app that seems to be mainly types. I'm assuming that is actually by folder?
  - Semaphor 5y ago
    > I thought phones had moved to an "ownership" model where access to files goes through the applications. There are many file explorers for Android, and they all have a normal browsing interface. The only "ownership" thing I’ve encountered is with the Nextcloud client, where it makes sense as those directories and files are not necessarily on the phone. I’m only on Android 10, but I think I would have heard if Android 12 completely changed things.
    - taeric 5y ago
      Yeah. I can only think I was confused by the few times I've tried to mess with the data. I thought I read they were trying to not go folders based.

jasfi 5y ago

What is the equivalent of a directory, and of a file? From the relation point of view, what would your tables be? It doesn't map that well, the idea sounds intriguing, but in practice a filesystem seems like a better model for files.

dagss 5y ago
You would not have a "directory" in the sense you use the word (why would directory be required in a FILE system?) Instead a heap of files and you search for them using metadata (e.g. each file has an associated key value store that you search on). Music catalogues is a good example. In the 90s one vould wonder if the music sitting on HD should be organized as "Music/(genre)/(artist)" or "Music/(artist)/(genre)" as directories. Different choices was best for different persons. Eventually many music players for desktop (e.g. iTunes) just made a different metaphor. And Spotify, Netflix etc do not use a hierarchy but you search for items using metadata. Another example is e.g. /lib/libfoo-1.2.so Where one could instead have libraryname=foo scope=system version=1.2 sharedlibrary=true Or similar
- mnsc 5y ago
  > Music/(artist)/(genre) Well that's only necessary when your music library contains Zappa.
- jasfi 5y ago
  That's not relational, that's closer to a NoSQL store.
  - dagss 5y ago
    That is orthogonal, NoSQL can be used as a relational DB too. SQL is one way to do a relational DB. Also what I describe is close to SQL if e.g. you declare a set of properties and property types for each "filetype". Relational is one aspect of a DB. Schemaless or not is another aspect.
zozbot234 5y ago
A "file" is just a key-value entry where the "key" is some label in an arbitrary namespace, and the "value" is a blob of bytes. The "directories" of a relational FS would be dynamically generated: they would arise as query result sets based on user input, as opposed to being fixed and materialized on-disk. This is how the relational FS worked in BeOS, but the exact same "virtual directory" feature was also ported to some versions of Windows.

mftb 5y ago

Microsoft Sharepoint uses SQL Server for all user Item storage, so it's an example of an all relational file store. They did this around the time they were also experimenting with the Cairo a sibling comment mentions. Also as another commenter noted you query on the metadata. Folders/directories for instance are represented as metadata. Installations can also describe very detailed ontologies including ones provided by third parties. It's a lot of work. Edit: I think I was actually thinking of WinFS which came out of Cairo later, around 2000.

kodemager 5y ago
That means onedrive for business uses sql server as it uses sharepoint online, interesting.
- mftb 5y ago
  That's an interesting point. As I recall (unfortunately it's been a while). There was tension at first who would be top dog. SharePoint or OneDrive, and OneDrive won, so SharePoint Online is actually mapped over OneDrive. MS could do that because ironically even though SharePoint On-prem had been entirely SQL Server underneath, SharePoint On-prem devs generally never wrote a line of SQL. They interacted with the SharePoint Object model and that was basically mapped over OneDrive. What OneDrive is underneath I don't think MS had to be as forthcoming about because of it's cloud-based nature. I was on my way out by that point though.

roenxi 5y ago

The only thing the relational models really gives you is the ability to join two relations, which lets data decompose for storage but re-compose for many different uses. I'm not immediately seeing how that would be useful in a file context, where generally you want to look up a specific blob of data (aka a file). A tree-based file-system is optimised for doing a search from the users perspective, finding a file takes log(files) steps and finding related files is trivially cheap. It is likely hard to outdo that with a relational model.

klodolph 5y ago
The filesystem is used by different people for different purposes. Any attempt to make sense about “what makes sense” for files is going to be colored very differently depending on what perspective you have. From an end-user perspective, files are documents that I create and I want to be able to find them in different ways. From the perspective of a typical app developer, the filesystem is a hierarchical key-value store. The perspective of a database developer, backup software developer, system administrator, etc. is going to be completely different yet.
- roenxi 5y ago
  It is easy to do that though - set up a database. I've met one, maybe 2, people who don't use file systems the conventional way. They're rare and they generally just want a search index as opposed to relational data.
  - klodolph 5y ago
    > It is easy to do that though - set up a database. I think we have really, fundamentally, failed to communicate here. I’m 100% sure we are talking about different things.
hvidgaard 5y ago
I would see it more as a different and better way to store the same and additional information. In a RDBMS an index is already storing references in a tree format. If you map that to files and folders you now have basically the same thing as a file system. Doing it more as a DB enables the OS to use the knowledge from RDBMS for efficiency, which I'm sure rivals the best file systems and it's possible to create multiple indexes and views for other use cases. Our current view on file systems and the knowledge we have is heavily influenced by slow spinning disks, while RDBMS have leveraged RAM a lot more. With todays fast SSDs the file system operates in a reality that is more like RAM than a slow spinning disk.

dboreham 5y ago

> Maybe the File System shouldn’t be hierarchical but rather relational as well How do you know it isn't?