6 ms·

They say file format problem, but one of the reasons why there are so many different file formats is because current file system abstractions are fundamentally

by tgbugs 6y ago

They say file format problem, but one of the reasons why there are so many different file formats is because current file system abstractions are fundamentally insufficient for a wide variety of use cases, particularly ones where you need to ensure that a user can't accidentally separate or delete a file. How do you keep a cache directory that backs a file with the file if the file system doesn't let you put folders "inside" of files? This is only one of many issues that have resulted in the proliferation of file formats to meet use cases with different performance characteristics that all have the common requirement for data to be bundled together an not be separable. File systems that can decouple the notion of access and data locality from inseparability simply do not exist right now, so we are left with the horrible abstraction that is the archive or zip file.

Ericson2314 6y ago
Here's to hoping https://docs.ipld.io/ https://docs.ipld.io/ helps for acylic data. Not sure what's needed for cyclic (including relational) data.
skissane 6y ago
> particularly ones where you need to ensure that a user can't accidentally separate or delete a file. How do you keep a cache directory that backs a file with the file if the file system doesn't let you put folders "inside" of files? You are worried that if an application has a directory "cache" with files in it, a user will go into that directory and randomly delete the file "cache/3F3819A1.dat", and that might confuse the application. But, suppose instead you had a single file "cache.sqlite", containing a table "cache" with an "id" column. What's to stop a user from opening that file with the sqlite command line tool and running "delete cache where id='3F3819A1'" If a user really wants to muck with your app's data by hand, you can't stop them. (Unless you go with some DRM-like solution in which your app runs in a secure enclave such as Intel SGX) Whether it is a single file or a directory of files doesn't really make a difference
- tgbugs 6y ago
  Even with current abstractions this is the case. I'm not worried about users that have sufficient knowledge to intentionally modify a file. Users who know enough to find, much less modify an sqlite database can make their own mess if they want to. I'm more worried about cases where something like a sidecar file gets deleted or overwritten, or fails to be copied along with the main file. Some programs and file formats come up with their own way of embedding sidecar metadata in the main file, but that is a complete hack that has to be used to work around the fact that the sidecar file cannot be made inseparable from the main file while still being just another file. Alternate data streams on windows can sort of do this (xattrs on nixen aren't really up to the task) but they are completely non-portable.
  - skissane 6y ago
    > Alternate data streams on windows can sort of do this (xattrs on nixen aren't really up to the task) but they are completely non-portable. On Solaris, xattrs really are up to the task; they are basically Windows alternate data streams by another name. Unlike Linux xattrs, their maximum size is the same as that of the main stream of the file. (Likewise, on macOS the maximum xattr size is the same as that of a file's main stream.) I think if Linux would lift its limit on maximum xattr size then you would have the same basic function across most major operating systems, just with some differences in API. Then if someone could devise a standard API (even as just a portability layer), your problem would be mostly solved. Just also making sure that tools like cp, scp, sftp, zip, tar, etc, know about the xattrs and transport them. (Which is actually already true for some of those tools on some platforms.)
    - tgbugs 6y ago
      I have used python's pathlib to create a common abstraction across ADS and linux xattrs, but have had to stop at the lowest common denominator that is the linux implementation. It is good to know that some of the other nixen don't have the same restrictions because it means there might be a shorter route to interoperability (especially in light of some of the other threads on the linux file system that have popped up over the last couple of days). The second piece of the puzzle is also exactly as you point out, which is xattr support across all the common tools. Many of them have it already, however it is usually not enabled by default, so it is extremely easy for a user to accidentally copy only the main file data. To give an example of the issue, vim's default save implementation copies files in a way that removes xattrs. Thus, even if xattrs can get closer, the fact that they still require tools to implement correct behaviour makes is a bit like the situation with cooperative multitasking.
- TheSoftwareGuy 6y ago
  More like you want users to be able to move their data around, send it, email it, etc, without thinking about 50 files in 10 directories that need to be placed in the right way. Rather just have 1 file with all the relevant data, and any structure you need for that data is embedded in the file itself
  - superdisk 6y ago
    So, a zip file?
    - jdc 6y ago
      I think the problem is the user needs to know where to look, and that gets difficult when you bring in sprawling, deeply nested directory trees.
- Aerroon 6y ago
  >You are worried that if an application has a directory "cache" with files in it, a user will go into that directory and randomly delete the file "cache/3F3819A1.dat", and that might confuse the application. I have done exactly this for a few applications. I likely wouldn't have done it for the database. The reason I've done this is disk space. Some software is bad at getting rid of things that they've cached (eg Chrome). Theyend up taking more and more and more and more disk space. I don't understand why my browser needs to take up 10-20 GB of disk space, if it can't even retain 7 months of history.
  - carlmr 6y ago
    >The reason I've done this is disk space. Some software is bad at getting rid of things that they've cached (eg Chrome). Theyend up taking more and more and more and more disk space. I don't understand why my browser needs to take up 10-20 GB of disk space, if it can't even retain 7 months of history. But then you just hide your bad caching design from users by only "showing" what you're doing wrong to those that understand SQL. And those users can still change it. You should test your app for random cache permutations (user changes .dat file in the wrong way) and partial cache loss (user deletes random file). You have to assume that anything that can be broken will be broken.
    - Aerroon 6y ago
      Oh, I'm sorry. I meant that I've deleted these "random" files as the user. As a user I have no faith that application developers will properly manage disk space usage for what they cache. If an application like Chrome has no problem taking up 10 GB of space mostly due to caching, then I have no expectation that most other software will do better. Perhaps this information gets communicated to the user somewhere, but it tends to not be done in an easy to understand manner.
      - tinus_hn 6y ago
        There should be an operating system solution for cache so the storage used can be discounted and the space can just be freed as required, so the operating system can delete caches belonging to deleted apps and that apps have failed to clean up, and so the system can show the user which app is hogging space.
deleted 6y ago
[deleted]
sriku 6y ago
For a good lot of cases, sqlite can serve as a perfectly good file format. I actually find it hard to think of a case where it would be hard for sqlite .. perhaps nested markup requirements are the only thing that can necessitate additional structure in the data. If your application needs a file format, choose sqlite and move on.
- Kalium 6y ago
  Using a database is a partial answer at best. File formats don't just dictate the structure of data, they carry semantic info. Using sqlite gives you structure without semantics.
  - jaytaylor 6y ago
    Unless you also store contextual information alongside the raw bytes. Curious if you can clarify / specify / examplify what "semantic" means in this case?
    - Kalium 6y ago
      Understanding a file format means knowing the significance of a given bit. Not just that X has value 13, but that 13 means something and might influence the significance of Y or Z. Having a table with columns A, B, and C doesn't say a ton about what the values in them mean. Or how they relate to one another. If it's a bunch of decimal numbers, it could be financial data, and the sums expected to sum depending on how another column that indicates a type is set. Or it could be temperature readings. Or distances, with the unit implied or tracked elsewhere. Generally this is the kind of data and rules for reasoning about data that isn't stored in a database. Certainly not a sqlite database. Column and table names are often vague or unclear, and in my experience it's an exceptionally rare developer who leaves detailed commentary on the relationships between columns and tables and their possible values in their create statements.
      - mongol 6y ago
        Can you give an example where the semantic data you refer to is in the file format in a way that is better?
helloycombinat 6y ago
What's wrong with zip? It's fine. Maybe more than fine if it managed to be this popular for so many applications. From my understanding it's very good on floppy disk too
- tgbugs 6y ago
  Consider a use case where you need high concurrent write performance and you don't want the things you are writing to be accidentally dissociated from the file. If you have to write to a zip file there will be overhead every time you append a new "file" because you have to rewrite the central registry at the end of the file. Basically there is no way to say to you operating system "hey, can I get a no-nonsense, don't what to have to come up with some other conventions that will be broken, way to have some additional storage and file system paths that are just for this file so I don't have to worry about name collisions?" I already have a perfectly good unique name under which I could stash information, but the posix standard basically prevents me from using that name as a way to safely namespace things that I need per file. However, to your original point I have partially implemented this [0] using zip files on the read side. I treat the internal paths inside the zip file as if they are rooted at the zip file itself. I haven't implemented rewriting though. 0. https://github.com/tgbugs/augpathlib/blob/master/augpathlib/zip.py https://github.com/tgbugs/augpathlib/blob/master/augpathlib/...
  - helloycombinat 6y ago
    This legit makes no sense to me. Why would you need concurrent write on a zip file? How would you even know where to write when you depend on compressed sizes that haven't been done on other cores? From what I remember you basically can create a zip by putting a header than adding compressed files back to back (no idea why you couldn't use multiple cores to compress each and add when your ready) then add the filelist at the end. Like I have no idea what your issue is
zupa-hu 6y ago
> File systems that can decouple the notion of access and data locality from inseparability simply do not exist right now For the record, it does. The Boomla OS has exactly that, the ability to store files in files. As in, files and directories are the same concept. It is not a posix filesystem of course and the OS is built specifically for web development. (Disclosure: I’m working on the project.) https://boomla.com/docs/how-it-works/anatomy-of-the-boomla-filesystem/filesystem-tree https://boomla.com/docs/how-it-works/anatomy-of-the-boomla-f...

by tgbugs 6y ago

Ericson2314 6y ago

Here's to hoping https://docs.ipld.io/ https://docs.ipld.io/ helps for acylic data. Not sure what's needed for cyclic (including relational) data.

skissane 6y ago

> particularly ones where you need to ensure that a user can't accidentally separate or delete a file. How do you keep a cache directory that backs a file with the file if the file system doesn't let you put folders "inside" of files? You are worried that if an application has a directory "cache" with files in it, a user will go into that directory and randomly delete the file "cache/3F3819A1.dat", and that might confuse the application. But, suppose instead you had a single file "cache.sqlite", containing a table "cache" with an "id" column. What's to stop a user from opening that file with the sqlite command line tool and running "delete cache where id='3F3819A1'" If a user really wants to muck with your app's data by hand, you can't stop them. (Unless you go with some DRM-like solution in which your app runs in a secure enclave such as Intel SGX) Whether it is a single file or a directory of files doesn't really make a difference

tgbugs 6y ago
Even with current abstractions this is the case. I'm not worried about users that have sufficient knowledge to intentionally modify a file. Users who know enough to find, much less modify an sqlite database can make their own mess if they want to. I'm more worried about cases where something like a sidecar file gets deleted or overwritten, or fails to be copied along with the main file. Some programs and file formats come up with their own way of embedding sidecar metadata in the main file, but that is a complete hack that has to be used to work around the fact that the sidecar file cannot be made inseparable from the main file while still being just another file. Alternate data streams on windows can sort of do this (xattrs on nixen aren't really up to the task) but they are completely non-portable.
- skissane 6y ago
  > Alternate data streams on windows can sort of do this (xattrs on nixen aren't really up to the task) but they are completely non-portable. On Solaris, xattrs really are up to the task; they are basically Windows alternate data streams by another name. Unlike Linux xattrs, their maximum size is the same as that of the main stream of the file. (Likewise, on macOS the maximum xattr size is the same as that of a file's main stream.) I think if Linux would lift its limit on maximum xattr size then you would have the same basic function across most major operating systems, just with some differences in API. Then if someone could devise a standard API (even as just a portability layer), your problem would be mostly solved. Just also making sure that tools like cp, scp, sftp, zip, tar, etc, know about the xattrs and transport them. (Which is actually already true for some of those tools on some platforms.)
  - tgbugs 6y ago
    I have used python's pathlib to create a common abstraction across ADS and linux xattrs, but have had to stop at the lowest common denominator that is the linux implementation. It is good to know that some of the other nixen don't have the same restrictions because it means there might be a shorter route to interoperability (especially in light of some of the other threads on the linux file system that have popped up over the last couple of days). The second piece of the puzzle is also exactly as you point out, which is xattr support across all the common tools. Many of them have it already, however it is usually not enabled by default, so it is extremely easy for a user to accidentally copy only the main file data. To give an example of the issue, vim's default save implementation copies files in a way that removes xattrs. Thus, even if xattrs can get closer, the fact that they still require tools to implement correct behaviour makes is a bit like the situation with cooperative multitasking.
TheSoftwareGuy 6y ago
More like you want users to be able to move their data around, send it, email it, etc, without thinking about 50 files in 10 directories that need to be placed in the right way. Rather just have 1 file with all the relevant data, and any structure you need for that data is embedded in the file itself
- superdisk 6y ago
  So, a zip file?
  - jdc 6y ago
    I think the problem is the user needs to know where to look, and that gets difficult when you bring in sprawling, deeply nested directory trees.
Aerroon 6y ago
>You are worried that if an application has a directory "cache" with files in it, a user will go into that directory and randomly delete the file "cache/3F3819A1.dat", and that might confuse the application. I have done exactly this for a few applications. I likely wouldn't have done it for the database. The reason I've done this is disk space. Some software is bad at getting rid of things that they've cached (eg Chrome). Theyend up taking more and more and more and more disk space. I don't understand why my browser needs to take up 10-20 GB of disk space, if it can't even retain 7 months of history.
- carlmr 6y ago
  >The reason I've done this is disk space. Some software is bad at getting rid of things that they've cached (eg Chrome). Theyend up taking more and more and more and more disk space. I don't understand why my browser needs to take up 10-20 GB of disk space, if it can't even retain 7 months of history. But then you just hide your bad caching design from users by only "showing" what you're doing wrong to those that understand SQL. And those users can still change it. You should test your app for random cache permutations (user changes .dat file in the wrong way) and partial cache loss (user deletes random file). You have to assume that anything that can be broken will be broken.
  - Aerroon 6y ago
    Oh, I'm sorry. I meant that I've deleted these "random" files as the user. As a user I have no faith that application developers will properly manage disk space usage for what they cache. If an application like Chrome has no problem taking up 10 GB of space mostly due to caching, then I have no expectation that most other software will do better. Perhaps this information gets communicated to the user somewhere, but it tends to not be done in an easy to understand manner.
    - tinus_hn 6y ago
      There should be an operating system solution for cache so the storage used can be discounted and the space can just be freed as required, so the operating system can delete caches belonging to deleted apps and that apps have failed to clean up, and so the system can show the user which app is hogging space.

deleted 6y ago

[deleted]

sriku 6y ago

For a good lot of cases, sqlite can serve as a perfectly good file format. I actually find it hard to think of a case where it would be hard for sqlite .. perhaps nested markup requirements are the only thing that can necessitate additional structure in the data. If your application needs a file format, choose sqlite and move on.

Kalium 6y ago
Using a database is a partial answer at best. File formats don't just dictate the structure of data, they carry semantic info. Using sqlite gives you structure without semantics.
- jaytaylor 6y ago
  Unless you also store contextual information alongside the raw bytes. Curious if you can clarify / specify / examplify what "semantic" means in this case?
  - Kalium 6y ago
    Understanding a file format means knowing the significance of a given bit. Not just that X has value 13, but that 13 means something and might influence the significance of Y or Z. Having a table with columns A, B, and C doesn't say a ton about what the values in them mean. Or how they relate to one another. If it's a bunch of decimal numbers, it could be financial data, and the sums expected to sum depending on how another column that indicates a type is set. Or it could be temperature readings. Or distances, with the unit implied or tracked elsewhere. Generally this is the kind of data and rules for reasoning about data that isn't stored in a database. Certainly not a sqlite database. Column and table names are often vague or unclear, and in my experience it's an exceptionally rare developer who leaves detailed commentary on the relationships between columns and tables and their possible values in their create statements.
    - mongol 6y ago
      Can you give an example where the semantic data you refer to is in the file format in a way that is better?

helloycombinat 6y ago

What's wrong with zip? It's fine. Maybe more than fine if it managed to be this popular for so many applications. From my understanding it's very good on floppy disk too

tgbugs 6y ago
Consider a use case where you need high concurrent write performance and you don't want the things you are writing to be accidentally dissociated from the file. If you have to write to a zip file there will be overhead every time you append a new "file" because you have to rewrite the central registry at the end of the file. Basically there is no way to say to you operating system "hey, can I get a no-nonsense, don't what to have to come up with some other conventions that will be broken, way to have some additional storage and file system paths that are just for this file so I don't have to worry about name collisions?" I already have a perfectly good unique name under which I could stash information, but the posix standard basically prevents me from using that name as a way to safely namespace things that I need per file. However, to your original point I have partially implemented this [0] using zip files on the read side. I treat the internal paths inside the zip file as if they are rooted at the zip file itself. I haven't implemented rewriting though. 0. https://github.com/tgbugs/augpathlib/blob/master/augpathlib/zip.py https://github.com/tgbugs/augpathlib/blob/master/augpathlib/...
- helloycombinat 6y ago
  This legit makes no sense to me. Why would you need concurrent write on a zip file? How would you even know where to write when you depend on compressed sizes that haven't been done on other cores? From what I remember you basically can create a zip by putting a header than adding compressed files back to back (no idea why you couldn't use multiple cores to compress each and add when your ready) then add the filelist at the end. Like I have no idea what your issue is

zupa-hu 6y ago

> File systems that can decouple the notion of access and data locality from inseparability simply do not exist right now For the record, it does. The Boomla OS has exactly that, the ability to store files in files. As in, files and directories are the same concept. It is not a posix filesystem of course and the OS is built specifically for web development. (Disclosure: I’m working on the project.) https://boomla.com/docs/how-it-works/anatomy-of-the-boomla-filesystem/filesystem-tree https://boomla.com/docs/how-it-works/anatomy-of-the-boomla-f...