6 ms·

I think this is really interesting, but I don't see any reason why you'd abandon the idea of a filesystem. UNIX has been consistently useful for all sorts of p

I think this is really interesting, but I don't see any reason why you'd abandon the idea of a filesystem. UNIX has been consistently useful for all sorts of purposes largely because of the everything-is-a-file concept. Are we going to keep only a single copy of everything in memresister or whatever? someone will reinvent raid, and someone will reinvent the disk controller. i'm sure this tech will change all kinds of things but why wouldn't i want a simple hierarchial naming system to refer to all of the different blobs i have? that said, i'm sure i'll watch this!

klodolph 6y ago
The filesystem, at least the Unix concept of it, is nice in theory but surprisingly ill-suited for anything besides, say, batch processing, once you start to look more closely at it. If you look at projects like SQLite or PostgreSQL, you’ll see all the fantastic, precise logic you need in order to ensure expected levels of consistency when you are using real filesystems to store data in a database. These problems come with serious enough pitfalls and traps that it’s common to see blanket recommendations to avoid directly interacting with the filesystem altogether, and simply do everything through SQLite or another library when possible. At the other end, if you look at GUI applications, the filesystem is a bit lower-level than it should be. At least, the Unix concept of filesystem is too low-level. If I open a file in a word processor, save it, shouldn’t I be able to move around the file and rename it in the file browser, without affecting the relationship of that file to the word processor? This is possible on a Mac, but this is done with APIs inherited from the pre-Unix days of macOS.
- johnvaluk 6y ago
  A file is more than simply a unit or collection of data in a local hierarchy. It's an abstraction that supports the copying, exchange and even possession of data. How is one supposed to share part of a database with another system or user without ultimately resorting to the file metaphor?
  - klodolph 6y ago
    > How is one supposed to share part of a database with another system or user without ultimately resorting to the file metaphor? It is exceptionally difficult to share a database using files. This is probably one of the worst ways to share a database. Most of the time, the best you can do is take a snapshot of the database, dump it to a file, and import it into another system. Generally you would want to share a database by granting multiple users access to a single running instance.
    - sizt 6y ago
      Well, SQLite is the most prevalent database on the planet, and its a file so ... https://sqlite.org/fileformat2.html https://sqlite.org/fileformat2.html
  - webmaven 6y ago
    > How is one supposed to share part of a database with another system or user without ultimately resorting to the file metaphor? With an access-controlled view, perhaps? Your question isn't entirely academic, incidentally, the Open Data movement prompted a lot of organizations (particularly government or government-like ones) to address this sort of issue (especially with geodata) en masse starting about a decade ago, and providing snapshots in the form of downloadable files is generally seen as an important but generally inferior when-all-else-fails solution for most use-cases that don't involve forking the data and/or bulk-processing it into a new form. Live read-only access to the system of record or a DMZ'd proxy or cache is usually preferred. And the Semantic Web / Linked Open Data movement addressed a similar set of issues two decades ago. I suppose it's time for another turn if the wheel, perhaps with a distributed web twist.
- emptyparadise 6y ago
  I think the most important thing about files is being able to take a unit of work - a picture or a document - and interact with it as a single object, doing things like copying it or sharing it with others. Any replacement system that doesn't let me take my work and simply drag it to a flash drive is flawed. This doesn't mean we can't use a database, but it is something important to remember.
  - klodolph 6y ago
    > I think the most important thing about files is being able to take a unit of work - a picture or a document - and interact with it as a single object, doing things like copying it or sharing it with others. Yes, we want this, but the filesystem is actually quite bad at this! We have just gotten used to how bad files and filesystems are, and ignore the fact that they are not good matches for our metaphors about how they work. Just to be clear—I completely disagree with your claims about how files work. You can’t really interact much with a file as a single object. When you open a document, a copy of the file is loaded into memory. This separate, invisible copy does not show up on the filesystem and will be out of synch with the filesystem copy. For example, 1. Open a text file in Vim, Nano, or Emacs. 2. In another terminal window, rename the file. 3. Make changes to the text file and save it. What happens? You end up with two different files! This is not some weird hackery, this is just renaming a file, one of the most boring things you can do. Something I do all the time. It shows that files are not single objects, and you do not interact with them that way. This is one of the failings of the Unix filesystem model—files are not single objects. There is a ton of room for improvement. Just as a point of comparison, Windows is a bit worse (usually won’t even let you rename the file) and the Mac is a bit better (when you rename a file, the open document will follow the rename).
    - emptyparadise 6y ago
      I'm speaking more of files as a metaphor than as a file system implementation detail. I'm pointing this out because the mobile first and web first world dropped the ball on this incredibly badly. Every webapp is an isolated silo with weird logins and no proper offline usage whatsoever. To share a small anecdote, I recently asked a non-technical acquaintance how they would share a Word document from their desktop computer: they said they'd email the file. Then I asked how they would share a note from their phone's notes app: they had no clue. Yes, implementation details bleed through: I can tell that Word operates on files and the notes app probably has some sqlite database under the hood, but ultimately it's about how we interact with data. Word operates on content you have, while the notes app operates on content it has. And this is where my flash drive test came from, not because I'm attached to inodes or directories, but because of the interaction model that decouples apps and data. If tomorrow I wake up in a world where ext4 has been replaced by Postgres as its file system, I wouldn't miss chmod or fsck. But I'd want to be sure I can still take a single object (a file!) and do actions with it - send it to somebody, delete it, copy it, all without it having some inseparable link to an app or a website.

klodolph 6y ago

The filesystem, at least the Unix concept of it, is nice in theory but surprisingly ill-suited for anything besides, say, batch processing, once you start to look more closely at it. If you look at projects like SQLite or PostgreSQL, you’ll see all the fantastic, precise logic you need in order to ensure expected levels of consistency when you are using real filesystems to store data in a database. These problems come with serious enough pitfalls and traps that it’s common to see blanket recommendations to avoid directly interacting with the filesystem altogether, and simply do everything through SQLite or another library when possible. At the other end, if you look at GUI applications, the filesystem is a bit lower-level than it should be. At least, the Unix concept of filesystem is too low-level. If I open a file in a word processor, save it, shouldn’t I be able to move around the file and rename it in the file browser, without affecting the relationship of that file to the word processor? This is possible on a Mac, but this is done with APIs inherited from the pre-Unix days of macOS.

johnvaluk 6y ago
A file is more than simply a unit or collection of data in a local hierarchy. It's an abstraction that supports the copying, exchange and even possession of data. How is one supposed to share part of a database with another system or user without ultimately resorting to the file metaphor?
- klodolph 6y ago
  > How is one supposed to share part of a database with another system or user without ultimately resorting to the file metaphor? It is exceptionally difficult to share a database using files. This is probably one of the worst ways to share a database. Most of the time, the best you can do is take a snapshot of the database, dump it to a file, and import it into another system. Generally you would want to share a database by granting multiple users access to a single running instance.
  - sizt 6y ago
    Well, SQLite is the most prevalent database on the planet, and its a file so ... https://sqlite.org/fileformat2.html https://sqlite.org/fileformat2.html
- webmaven 6y ago
  > How is one supposed to share part of a database with another system or user without ultimately resorting to the file metaphor? With an access-controlled view, perhaps? Your question isn't entirely academic, incidentally, the Open Data movement prompted a lot of organizations (particularly government or government-like ones) to address this sort of issue (especially with geodata) en masse starting about a decade ago, and providing snapshots in the form of downloadable files is generally seen as an important but generally inferior when-all-else-fails solution for most use-cases that don't involve forking the data and/or bulk-processing it into a new form. Live read-only access to the system of record or a DMZ'd proxy or cache is usually preferred. And the Semantic Web / Linked Open Data movement addressed a similar set of issues two decades ago. I suppose it's time for another turn if the wheel, perhaps with a distributed web twist.
emptyparadise 6y ago
I think the most important thing about files is being able to take a unit of work - a picture or a document - and interact with it as a single object, doing things like copying it or sharing it with others. Any replacement system that doesn't let me take my work and simply drag it to a flash drive is flawed. This doesn't mean we can't use a database, but it is something important to remember.
- klodolph 6y ago
  > I think the most important thing about files is being able to take a unit of work - a picture or a document - and interact with it as a single object, doing things like copying it or sharing it with others. Yes, we want this, but the filesystem is actually quite bad at this! We have just gotten used to how bad files and filesystems are, and ignore the fact that they are not good matches for our metaphors about how they work. Just to be clear—I completely disagree with your claims about how files work. You can’t really interact much with a file as a single object. When you open a document, a copy of the file is loaded into memory. This separate, invisible copy does not show up on the filesystem and will be out of synch with the filesystem copy. For example, 1. Open a text file in Vim, Nano, or Emacs. 2. In another terminal window, rename the file. 3. Make changes to the text file and save it. What happens? You end up with two different files! This is not some weird hackery, this is just renaming a file, one of the most boring things you can do. Something I do all the time. It shows that files are not single objects, and you do not interact with them that way. This is one of the failings of the Unix filesystem model—files are not single objects. There is a ton of room for improvement. Just as a point of comparison, Windows is a bit worse (usually won’t even let you rename the file) and the Mac is a bit better (when you rename a file, the open document will follow the rename).
  - emptyparadise 6y ago
    I'm speaking more of files as a metaphor than as a file system implementation detail. I'm pointing this out because the mobile first and web first world dropped the ball on this incredibly badly. Every webapp is an isolated silo with weird logins and no proper offline usage whatsoever. To share a small anecdote, I recently asked a non-technical acquaintance how they would share a Word document from their desktop computer: they said they'd email the file. Then I asked how they would share a note from their phone's notes app: they had no clue. Yes, implementation details bleed through: I can tell that Word operates on files and the notes app probably has some sqlite database under the hood, but ultimately it's about how we interact with data. Word operates on content you have, while the notes app operates on content it has. And this is where my flash drive test came from, not because I'm attached to inodes or directories, but because of the interaction model that decouples apps and data. If tomorrow I wake up in a world where ext4 has been replaced by Postgres as its file system, I wouldn't miss chmod or fsck. But I'd want to be sure I can still take a single object (a file!) and do actions with it - send it to somebody, delete it, copy it, all without it having some inseparable link to an app or a website.