3 ms·

I've been working in the collaborative editing space for a decade now (!) and I think this problem is simply a consequence of an age old architectural mistake t

I've been working in the collaborative editing space for a decade now (!) and I think this problem is simply a consequence of an age old architectural mistake that we persist because we don't know any better. The problem is that the filesystem acts as a dumb blob store that doesn't understand changes. And it doesn't understand any of the semantics of what its storing. Its a lovely abstraction - but it has bullied us into the age old "load / edit / save" workflow. And that workflow just isn't very good: - Its slower for the computer - because no matter how incremental, the computer has to rewrite the entire file to disk - Its buggy by default. A power loss or malfunction can and will result in a corrupt save file. You can fix this with careful use of fsync() and renames, but almost nobody does that correctly because its complicated and confusing. And it makes saving slower. You see this all the time in video games - "When you see this symbol, don't turn off your console". - Because you only give the OS snapshots in time, you end up discarding information about the changes that have happened along the way. With code editing we work around this with diff-match-patch, but its lossy, not realtime, and not good enough for collaborative editing. With more complex data (like 3d models) this problem is much more egregious. There's no reason the modelling program couldn't output a fine grained sequence of changes as they're made by the user. Any collaborative editor already does this. Having a rich set of changes gives us way more options for merging - eg, they could be merged in a conflict-free way if we want. Or with a custom UI or whatever. Its strictly more information that you can generate with diff-match-patch. The problem is there's no obvious place on the filesystem to put that information, or way to tell the OS about changes as they happen. And so long as the file system is the way that disparate programs talk to each other, and the file system can't handle this information, programs ends up just discarding any detailed change sets. And we end up with ugly JSON merge conflicts in 3d files. Collaborative editing (if it happens at all) ends up done on a per-app basis rather than just working by default across the whole OS - like it should. And the result of all that is that the desktop is being slowly sidelined in favour of proprietary web apps (like figma or google docs). Why? Because web apps aren't constrained to the paradigm of saving and loading files. It won't be easy, but it seems to me like a no-brainer for us to fix.

eternalban 5y ago
> web apps aren't constrained to the paradigm of saving and loading files. How is reading and writing bytes to FS that different from doing the same on a network stack? In fact I'm puzzled by your post which seems to imply that a 'database' -- a store for structured data -- can't be written over a file system. > The problem is there's no obvious place on the filesystem to put that information, or way to tell the OS about changes as they happen. A thin layer of abstraction over FS seems reasonable, possible, and frankly ancient tech: they are called databases.
- rvense 5y ago
  Of course you can build things on top of files. But most people don't, they use files directly, and therefore get "bullied" (which I think is a very apt term) into load/edit/save cycles of file-shaped chunks. Application programmers could be provided with a better storage API, either as a library or as a standard from the platform. The result would be better software with little work from the application programmer's side. I'm not a Mac person anymore, but I believe they have something like this in the form of CoreData (no idea of it's good or not), and I imagine Windows has seven things that you can either use or not. An analogy would be how it's possible to build something that resembles concurrent processing and top of any single-tasking system, but a platform that doesn't provide it as a built-in will be very different to a platform that does. Non-UNIX operating systems had structured data as a core part of their libraries. This was probably a good idea that got lost thrown aside by some people call "the minimalist elegance of UNIX", which other people, including I imagine my GP post, have less endearing terms for.
  - eternalban 5y ago
    The GP I replied to starts with "I've been working in the collaborative editing space for a decade now (!)" and claiming a network interface somehow magically does something that an FS can not (and speaking of Unix, entirely ignoring that both mechanisms use file descriptors). I mean, what to make of statements such as "[t]he problem is there's no obvious place on the filesystem to put that [meta-data] information"? As for the generic "application programmers", possibly you are right, and doing 'literal' mappings between their memory and persistent models is to be expected.

eternalban 5y ago

> web apps aren't constrained to the paradigm of saving and loading files. How is reading and writing bytes to FS that different from doing the same on a network stack? In fact I'm puzzled by your post which seems to imply that a 'database' -- a store for structured data -- can't be written over a file system. > The problem is there's no obvious place on the filesystem to put that information, or way to tell the OS about changes as they happen. A thin layer of abstraction over FS seems reasonable, possible, and frankly ancient tech: they are called databases.

rvense 5y ago
Of course you can build things on top of files. But most people don't, they use files directly, and therefore get "bullied" (which I think is a very apt term) into load/edit/save cycles of file-shaped chunks. Application programmers could be provided with a better storage API, either as a library or as a standard from the platform. The result would be better software with little work from the application programmer's side. I'm not a Mac person anymore, but I believe they have something like this in the form of CoreData (no idea of it's good or not), and I imagine Windows has seven things that you can either use or not. An analogy would be how it's possible to build something that resembles concurrent processing and top of any single-tasking system, but a platform that doesn't provide it as a built-in will be very different to a platform that does. Non-UNIX operating systems had structured data as a core part of their libraries. This was probably a good idea that got lost thrown aside by some people call "the minimalist elegance of UNIX", which other people, including I imagine my GP post, have less endearing terms for.
- eternalban 5y ago
  The GP I replied to starts with "I've been working in the collaborative editing space for a decade now (!)" and claiming a network interface somehow magically does something that an FS can not (and speaking of Unix, entirely ignoring that both mechanisms use file descriptors). I mean, what to make of statements such as "[t]he problem is there's no obvious place on the filesystem to put that [meta-data] information"? As for the generic "application programmers", possibly you are right, and doing 'literal' mappings between their memory and persistent models is to be expected.