3 ms·

> Locus allowed live Unix processes to migrate not just from one machine to another on a network, but to a variety of machine types. My personal take on this i

> Locus allowed live Unix processes to migrate not just from one machine to another on a network, but to a variety of machine types. My personal take on this is that need to make the data layer in computers transparent. Files are currently opaque sets of bytes which need an application specific parser to interact with them. And when you have programs that can interact with a file, the only operations are read and write. Realistically, you can only have one program editing a file at once. Even having multiple programs reading a file at the same time is often a bit broken. Instead, what if files were structured (think JSON or SQL). And instead of overwriting the file when you hit save, the application could send granular updates to the files’ content as edits happened. (Eg update Foo.bar.baz = 22, or the sql equivalent). And applications listen for changes in the data model, so another application (maybe on a different computer)! Can edit something in the file, and the changes get propagated live. There’s all sorts of benefits to this: - We can enable collaborative editing in all applications for free - It’s super easy to make plugins for applications - just make another program which interacts with the data. No plug-in api needed. - The Unix philosophy is strengthened because multiple small programs can work with the same data at once. We can split out IDEs into a lot more small tools. - Users have lots of devices. This sort of thing would make data able to move between their devices in a way that’s transparent to the application. Like live collaborative editing Dropbox. We need CRDTs to do it properly. But the technology behind that is more or less solved at this point. Honestly I think the hardest thing will be getting OS people on board. I think it’s about time we shook up how the filesystem works. open/read/write basically haven’t changed in 50 years.

pavlov 4y ago
Microsoft made a huge bet on this exact idea in the mid-1990s as part of their Cairo OS project. When Cairo fizzled, the database file system became a separate component called WinFS but didn’t result in any end-user product: https://en.wikipedia.org/wiki/WinFS https://en.wikipedia.org/wiki/WinFS
- josephg 4y ago
  What went wrong? Was it just a failure to deliver?
  - pavlov 4y ago
    My impression is that it was a combination of three things: internal Windows team dysfunction, ambition too far ahead of the storage and network technologies of the day, and lack of a clear customer use case at a time when the focus of Windows+Office shifted towards enterprise. The OO database storage project was apparently Bill Gates's personal baby, so it didn't lack in executive support. That probably explains why the dream was kept alive for over a decade (Cairo started in 1991, WinFS was finally left out of Vista in 2006). Steven Sinofsky's interesting online book "Hardcore Software" has some chapters about Cairo and later WinFS: https://hardcoresoftware.learningbyshipping.com/p/020-innovation-versus-shipping-the https://hardcoresoftware.learningbyshipping.com/p/020-innova... "Cairo aimed to advance personal computing with dramatic changes in how we thought of files—rather than single files and folders, Cairo intended for files to have the capabilities of a database. Everything on your PC was to be stored in a database to easily search, find, and show relationships between items: files, email, contacts, photos, documents and more. Advancing storage was a long arc of innovation Bill favored. "Ultimately, the human toll of Cairo was high in the sense that so many people spent so much time early in career working on a project that not only didn’t ship but was viewed as squandering resources, at best, and misguided at worst. It was a bit of a black eye for Microsoft among the press and analysts who believed Microsoft would deliver on the idea of object-oriented the way that NeXT had done but at scale. The magnitude of the project would leave many people with Cairo war stories for years to come. I wish I could say that the lessons learned would prevent another experience like this from happening, but that isn’t the case as we will see."
- rahoulb 4y ago
  Apple too, with OpenDoc (that His Steveness killed on his return)
mike_hearn 4y ago
You're on the right track. A good place to start is by defining some behaviours for cases which are errors in POSIX, because you know apps can't be using them. The most obvious move is to let directories also be files. This isn't allowed today mostly for historical reasons - directories were originally implemented as files internally, so the OS couldn't let you store your own stuff there and the side effects of this ripple all the way up and down the tech stack. If you fix this then you can "mount" a file onto itself using FUSE, such that apps that want to read the custom binary format or JSON as text can do so, and other tools that want to explore a directory structure can do that too (e.g. a shell), and the two are linked by a collection of user space format-specific filesystems. Collaborative editing is too much to bite off at once. The semantics of edit conflicts are very app specific and high level. You can get a long way by just hooking file IO in userspace.
- josephg 4y ago
  > Collaborative editing is too much to bite off at once. The semantics of edit conflicts are very app specific and high level. You can get a long way by just hooking file IO in userspace. I've been working in the collaborative editing space for over a decade at this point. So (predictably) I think it should be a day 1 feature. In my mind realtime collaborative editing is one of the main motivating use cases for this whole system - its one of the things we can't do with traditional local file based software architecture. Based on my experience with collaborative editing I don't think you need a special app-specific conflict resolution strategy most of the time. I think about 95% of applications just need CRDTs for: - Tables with editable values. (Or, equivalently, collections of JSON objects). I think an MVRegister is the way to go here. - Text And then the longer tail of apps need: - Generic sequences (eg for the list of layers in photoshop) - Rich text documents - Text with conflicts (eg for Git) For editing regular data, MVRegisters are the star of the show. MVRegisters - aka Multi-Value Registers usually just store a single value. If two users concurrently set it to different values, when you do a read operation you can either (consistently) choose a single winner, or present the application with all of the conflicting values and let it choose how to resolve the conflict. Its the best of all worlds in my mind - because conflicts are rare in practice, and this lets application developers start simple and make their apps more complex over time as they desire. MVRegisters are also crazy simple to implement.

pavlov 4y ago

Microsoft made a huge bet on this exact idea in the mid-1990s as part of their Cairo OS project. When Cairo fizzled, the database file system became a separate component called WinFS but didn’t result in any end-user product: https://en.wikipedia.org/wiki/WinFS https://en.wikipedia.org/wiki/WinFS

josephg 4y ago
What went wrong? Was it just a failure to deliver?
- pavlov 4y ago
  My impression is that it was a combination of three things: internal Windows team dysfunction, ambition too far ahead of the storage and network technologies of the day, and lack of a clear customer use case at a time when the focus of Windows+Office shifted towards enterprise. The OO database storage project was apparently Bill Gates's personal baby, so it didn't lack in executive support. That probably explains why the dream was kept alive for over a decade (Cairo started in 1991, WinFS was finally left out of Vista in 2006). Steven Sinofsky's interesting online book "Hardcore Software" has some chapters about Cairo and later WinFS: https://hardcoresoftware.learningbyshipping.com/p/020-innovation-versus-shipping-the https://hardcoresoftware.learningbyshipping.com/p/020-innova... "Cairo aimed to advance personal computing with dramatic changes in how we thought of files—rather than single files and folders, Cairo intended for files to have the capabilities of a database. Everything on your PC was to be stored in a database to easily search, find, and show relationships between items: files, email, contacts, photos, documents and more. Advancing storage was a long arc of innovation Bill favored. "Ultimately, the human toll of Cairo was high in the sense that so many people spent so much time early in career working on a project that not only didn’t ship but was viewed as squandering resources, at best, and misguided at worst. It was a bit of a black eye for Microsoft among the press and analysts who believed Microsoft would deliver on the idea of object-oriented the way that NeXT had done but at scale. The magnitude of the project would leave many people with Cairo war stories for years to come. I wish I could say that the lessons learned would prevent another experience like this from happening, but that isn’t the case as we will see."
rahoulb 4y ago
Apple too, with OpenDoc (that His Steveness killed on his return)

mike_hearn 4y ago

You're on the right track. A good place to start is by defining some behaviours for cases which are errors in POSIX, because you know apps can't be using them. The most obvious move is to let directories also be files. This isn't allowed today mostly for historical reasons - directories were originally implemented as files internally, so the OS couldn't let you store your own stuff there and the side effects of this ripple all the way up and down the tech stack. If you fix this then you can "mount" a file onto itself using FUSE, such that apps that want to read the custom binary format or JSON as text can do so, and other tools that want to explore a directory structure can do that too (e.g. a shell), and the two are linked by a collection of user space format-specific filesystems. Collaborative editing is too much to bite off at once. The semantics of edit conflicts are very app specific and high level. You can get a long way by just hooking file IO in userspace.

josephg 4y ago
> Collaborative editing is too much to bite off at once. The semantics of edit conflicts are very app specific and high level. You can get a long way by just hooking file IO in userspace. I've been working in the collaborative editing space for over a decade at this point. So (predictably) I think it should be a day 1 feature. In my mind realtime collaborative editing is one of the main motivating use cases for this whole system - its one of the things we can't do with traditional local file based software architecture. Based on my experience with collaborative editing I don't think you need a special app-specific conflict resolution strategy most of the time. I think about 95% of applications just need CRDTs for: - Tables with editable values. (Or, equivalently, collections of JSON objects). I think an MVRegister is the way to go here. - Text And then the longer tail of apps need: - Generic sequences (eg for the list of layers in photoshop) - Rich text documents - Text with conflicts (eg for Git) For editing regular data, MVRegisters are the star of the show. MVRegisters - aka Multi-Value Registers usually just store a single value. If two users concurrently set it to different values, when you do a read operation you can either (consistently) choose a single winner, or present the application with all of the conflicting values and let it choose how to resolve the conflict. Its the best of all worlds in my mind - because conflicts are rare in practice, and this lets application developers start simple and make their apps more complex over time as they desire. MVRegisters are also crazy simple to implement.