5 ms·

I love all the cool backend remote storage projects we have, but what I really want is a simple, well-specified app protocol for filesystem operations. WebDAV i

I love all the cool backend remote storage projects we have, but what I really want is a simple, well-specified app protocol for filesystem operations. WebDAV is too crufty. I'm bullish on Solid but I think there's room for something much simpler. I've done a bit of work[0] in this space but keep getting distracted with other parts of the self-hosted storage stack. [0]: https://gemdrive.io/ https://gemdrive.io/

qbasic_forever 4y ago
9p? https://en.wikipedia.org/wiki/9P_(protocol) https://en.wikipedia.org/wiki/9P_(protocol) I believe most VMs use 9p as a low level protocol for syncing filesystems between host and VM or container.
- anderspitman 4y ago
  9p won't work for web apps. Personally I think we just need a small layer on top of standard HTTP for most use cases. It's really not a very hard technical problem, we just need to agree on a protocol. But it's a chicken-egg situation. App developers aren't going to develop against a protocol that no one uses, and no one's going to use a protocol that doesn't have any apps.
  - qbasic_forever 4y ago
    I'd tunnel it over a websocket, or there's no reason why you couldn't translate its verbs into a REST API. It's a super simple protocol with not a lot of complexity. It sounds like you're after a federated storage protocol though, i.e. something that two totally different apps can use to allow a user to transfer data between them. This is a much, much bigger leap in complexity and requires a very strong authentication and authorization story. IMHO look into the fediverse and whatever is going on with storage and files there.
    - mike_hearn 4y ago
      Isn't translating filesystem operation verbs into an HTTP based protocol what WebDAV does?
      - anderspitman 4y ago
        Yes, and it's almost what we need. I think it's just too ossified to adapt to the few remaining shortcomings. Here's a couple: * Doesn't specify a federated authentication/authorization mechanism to allow self-hosted instances to work with each other. This includes things like how to have an app delegate permissions to other apps, ie for things like launcher apps which can open external apps and give them access to only specific files. * Doesn't specify a way to implement image thumbnails/previews, which is necessary for any sort of remote file explorer or gallery app.
  - monocasa 4y ago
    Plan 9 (well, 9 front at least) natively supports 9p over websockets. https://p9f.org/magic/man2html/8/websocket https://p9f.org/magic/man2html/8/websocket
    - anderspitman 4y ago
      That's pretty sweet, but it's an entire additional layer of unnecessary complexity. Plus it still requires everyone to implement a new protocol into their apps. Why not use one purpose-built for it? Also, WebSockets bring some nontrivial complexity. This can bite you with things like proxies and custom backends, which have to implement a special code path from normal HTTP to handle WebSockets.
      - mike_d 4y ago
        Yeah... computers are hard sometimes. If you want to read and write files, there are plenty of operating system APIs to do it super simple. If you want low level access to file systems and to get into the complex parts... well WebDAV sucks because the underlying problems of file systems suck.
        anderspitman 4y ago
        But you don't need that level of complexity for large classes of problems.
      - monocasa 4y ago
        I basically view WebDAV as the weird one off protocol here. 9p runs over just about any stream oriented transport (including web technologies), while WebDAV is planted firmly in web technologies.
spicybright 4y ago
What's crufty about WebDAV? I've never used it, but it seems good enough to just use a useful subset of features to avoid the cruft.
- anderspitman 4y ago
  It uses XML, which requires a library for browser JavaScript (EDIT: not true, see comment below), when filesystem operations don't really need to be more complicated than vanilla fetch requests. Also uses special HTTP methods, which isn't ideal. I've heard people say that various implementations are often incompatible with each other in some ways. I'm not sure how big of a problem this actually is in practice, but see for example this readme[0] which explicitly says the library doesn't seek to be compatible with RFCs, but rather with implementations in the wild. That's a red flag to me. See also [1]. Trying to use a subset is an interesting idea, but the technical problem being solved is simple enough that I don't see a strong reason not to just greenfield it and make a bunch of nice improvements along with the critical ones. [0]: https://github.com/perry-mitchell/webdav-client#about https://github.com/perry-mitchell/webdav-client#about [1]: https://news.ycombinator.com/item?id=10213657 https://news.ycombinator.com/item?id=10213657
  - hunterb123 4y ago
    > It uses XML, which requires a library for browser JavaScript, What? XMLHTTPRequest (XHR) People forget XML parsing is baked into JS as is XPath But I do agree with your premise otherwise.
    - anderspitman 4y ago
      Ha. Honestly, you're totally right. I've used XHR for years but never actually for XML. When looking at WebDAV in the past I couldn't find a way to serialize/deserialize XML the same way you can with JSON (is there a way?), but I didn't realize XHR handles that natively. TIL, thanks.
      - easrng 4y ago
vineyardmike 4y ago
> what I really want is a simple, well-specified app protocol for filesystem operations. Genuine question, what’s wrong with existing Operating System FS + FUSE with any remote protocol? Eg there exists S3fuse if you want object/bucket storage, sshfs for using ssh to a host, and the fuse api is easy enough to port to custom backends (in memory, db, etc). Don’t want to be the guy saying “who needs Dropbox we have sftp” but you’re asking for a technical spec and protocol… and one exists? I haven’t worked too much in this space so I don’t know what tasks are hard to do, but it seems like leaning on existing file system tools/api at the app level makes sense? What’s missing, does “over the network” need to be a primary consideration for these uses?
- anderspitman 4y ago
  The page I linked lists quite a uses cases that would become possible/easier with a new protocol. To your specific concerns, object storage systems like S3 don't support partial writes to files, which makes FUSE mounts a leaky abstraction. SSHFS is much better, but obviously can't be used in web apps. I'm not saying we need to replace existing protocols, but we need one that can work in a more diverse set of use cases.

qbasic_forever 4y ago

9p? https://en.wikipedia.org/wiki/9P_(protocol) https://en.wikipedia.org/wiki/9P_(protocol) I believe most VMs use 9p as a low level protocol for syncing filesystems between host and VM or container.

anderspitman 4y ago
9p won't work for web apps. Personally I think we just need a small layer on top of standard HTTP for most use cases. It's really not a very hard technical problem, we just need to agree on a protocol. But it's a chicken-egg situation. App developers aren't going to develop against a protocol that no one uses, and no one's going to use a protocol that doesn't have any apps.
- qbasic_forever 4y ago
  I'd tunnel it over a websocket, or there's no reason why you couldn't translate its verbs into a REST API. It's a super simple protocol with not a lot of complexity. It sounds like you're after a federated storage protocol though, i.e. something that two totally different apps can use to allow a user to transfer data between them. This is a much, much bigger leap in complexity and requires a very strong authentication and authorization story. IMHO look into the fediverse and whatever is going on with storage and files there.
  - mike_hearn 4y ago
    Isn't translating filesystem operation verbs into an HTTP based protocol what WebDAV does?
    - anderspitman 4y ago
      Yes, and it's almost what we need. I think it's just too ossified to adapt to the few remaining shortcomings. Here's a couple: * Doesn't specify a federated authentication/authorization mechanism to allow self-hosted instances to work with each other. This includes things like how to have an app delegate permissions to other apps, ie for things like launcher apps which can open external apps and give them access to only specific files. * Doesn't specify a way to implement image thumbnails/previews, which is necessary for any sort of remote file explorer or gallery app.
- monocasa 4y ago
  Plan 9 (well, 9 front at least) natively supports 9p over websockets. https://p9f.org/magic/man2html/8/websocket https://p9f.org/magic/man2html/8/websocket
  - anderspitman 4y ago
    That's pretty sweet, but it's an entire additional layer of unnecessary complexity. Plus it still requires everyone to implement a new protocol into their apps. Why not use one purpose-built for it? Also, WebSockets bring some nontrivial complexity. This can bite you with things like proxies and custom backends, which have to implement a special code path from normal HTTP to handle WebSockets.
    - mike_d 4y ago
      Yeah... computers are hard sometimes. If you want to read and write files, there are plenty of operating system APIs to do it super simple. If you want low level access to file systems and to get into the complex parts... well WebDAV sucks because the underlying problems of file systems suck.
      - anderspitman 4y ago
        But you don't need that level of complexity for large classes of problems.
    - monocasa 4y ago
      I basically view WebDAV as the weird one off protocol here. 9p runs over just about any stream oriented transport (including web technologies), while WebDAV is planted firmly in web technologies.

spicybright 4y ago

What's crufty about WebDAV? I've never used it, but it seems good enough to just use a useful subset of features to avoid the cruft.

anderspitman 4y ago
It uses XML, which requires a library for browser JavaScript (EDIT: not true, see comment below), when filesystem operations don't really need to be more complicated than vanilla fetch requests. Also uses special HTTP methods, which isn't ideal. I've heard people say that various implementations are often incompatible with each other in some ways. I'm not sure how big of a problem this actually is in practice, but see for example this readme[0] which explicitly says the library doesn't seek to be compatible with RFCs, but rather with implementations in the wild. That's a red flag to me. See also [1]. Trying to use a subset is an interesting idea, but the technical problem being solved is simple enough that I don't see a strong reason not to just greenfield it and make a bunch of nice improvements along with the critical ones. [0]: https://github.com/perry-mitchell/webdav-client#about https://github.com/perry-mitchell/webdav-client#about [1]: https://news.ycombinator.com/item?id=10213657 https://news.ycombinator.com/item?id=10213657
- hunterb123 4y ago
  > It uses XML, which requires a library for browser JavaScript, What? XMLHTTPRequest (XHR) People forget XML parsing is baked into JS as is XPath But I do agree with your premise otherwise.
  - anderspitman 4y ago
    Ha. Honestly, you're totally right. I've used XHR for years but never actually for XML. When looking at WebDAV in the past I couldn't find a way to serialize/deserialize XML the same way you can with JSON (is there a way?), but I didn't realize XHR handles that natively. TIL, thanks.
    - easrng 4y ago

vineyardmike 4y ago

> what I really want is a simple, well-specified app protocol for filesystem operations. Genuine question, what’s wrong with existing Operating System FS + FUSE with any remote protocol? Eg there exists S3fuse if you want object/bucket storage, sshfs for using ssh to a host, and the fuse api is easy enough to port to custom backends (in memory, db, etc). Don’t want to be the guy saying “who needs Dropbox we have sftp” but you’re asking for a technical spec and protocol… and one exists? I haven’t worked too much in this space so I don’t know what tasks are hard to do, but it seems like leaning on existing file system tools/api at the app level makes sense? What’s missing, does “over the network” need to be a primary consideration for these uses?

anderspitman 4y ago
The page I linked lists quite a uses cases that would become possible/easier with a new protocol. To your specific concerns, object storage systems like S3 don't support partial writes to files, which makes FUSE mounts a leaky abstraction. SSHFS is much better, but obviously can't be used in web apps. I'm not saying we need to replace existing protocols, but we need one that can work in a more diverse set of use cases.