6 ms·

Considering how many things have been implemented on top of HTTP, I find it interesting that something like WebDAV (or Solid[0], or remoteStorage[1]) hasn't nea

Considering how many things have been implemented on top of HTTP, I find it interesting that something like WebDAV (or Solid[0], or remoteStorage[1]) hasn't nearly completely supplanted NFS/SFTP/etc. Not saying that would be ideal from a technical perspective (WebDAV at least has some issues), just surprised the ability to access remote filesystems from the browser hasn't been a bigger driver. For example, there are a lot of interesting apps built on top of Google Drive as the storage backend, but overall the concept doesn't seem to have gained much traction. [0]: https://inrupt.com/solid https://inrupt.com/solid [1]: https://remotestorage.io/ https://remotestorage.io/

mprovost 6y ago
Sun developed WebNFS [0] specifically so Java applets (remember them?) could access remote filesystems through firewalls from the browser. It's basically the same protocol as NFSv3 with some clever workarounds so that it doesn't require the separate MOUNT protocol to set up a connection (or the portmapper). That way everything runs on a single port. [0] https://en.wikipedia.org/wiki/WebNFS https://en.wikipedia.org/wiki/WebNFS
protomyth 6y ago
Because WebDAV implementations aren't that good and a true pain in the butt for system admins to setup. If someone made one that was half as easy to setup as say PHP, it would be doing really well right now.
api 6y ago
NFS is a lot faster. A custom protocol will almost always beat a kludge on top of HTTP
- fulafel 6y ago
  But NFS isn't a custom protocol for any specific app either, and apps that need good fs performance (eg databases, video editing platforms, "big data" compute platforms etc) avoid it. Implementation-wise HTTP has the advantage that the apps can tune the client code because the protocol client implementation isn't in the kernel. In addition HTTP security (transport & user authentication) is works and everyone understands it, whereas on NFS it's a huge mess and enemy of interoperability. In practical popularity NFS fares extremely poorly eg on AWS (how many apps use EFS vs S3).
  - ralph84 6y ago
    EFS is much more expensive than S3, so there is a strong incentive to make your app work with S3 unless you absolutely need a filesystem.
    - kondro 6y ago
      As with all things AWS, it depends. Yes EFS is 30¢ per GB and S3 is 2.3¢. But S3 also charges $5/million object writes (and 40¢ for reads), but EFS is "free" (factoring burst capacity, per-instance maximum throughput limits, etc). Couple that with EFS' lifecycle migration allowing you to get storage costs as low as 2.5¢/GB for many write-once-read-(almost)never after 7 day applications, things get a lot more complicated.
- anderspitman 6y ago
  Yeah, that's what I was alluding to by saying it would likely not be ideal from a technical perspective. But raw TCP with a custom protocol is faster also, and yet many (most? all?) mobile apps use HTTP+JSON, even if they don't have a web app to support.
  - peterwwillis 6y ago
    Most internet applications use HTTP because real networking is a pain, designing a custom protocol correctly is difficult, and HTTP provides enough functionality and flexibility to munge network traffic any way you want. They use JSON because it's the least pain in the ass of all the data formats. Custom protocol over UDP is the fastest, which NFS supports, so NFS is basically the fastest (unless you get into wonky multiplex-streamed custom apps). However, UDP apps do not often work well through firewalls.
    - skissane 6y ago
      I'm wondering if we are going to see NFS-over-QUIC. Widespread adoption of QUIC may make more firewalls become UDP-friendly.
      - ComputerGuru 6y ago
        NFS is a really slow churning wheel. I think you’ll probably first see an inefficient generic option to tunnel tcp over quic (on both ends) that someone will hack to support nfs before you see an NFSv5 with quic support.
      - fulafel 6y ago
        NFS is not particularly married to UDP though, unless we're talking about old timey NFSv2. There was even a WebNFS spec back in the day designed to let Java applets mount stuff over the internet.
    - anderspitman 6y ago
      I feel like I'm fairly familiar with the tradeoffs. I'm simply making an observation that it's interesting to me that filesystem access over HTTP isn't bigger than it is.
reincarnate0x14 6y ago
Mostly target audience, I'd imagine. Anyone still using NFS or iSCSI probably has specific performance or integrity requirements that those services aren't geared for, plus multiple concurrent access that makes locking and cache coherence significant questions. In terms of low-urgency storage though, the Dropbox/Google Drive/Syncthing model of directory replication largely has supplanted SFTP, though. There is probably some middle group of S3/B2 type storage backing some FUSE-style integrated filesystem, but I'd suspect the same problem that makes WebDAV unpopular hampers that, namely that it almost invariably sucks horribly on Windows when it works at all.
- dnautics 6y ago
  Nfs is a protocol, and there are many other interesting uses for it, for example nfs over vsock.
peterwwillis 6y ago
Almost all WebDAV implementations are a big pile of crap, and network filesystems in general are much more tricky than people realize. If somebody invents a brand new protocol that that actually works it would probably be adopted, but it would face at least as many problems as NFS does. I don't really understand why people want network filesystems most of the time. If you try to pretend it's like real local i/o, and it's not on a perfect network, you will have a bad time. Protocols designed specifically just to transfer a file tend to be more resilient and also not expose people to the problems of programs trying to do file i/o.
- anderspitman 6y ago
  I don't think you need to go full filesystem abstraction, a la FUSE. I completely agree trying to do something like access a git repo on a FUSE FS which is actually remote is going to be painful. But just having a standardized way to do directory listings, uploads, partial writes, auth, etc over HTTP would be useful to me.
  - ComputerGuru 6y ago
    Windows 10 has a new API just for this, the integration with OneDrive (apart from being extremely obtrusive/pressuring me into use something I don’t want) works great. It’s basically a shell component that abstracts over file selection for open/save and an api for providing the list of files, reading or writing files/parts of files, and shims around common fs operations like copy and move.
cbhl 6y ago
I remember trying a few WebDAV implementations back in the day -- the performance was awful, and they were difficult to configure compared to a vanilla SSH or Apache/Nginx install. Implementing on top of HTTP is usually to make things easier for users behind firewalls, squid proxies, and the like. In my experience, those environments (schools, offices) usually had to open port 21 and/or 22 for FTP/SFTP for a web maintainer anyway...

mprovost 6y ago

Sun developed WebNFS [0] specifically so Java applets (remember them?) could access remote filesystems through firewalls from the browser. It's basically the same protocol as NFSv3 with some clever workarounds so that it doesn't require the separate MOUNT protocol to set up a connection (or the portmapper). That way everything runs on a single port. [0] https://en.wikipedia.org/wiki/WebNFS https://en.wikipedia.org/wiki/WebNFS

protomyth 6y ago

Because WebDAV implementations aren't that good and a true pain in the butt for system admins to setup. If someone made one that was half as easy to setup as say PHP, it would be doing really well right now.

api 6y ago

NFS is a lot faster. A custom protocol will almost always beat a kludge on top of HTTP

fulafel 6y ago
But NFS isn't a custom protocol for any specific app either, and apps that need good fs performance (eg databases, video editing platforms, "big data" compute platforms etc) avoid it. Implementation-wise HTTP has the advantage that the apps can tune the client code because the protocol client implementation isn't in the kernel. In addition HTTP security (transport & user authentication) is works and everyone understands it, whereas on NFS it's a huge mess and enemy of interoperability. In practical popularity NFS fares extremely poorly eg on AWS (how many apps use EFS vs S3).
- ralph84 6y ago
  EFS is much more expensive than S3, so there is a strong incentive to make your app work with S3 unless you absolutely need a filesystem.
  - kondro 6y ago
    As with all things AWS, it depends. Yes EFS is 30¢ per GB and S3 is 2.3¢. But S3 also charges $5/million object writes (and 40¢ for reads), but EFS is "free" (factoring burst capacity, per-instance maximum throughput limits, etc). Couple that with EFS' lifecycle migration allowing you to get storage costs as low as 2.5¢/GB for many write-once-read-(almost)never after 7 day applications, things get a lot more complicated.
anderspitman 6y ago
Yeah, that's what I was alluding to by saying it would likely not be ideal from a technical perspective. But raw TCP with a custom protocol is faster also, and yet many (most? all?) mobile apps use HTTP+JSON, even if they don't have a web app to support.
- peterwwillis 6y ago
  Most internet applications use HTTP because real networking is a pain, designing a custom protocol correctly is difficult, and HTTP provides enough functionality and flexibility to munge network traffic any way you want. They use JSON because it's the least pain in the ass of all the data formats. Custom protocol over UDP is the fastest, which NFS supports, so NFS is basically the fastest (unless you get into wonky multiplex-streamed custom apps). However, UDP apps do not often work well through firewalls.
  - skissane 6y ago
    I'm wondering if we are going to see NFS-over-QUIC. Widespread adoption of QUIC may make more firewalls become UDP-friendly.
    - ComputerGuru 6y ago
      NFS is a really slow churning wheel. I think you’ll probably first see an inefficient generic option to tunnel tcp over quic (on both ends) that someone will hack to support nfs before you see an NFSv5 with quic support.
    - fulafel 6y ago
      NFS is not particularly married to UDP though, unless we're talking about old timey NFSv2. There was even a WebNFS spec back in the day designed to let Java applets mount stuff over the internet.
  - anderspitman 6y ago
    I feel like I'm fairly familiar with the tradeoffs. I'm simply making an observation that it's interesting to me that filesystem access over HTTP isn't bigger than it is.

reincarnate0x14 6y ago

Mostly target audience, I'd imagine. Anyone still using NFS or iSCSI probably has specific performance or integrity requirements that those services aren't geared for, plus multiple concurrent access that makes locking and cache coherence significant questions. In terms of low-urgency storage though, the Dropbox/Google Drive/Syncthing model of directory replication largely has supplanted SFTP, though. There is probably some middle group of S3/B2 type storage backing some FUSE-style integrated filesystem, but I'd suspect the same problem that makes WebDAV unpopular hampers that, namely that it almost invariably sucks horribly on Windows when it works at all.

dnautics 6y ago
Nfs is a protocol, and there are many other interesting uses for it, for example nfs over vsock.

peterwwillis 6y ago

Almost all WebDAV implementations are a big pile of crap, and network filesystems in general are much more tricky than people realize. If somebody invents a brand new protocol that that actually works it would probably be adopted, but it would face at least as many problems as NFS does. I don't really understand why people want network filesystems most of the time. If you try to pretend it's like real local i/o, and it's not on a perfect network, you will have a bad time. Protocols designed specifically just to transfer a file tend to be more resilient and also not expose people to the problems of programs trying to do file i/o.

anderspitman 6y ago
I don't think you need to go full filesystem abstraction, a la FUSE. I completely agree trying to do something like access a git repo on a FUSE FS which is actually remote is going to be painful. But just having a standardized way to do directory listings, uploads, partial writes, auth, etc over HTTP would be useful to me.
- ComputerGuru 6y ago
  Windows 10 has a new API just for this, the integration with OneDrive (apart from being extremely obtrusive/pressuring me into use something I don’t want) works great. It’s basically a shell component that abstracts over file selection for open/save and an api for providing the list of files, reading or writing files/parts of files, and shims around common fs operations like copy and move.

cbhl 6y ago

I remember trying a few WebDAV implementations back in the day -- the performance was awful, and they were difficult to configure compared to a vanilla SSH or Apache/Nginx install. Implementing on top of HTTP is usually to make things easier for users behind firewalls, squid proxies, and the like. In my experience, those environments (schools, offices) usually had to open port 21 and/or 22 for FTP/SFTP for a web maintainer anyway...