4 ms·

Mounting WebDAV -- if you are in a situation, where you have to do it (e.g. own^W^W^Wnextcloud) is such an adventure. Everything - mac, win, linux - supports We

by 112233 9mo ago

Mounting WebDAV -- if you are in a situation, where you have to do it (e.g. own^W^W^Wnextcloud) is such an adventure. Everything - mac, win, linux - supports WebDAV. You mount and it works! Then you notice HOW it works: files are downloaded in full before program can access them, some operations are super slow, some fail or time out, plaintext credentials in mysterious places... I heard DeltaV is very advanced, and Subversion supported it. I'm afraid to ask.

n3storm 9mo ago
Subversion works ok over webdav, it has done it for decades. Mounting a directory through nfs, smb or ssh and files are downloaded in full before program access them. What you mean? Listing a directory or accessing file properties, like size for example do not need full download.
- 112233 9mo ago
  I am confused, what do you mean? What OS forces you to download whole file over NFS or SMB before serving read()? Even SFTP does support reading and writing at an offset.
  - shellac 9mo ago
    Are you saying WebDAV doesn't support range requests?
    - 112233 9mo ago
      That's the beauty of working with WebDAV, also captured vividly in the above article -- any particular server/client combination feels no obligation to try and act like some "standards" prescribe, or make use of facilities available. I might be wrong, but when I last mounted webdav from windows, it did the same dumb thing too.
    - blacklion 9mo ago
      WebDAV as standard? Supports. This particular combination of client and server? Who knows, good luck.
  - n3storm 9mo ago
    If I open a nfs doc with, let's say Libreoffice, will I not download whole file? On a second thought, I think you are looking at webdav as sysadmins not as developers. Webdav was designed for document authoring, and you cannot author a document, version it, merge other authors changes, track changes without fully controlling resources. Conceptually is much like git needs a local copy. I can't imagine how to have an editor editing a file and file is changed at any offset at any time by any unknown agent whitouth any type of orchestration.
    - wbl 9mo ago
      NFS infamously proxies reads and writes. Obviously there is some caching but that just makes the behavior funner.
    - rkeene2 9mo ago
      If you open a file with LibreOffice will read the whole thing regardless of whether or not the file is on NFS or not. The parent comment was stating that if you use the open(2) system call on a WebDAV mounted filesystem, which doesn't perform any read operation, the entire file will be downloaded locally before that system call completes. This is not true for NFS which has more granular access patterns using the READ operation (e.g., READ3) and file locking operations. It may be the case that you're using an application that isn't LibreOffice on files that aren't as small as documents -- for example if you wanted to watch a video via a remote filesystem. If that filesystem is WebDAV (davfs2) then before the first piece of metadata can be displayed the entire file would be downloaded locally, versus if it was NFS each 4KiB (or whatever your block size is) chunk would be fetched independently.
    - theamk 9mo ago
      Libreoffice will likely download the whole file. But many others clients won't. In particular, any video player will _not_ download entire file before accessing it. And for images, many viewers start showing image before whole thing is downloaded. And to look at zip files, you don't need the whole thing - just index at the end. And for music, you stream data... Requiring that file is "downloaded in full before program access them" is a pretty bad degradation in a lot of cases. I've used smb and nfs and sshfs and they all let you read any range of file, and start giving the data immediately, even before the full download.
- goodthink 9mo ago
  > Subversion works ok over webdav, it has done it for decades. Thank you!!!!
jjkaczor 9mo ago
Actually - I believe - within Windows 11 - the "WebClient" service is now deprecated (which is what - IIRC, actually implements the WebDAV client protocol so that it works with Windows File Explorer, drive mappings, etc.)... Played around with WebDAV alot... a long time ago... (Exchange Webstore/Webstorage System, STS/SharePoint early editions)...
hurflmurfl 9mo ago
I'm using the nextcloud app on my android, and for my Linux systems I mount WebDAV using rclone, with VFS cache mode set to FULL. This way I can: 1. Have the file structure etc synced to local without downloading the files 2. Have it fetch files automatically when I try to read them. Also supports range requests, so if I want to play a video, it sort of streams it, no need to wait for download. 3. If a file has been accessed locally, it's going to be cached for a while, so even if I'm offline, I can still access the cached version without having to verify that it's the latest. If I'm online, then it will verify if it's the latest version. Overall, this has worked great for me, but it did take me a while before I set it up correctly. Now I have a cache of files I use, and the rest of the stuff that I just keep there for backup or hogging purposes doesn't take disk space and stays in the cloud until I sync it.
- sureglymop 9mo ago
  Sine you are mounting and not syncing the files, what happens when you edit a file offline? And what if on another offline device the file is also edited?
  - hurflmurfl 9mo ago
    Fair question. Conflicts happen, which I'm fine with. Realistically speaking, most files I have in my cloud are read-only. The most common file that I read-write on multiple devices is my keepass file, which supports conflict resolution (by merging changes) in clients. Also used to happen when I tried editing some markdown notes using obsidian on PC, and then using text editor (or maybe obsidian again?) on android, but I eventually sort of gave up on that use-case. Editing my notes from my phone is sort of inconvenient anyway, so I mostly just create new short notes that I can later edit into some larger note, but honestly can't remember the last time this happened. But yes, if not careful, you could run into your laptop overwriting the file when it comes online. In my case, it doesn't really happen, and when it does, Nextcloud will have the "overwritten version" saved, so I can always check what was overwritten and manually merge the changes. P.S. If anyone wants to set this up, here's my nixos config for the service, feel free to comment on it: # don't forget to run `rclone config` beforehand # to create the "nextcloud:" remote # some day I may do this declaratively, but not today systemd.services.rclone-nextcloud-mount = { # Ensure the service starts after the network is up wantedBy = [ "multi-user.target" ]; after = [ "network-online.target" ]; requires = [ "network-online.target" ]; # Service configuration serviceConfig = let ncDir = "/home/username/nextcloud"; mountOptions = "--vfs-cache-mode full --dir-cache-time 1w --vfs-cache-max-age 1w"; in { Type = "simple"; ExecStartPre = "/run/current-system/sw/bin/mkdir -p ${ncDir}"; # Creates folder if didn't exist ExecStart = "${pkgs.rclone}/bin/rclone mount ${mountOptions} nextcloud: ${ncDir}"; # Mounts ExecStop = "/run/current-system/sw/bin/fusermount -u ${ncDir}"; # Dismounts Restart = "on-failure"; RestartSec = "10s"; User = "username"; Group = "users"; Environment = [ "PATH=/run/wrappers/bin/:$PATH" ]; }; };
Fnoord 9mo ago
> own^W^W^Wnextcloud own^H^H^Hnextcloud or own^Wnextcloud You might wanna look into OpenCloud (formerly known as nextcloud-go) [1]. I still use Nextcloud for the uploading of files and the calendar (though I may switch the latter), but I now sync the dir with Immich. Performance-wise a relief. I also swapped Airsonic Advanced (Java) with Navidrome (Go). Same story. [1] https://github.com/opencloud-eu/opencloud https://github.com/opencloud-eu/opencloud
- solarkraft 9mo ago
  > but I now sync the dir with Immich Do you use this for anything other than photos and videos?
  - Fnoord 9mo ago
    Good point, I have to migrate that away and clean it up, but the main user is my wife, not me. I might completely get rid of Nextcloud. I mean, the name is tarnished for my wife.
blacklion 9mo ago
Windows officialy removed support for WebDAV. It still works, but nothing is guaranteed. It has stupid limitation on file size of 10MB, it can be lifted to 2GB (max signed 32 bit number) in Registry, but it is still not very much in modern world (I wanted to share my medial library via WebDAV and failed due to this limitation). It lose credentials on regular basis, errors are too vague («Wrong credentials» means both mistyped password AND expired server certificate), etc.
- bigfatkitten 9mo ago
  It’s also a bit of a disaster from a security perspective. https://www.thehacker.recipes/ad/movement/mitm-and-coerced-authentications/webclient https://www.thehacker.recipes/ad/movement/mitm-and-coerced-a...
heavyset_go 9mo ago
Regarding Linux, WebDAV has been partially working/broken in Dolphin/kio since Plasma 5 on KDE. I've found the davfs2 FUSE module to be more reliable.
- QuercusMax 9mo ago
  I just imagined implementing webdav as a kernel module and I think I just broke my brain
  - heavyset_go 9mo ago
    Here's some prior art for your cursed journey https://github.com/sysprog21/khttpd https://github.com/sysprog21/khttpd
- 112233 9mo ago
  Sibling comment mentioned rclone, which is enabling piece of software and much better at webdaw than davfs2
  - esperent 9mo ago
    As it happens, I've been deep in the with rclone, webdav, and Nextcloud over the last few days. It works. It's slow when synching a ton of small files but besides that I haven't run into any unexpected problems. If I did it again I'd probably use the Nextcloud client in API mode. But this was originally supposed to be a small project... Oh well.
  - Avamander 9mo ago
    Rclone works but is still not very good, either it doesn't allow a lot of the things you'd expect from a filesystem or it'll just consume tremendous amount of disk space while it slowly transfers the files to the remote host (even on a really fast link).
8fingerlouie 9mo ago
WebDAV is like the 12V plug in cars. Its certainly not the optimal design, but it exists in pretty much all cars, so we use it because it's there, and because of it's universal presence, its also hard to replace. The sad part is, in a world that is increasingly mobile first, and computing devices move in and out of network coverage, HTTP based protocols actually handle frequent disconnects/reconnects much better than something like SMB. For my personal backup needs, running from my phone, WebDAV is king. S3 would probably be better, protocol wise, but i can't have that in a simple "wrapper" that simply exposes existing files, and WebDAV works perfectly fine for LAN anyway.

by 112233 9mo ago

n3storm 9mo ago

Subversion works ok over webdav, it has done it for decades. Mounting a directory through nfs, smb or ssh and files are downloaded in full before program access them. What you mean? Listing a directory or accessing file properties, like size for example do not need full download.

112233 9mo ago
I am confused, what do you mean? What OS forces you to download whole file over NFS or SMB before serving read()? Even SFTP does support reading and writing at an offset.
- shellac 9mo ago
  Are you saying WebDAV doesn't support range requests?
  - 112233 9mo ago
    That's the beauty of working with WebDAV, also captured vividly in the above article -- any particular server/client combination feels no obligation to try and act like some "standards" prescribe, or make use of facilities available. I might be wrong, but when I last mounted webdav from windows, it did the same dumb thing too.
  - blacklion 9mo ago
    WebDAV as standard? Supports. This particular combination of client and server? Who knows, good luck.
- n3storm 9mo ago
  If I open a nfs doc with, let's say Libreoffice, will I not download whole file? On a second thought, I think you are looking at webdav as sysadmins not as developers. Webdav was designed for document authoring, and you cannot author a document, version it, merge other authors changes, track changes without fully controlling resources. Conceptually is much like git needs a local copy. I can't imagine how to have an editor editing a file and file is changed at any offset at any time by any unknown agent whitouth any type of orchestration.
  - wbl 9mo ago
    NFS infamously proxies reads and writes. Obviously there is some caching but that just makes the behavior funner.
  - rkeene2 9mo ago
    If you open a file with LibreOffice will read the whole thing regardless of whether or not the file is on NFS or not. The parent comment was stating that if you use the open(2) system call on a WebDAV mounted filesystem, which doesn't perform any read operation, the entire file will be downloaded locally before that system call completes. This is not true for NFS which has more granular access patterns using the READ operation (e.g., READ3) and file locking operations. It may be the case that you're using an application that isn't LibreOffice on files that aren't as small as documents -- for example if you wanted to watch a video via a remote filesystem. If that filesystem is WebDAV (davfs2) then before the first piece of metadata can be displayed the entire file would be downloaded locally, versus if it was NFS each 4KiB (or whatever your block size is) chunk would be fetched independently.
  - theamk 9mo ago
    Libreoffice will likely download the whole file. But many others clients won't. In particular, any video player will _not_ download entire file before accessing it. And for images, many viewers start showing image before whole thing is downloaded. And to look at zip files, you don't need the whole thing - just index at the end. And for music, you stream data... Requiring that file is "downloaded in full before program access them" is a pretty bad degradation in a lot of cases. I've used smb and nfs and sshfs and they all let you read any range of file, and start giving the data immediately, even before the full download.
goodthink 9mo ago
> Subversion works ok over webdav, it has done it for decades. Thank you!!!!

jjkaczor 9mo ago

Actually - I believe - within Windows 11 - the "WebClient" service is now deprecated (which is what - IIRC, actually implements the WebDAV client protocol so that it works with Windows File Explorer, drive mappings, etc.)... Played around with WebDAV alot... a long time ago... (Exchange Webstore/Webstorage System, STS/SharePoint early editions)...

hurflmurfl 9mo ago

I'm using the nextcloud app on my android, and for my Linux systems I mount WebDAV using rclone, with VFS cache mode set to FULL. This way I can: 1. Have the file structure etc synced to local without downloading the files 2. Have it fetch files automatically when I try to read them. Also supports range requests, so if I want to play a video, it sort of streams it, no need to wait for download. 3. If a file has been accessed locally, it's going to be cached for a while, so even if I'm offline, I can still access the cached version without having to verify that it's the latest. If I'm online, then it will verify if it's the latest version. Overall, this has worked great for me, but it did take me a while before I set it up correctly. Now I have a cache of files I use, and the rest of the stuff that I just keep there for backup or hogging purposes doesn't take disk space and stays in the cloud until I sync it.

sureglymop 9mo ago
Sine you are mounting and not syncing the files, what happens when you edit a file offline? And what if on another offline device the file is also edited?
- hurflmurfl 9mo ago
  Fair question. Conflicts happen, which I'm fine with. Realistically speaking, most files I have in my cloud are read-only. The most common file that I read-write on multiple devices is my keepass file, which supports conflict resolution (by merging changes) in clients. Also used to happen when I tried editing some markdown notes using obsidian on PC, and then using text editor (or maybe obsidian again?) on android, but I eventually sort of gave up on that use-case. Editing my notes from my phone is sort of inconvenient anyway, so I mostly just create new short notes that I can later edit into some larger note, but honestly can't remember the last time this happened. But yes, if not careful, you could run into your laptop overwriting the file when it comes online. In my case, it doesn't really happen, and when it does, Nextcloud will have the "overwritten version" saved, so I can always check what was overwritten and manually merge the changes. P.S. If anyone wants to set this up, here's my nixos config for the service, feel free to comment on it: # don't forget to run `rclone config` beforehand # to create the "nextcloud:" remote # some day I may do this declaratively, but not today systemd.services.rclone-nextcloud-mount = { # Ensure the service starts after the network is up wantedBy = [ "multi-user.target" ]; after = [ "network-online.target" ]; requires = [ "network-online.target" ]; # Service configuration serviceConfig = let ncDir = "/home/username/nextcloud"; mountOptions = "--vfs-cache-mode full --dir-cache-time 1w --vfs-cache-max-age 1w"; in { Type = "simple"; ExecStartPre = "/run/current-system/sw/bin/mkdir -p ${ncDir}"; # Creates folder if didn't exist ExecStart = "${pkgs.rclone}/bin/rclone mount ${mountOptions} nextcloud: ${ncDir}"; # Mounts ExecStop = "/run/current-system/sw/bin/fusermount -u ${ncDir}"; # Dismounts Restart = "on-failure"; RestartSec = "10s"; User = "username"; Group = "users"; Environment = [ "PATH=/run/wrappers/bin/:$PATH" ]; }; };

Fnoord 9mo ago

> own^W^W^Wnextcloud own^H^H^Hnextcloud or own^Wnextcloud You might wanna look into OpenCloud (formerly known as nextcloud-go) [1]. I still use Nextcloud for the uploading of files and the calendar (though I may switch the latter), but I now sync the dir with Immich. Performance-wise a relief. I also swapped Airsonic Advanced (Java) with Navidrome (Go). Same story. [1] https://github.com/opencloud-eu/opencloud https://github.com/opencloud-eu/opencloud

solarkraft 9mo ago
> but I now sync the dir with Immich Do you use this for anything other than photos and videos?
- Fnoord 9mo ago
  Good point, I have to migrate that away and clean it up, but the main user is my wife, not me. I might completely get rid of Nextcloud. I mean, the name is tarnished for my wife.

blacklion 9mo ago

Windows officialy removed support for WebDAV. It still works, but nothing is guaranteed. It has stupid limitation on file size of 10MB, it can be lifted to 2GB (max signed 32 bit number) in Registry, but it is still not very much in modern world (I wanted to share my medial library via WebDAV and failed due to this limitation). It lose credentials on regular basis, errors are too vague («Wrong credentials» means both mistyped password AND expired server certificate), etc.

bigfatkitten 9mo ago
It’s also a bit of a disaster from a security perspective. https://www.thehacker.recipes/ad/movement/mitm-and-coerced-authentications/webclient https://www.thehacker.recipes/ad/movement/mitm-and-coerced-a...

heavyset_go 9mo ago

Regarding Linux, WebDAV has been partially working/broken in Dolphin/kio since Plasma 5 on KDE. I've found the davfs2 FUSE module to be more reliable.

QuercusMax 9mo ago
I just imagined implementing webdav as a kernel module and I think I just broke my brain
- heavyset_go 9mo ago
  Here's some prior art for your cursed journey https://github.com/sysprog21/khttpd https://github.com/sysprog21/khttpd
112233 9mo ago
Sibling comment mentioned rclone, which is enabling piece of software and much better at webdaw than davfs2
- esperent 9mo ago
  As it happens, I've been deep in the with rclone, webdav, and Nextcloud over the last few days. It works. It's slow when synching a ton of small files but besides that I haven't run into any unexpected problems. If I did it again I'd probably use the Nextcloud client in API mode. But this was originally supposed to be a small project... Oh well.
- Avamander 9mo ago
  Rclone works but is still not very good, either it doesn't allow a lot of the things you'd expect from a filesystem or it'll just consume tremendous amount of disk space while it slowly transfers the files to the remote host (even on a really fast link).

8fingerlouie 9mo ago

WebDAV is like the 12V plug in cars. Its certainly not the optimal design, but it exists in pretty much all cars, so we use it because it's there, and because of it's universal presence, its also hard to replace. The sad part is, in a world that is increasingly mobile first, and computing devices move in and out of network coverage, HTTP based protocols actually handle frequent disconnects/reconnects much better than something like SMB. For my personal backup needs, running from my phone, WebDAV is king. S3 would probably be better, protocol wise, but i can't have that in a simple "wrapper" that simply exposes existing files, and WebDAV works perfectly fine for LAN anyway.