23 ms·

I hope WebDAV dies

102 points by kristianp 11y ago

Tharkun 11y ago
OP seems to think that parsing XML is slow. If parsing XML is slower than parsing JSON, you're doing something wrong.
- untitaker_ 11y ago
  Parsing was never the bottleneck, I admit. But I can't see how parsing XML is as easy a task as parsing JSON either? Of course you can show me some benchmarks that show that libxml2 is much faster than the avg. JSON implementation. But you can do that with any format that is verbose, complex and therefore hard to parse, yet has been around for so long that it paid off to write well-optimized C libraries for it.
  - Tharkun 11y ago
    Exactly. You can point to benchmarks that favour one or the other. Parsing is almost never the bottleneck.
    - untitaker_ 11y ago
      The point I'm trying to make is that JSON is generally easier to parse, therefore theoretically faster to parse, even though XML implementations might be faster at the moment.
      - tveita 11y ago
        A theoretical microsecond difference that doesn't even exist in practice doesn't seem worth mentioning when there is network traffic involved. The article says: "Yes, it's slow to fetch all events, but so is parsing XML. " That's several orders of magnitude difference glossed over.
        anentropic 11y ago
        of course, being verbose, XML will be slower to fetch across the network as well
        untitaker_ 11y ago
        Meh, I have to agree that it isn't a good point. I guess I was merely trying to point out that both have its performance deficiencies. At least the ones of remoteStorage are avoidable, while you can't just avoid XML in WebDAV.
  - mikekchar 11y ago
    I recommend that you simply write a JSON parser and an XML parser using a parser generator. Then you will see. They are both trivial. I would be surprised if it took you more than an afternoon to do both. Ease of parsing was the whole point of XML. It was actually a beautiful concept that because it was a subset of SGML, you could validate documents and build smart editors, etc, etc on file formats in XML. Then when you wanted to simply read the file, the parser was mind blowingly simple to write. Of course people decided that it would be great for communications protocols, and inter-language object representations and... all the stuff that it really wasn't great for. And people decided that you would want to write event driven parsers and do DOM traversals and crazy things like that. Yeah, that stuff is complicated. (and dare I offend people by saying that it wasn't really a good idea after all). But parsing is dead easy. Personally prefer JSON for most things because it is easier to type :-)
    - untitaker_ 11y ago
      I said "easy parsing", not "easy implementation". "Computationally cheap" is probably a better term. Everything is trivial with a parser generator. This is besides the point.
- debacle 11y ago
  JSON is a much more simple specification than XML. If parsing XML is as fast as parsing JSON, you aren't parsing XML, just a subset of it.
  - olavk 11y ago
    Parsing speed does not necessarily correspond to the complexity of the spec. For example a lot of complexity in the XML spec is the DTD part. But if you parse a document without a DTD, this would presumably not have a performance cost.
    - debacle 11y ago
      There's a massive difference between what makes a valid XML document and what makes a valid JSON document.
- yarrel 11y ago
  XML will be slightly slower than parsing JSON simply because you have to scan more characters for each item.
- brongondwana 11y ago
  I guess most of the world has been doing something wrong for most of the time we've been working with XML then. (actually, for us at FastMail the major cost was always passing the bloody VCARDs themselves - I wound up writing https://github.com/brong/Text-VCardFast https://github.com/brong/Text-VCardFast)
anc84 11y ago
In a footnote XML is mentioned as being wasteful on bandwidth but doesn't it compress very well?
- untitaker_ 11y ago
  That's a result of it being bloated, not a feature. By that logic it doesn't matter how bloated dataformats are. EDIT: The big problem with WebDAV is that it's overall so complex that it's hard to implement, and XML doesn't help with that.
- zamalek 11y ago
  It compresses extremely well; even if you use a predetermined dictionary (reducing CPU usage). However, because of CPU usage considerations it probably won't ever do as well as high-entropy encodings such as Protobuf because to get truly great compression you simply have to spend the cycles. If you are using XML for the right applications (i.e. not a data firehose) the compression/CPU characteristics shouldn't matter at all. It's when you start using it for high bandwidth scenarios that things become sketchy: you should be negotiating an out-of-band stream (such as SI, Jingle or just a plain old socket) and using that for the firehose. That being said, people like AeroFS supposedly used it as a firehose for years[1] (in the form of XMPP) before having to replace it with a simpler protocol: so there does seem to be some elasticity to that assertion about not using it for a firehose. [1]: https://www.aerofs.com/blog/open-sourcing-the-stupid-simple-messaging-protocol/ https://www.aerofs.com/blog/open-sourcing-the-stupid-simple-...
untitaker_ 11y ago
Weirdly this post seems to have been submitted multiple times: https://hn.algolia.com/?query=webdav%20dies&sort=byPopularit.. https://hn.algolia.com/?query=webdav%20dies&sort=byPopularit.... I thought HN prevents this? It seems to be the same exact URL.
fkooman 11y ago
Full disclosure: I contributed to the remoteStorage specification [0]. Instead of (or in addition to) considering remoteStorage, one could also go back to "normal" WebDAV, if such a thing exists, and completely forget about the CalDAV/CardDAV servers. I'd say WebDAV is more mature, it has litmus[1] for testing implementations. remoteStorage has as far as I know only one production instance running at 5apps [2]. The benefits of using the RS protocol are mostly due to the CORS headers (which could be implemented easily for WebDAV) and the use of OAuth/Bearer, for which a PR exists for SabreDAV [3]. One thing missing from WebDAV is the (implicit) mapping of OAuth scopes to ACLs, which should not be too difficult to implement... The discovery, by depending on Webfinger, is also not one of my favorites. I'd prefer something like OAuth authorization server discovery [4]. I mean, I am not opposed to using remoteStorage and love JSON as much as the next guy, but it just doesn't bring (in my opinion) many benefits and loses interop with existing WebDAV clients for no good reason... [0] https://datatracker.ietf.org/doc/draft-dejong-remotestorage/ https://datatracker.ietf.org/doc/draft-dejong-remotestorage/ [1] http://www.webdav.org/neon/litmus/ http://www.webdav.org/neon/litmus/ [2] https://5apps.com/ https://5apps.com/ [3] https://github.com/fruux/sabre-dav/pull/710 https://github.com/fruux/sabre-dav/pull/710 [4] https://www.tuxed.net/fkooman/blog/as_discovery.html https://www.tuxed.net/fkooman/blog/as_discovery.html
- binbasti 11y ago
  (Co-founder of 5apps and RS core contributor here.) > remoteStorage has as far as I know only one production instance running at 5apps That's not true. 5apps is running the only public service for end users at the moment, but there are certainly more production instances running. > The benefits of using the RS protocol are mostly due to the CORS headers (which could be implemented easily for WebDAV) and the use of OAuth/Bearer, for which a PR exists for SabreDAV [3]. As both of these would be optional additions to WebDAV servers, all of WebDAV's benefits parish with most servers not supporting these new extensions. That's the very critique in the article as far as I understand. WebDAV alone is not good enough, and optional additions lead to a world of incompatibility and pain. > One thing missing from WebDAV is the (implicit) mapping of OAuth scopes to ACLs, which should not be too difficult to implement And another addition. > I'd prefer something like OAuth authorization server discovery And another one. Counting 4 now. :) > but it just doesn't bring (in my opinion) many benefits and loses interop with existing WebDAV clients for no good reason You just mentioned that to get to feature parity with remoteStorage, a WebDAV server needs 4 optional additions, for only one of which an unmerged PR to a single server implementation exists. Maybe I miss something, but it doesn't sound like interop is WebDAV's benefit in this scenario.
robmueller 11y ago
It's not just WebDAV that's an abomination (returning HTTP status codes and messages in XML, WTF were they thinking), but vCard and iCal as well. The vCard "group construct" (see rfc6350) is one of the dumbest things ever added to a spec. It seems a trivial thing to add, but completely screws up your internal storage and manipulation formats. It's horrible, on top of all the other horribleness that is vCard. Of course, I'm biased, given we're trying to push alternatives to IMAP, CardDAV and CalDAV - http://jmap.io/ http://jmap.io/ - but please do check it out. Having actually worked on IMAP servers and clients, CalDAV servers and clients and CardDAV servers and clients, we've learned a lot about creating a saner alternative.
- untitaker_ 11y ago
  >returning HTTP status codes and messages in XML, WTF were they thinking You're giving me so many ideas right now for a follow-up post. >but vCard and iCal as well The group construct is bad, the syntax is bad, but other than that, I never had to implement or parse it properly. Vdirsyncer is mostly passing strings through. I suspect that this part is much harder to replace than CardDAV and CalDAV, since it's not just a file structure with a ton of bullcrap on top of it. For that reason I currently have those file formats in the remoteStorage folder structure i'm syncing to, but it's still a step forward. I might switch to jCard and jCal, but I don't think it'd be worth the breakage.
- binbasti 11y ago
  I didn't notice that JMAP includes calendars and contacts. Looks good! Personally, I'd prefer a dumb storage with a generic API over yet another custom protocol/API just for these data formats alone. Worked great on people's local drives before the Web, and I think it should work similar on the Web.
  - untitaker_ 11y ago
    I guess you can dump a [Maildir](http://cr.yp.to/proto/maildir.html http://cr.yp.to/proto/maildir.html) (or at least a modified version) into a remoteStorage, but this is not really an acceptable protocol for mobile clients with limited space. As mentioned in the blogpost I have the same problems with storing calendars in remoteStorage.
rebelde 11y ago
What I loved about WebDAV is that it let me edit a file on a remote server using my personal PC. When I wanted to save to the remote computer, I just saved it. Is there a good replacement out there?
- untitaker_ 11y ago
  SFTP, or really any other file protocol. I guess it comes with its own quirks but at least it's faster. There used to be a remoteStorage FUSE module, not sure if it still works.
  - binbasti 11y ago
    It was never quite finished, as the protocol was missing some features at the time (content types in folder listings e.g.). Should be rather easy to implement, though. Here's the old repo: https://github.com/remotestorage/fuse https://github.com/remotestorage/fuse
- ikurei 11y ago
  What about samba/shared folders, ftp, or even sshfs if you are a unix user. If you like Emacs, there is Tramp, and I know Vim has something equivalent.
  - rebelde 11y ago
    I have tried a number of the ones that you and others mention. The security was awful. They would create the remote drive (or whatever) automatically, all the time. eek! I want to have to log in first. It has been a few years since I have tried. I hope something has improved. Again, basically, I would prefer to edit in Notepad++ or Sublime instead of vi. Personal preference.
- icebraining 11y ago
  Samba, NFS, sshfs, 9P, ElasticDrive, CurlFtpFS and others.
- mrcsparker 11y ago
0x0 11y ago
The two (yes, two!) implementations that Microsoft ship are also full of crazy bugs that come and go between versions. There's a list compiling known bugs here: http://www.greenbytes.de/tech/webdav/webdav-redirector-list.html http://www.greenbytes.de/tech/webdav/webdav-redirector-list.... and here: http://www.greenbytes.de/tech/webdav/webfolder-client-list.html http://www.greenbytes.de/tech/webdav/webfolder-client-list.h... It's almost like they sabotaged the clients on purpose to break interoperability.
- bazzargh 11y ago
  I worked on a document management system 10 years ago, we would loved to have used webdav to present a virtual filesystem from our (linux) servers to Office clients for document retrieval/conversion. It was just impossible. As you say, every version of office & windows had different, severe bugs that meant we couldn't get something as simple as browsing a directory, loading & saving files to work across all versions, and it wasn't like recent versions were better - it was completely unreliable. It turned out to be easier to write plugins for all versions of office than it was to rely on office's own code actually working.
- untitaker_ 11y ago
  I only have experiences with DavMail in front of an Outlook server. It actually works well, except when it doesn't: http://sourceforge.net/p/davmail/bugs/598/ http://sourceforge.net/p/davmail/bugs/598/ When it rains it pours.
- e12e 11y ago
  FWIW last time I looked, the sanest way I could find to actually access WebDAV from Microsoft Windows was with Cyberduck: https://cyberduck.io/ https://cyberduck.io/
  - 0x0 11y ago
    One of the big "sells" for WebDAV was transparent access like a network drive in Windows clients without the need to install additional software. As a power user, accessing WebDAV isn't much of an issue. But as, say, a software developer wanting to provide a WebDAV service, it's another story :-/
INTPenis 11y ago
For a self-hosted, wide area network file sharing solution I think WebDAV is a very simple and cheap choice. My experience with WebDAV has been just that, its simplest form. Setup an Apache web server for project managers to share files with a client for example.
- Touche 11y ago
  IIRC doesn't Apache require that the Apache user owns the files? This made it annoying when you wanted to use those files elsewhere on the server.
  - e12e 11y ago
    Well, there's just three basic options: have the WebDAV server run as root, to allow changing uids. It's probably a bad idea. Have a group that both the server and uid is member of, have that group own the files (better, but breaks down when you need more than one user, unless you have dav-uid1, dav-uid2 ... groups for every uidN, and have the server be member of each -- and even then you're not much better off -- the server can still access all files -- at least uid1 logged in locally won't have access to files owned by dav-uid2 directly). Finally, just run the server as your uid, on an alternate port. That's probably more sane, optionally have a proxy stand in so you can connect over 443. The state of dav clients is a mess, FWIW last I checked, getting it to work sanely on Linux, either with Gnome, or davfs -- was easy enough. Trying to get it to work reasonably from Windows was hopeless -- I never tried OS X -- I'd hoped that they actually had a usable client. Either way, if you basically end up just being able to access it reasonably from Linux, you might as well just go with sshfs. For a handful of users, setting up reasonably secure[1], key-based file sharing -- ssh(fs) is hard to beat. SAMBA/CIFS isn't fit to be used over the Internet, NFSv4/pNFS w/kerberos and encryption might be, but it, along with AFS is complex to set up. The second easiest thing with some hope of being both secure and not a nightmare to get working for most (after sftp/sshfs) is probably Samba/CIFS over a VPN. Not sure about the current state of the webdav-server/client stuff for python -- that might be a viable way to get a pair of multiplatform client/server things working. Last I looked the WebDAV part of Zope/Plone worked fine with davfs (sadly the rest of the thing, for editing content etc, needs/needed work). On paper WebDAV har many things going for it: it works over standard TLS ports, should go through most firewalls, is "trivial" to secure using certificates (not sure about 2fa -- would probably depend on clients having a sane UI/UX for "require certificate and OTP/password" etc). Does anyone have any experience with actually using something derived from Plan9 9p for sharing files? I've only been able to find broken, abandoned implementations, and no recent how-to's or tutorials. Again, on paper, it would seem that 9p wrapped in some kind of certificate based protocol (either built from nacl primitives, or simply TLS/SSH) should be a reasonable candidate. A nice combination of not reinventing the wheel, and keeping things as simple as possible (as opposed to, NFSv4, CIFS, AFS). [1] Ssh breaks down when it comes to revoking keys etc. This is in theory solved by using ssh certificates (which can expire, be revoked etc) -- but AFAIK there aren't any sane cli/ui/management tools yet. In that case managing x509 via AD or some other CA tool is probably easier -- but still ridiculously painful.
amelius 11y ago
I tried WebDAV a couple of years back on MacOS. Turned out that MacOS had a bug in its WebDAV implementation, causing files to be lost. Never looked at it since.
spiralpolitik 11y ago
If only there was a process whereby you could propose corrections to existing standards or even propose a new standard to replace the existing standard instead of whining about it on Hacker News...oh wait. Seriously if these things bother you then and you think you have a better idea write it up in an RFC.
- adrusi 11y ago
  Writing a good RFC is hard work, does it not make sense to write informally about your ideas first to see if they have any appeal before investing the time into proposing a new standard? Writing about it this way also exposes the problems to people who otherwise wouldn't give two fucks about webdav. If the author had simply submited an RFC, only the people working on calendar/contact syncing tech would take notice. I find it sad that you consider pointing out the flaws in something to be "whining".
  - spiralpolitik 11y ago
    Isn't that the point of the RFC process ? It's right there in the name "Request for Comment". The whole point of the RFC structure is that it forces you to lay out what someone would need to implement your RFC, which is what you need to even begin the discussion on a new standard. A lot of the successful early internet standards followed this path. They weren't written by committee, they were written by one person with an idea to make things better. And I only consider it whining if you point out the flaws, then don't do something about the flaws using the process for correcting them. The internet was built by the RFC process and its poorer these days because people feel to follow it is "a bit of hard work".
    - Anthony-G 11y ago
      The documents may be called “Requests for Comments” but in reality, most of the modern RFCs dealing with Internet protocols simply became de facto standards – until they were superseded by subsequent RFCs.
    - adrusi 11y ago
mkesper 11y ago
The hassle of syncing contacts, spam filters, calendar events, GPG keys (and emails) makes me think about replacing it with something like mailpile. Did anyone here try that?
zeveb 11y ago
XML really is the mindkiller (and the bandwidth-killer). Let's compare & contrast XML, S-expressions and JSON for the two examples in the article: <?xml version="1.0" encoding="utf-8" ?> <D:propfind xmlns:D="DAV:"> <D:prop> <D:getcontenttype/> <D:getetag/> </D:prop> </D:propfind> becomes: (propfind (prop getcontenttype getetag)) or: { "op": "propfind", "prop": ["getcontenttype", "getetag"] } And: <?xml version="1.0" encoding="utf-8" ?> <C:calendar-query xmlns:D="DAV:" xmlns:C="urn:ietf:params:xml:ns:caldav"> <D:prop> <D:getcontenttype/> <D:getetag/> </D:prop> <C:filter> <C:comp-filter name="VCALENDAR"> <C:comp-filter name="VEVENT"> <C:time-range start="20150909T000000Z" end="20300909T000000Z"/> </C:comp-filter> </C:comp-filter> </C:filter> </C:calendar-query> becomes: (calendar-query (prop getcontenttype getetag) (filter (component "VCALENDAR") (component "VEVENT" (time-range "2015-09-09T00:00:00Z" "2030-09-09T00:00:00Z")))) or: { "op": "calendar-query", "prop": ["getcontenttype", "getetag"], "filter": [ { "component": "VCALENDAR" }, { "component": "VEVENT", "filter": { "time-range": ["2015-09-09T00:00:00Z", "2030-09-09T00:00:00Z"] } } ] } I think it's pretty clear which is least readable and least concise. So, what does XML buy one in exchange for its lack of concision? Well, a pre-processing tool which knows nothing about WebDAV or CalDAV could use a schema definition to sanity-check the formatting of a query. It could probably be set up to detect if one used an invalid date format; it may or may not be configurable to detect if one searched for a VCALENDAR component inside a VEVENT. Regardless, there are semantic constraints in a data model which cannot be expressed with, e.g., a DTD or XSD. How about JSON? What does it buy? Well, it has built-in hash tables (as opposed to faking them with (table (key val) (key2 val2))), which is a huge win, and it has a ton of libraries in every language one can imagine. How about S-expressions? I think that they win on clarity, readability and concision. But there are not a ton of parsing libraries available for every language (OTOH, a canonical-S-expression parser can be written in a few hours for any language). So, who wins? I think that for most projects nowadays, JSON's clearly the answer. The pain of XML simply doesn't pay off in practice, while the elegance of S-expressions doesn't outweigh the unfamiliarity of the great mass of enterprise software developers. But if you have a choice, use S-expressions.
rsync 11y ago
We (rsync.net) supported webDAV from ... about 2006-2012. It was a total mess and headache the entire time - mostly because the standard implementations that you would attempt to use in a FOSS environment are complete abandonware. I am speaking of mod_dav on apache. It's abandonware. The original authors cannot be contacted, it lies stagnant ... it's broken. One of the original reasons that we supported webDAV was that the mac finder, with it's "Connect to Server" choice in the "Go" menu supported webDAV - but their webDAV was really, really quirky and non-standard and did not function well with anything. We had to reverse engineer Apples weirdness and even then it rarely worked well. MS DAV in office, etc. - also weird. And again, very difficult to even work around the weirdness because mod_dav was abandoned. Giving up on DAV was a win-win - not only did we stop wasting time on these bizarro home-grown dialects of DAV, but we also removed a ton of attack surface by removing apache entirely from rsync.net storage arrays. Nowadays it's just OpenSSH and I think it will stay that way. wistful: If only (if only!) Apple would support SFTP in the "connect to server" function of the Finder.
- jfb 11y ago
  You might be happy to know that WebDAV didn't work any better for those of us at Apple, either.
  - rsync 11y ago
    Do you have any insight into why SFTP support was never built into "connect to server" ? It seems so simple and obvious ... would have saved humanity millions of man hours with people mucking around with sshFS/FUSE on their macs, which barely worked back then ...
    - jfb 11y ago
      No insight whatsoever. Probably nobody ever asked for it, to be honest. "Connect to Server ..." was overwhelmingly used to connect via AFP or, shudder NFS at the time, both of which expose a filesystem like interface. How much like a filesystem is a SFTP connection, as opposed to a FTP-like one? That's probably the reason, right there: WebDAV purported to expose filesystem semantics, which meant that in theory adding it was simply defining a communications protocol to the Finder, rather than a complete translation layer between filesystem operations and FTP ones.
hobarrera 11y ago
JMAP[1] seems to be attempting to replace IMAP + CalDAV + CardDAV. Now that we have a replacement, I'd love to see these die. Especially the latter two. We're still missing a replacement for WebDAV though. :( [1]: http://jmap.io/ http://jmap.io/
- treve 11y ago
  The problem with Jmap is that it doesn't work well with the rest of the world. Unless your only use-case is personal sync, this can be problem. It's possible to ditch WebDAV, but also breaking compatibility with iCalendar and iTip makes this a no-go for most people. If they considered using jCal/jCard, compatiblity would be possible and then it actually has a chance. In fact, they could even ignore jCal/jCard and pick a format that can express the same data-model as iCalendar/vCard, and then they would still have my vote. The way it's currently developed, is that it's an entirely new format that maps to iCalendar in a lossy way. Any iCalendar/iTip extension that is currently exists or that is under development would have no way to be expressed in Jmap. I think the authors of Jmap have a really great understanding of Imap and what it would take to replace it, but their use-case of what they need from calendaring is narrow and their approach to replacing CalDAV is naive. It's basically ActiveSync v2. Just like ActiveSync it uses HTTP as a tunnel with a single endpoint and an RPC-like system using POST, except ActiveSync uses WBXML, and Jmap uses Json.
  - untitaker_ 11y ago
    I don't know. Compatibility with jCal and jCard would've been nice, yes, but I don't think the kind of properties they support in their datamodel differ too much from what the VCard and iCalendar support. They also have a proxy server that proxies client => JMAP => DAV, so if that works without a massive translation layer, it'll probably be fine. EDIT: Just saw that you edited your post to address those points. In that case I'd like to see a list of things that are required from a -DAV successor. JMAP's syntax strikes me as quirky, I agree, but the datamodel and the amount of methods provided seem sensible to me.
    - treve 11y ago
graniter 11y ago
After writing a custom CalDAV server I agree with the author. The number of requests it takes to get the data you need is insane, and most requests will involve a DB hit. My CalDAV server is much more "chatty" than my web server even though they serve essentially the same information. Plus each CalDAV client does things a little differently so it's very hard to debug, especially when most clients are black boxes. Permissions and feature and how they respond...a potentially different. I (unfortunately) consider the custom CalDAV server I wrote to be a bit a technical achievement, even though I'd rather I didn't have to do it. But I had to do it because there's really no other good option for writing calendar services that work with native calendar applications.
- untitaker_ 11y ago
  I bet it's easier to use SabreDAV (http://sabre.io/dav/ http://sabre.io/dav/) for your server than writing it all by yourself, even though it means writing PHP. I really feel like its author is the only one who understands what the hell is going on.
  - brongondwana 11y ago
    Ken Murchison from CMU has a pretty good grasp on it these days too, having written the Cal/CardDAV support for the Cyrus IMAP server. I'm slowly catching up, having rewritten parts of it :)
dragonwriter 11y ago
The big problem I have with WebDAV is that the specs for very pieces all do kind of "vertical" slices of functionality without separation of concerns between low-level protocol (what does HTTP provide and what is it missing, and how do we generically extend it to provide what is necessary for the applications we are trying to do) and content semantics, so that where it extended the HTTP protocol (with new methods, etc.), those extensions were tightly bound to content semantics. I think rethinking the same problem domain with a more RESTful approach, a lot better could be done, with a lot simpler set of generic protocol extensions covering the few real gaps in HTTP/1.1 and a simple set of metadata content types [0] to cover authoring/versioning information, with primary content just using any content type. [0] And, really, this is two problems that should be addressed somewhat separately, in terms of representation-neutral metadata content models first, and then in terms of various representations for those content models.
hobarrera 11y ago
> In practice, you can't use a WebDAV client library. In practice, you copy-paste XML from the examples in the RFC into your source code and hope for the best. I've written CalDav clients. This is completely true. Being both an extension and modification of WebDAV, CalDAV cannot be used with common WebDAV libraries.

I hope WebDAV dies

102 points by kristianp 11y ago

Tharkun 11y ago

OP seems to think that parsing XML is slow. If parsing XML is slower than parsing JSON, you're doing something wrong.

untitaker_ 11y ago
Parsing was never the bottleneck, I admit. But I can't see how parsing XML is as easy a task as parsing JSON either? Of course you can show me some benchmarks that show that libxml2 is much faster than the avg. JSON implementation. But you can do that with any format that is verbose, complex and therefore hard to parse, yet has been around for so long that it paid off to write well-optimized C libraries for it.
- Tharkun 11y ago
  Exactly. You can point to benchmarks that favour one or the other. Parsing is almost never the bottleneck.
  - untitaker_ 11y ago
    The point I'm trying to make is that JSON is generally easier to parse, therefore theoretically faster to parse, even though XML implementations might be faster at the moment.
    - tveita 11y ago
      A theoretical microsecond difference that doesn't even exist in practice doesn't seem worth mentioning when there is network traffic involved. The article says: "Yes, it's slow to fetch all events, but so is parsing XML. " That's several orders of magnitude difference glossed over.
      - anentropic 11y ago
        of course, being verbose, XML will be slower to fetch across the network as well
      - untitaker_ 11y ago
        Meh, I have to agree that it isn't a good point. I guess I was merely trying to point out that both have its performance deficiencies. At least the ones of remoteStorage are avoidable, while you can't just avoid XML in WebDAV.
- mikekchar 11y ago
  I recommend that you simply write a JSON parser and an XML parser using a parser generator. Then you will see. They are both trivial. I would be surprised if it took you more than an afternoon to do both. Ease of parsing was the whole point of XML. It was actually a beautiful concept that because it was a subset of SGML, you could validate documents and build smart editors, etc, etc on file formats in XML. Then when you wanted to simply read the file, the parser was mind blowingly simple to write. Of course people decided that it would be great for communications protocols, and inter-language object representations and... all the stuff that it really wasn't great for. And people decided that you would want to write event driven parsers and do DOM traversals and crazy things like that. Yeah, that stuff is complicated. (and dare I offend people by saying that it wasn't really a good idea after all). But parsing is dead easy. Personally prefer JSON for most things because it is easier to type :-)
  - untitaker_ 11y ago
    I said "easy parsing", not "easy implementation". "Computationally cheap" is probably a better term. Everything is trivial with a parser generator. This is besides the point.
debacle 11y ago
JSON is a much more simple specification than XML. If parsing XML is as fast as parsing JSON, you aren't parsing XML, just a subset of it.
- olavk 11y ago
  Parsing speed does not necessarily correspond to the complexity of the spec. For example a lot of complexity in the XML spec is the DTD part. But if you parse a document without a DTD, this would presumably not have a performance cost.
  - debacle 11y ago
    There's a massive difference between what makes a valid XML document and what makes a valid JSON document.
yarrel 11y ago
XML will be slightly slower than parsing JSON simply because you have to scan more characters for each item.
brongondwana 11y ago
I guess most of the world has been doing something wrong for most of the time we've been working with XML then. (actually, for us at FastMail the major cost was always passing the bloody VCARDs themselves - I wound up writing https://github.com/brong/Text-VCardFast https://github.com/brong/Text-VCardFast)

anc84 11y ago

In a footnote XML is mentioned as being wasteful on bandwidth but doesn't it compress very well?

untitaker_ 11y ago
That's a result of it being bloated, not a feature. By that logic it doesn't matter how bloated dataformats are. EDIT: The big problem with WebDAV is that it's overall so complex that it's hard to implement, and XML doesn't help with that.
zamalek 11y ago
It compresses extremely well; even if you use a predetermined dictionary (reducing CPU usage). However, because of CPU usage considerations it probably won't ever do as well as high-entropy encodings such as Protobuf because to get truly great compression you simply have to spend the cycles. If you are using XML for the right applications (i.e. not a data firehose) the compression/CPU characteristics shouldn't matter at all. It's when you start using it for high bandwidth scenarios that things become sketchy: you should be negotiating an out-of-band stream (such as SI, Jingle or just a plain old socket) and using that for the firehose. That being said, people like AeroFS supposedly used it as a firehose for years[1] (in the form of XMPP) before having to replace it with a simpler protocol: so there does seem to be some elasticity to that assertion about not using it for a firehose. [1]: https://www.aerofs.com/blog/open-sourcing-the-stupid-simple-messaging-protocol/ https://www.aerofs.com/blog/open-sourcing-the-stupid-simple-...

untitaker_ 11y ago

Weirdly this post seems to have been submitted multiple times: https://hn.algolia.com/?query=webdav%20dies&sort=byPopularit.. https://hn.algolia.com/?query=webdav%20dies&sort=byPopularit.... I thought HN prevents this? It seems to be the same exact URL.

fkooman 11y ago

Full disclosure: I contributed to the remoteStorage specification [0]. Instead of (or in addition to) considering remoteStorage, one could also go back to "normal" WebDAV, if such a thing exists, and completely forget about the CalDAV/CardDAV servers. I'd say WebDAV is more mature, it has litmus[1] for testing implementations. remoteStorage has as far as I know only one production instance running at 5apps [2]. The benefits of using the RS protocol are mostly due to the CORS headers (which could be implemented easily for WebDAV) and the use of OAuth/Bearer, for which a PR exists for SabreDAV [3]. One thing missing from WebDAV is the (implicit) mapping of OAuth scopes to ACLs, which should not be too difficult to implement... The discovery, by depending on Webfinger, is also not one of my favorites. I'd prefer something like OAuth authorization server discovery [4]. I mean, I am not opposed to using remoteStorage and love JSON as much as the next guy, but it just doesn't bring (in my opinion) many benefits and loses interop with existing WebDAV clients for no good reason... [0] https://datatracker.ietf.org/doc/draft-dejong-remotestorage/ https://datatracker.ietf.org/doc/draft-dejong-remotestorage/ [1] http://www.webdav.org/neon/litmus/ http://www.webdav.org/neon/litmus/ [2] https://5apps.com/ https://5apps.com/ [3] https://github.com/fruux/sabre-dav/pull/710 https://github.com/fruux/sabre-dav/pull/710 [4] https://www.tuxed.net/fkooman/blog/as_discovery.html https://www.tuxed.net/fkooman/blog/as_discovery.html

binbasti 11y ago
(Co-founder of 5apps and RS core contributor here.) > remoteStorage has as far as I know only one production instance running at 5apps That's not true. 5apps is running the only public service for end users at the moment, but there are certainly more production instances running. > The benefits of using the RS protocol are mostly due to the CORS headers (which could be implemented easily for WebDAV) and the use of OAuth/Bearer, for which a PR exists for SabreDAV [3]. As both of these would be optional additions to WebDAV servers, all of WebDAV's benefits parish with most servers not supporting these new extensions. That's the very critique in the article as far as I understand. WebDAV alone is not good enough, and optional additions lead to a world of incompatibility and pain. > One thing missing from WebDAV is the (implicit) mapping of OAuth scopes to ACLs, which should not be too difficult to implement And another addition. > I'd prefer something like OAuth authorization server discovery And another one. Counting 4 now. :) > but it just doesn't bring (in my opinion) many benefits and loses interop with existing WebDAV clients for no good reason You just mentioned that to get to feature parity with remoteStorage, a WebDAV server needs 4 optional additions, for only one of which an unmerged PR to a single server implementation exists. Maybe I miss something, but it doesn't sound like interop is WebDAV's benefit in this scenario.

robmueller 11y ago

It's not just WebDAV that's an abomination (returning HTTP status codes and messages in XML, WTF were they thinking), but vCard and iCal as well. The vCard "group construct" (see rfc6350) is one of the dumbest things ever added to a spec. It seems a trivial thing to add, but completely screws up your internal storage and manipulation formats. It's horrible, on top of all the other horribleness that is vCard. Of course, I'm biased, given we're trying to push alternatives to IMAP, CardDAV and CalDAV - http://jmap.io/ http://jmap.io/ - but please do check it out. Having actually worked on IMAP servers and clients, CalDAV servers and clients and CardDAV servers and clients, we've learned a lot about creating a saner alternative.

untitaker_ 11y ago
>returning HTTP status codes and messages in XML, WTF were they thinking You're giving me so many ideas right now for a follow-up post. >but vCard and iCal as well The group construct is bad, the syntax is bad, but other than that, I never had to implement or parse it properly. Vdirsyncer is mostly passing strings through. I suspect that this part is much harder to replace than CardDAV and CalDAV, since it's not just a file structure with a ton of bullcrap on top of it. For that reason I currently have those file formats in the remoteStorage folder structure i'm syncing to, but it's still a step forward. I might switch to jCard and jCal, but I don't think it'd be worth the breakage.
binbasti 11y ago
I didn't notice that JMAP includes calendars and contacts. Looks good! Personally, I'd prefer a dumb storage with a generic API over yet another custom protocol/API just for these data formats alone. Worked great on people's local drives before the Web, and I think it should work similar on the Web.
- untitaker_ 11y ago
  I guess you can dump a [Maildir](http://cr.yp.to/proto/maildir.html http://cr.yp.to/proto/maildir.html) (or at least a modified version) into a remoteStorage, but this is not really an acceptable protocol for mobile clients with limited space. As mentioned in the blogpost I have the same problems with storing calendars in remoteStorage.

rebelde 11y ago

What I loved about WebDAV is that it let me edit a file on a remote server using my personal PC. When I wanted to save to the remote computer, I just saved it. Is there a good replacement out there?

untitaker_ 11y ago
SFTP, or really any other file protocol. I guess it comes with its own quirks but at least it's faster. There used to be a remoteStorage FUSE module, not sure if it still works.
- binbasti 11y ago
  It was never quite finished, as the protocol was missing some features at the time (content types in folder listings e.g.). Should be rather easy to implement, though. Here's the old repo: https://github.com/remotestorage/fuse https://github.com/remotestorage/fuse
ikurei 11y ago
What about samba/shared folders, ftp, or even sshfs if you are a unix user. If you like Emacs, there is Tramp, and I know Vim has something equivalent.
- rebelde 11y ago
  I have tried a number of the ones that you and others mention. The security was awful. They would create the remote drive (or whatever) automatically, all the time. eek! I want to have to log in first. It has been a few years since I have tried. I hope something has improved. Again, basically, I would prefer to edit in Notepad++ or Sublime instead of vi. Personal preference.
icebraining 11y ago
Samba, NFS, sshfs, 9P, ElasticDrive, CurlFtpFS and others.
mrcsparker 11y ago

0x0 11y ago

The two (yes, two!) implementations that Microsoft ship are also full of crazy bugs that come and go between versions. There's a list compiling known bugs here: http://www.greenbytes.de/tech/webdav/webdav-redirector-list.html http://www.greenbytes.de/tech/webdav/webdav-redirector-list.... and here: http://www.greenbytes.de/tech/webdav/webfolder-client-list.html http://www.greenbytes.de/tech/webdav/webfolder-client-list.h... It's almost like they sabotaged the clients on purpose to break interoperability.

bazzargh 11y ago
I worked on a document management system 10 years ago, we would loved to have used webdav to present a virtual filesystem from our (linux) servers to Office clients for document retrieval/conversion. It was just impossible. As you say, every version of office & windows had different, severe bugs that meant we couldn't get something as simple as browsing a directory, loading & saving files to work across all versions, and it wasn't like recent versions were better - it was completely unreliable. It turned out to be easier to write plugins for all versions of office than it was to rely on office's own code actually working.
untitaker_ 11y ago
I only have experiences with DavMail in front of an Outlook server. It actually works well, except when it doesn't: http://sourceforge.net/p/davmail/bugs/598/ http://sourceforge.net/p/davmail/bugs/598/ When it rains it pours.
e12e 11y ago
FWIW last time I looked, the sanest way I could find to actually access WebDAV from Microsoft Windows was with Cyberduck: https://cyberduck.io/ https://cyberduck.io/
- 0x0 11y ago
  One of the big "sells" for WebDAV was transparent access like a network drive in Windows clients without the need to install additional software. As a power user, accessing WebDAV isn't much of an issue. But as, say, a software developer wanting to provide a WebDAV service, it's another story :-/

INTPenis 11y ago

For a self-hosted, wide area network file sharing solution I think WebDAV is a very simple and cheap choice. My experience with WebDAV has been just that, its simplest form. Setup an Apache web server for project managers to share files with a client for example.

Touche 11y ago
IIRC doesn't Apache require that the Apache user owns the files? This made it annoying when you wanted to use those files elsewhere on the server.
- e12e 11y ago
  Well, there's just three basic options: have the WebDAV server run as root, to allow changing uids. It's probably a bad idea. Have a group that both the server and uid is member of, have that group own the files (better, but breaks down when you need more than one user, unless you have dav-uid1, dav-uid2 ... groups for every uidN, and have the server be member of each -- and even then you're not much better off -- the server can still access all files -- at least uid1 logged in locally won't have access to files owned by dav-uid2 directly). Finally, just run the server as your uid, on an alternate port. That's probably more sane, optionally have a proxy stand in so you can connect over 443. The state of dav clients is a mess, FWIW last I checked, getting it to work sanely on Linux, either with Gnome, or davfs -- was easy enough. Trying to get it to work reasonably from Windows was hopeless -- I never tried OS X -- I'd hoped that they actually had a usable client. Either way, if you basically end up just being able to access it reasonably from Linux, you might as well just go with sshfs. For a handful of users, setting up reasonably secure[1], key-based file sharing -- ssh(fs) is hard to beat. SAMBA/CIFS isn't fit to be used over the Internet, NFSv4/pNFS w/kerberos and encryption might be, but it, along with AFS is complex to set up. The second easiest thing with some hope of being both secure and not a nightmare to get working for most (after sftp/sshfs) is probably Samba/CIFS over a VPN. Not sure about the current state of the webdav-server/client stuff for python -- that might be a viable way to get a pair of multiplatform client/server things working. Last I looked the WebDAV part of Zope/Plone worked fine with davfs (sadly the rest of the thing, for editing content etc, needs/needed work). On paper WebDAV har many things going for it: it works over standard TLS ports, should go through most firewalls, is "trivial" to secure using certificates (not sure about 2fa -- would probably depend on clients having a sane UI/UX for "require certificate and OTP/password" etc). Does anyone have any experience with actually using something derived from Plan9 9p for sharing files? I've only been able to find broken, abandoned implementations, and no recent how-to's or tutorials. Again, on paper, it would seem that 9p wrapped in some kind of certificate based protocol (either built from nacl primitives, or simply TLS/SSH) should be a reasonable candidate. A nice combination of not reinventing the wheel, and keeping things as simple as possible (as opposed to, NFSv4, CIFS, AFS). [1] Ssh breaks down when it comes to revoking keys etc. This is in theory solved by using ssh certificates (which can expire, be revoked etc) -- but AFAIK there aren't any sane cli/ui/management tools yet. In that case managing x509 via AD or some other CA tool is probably easier -- but still ridiculously painful.

amelius 11y ago

I tried WebDAV a couple of years back on MacOS. Turned out that MacOS had a bug in its WebDAV implementation, causing files to be lost. Never looked at it since.

spiralpolitik 11y ago

If only there was a process whereby you could propose corrections to existing standards or even propose a new standard to replace the existing standard instead of whining about it on Hacker News...oh wait. Seriously if these things bother you then and you think you have a better idea write it up in an RFC.

adrusi 11y ago
Writing a good RFC is hard work, does it not make sense to write informally about your ideas first to see if they have any appeal before investing the time into proposing a new standard? Writing about it this way also exposes the problems to people who otherwise wouldn't give two fucks about webdav. If the author had simply submited an RFC, only the people working on calendar/contact syncing tech would take notice. I find it sad that you consider pointing out the flaws in something to be "whining".
- spiralpolitik 11y ago
  Isn't that the point of the RFC process ? It's right there in the name "Request for Comment". The whole point of the RFC structure is that it forces you to lay out what someone would need to implement your RFC, which is what you need to even begin the discussion on a new standard. A lot of the successful early internet standards followed this path. They weren't written by committee, they were written by one person with an idea to make things better. And I only consider it whining if you point out the flaws, then don't do something about the flaws using the process for correcting them. The internet was built by the RFC process and its poorer these days because people feel to follow it is "a bit of hard work".
  - Anthony-G 11y ago
    The documents may be called “Requests for Comments” but in reality, most of the modern RFCs dealing with Internet protocols simply became de facto standards – until they were superseded by subsequent RFCs.
  - adrusi 11y ago

mkesper 11y ago

The hassle of syncing contacts, spam filters, calendar events, GPG keys (and emails) makes me think about replacing it with something like mailpile. Did anyone here try that?

zeveb 11y ago

XML really is the mindkiller (and the bandwidth-killer). Let's compare & contrast XML, S-expressions and JSON for the two examples in the article: <?xml version="1.0" encoding="utf-8" ?> <D:propfind xmlns:D="DAV:"> <D:prop> <D:getcontenttype/> <D:getetag/> </D:prop> </D:propfind> becomes: (propfind (prop getcontenttype getetag)) or: { "op": "propfind", "prop": ["getcontenttype", "getetag"] } And: <?xml version="1.0" encoding="utf-8" ?> <C:calendar-query xmlns:D="DAV:" xmlns:C="urn:ietf:params:xml:ns:caldav"> <D:prop> <D:getcontenttype/> <D:getetag/> </D:prop> <C:filter> <C:comp-filter name="VCALENDAR"> <C:comp-filter name="VEVENT"> <C:time-range start="20150909T000000Z" end="20300909T000000Z"/> </C:comp-filter> </C:comp-filter> </C:filter> </C:calendar-query> becomes: (calendar-query (prop getcontenttype getetag) (filter (component "VCALENDAR") (component "VEVENT" (time-range "2015-09-09T00:00:00Z" "2030-09-09T00:00:00Z")))) or: { "op": "calendar-query", "prop": ["getcontenttype", "getetag"], "filter": [ { "component": "VCALENDAR" }, { "component": "VEVENT", "filter": { "time-range": ["2015-09-09T00:00:00Z", "2030-09-09T00:00:00Z"] } } ] } I think it's pretty clear which is least readable and least concise. So, what does XML buy one in exchange for its lack of concision? Well, a pre-processing tool which knows nothing about WebDAV or CalDAV could use a schema definition to sanity-check the formatting of a query. It could probably be set up to detect if one used an invalid date format; it may or may not be configurable to detect if one searched for a VCALENDAR component inside a VEVENT. Regardless, there are semantic constraints in a data model which cannot be expressed with, e.g., a DTD or XSD. How about JSON? What does it buy? Well, it has built-in hash tables (as opposed to faking them with (table (key val) (key2 val2))), which is a huge win, and it has a ton of libraries in every language one can imagine. How about S-expressions? I think that they win on clarity, readability and concision. But there are not a ton of parsing libraries available for every language (OTOH, a canonical-S-expression parser can be written in a few hours for any language). So, who wins? I think that for most projects nowadays, JSON's clearly the answer. The pain of XML simply doesn't pay off in practice, while the elegance of S-expressions doesn't outweigh the unfamiliarity of the great mass of enterprise software developers. But if you have a choice, use S-expressions.

rsync 11y ago

We (rsync.net) supported webDAV from ... about 2006-2012. It was a total mess and headache the entire time - mostly because the standard implementations that you would attempt to use in a FOSS environment are complete abandonware. I am speaking of mod_dav on apache. It's abandonware. The original authors cannot be contacted, it lies stagnant ... it's broken. One of the original reasons that we supported webDAV was that the mac finder, with it's "Connect to Server" choice in the "Go" menu supported webDAV - but their webDAV was really, really quirky and non-standard and did not function well with anything. We had to reverse engineer Apples weirdness and even then it rarely worked well. MS DAV in office, etc. - also weird. And again, very difficult to even work around the weirdness because mod_dav was abandoned. Giving up on DAV was a win-win - not only did we stop wasting time on these bizarro home-grown dialects of DAV, but we also removed a ton of attack surface by removing apache entirely from rsync.net storage arrays. Nowadays it's just OpenSSH and I think it will stay that way. wistful: If only (if only!) Apple would support SFTP in the "connect to server" function of the Finder.

jfb 11y ago
You might be happy to know that WebDAV didn't work any better for those of us at Apple, either.
- rsync 11y ago
  Do you have any insight into why SFTP support was never built into "connect to server" ? It seems so simple and obvious ... would have saved humanity millions of man hours with people mucking around with sshFS/FUSE on their macs, which barely worked back then ...
  - jfb 11y ago
    No insight whatsoever. Probably nobody ever asked for it, to be honest. "Connect to Server ..." was overwhelmingly used to connect via AFP or, shudder NFS at the time, both of which expose a filesystem like interface. How much like a filesystem is a SFTP connection, as opposed to a FTP-like one? That's probably the reason, right there: WebDAV purported to expose filesystem semantics, which meant that in theory adding it was simply defining a communications protocol to the Finder, rather than a complete translation layer between filesystem operations and FTP ones.

hobarrera 11y ago

JMAP[1] seems to be attempting to replace IMAP + CalDAV + CardDAV. Now that we have a replacement, I'd love to see these die. Especially the latter two. We're still missing a replacement for WebDAV though. :( [1]: http://jmap.io/ http://jmap.io/

treve 11y ago
The problem with Jmap is that it doesn't work well with the rest of the world. Unless your only use-case is personal sync, this can be problem. It's possible to ditch WebDAV, but also breaking compatibility with iCalendar and iTip makes this a no-go for most people. If they considered using jCal/jCard, compatiblity would be possible and then it actually has a chance. In fact, they could even ignore jCal/jCard and pick a format that can express the same data-model as iCalendar/vCard, and then they would still have my vote. The way it's currently developed, is that it's an entirely new format that maps to iCalendar in a lossy way. Any iCalendar/iTip extension that is currently exists or that is under development would have no way to be expressed in Jmap. I think the authors of Jmap have a really great understanding of Imap and what it would take to replace it, but their use-case of what they need from calendaring is narrow and their approach to replacing CalDAV is naive. It's basically ActiveSync v2. Just like ActiveSync it uses HTTP as a tunnel with a single endpoint and an RPC-like system using POST, except ActiveSync uses WBXML, and Jmap uses Json.
- untitaker_ 11y ago
  I don't know. Compatibility with jCal and jCard would've been nice, yes, but I don't think the kind of properties they support in their datamodel differ too much from what the VCard and iCalendar support. They also have a proxy server that proxies client => JMAP => DAV, so if that works without a massive translation layer, it'll probably be fine. EDIT: Just saw that you edited your post to address those points. In that case I'd like to see a list of things that are required from a -DAV successor. JMAP's syntax strikes me as quirky, I agree, but the datamodel and the amount of methods provided seem sensible to me.
  - treve 11y ago

graniter 11y ago

After writing a custom CalDAV server I agree with the author. The number of requests it takes to get the data you need is insane, and most requests will involve a DB hit. My CalDAV server is much more "chatty" than my web server even though they serve essentially the same information. Plus each CalDAV client does things a little differently so it's very hard to debug, especially when most clients are black boxes. Permissions and feature and how they respond...a potentially different. I (unfortunately) consider the custom CalDAV server I wrote to be a bit a technical achievement, even though I'd rather I didn't have to do it. But I had to do it because there's really no other good option for writing calendar services that work with native calendar applications.

untitaker_ 11y ago
I bet it's easier to use SabreDAV (http://sabre.io/dav/ http://sabre.io/dav/) for your server than writing it all by yourself, even though it means writing PHP. I really feel like its author is the only one who understands what the hell is going on.
- brongondwana 11y ago
  Ken Murchison from CMU has a pretty good grasp on it these days too, having written the Cal/CardDAV support for the Cyrus IMAP server. I'm slowly catching up, having rewritten parts of it :)

dragonwriter 11y ago

The big problem I have with WebDAV is that the specs for very pieces all do kind of "vertical" slices of functionality without separation of concerns between low-level protocol (what does HTTP provide and what is it missing, and how do we generically extend it to provide what is necessary for the applications we are trying to do) and content semantics, so that where it extended the HTTP protocol (with new methods, etc.), those extensions were tightly bound to content semantics. I think rethinking the same problem domain with a more RESTful approach, a lot better could be done, with a lot simpler set of generic protocol extensions covering the few real gaps in HTTP/1.1 and a simple set of metadata content types [0] to cover authoring/versioning information, with primary content just using any content type. [0] And, really, this is two problems that should be addressed somewhat separately, in terms of representation-neutral metadata content models first, and then in terms of various representations for those content models.

hobarrera 11y ago

> In practice, you can't use a WebDAV client library. In practice, you copy-paste XML from the examples in the RFC into your source code and hope for the best. I've written CalDav clients. This is completely true. Being both an extension and modification of WebDAV, CalDAV cannot be used with common WebDAV libraries.