3 ms·

Two lines from a song I and rjbs wrote but never quite published (set to the tune of Major-General’s song): > Its JSON-centred roots and HTTP are conventional,

Two lines from a song I and rjbs wrote but never quite published (set to the tune of Major-General’s song): > Its JSON-centred roots and HTTP are conventional, > so special parsers are not needed: this is quite intentional. SMTP and IMAP are a pain and problematic to deal with for various reasons. Building on popular web tech primitives makes everyone’s lives much easier, even if you’re not making a webmail client.

kazinator 3y ago
SMTP and IMAP are a solved problem. They are problematic for only one kind of entity: someone who has delusions of becoming the next gmail. Everyone else can just set up an open source SMTP and IMAP setup using existing programs that work, without writing a single line of parsing code. I don't trust this "open standard" bullshit. When the only thing running JMAP is Fastmail, it's a Fastmail-specific protocol. If you want developers to interface with it, you can't hide it: you have to document what the requests and responses look like. The "open standard" rhetoric has good optics; even more so with the IETF rubber stamp. The underlying idea behind JMAP is that the future consists of only large mail operators who somehow forward mail among themselves. Routing to SMTP servers is gone, and so is self-hosting, mostly. If you want your own mail domain, you can do that, but to operate it, your instance must connect over some web protocol to an e-mail monopolist. The IETF may have signed off on this, but I look forward to what the EFF have to say.
- chrismorgan 3y ago
  Speaking as one who has drunk the JMAP kool-aid (I worked for Fastmail for a few years): you sound like you’ve never tried to write an email client, or done much with email in general. Because the conclusions you’re coming to are completely back-to-front. Suppose you want to write a client that can do email sending (SMTP), email reading (IMAP), contact book (CardDAV) and calendar (CalDAV) interactions. SMTP/IMAP autodiscovery is limited at best, and CardDAV/CalDAV autodiscovery nonexistent, I think, so new users will have to find the details for several things, which is a big hurdle—or else you’ll have to recognise and special-case individual providers, which basically means Google and Microsoft get a massive unfair advantage. By contrast, JMAP handles autodiscovery of all this: type in an email address, and it finds the JMAP endpoint, which handles all of the stuff. (I will admit that this is currently more aspirational than real: to begin with it requires a DNS SRV record and/or a /.well-known/jmap HTTP response; and the means of authentication is deliberately unspecified at this stage, so we kinda need to wait around to see what providers settle on for authentication, with some kind of OAuth arrangement probably most likely.) And then perhaps your client is webmail; if so, you simply can’t use IMAP or SMTP. I guess you could try talking them over WebSocket or something like that (still requiring a proxy of some kind, even if a pass-through one), but that’d be painful; instead, everyone rolls their own thing, and so there’s never any webmail cross-compatibility. In JMAP, by contrast, the only thing you can’t do from a browser is looking up DNS SRV records. Everything else works, so that you can run your own webmail client that talks directly to the origin server. This allows you to disentangle the provision of service and software, which is very desirable for decentralisation. Well, now you want your client to produce notifications when new messages come in. If you’re writing for a desktop platform, fine: you can keep your IMAP connection open. But if you’re on a mobile platform, you probably can’t do that, and certainly can’t reliably, so… you’re stuck needing some kind of proxy and server component again. No good. But with JMAP, you set up a push subscription and it works just like any other app, without needing to keep a connection open. So there are three simple and concrete advantages of JMAP over the legacy protocols. JMAP actively bolsters decentralisation, in email providers, in domain names, in contact book and calendar support, in webmail, and in mobile notifications. There are other functional advantages too. JMAP enables some moderately complex batching that makes some sorts of realistic operations vastly faster than IMAP due to being able to skip many round trips. And JMAP’s pleasantness to work with directly should not be understated: no one wants to work with SMTP or IMAP directly because they’re painful protocols, and so of course all of that stuff tends to get extracted into a single library that protects you from them; by contrast, working in raw JMAP is pleasant and easy, which can genuinely open you up to doing things that would have been too painful to do through your IMAP library. And if you use JMAP, your mail client can skip worrying about MIME parsing (which is landmine-filled territory) if you choose. You can write a useful and complex standalone email workflow with nothing more than HTTP and JSON libraries. > The underlying idea behind JMAP is that the future consists of only large mail operators who somehow forward mail among themselves. Routing to SMTP servers is gone, and so is self-hosting, mostly. This is all completely wrong: JMAP is for client–server communications and doesn’t replace server–server communications, which still use SMTP. JMAP does not prop up email monopolists. > If you want developers to interface with it, you can't hide it: you have to document what the requests and responses look like. I have no idea what this is about. The spec shows examples. > The "open standard" rhetoric has good optics; even more so with the IETF rubber stamp. There was no IETF rubber stamp (which I take you to be using as a pejorative). There was collaborative process with people from a variety of companies who took what Fastmail had designed for their own use and wanted to share, and made it considerably better.
  - kazinator 3y ago
    I've worked in the mail area, though not recently. Twenty years ago I worked on an IMAP4 transport, optimized for wireless, for Windows CE ("Microsoft Mobile OS" or whatever it was called). I don't find your arguments rationally convincing. You're comparing developing mail software using libraries for JSON, HTTPS and other pieces, versus doing the same thing while developing IMAP from scratch. It boils down to "web developers don't have nice libraries for IMAP access and are too busy making another clone of React or Typescript to make one". It's not realistic to develop a new mail client without IMAP (complete non-starter) even if it suports JMAP. This shit only benefits people who are trying to be the next Gmail. On the surface, it's somewhat good for users who rely on monolithic mail service in that if there are more such services, they have more choice. It's bad for everyone is not using a monolithic mail service, who doesn't want a new mail client speaking a new protocol or to install services for that or anything else. > And then perhaps your client is webmail; if so, you simply can’t use IMAP or SMTP. That claim is amazing; in another browser tab right here I'm logged into my self-hosted RoundCube webmail which is accessing my inbox with IMAP, and sends with SMTP. Of course, it does those things at the PHP back end. I think you may be conflating "webmail" with a "browser-based, purely local mail client application not accessed over the network". If such a thing is not able to speak SMTP or IMAP, maybe the Javscript world should tool up?
    - chrismorgan 3y ago
      > You're comparing developing mail software using libraries for JSON, HTTPS and other pieces, versus doing the same thing while developing IMAP from scratch. This is nonsense. Most of my comment was describing protocol-level advantages of JMAP over IMAP: things that you can’t get with IMAP (+ SMTP + CalDAV + CardDAV + …), and things that affect the user experience. I did mention at one point that JMAP is such that you don’t even need any library to achieve useful things with it, whereas you wouldn’t tend to get far without such libraries on SMTP/IMAP, but I wasn’t making that comparison in any other way. > It's not realistic to develop a new mail client without IMAP (complete non-starter) even if it suports JMAP. Not true. It depends on who you’re targeting and what you’re trying to achieve. Sure, most will still want to support IMAP/SMTP due to JMAP’s current lack of widespread adoption, but honestly going native JMAP only carries some pretty compelling technical and functional advantages. > This only benefits people who are trying to be the next Gmail. I rebutted this fairly clearly in my previous comment. You seem to be fixing on first-party software, where service and software are provided by the same entity, but this is something that JMAP is better about compared to IMAP/SMTP, removing some of the unfair advantage first-party software had and levelling the field for all service providers and domain names, due to autodiscovery that works, and exposing the server in a way that even a web frontend can directly access. Just because Fastmail is currently the only notable provider shipping JMAP doesn’t mean that JMAP is about Fastmail. There’s a fair bit of interest from other providers, it just takes time. Also I would point out that Fastmail has not the slightest ambition of being “the next Gmail”; their business model, history and market positioning should make that very clear. > I'm logged into my self-hosted RoundCube webmail which is accessing my inbox with IMAP, and sends with SMTP. Of course, it does those things at the PHP back end. That RoundCube requires a backend, and its frontend has to talk something other than IMAP and SMTP, is a problem. Until JMAP, every single webmail client has done its own thing for a client–server protocol, and they’ve almost all been terrible, and so you’ve kinda had to learn two things to develop it, and you’ll have functional limitations all over the place that make developing the frontend hard. By contrast, with JMAP, a webmail frontend and a desktop app are talking the same protocol. That’s very valuable. I did go a little too hard on this in that a backend can allow you to connect to arbitrary mail sources (though in practice webmail is almost always first-party, service and software coming from the same provider), but that fact that you need that extra backend is still a problem, for performance and for privacy: your software vendor shouldn’t need to be able to access all your mail; it’s much better if the software can talk directly to the service provider. > If such a thing is not able to speak SMTP or IMAP, maybe the Javscript world should tool up? It’s not possible due to the web’s security model; it’d be a fiasco if browsers let JS initiate arbitrary TCP connections—far too much software assumes that only trusted software can talk on the local network. From time to time there are ideas about relaxing this limitation by server opt-in (along the lines of CORS preflight requests), but they’ve never gone anywhere yet and I don’t think they will any time soon, and certainly it will never be fully relaxed.

kazinator 3y ago

SMTP and IMAP are a solved problem. They are problematic for only one kind of entity: someone who has delusions of becoming the next gmail. Everyone else can just set up an open source SMTP and IMAP setup using existing programs that work, without writing a single line of parsing code. I don't trust this "open standard" bullshit. When the only thing running JMAP is Fastmail, it's a Fastmail-specific protocol. If you want developers to interface with it, you can't hide it: you have to document what the requests and responses look like. The "open standard" rhetoric has good optics; even more so with the IETF rubber stamp. The underlying idea behind JMAP is that the future consists of only large mail operators who somehow forward mail among themselves. Routing to SMTP servers is gone, and so is self-hosting, mostly. If you want your own mail domain, you can do that, but to operate it, your instance must connect over some web protocol to an e-mail monopolist. The IETF may have signed off on this, but I look forward to what the EFF have to say.

chrismorgan 3y ago
Speaking as one who has drunk the JMAP kool-aid (I worked for Fastmail for a few years): you sound like you’ve never tried to write an email client, or done much with email in general. Because the conclusions you’re coming to are completely back-to-front. Suppose you want to write a client that can do email sending (SMTP), email reading (IMAP), contact book (CardDAV) and calendar (CalDAV) interactions. SMTP/IMAP autodiscovery is limited at best, and CardDAV/CalDAV autodiscovery nonexistent, I think, so new users will have to find the details for several things, which is a big hurdle—or else you’ll have to recognise and special-case individual providers, which basically means Google and Microsoft get a massive unfair advantage. By contrast, JMAP handles autodiscovery of all this: type in an email address, and it finds the JMAP endpoint, which handles all of the stuff. (I will admit that this is currently more aspirational than real: to begin with it requires a DNS SRV record and/or a /.well-known/jmap HTTP response; and the means of authentication is deliberately unspecified at this stage, so we kinda need to wait around to see what providers settle on for authentication, with some kind of OAuth arrangement probably most likely.) And then perhaps your client is webmail; if so, you simply can’t use IMAP or SMTP. I guess you could try talking them over WebSocket or something like that (still requiring a proxy of some kind, even if a pass-through one), but that’d be painful; instead, everyone rolls their own thing, and so there’s never any webmail cross-compatibility. In JMAP, by contrast, the only thing you can’t do from a browser is looking up DNS SRV records. Everything else works, so that you can run your own webmail client that talks directly to the origin server. This allows you to disentangle the provision of service and software, which is very desirable for decentralisation. Well, now you want your client to produce notifications when new messages come in. If you’re writing for a desktop platform, fine: you can keep your IMAP connection open. But if you’re on a mobile platform, you probably can’t do that, and certainly can’t reliably, so… you’re stuck needing some kind of proxy and server component again. No good. But with JMAP, you set up a push subscription and it works just like any other app, without needing to keep a connection open. So there are three simple and concrete advantages of JMAP over the legacy protocols. JMAP actively bolsters decentralisation, in email providers, in domain names, in contact book and calendar support, in webmail, and in mobile notifications. There are other functional advantages too. JMAP enables some moderately complex batching that makes some sorts of realistic operations vastly faster than IMAP due to being able to skip many round trips. And JMAP’s pleasantness to work with directly should not be understated: no one wants to work with SMTP or IMAP directly because they’re painful protocols, and so of course all of that stuff tends to get extracted into a single library that protects you from them; by contrast, working in raw JMAP is pleasant and easy, which can genuinely open you up to doing things that would have been too painful to do through your IMAP library. And if you use JMAP, your mail client can skip worrying about MIME parsing (which is landmine-filled territory) if you choose. You can write a useful and complex standalone email workflow with nothing more than HTTP and JSON libraries. > The underlying idea behind JMAP is that the future consists of only large mail operators who somehow forward mail among themselves. Routing to SMTP servers is gone, and so is self-hosting, mostly. This is all completely wrong: JMAP is for client–server communications and doesn’t replace server–server communications, which still use SMTP. JMAP does not prop up email monopolists. > If you want developers to interface with it, you can't hide it: you have to document what the requests and responses look like. I have no idea what this is about. The spec shows examples. > The "open standard" rhetoric has good optics; even more so with the IETF rubber stamp. There was no IETF rubber stamp (which I take you to be using as a pejorative). There was collaborative process with people from a variety of companies who took what Fastmail had designed for their own use and wanted to share, and made it considerably better.
- kazinator 3y ago
  I've worked in the mail area, though not recently. Twenty years ago I worked on an IMAP4 transport, optimized for wireless, for Windows CE ("Microsoft Mobile OS" or whatever it was called). I don't find your arguments rationally convincing. You're comparing developing mail software using libraries for JSON, HTTPS and other pieces, versus doing the same thing while developing IMAP from scratch. It boils down to "web developers don't have nice libraries for IMAP access and are too busy making another clone of React or Typescript to make one". It's not realistic to develop a new mail client without IMAP (complete non-starter) even if it suports JMAP. This shit only benefits people who are trying to be the next Gmail. On the surface, it's somewhat good for users who rely on monolithic mail service in that if there are more such services, they have more choice. It's bad for everyone is not using a monolithic mail service, who doesn't want a new mail client speaking a new protocol or to install services for that or anything else. > And then perhaps your client is webmail; if so, you simply can’t use IMAP or SMTP. That claim is amazing; in another browser tab right here I'm logged into my self-hosted RoundCube webmail which is accessing my inbox with IMAP, and sends with SMTP. Of course, it does those things at the PHP back end. I think you may be conflating "webmail" with a "browser-based, purely local mail client application not accessed over the network". If such a thing is not able to speak SMTP or IMAP, maybe the Javscript world should tool up?
  - chrismorgan 3y ago
    > You're comparing developing mail software using libraries for JSON, HTTPS and other pieces, versus doing the same thing while developing IMAP from scratch. This is nonsense. Most of my comment was describing protocol-level advantages of JMAP over IMAP: things that you can’t get with IMAP (+ SMTP + CalDAV + CardDAV + …), and things that affect the user experience. I did mention at one point that JMAP is such that you don’t even need any library to achieve useful things with it, whereas you wouldn’t tend to get far without such libraries on SMTP/IMAP, but I wasn’t making that comparison in any other way. > It's not realistic to develop a new mail client without IMAP (complete non-starter) even if it suports JMAP. Not true. It depends on who you’re targeting and what you’re trying to achieve. Sure, most will still want to support IMAP/SMTP due to JMAP’s current lack of widespread adoption, but honestly going native JMAP only carries some pretty compelling technical and functional advantages. > This only benefits people who are trying to be the next Gmail. I rebutted this fairly clearly in my previous comment. You seem to be fixing on first-party software, where service and software are provided by the same entity, but this is something that JMAP is better about compared to IMAP/SMTP, removing some of the unfair advantage first-party software had and levelling the field for all service providers and domain names, due to autodiscovery that works, and exposing the server in a way that even a web frontend can directly access. Just because Fastmail is currently the only notable provider shipping JMAP doesn’t mean that JMAP is about Fastmail. There’s a fair bit of interest from other providers, it just takes time. Also I would point out that Fastmail has not the slightest ambition of being “the next Gmail”; their business model, history and market positioning should make that very clear. > I'm logged into my self-hosted RoundCube webmail which is accessing my inbox with IMAP, and sends with SMTP. Of course, it does those things at the PHP back end. That RoundCube requires a backend, and its frontend has to talk something other than IMAP and SMTP, is a problem. Until JMAP, every single webmail client has done its own thing for a client–server protocol, and they’ve almost all been terrible, and so you’ve kinda had to learn two things to develop it, and you’ll have functional limitations all over the place that make developing the frontend hard. By contrast, with JMAP, a webmail frontend and a desktop app are talking the same protocol. That’s very valuable. I did go a little too hard on this in that a backend can allow you to connect to arbitrary mail sources (though in practice webmail is almost always first-party, service and software coming from the same provider), but that fact that you need that extra backend is still a problem, for performance and for privacy: your software vendor shouldn’t need to be able to access all your mail; it’s much better if the software can talk directly to the service provider. > If such a thing is not able to speak SMTP or IMAP, maybe the Javscript world should tool up? It’s not possible due to the web’s security model; it’d be a fiasco if browsers let JS initiate arbitrary TCP connections—far too much software assumes that only trusted software can talk on the local network. From time to time there are ideas about relaxing this limitation by server opt-in (along the lines of CORS preflight requests), but they’ve never gone anywhere yet and I don’t think they will any time soon, and certainly it will never be fully relaxed.