3 ms·

And I keep saying that SMTP is a perfectly good message-queue broker message submission protocol (and POP a perfectly good message-queue broker:consumer protoco

by derefr 4y ago

And I keep saying that SMTP is a perfectly good message-queue broker message submission protocol (and POP a perfectly good message-queue broker:consumer protocol), but nobody ever listens. :P

tleilaxu 4y ago
I listened, derefr. I listened. ;)
vidarh 4y ago
I've not just used SMTP the protocol as a submission protocol, but used actual mail servers as the broker, in a production system. Back in '99 I co-founded a mail provider, and we got so used to abusing qmail (+ rewritten replacement components - any qmail installation becomes a Ship of Theseus thanks to the clear API between small components) that when we needed a message broker we used our internal DNS server for service discovery and our mail infra as the broker (and we had a DNS server where the zones could be updated via SMTP). It worked well, and let us reuse a lot of tooling.
- derefr 4y ago
  > and we had a DNS server where the zones could be updated via SMTP Interesting — what was the message format (MIME type) you chose to use to describe the zone changes? Actually, were they even patch-files at all, or did you just fetch the zone data file from your DNS server, update it client-side, and then send an updated complete copy back in the body of the email message, to get de-enveloped directly into /var/named? With the benefit of hindsight, I'm guessing you'd agree that it would have been simpler to use an LDAP directory for service discovery; then you'd at least have a specced-out message format for updates (LDIF) to be sending to the queue. But then, OpenLDAP as a project only got started in 1998, so maybe you weren't even aware of its existence back then, and were stuck on YP/NIS for Intranet directory management (which has no similar useful extensibility.)
  - cryptonector 4y ago
    In hindsight everything should be HTTP, and there should be JSON schemas as an option for everything. Not needing the bloat of an LDAP implementation is very nice. Schema differences between sites and schema metastasis into code in ways that complicate schema evolution is a big deal too, so "stored procedures" are really a good idea for containing schema metastasis. I'm not sure why GP would need service discovery for DNS zone updates via SMTP though.
    - vidarh 4y ago
      Mail servers have a bunch of tooling for routing, retries, queue management etc. that already exists. If you do that via HTTP you end up having to reinvent a full message broker stack just for the sake of using a different protocol that lacks the abundance of components you can just drop in to do things like reflection (mailing lists), forwarding, filtering, retrieval, deletion, access control that you get for free with a mail server. Maybe if we that entire ecosystem existed on top of HTTP it'd have been ok, but the actual transport is the tiniest little uninteresting sliver of what we got by using a mail server for queuing. With respect to the DNS servers, the service discovery was for addressing, so we/a client component could e.g. e-mail "update@dns.live.local" or similar. The SMTP updated one was not the service discovery one, but feeding our customer facing DNS infrastructure.
      - cryptonector 4y ago
        There's several ways of doing service discovery for HTTP services like this in an enterprise, using 3xx replies to redirect if you move the service. The simplest is to just use a URI authority that you can change as needed. The next simplest is to publish URI RRs in DNS. The next simplest is to use the .well-known URI local-part namespace to include information for client-side routing. Granted, the last two are recent developments.
        vidarh 4y ago
        Or we could just use DNS A or CNAME records and not have to do any of those, the way e.g. CoreDNS w/k8s or ldapdns on top of openldap, and similar does. There are use cases for more complicated service discovery, especially discovery of external services, but for the scenario I described, which in involved discovery within our own network this all just would have added extra complexity for no benefit that would have made any difference to us. Note that service discovery was not listed as one of the reasons I think http doesn't solve the issues that made us choose a mail server back then..the exact same service discovery mechanism would have worked just fine for http too. But we'd still have had to build all the message broker code from scratch instead of picking and choosing from pre-existing components.

by derefr 4y ago

And I keep saying that SMTP is a perfectly good message-queue broker message submission protocol (and POP a perfectly good message-queue broker:consumer protocol), but nobody ever listens. :P

tleilaxu 4y ago

I listened, derefr. I listened. ;)

vidarh 4y ago

I've not just used SMTP the protocol as a submission protocol, but used actual mail servers as the broker, in a production system. Back in '99 I co-founded a mail provider, and we got so used to abusing qmail (+ rewritten replacement components - any qmail installation becomes a Ship of Theseus thanks to the clear API between small components) that when we needed a message broker we used our internal DNS server for service discovery and our mail infra as the broker (and we had a DNS server where the zones could be updated via SMTP). It worked well, and let us reuse a lot of tooling.

derefr 4y ago
> and we had a DNS server where the zones could be updated via SMTP Interesting — what was the message format (MIME type) you chose to use to describe the zone changes? Actually, were they even patch-files at all, or did you just fetch the zone data file from your DNS server, update it client-side, and then send an updated complete copy back in the body of the email message, to get de-enveloped directly into /var/named? With the benefit of hindsight, I'm guessing you'd agree that it would have been simpler to use an LDAP directory for service discovery; then you'd at least have a specced-out message format for updates (LDIF) to be sending to the queue. But then, OpenLDAP as a project only got started in 1998, so maybe you weren't even aware of its existence back then, and were stuck on YP/NIS for Intranet directory management (which has no similar useful extensibility.)
- cryptonector 4y ago
  In hindsight everything should be HTTP, and there should be JSON schemas as an option for everything. Not needing the bloat of an LDAP implementation is very nice. Schema differences between sites and schema metastasis into code in ways that complicate schema evolution is a big deal too, so "stored procedures" are really a good idea for containing schema metastasis. I'm not sure why GP would need service discovery for DNS zone updates via SMTP though.
  - vidarh 4y ago
    Mail servers have a bunch of tooling for routing, retries, queue management etc. that already exists. If you do that via HTTP you end up having to reinvent a full message broker stack just for the sake of using a different protocol that lacks the abundance of components you can just drop in to do things like reflection (mailing lists), forwarding, filtering, retrieval, deletion, access control that you get for free with a mail server. Maybe if we that entire ecosystem existed on top of HTTP it'd have been ok, but the actual transport is the tiniest little uninteresting sliver of what we got by using a mail server for queuing. With respect to the DNS servers, the service discovery was for addressing, so we/a client component could e.g. e-mail "update@dns.live.local" or similar. The SMTP updated one was not the service discovery one, but feeding our customer facing DNS infrastructure.
    - cryptonector 4y ago
      There's several ways of doing service discovery for HTTP services like this in an enterprise, using 3xx replies to redirect if you move the service. The simplest is to just use a URI authority that you can change as needed. The next simplest is to publish URI RRs in DNS. The next simplest is to use the .well-known URI local-part namespace to include information for client-side routing. Granted, the last two are recent developments.
      - vidarh 4y ago
        Or we could just use DNS A or CNAME records and not have to do any of those, the way e.g. CoreDNS w/k8s or ldapdns on top of openldap, and similar does. There are use cases for more complicated service discovery, especially discovery of external services, but for the scenario I described, which in involved discovery within our own network this all just would have added extra complexity for no benefit that would have made any difference to us. Note that service discovery was not listed as one of the reasons I think http doesn't solve the issues that made us choose a mail server back then..the exact same service discovery mechanism would have worked just fine for http too. But we'd still have had to build all the message broker code from scratch instead of picking and choosing from pre-existing components.