3 ms·

> and we had a DNS server where the zones could be updated via SMTP Interesting — what was the message format (MIME type) you chose to use to describe the zone

by derefr 4y ago

> and we had a DNS server where the zones could be updated via SMTP Interesting — what was the message format (MIME type) you chose to use to describe the zone changes? Actually, were they even patch-files at all, or did you just fetch the zone data file from your DNS server, update it client-side, and then send an updated complete copy back in the body of the email message, to get de-enveloped directly into /var/named? With the benefit of hindsight, I'm guessing you'd agree that it would have been simpler to use an LDAP directory for service discovery; then you'd at least have a specced-out message format for updates (LDIF) to be sending to the queue. But then, OpenLDAP as a project only got started in 1998, so maybe you weren't even aware of its existence back then, and were stuck on YP/NIS for Intranet directory management (which has no similar useful extensibility.)

cryptonector 4y ago
In hindsight everything should be HTTP, and there should be JSON schemas as an option for everything. Not needing the bloat of an LDAP implementation is very nice. Schema differences between sites and schema metastasis into code in ways that complicate schema evolution is a big deal too, so "stored procedures" are really a good idea for containing schema metastasis. I'm not sure why GP would need service discovery for DNS zone updates via SMTP though.
- vidarh 4y ago
  Mail servers have a bunch of tooling for routing, retries, queue management etc. that already exists. If you do that via HTTP you end up having to reinvent a full message broker stack just for the sake of using a different protocol that lacks the abundance of components you can just drop in to do things like reflection (mailing lists), forwarding, filtering, retrieval, deletion, access control that you get for free with a mail server. Maybe if we that entire ecosystem existed on top of HTTP it'd have been ok, but the actual transport is the tiniest little uninteresting sliver of what we got by using a mail server for queuing. With respect to the DNS servers, the service discovery was for addressing, so we/a client component could e.g. e-mail "update@dns.live.local" or similar. The SMTP updated one was not the service discovery one, but feeding our customer facing DNS infrastructure.
  - cryptonector 4y ago
    There's several ways of doing service discovery for HTTP services like this in an enterprise, using 3xx replies to redirect if you move the service. The simplest is to just use a URI authority that you can change as needed. The next simplest is to publish URI RRs in DNS. The next simplest is to use the .well-known URI local-part namespace to include information for client-side routing. Granted, the last two are recent developments.
    - vidarh 4y ago
      Or we could just use DNS A or CNAME records and not have to do any of those, the way e.g. CoreDNS w/k8s or ldapdns on top of openldap, and similar does. There are use cases for more complicated service discovery, especially discovery of external services, but for the scenario I described, which in involved discovery within our own network this all just would have added extra complexity for no benefit that would have made any difference to us. Note that service discovery was not listed as one of the reasons I think http doesn't solve the issues that made us choose a mail server back then..the exact same service discovery mechanism would have worked just fine for http too. But we'd still have had to build all the message broker code from scratch instead of picking and choosing from pre-existing components.
      - cryptonector 4y ago
        I'm not saying that you shouldn't have used SMTP. It really is appropriate for this sort of thing. If you wanted an HTTP interface you might still make it inject messages into SMTP, with HTTP being only used for a browser UI, say, or with the HTTP server as a proxy for some reason. My objection had been to LDAP. LDAP is too heavy-weight by comparison to HTTP w/ a JSON API.
vidarh 4y ago
The DNS for service discovery was fairly static, and not what we updated via SMTP. We treated the DNS server in question simply as a key-value store where a message would set/replace a specific record. The use case was that we we're setting a handful of records when people registered a domain, so those DNS servers were user visible, unlike the service discovery. The main reason for not considering LDAP at the time for the user facing DNS server besides the lack of maturity for OpenLDAP was that using SMTP gave us one single unified mechanism for dispatching messages to all parts of the system, which also meant it was trivial to proxy, intercept, and log the full message flow with the same mechanism as needed.
- derefr 4y ago
  To be clear, I'm not saying you should have used LDAP as a changeset submission protocol instead of SMTP; I'm saying you should have used LDIF as a changeset format for sending via SMTP, instead of inventing your own changeset description format to send over SMTP. :) > We treated the DNS server in question simply as a key-value store where a message would set/replace a specific record. Interesting! That doesn't sound like any DNS server I'm aware of. In BIND et al, a zone was a file, and to update a zone you needed to modify a zone file and SIGHUP the daemon; which in turn meant that whatever you were using to mechanistically modify zones, needed to be able to parse and regenerate an entire zone file at a time. (It's actually a lot like what's required to mechanistically update-patch Kubernetes YAML resource manifests, now that I think about it.) Or do you mean that you had exactly one record per zone in this key-value store? If so, how did that work exactly? Zones need at least an SOA record, no? Or, I guess, maybe not, if this nameserver was only exposed within the Intranet, without the requirement of being an authoritative source that other DNS servers could fetch and cache from.
  - vidarh 4y ago
    The DNS server was our own. Writing an authoritative-only DNS server is trivial (few hundred lines), and since it knew it was always authoritative, it'd synthesise a SOA record for every zone it answered for. The zones had a very minimal set of records: Either they had NS records pointing to where the user wanted the domain delegated, or to us. If they pointed to us they had an A and MX record which might point to us or to wherever the customer wanted. It was not exposed only within the intranet (the service discovery DNS was exposed only via the internet; separate thing)

by derefr 4y ago

cryptonector 4y ago

In hindsight everything should be HTTP, and there should be JSON schemas as an option for everything. Not needing the bloat of an LDAP implementation is very nice. Schema differences between sites and schema metastasis into code in ways that complicate schema evolution is a big deal too, so "stored procedures" are really a good idea for containing schema metastasis. I'm not sure why GP would need service discovery for DNS zone updates via SMTP though.

vidarh 4y ago
Mail servers have a bunch of tooling for routing, retries, queue management etc. that already exists. If you do that via HTTP you end up having to reinvent a full message broker stack just for the sake of using a different protocol that lacks the abundance of components you can just drop in to do things like reflection (mailing lists), forwarding, filtering, retrieval, deletion, access control that you get for free with a mail server. Maybe if we that entire ecosystem existed on top of HTTP it'd have been ok, but the actual transport is the tiniest little uninteresting sliver of what we got by using a mail server for queuing. With respect to the DNS servers, the service discovery was for addressing, so we/a client component could e.g. e-mail "update@dns.live.local" or similar. The SMTP updated one was not the service discovery one, but feeding our customer facing DNS infrastructure.
- cryptonector 4y ago
  There's several ways of doing service discovery for HTTP services like this in an enterprise, using 3xx replies to redirect if you move the service. The simplest is to just use a URI authority that you can change as needed. The next simplest is to publish URI RRs in DNS. The next simplest is to use the .well-known URI local-part namespace to include information for client-side routing. Granted, the last two are recent developments.
  - vidarh 4y ago
    Or we could just use DNS A or CNAME records and not have to do any of those, the way e.g. CoreDNS w/k8s or ldapdns on top of openldap, and similar does. There are use cases for more complicated service discovery, especially discovery of external services, but for the scenario I described, which in involved discovery within our own network this all just would have added extra complexity for no benefit that would have made any difference to us. Note that service discovery was not listed as one of the reasons I think http doesn't solve the issues that made us choose a mail server back then..the exact same service discovery mechanism would have worked just fine for http too. But we'd still have had to build all the message broker code from scratch instead of picking and choosing from pre-existing components.
    - cryptonector 4y ago
      I'm not saying that you shouldn't have used SMTP. It really is appropriate for this sort of thing. If you wanted an HTTP interface you might still make it inject messages into SMTP, with HTTP being only used for a browser UI, say, or with the HTTP server as a proxy for some reason. My objection had been to LDAP. LDAP is too heavy-weight by comparison to HTTP w/ a JSON API.

vidarh 4y ago

The DNS for service discovery was fairly static, and not what we updated via SMTP. We treated the DNS server in question simply as a key-value store where a message would set/replace a specific record. The use case was that we we're setting a handful of records when people registered a domain, so those DNS servers were user visible, unlike the service discovery. The main reason for not considering LDAP at the time for the user facing DNS server besides the lack of maturity for OpenLDAP was that using SMTP gave us one single unified mechanism for dispatching messages to all parts of the system, which also meant it was trivial to proxy, intercept, and log the full message flow with the same mechanism as needed.

derefr 4y ago
To be clear, I'm not saying you should have used LDAP as a changeset submission protocol instead of SMTP; I'm saying you should have used LDIF as a changeset format for sending via SMTP, instead of inventing your own changeset description format to send over SMTP. :) > We treated the DNS server in question simply as a key-value store where a message would set/replace a specific record. Interesting! That doesn't sound like any DNS server I'm aware of. In BIND et al, a zone was a file, and to update a zone you needed to modify a zone file and SIGHUP the daemon; which in turn meant that whatever you were using to mechanistically modify zones, needed to be able to parse and regenerate an entire zone file at a time. (It's actually a lot like what's required to mechanistically update-patch Kubernetes YAML resource manifests, now that I think about it.) Or do you mean that you had exactly one record per zone in this key-value store? If so, how did that work exactly? Zones need at least an SOA record, no? Or, I guess, maybe not, if this nameserver was only exposed within the Intranet, without the requirement of being an authoritative source that other DNS servers could fetch and cache from.
- vidarh 4y ago
  The DNS server was our own. Writing an authoritative-only DNS server is trivial (few hundred lines), and since it knew it was always authoritative, it'd synthesise a SOA record for every zone it answered for. The zones had a very minimal set of records: Either they had NS records pointing to where the user wanted the domain delegated, or to us. If they pointed to us they had an A and MX record which might point to us or to wherever the customer wanted. It was not exposed only within the intranet (the service discovery DNS was exposed only via the internet; separate thing)