5 ms·

It's not always DNS

56 points by todsacerdoti 1y ago

sim7c00 1y ago
it could also be gamma rays or a variety of problems that seem to appear and disappear between chairs and keyboards. memes are jokes. people taking jokes as something other is the problem.
bediger4000 1y ago
A lot of the time it's cabling.
kikoreis 1y ago
Resolver limitations, as opposed to server or protocol issues, are in my view the main reason why "it is always DNS".
jtbayly 1y ago
This is a beautifully designed page.
- lucasban 1y ago
  I wish it had a little bit more padding on mobile, but I agree otherwise
FuriouslyAdrift 1y ago
Well sure... it could be BGP
prmoustache 1y ago
No, sometimes it is just Spanish football as for everything behind Cloudflare. Which is the case for this blog being blocked right now and redirecting to another page: "El acceso a la presente dirección IP ha sido bloqueado en cumplimiento de lo dispuesto en la Sentencia de 18 de diciembre de 2024, dictada por el Juzgado de lo Mercantil nº 6 de Barcelona en el marco del procedimiento ordinario (Materia mercantil art. 249.1.4)-1005/2024-H instado por la Liga Nacional de Fútbol Profesional y por Telefónica Audiovisual Digital, S.L.U. https://www.laliga.com/noticias/nota-informativa-en-relacion-con-el-bloqueo-de-ips-durante-las-ultimas-jornadas-de-laliga-ea-sports-vinculadas-a-las-practicas-ilegales-de-cloudflare https://www.laliga.com/noticias/nota-informativa-en-relacion..."
- sodaclean 1y ago
  It's intentional- If people can't use the internet they're more likely to watch the "game." For once management might have learned something from employees- take a dive, cry foul.
- embedding-shape 1y ago
  At least you get some message about why. I'm on Vodafone and the only thing I saw was "Por causas ajenas a Vodafone, esta web no está disponible". Fucking censorship sucks, and seemingly people still see Spain as a modern democracy when shit like this happens in public and everyone knows about it, yet here we are. Because of football we can't browse the web when there are matches...
  - balaz 1y ago
    Who sees Spain as a modern democracy? Only those who benefit from the rampant clientelism.
    - embedding-shape 1y ago
      Most people outside of Spain? Just as one example, "The Economist Democracy Index" lists Spain at spot 21, yet we have rampant government censorship, make that make sense.
      - balaz 1y ago
        Those lists are a joke. They show how much the priorities of the government align with those of the newspaper they are printed on.
        embedding-shape 1y ago
        Right, another thing you can try (if you haven't) is traveling to any country except Spain and think what those people think of Spain, and you'll learn the same thing.
Spooky23 1y ago
Paul Tagliamonte sounds like a nice guy who has thought about these issues at length. He's reached the second level of DNS enlightenment: "There's no way it's DNS". Finality will arrive, and Paul will internalize the knowledge.
oliyoung 1y ago
Nope, the other times it's CORS
- jongjong 1y ago
  Though at least with CORS, once you actually get the damn thing working, it keeps working.
unilynx 1y ago
> but it is not the operational hazard it’s made out to be Until you flip that DNSSEC toggle
teddyh 1y ago
> a DNSSEC rollout bricking prod for hours He links to the Slack incident. But that problem wasn’t caused by a DNSSEC rollout; the problem was entirely caused by a completely botched attempt to back out of DNSSEC, by doing it the worst way possible.
- tptacek 1y ago
  What's your point?
  - teddyh 1y ago
    Truth. Unlike some people, I find it important.
    - tptacek 1y ago
      What decision would people make differently knowing the extra detail you just provided?
      - teddyh 1y ago
        It wasn’t an “extra detail”, it was a correction. Because accuracy matters.
ZebusJesus 1y ago
Tell that to AWS East 1
sshine 1y ago
I had the CEO and CTO of our ccTLD registry give a guest lecture to my CS students, and one question came up regarding the AWS incident. Prior to the question, the CEO boasted a 100% uptime (not just five nines), and the CTO said “We’re basically 30 people maintaining a 1GB text file.” So the question was, “How come 30 people can have 100% uptime, and the biggest cloud with all of its expertise can’t? Sure, it was DNS, but are you even doing the same thing?” And the answer was, (paraphrasing) “No, what we do is simple. They use DNS to solve all sorts of distributed problems.” As did the CTO with all of these new record types embedding authentication. But running CoreDNS in a Kubernetes megacluster is not “maintaining a 1GB text file”.
- hdgvhicv 1y ago
  Maintaining uptime on complex systems is hard. That’s why the best systems have as little complexity as possible But that doesn’t help boost your resume or get a bonus.
inopinatus 1y ago
The full maxim I was taught being, “it’s either DNS or permissions”. The fatal design flaw for the Domain Name System was failure to learn from SCSI, viz. that it should always be possible to sacrifice a goat to whatever gods are necessary to receive a blessing of stability. It hardly remains to observe that animal sacrifice is non-normative for IETF standards-track documents and the consequences for distributed systems everywhere are plainly evident. Goats notwithstanding, I think it is splitting hairs to suggest that the phrase “it’s always DNS” is erroneously reductive, merely because it does not explicitly convey that an adjacent control-plane mechanism updating the records may also be implicated. I don’t believe this aphorism drives a misconception that DNS itself is an inherently unreliable design. We’re not laughing it off to the extent of terminating further investigation, root-cause analysis, or subsequent reliability and consistency improvement. More constructively, also observe that the industry standard joke book has another one covering us for this circumstance, viz. “There are only two hard problems in distributed systems: 2. Exactly-once delivery 1. Guaranteed order of processing 2. Exactly-once delivery”
- throw0101a 1y ago
  A SCSI bus always needed three terminations: one at either end of the cable, and a black rooster.
- whatever1 1y ago
  Why Computers engineers refuse to talk with manufacturing graybeards that operate critical systems at scale ? The design shit I am seeing would not pass at a chemical plant not even a preliminary review.
  - 01HNNWZ0MV43FF 1y ago
    I don't know any manufacturing graybeards. Where could I meet some?
    - whatever1 1y ago
      Conferences! IEEE, AICHE, IMTS, Fabtech, Automate, Productronica
TacticalCoder 1y ago
[dead]
ricudis 1y ago
It's always DNS, except when it's BGP.
inlined 1y ago
Is this meant to be a defense of the DNS protocol? I’ve never assumed the meme was that the DNS protocol is flawed, but that these changes are particularly sensitive/dangerous. At Google we noticed the main cause of outages are config changes. Does that mean external config is dangerous? Of course not! But it does remind you to be vigilant
jiggawatts 1y ago
It’s DNS far too often in large part because Linux has the default behaviour of having a singular (“the”) name server. If you configure multiple, this is not the same as in Windows, MacOS, iOS, or even Android. It’s not a pair of redundant servers, it’s a sequential list of logically layered configurations, like an override. An outage of the primary will cause repeated timeouts until it comes back up. Contrast this with every other operating system that does not seek to emulate forever and ever the specific network setup of some computer lab at Berkeley in the 1970s, where failover is near instant (sub-second in Windows Vista and later) and persists, so that a small failure doesn’t become a big one. To compound things, thanks to response caching and the relatively good stability of typical DNS servers, this failure mode is rare enough that most admins have never encountered it and probably don’t even recognise the problem. Or worse, they’ll handwave it away saying things starting with “You can…” or “You should…”. Who’s this “You” person? It’s not me! It’s not most admins, none I’ve met are named You. None have changed this default in my experience. Not one. That would be a design decision requiring sign off, testing, rollout, etc… I’ve seen this take down corporations for a day. Defaults matter.
ranger207 1y ago
It's worth noting that the meme of "it was DNS", including the haiku[0], is from the old school sysadmin world, which has a lot more terrible DNS implementations than modern stuff (especially including Active Directory which has DNS attached to a massive complex system that does dozens of other things as well and because of which has its reliability suffer), so the meme is really a reflection of a harsher time. [0] the original source of the haiku: https://www.reddit.com/r/sysadmin/comments/4oj7pv/network_solutions_haiku/d4czk91 https://www.reddit.com/r/sysadmin/comments/4oj7pv/network_so...
phicoh 1y ago
This takes a rather narrow view at what is DNS. For DNS as a service to work, it has to be accessible and give the right answers. It doesn't matter why it is not accessible or why it doesn't give the right answers. If it doesn't, then the service is broken. DNS is in the unique position that it is relatively high up in the network stack, so lots of (network) failures affect DNS as a service. It is a big distributed database, which gives many possibilities for wrong data, it is used by almost all applications, so a failure of DNS as a service is highly noticeable. Finally, DNS has by nature (some what) centralized choke points. If you have a domain like company.com, then just about everything that company does has to go through the DNS servers for company.com. Any small failure there can have a huge effect. So DNS is a pretty exciting field to work in.

56 points by todsacerdoti 1y ago

it could also be gamma rays or a variety of problems that seem to appear and disappear between chairs and keyboards. memes are jokes. people taking jokes as something other is the problem.

bediger4000 1y ago

A lot of the time it's cabling.

kikoreis 1y ago

Resolver limitations, as opposed to server or protocol issues, are in my view the main reason why "it is always DNS".

jtbayly 1y ago

This is a beautifully designed page.

lucasban 1y ago
I wish it had a little bit more padding on mobile, but I agree otherwise

FuriouslyAdrift 1y ago

Well sure... it could be BGP

prmoustache 1y ago

No, sometimes it is just Spanish football as for everything behind Cloudflare. Which is the case for this blog being blocked right now and redirecting to another page: "El acceso a la presente dirección IP ha sido bloqueado en cumplimiento de lo dispuesto en la Sentencia de 18 de diciembre de 2024, dictada por el Juzgado de lo Mercantil nº 6 de Barcelona en el marco del procedimiento ordinario (Materia mercantil art. 249.1.4)-1005/2024-H instado por la Liga Nacional de Fútbol Profesional y por Telefónica Audiovisual Digital, S.L.U. https://www.laliga.com/noticias/nota-informativa-en-relacion-con-el-bloqueo-de-ips-durante-las-ultimas-jornadas-de-laliga-ea-sports-vinculadas-a-las-practicas-ilegales-de-cloudflare https://www.laliga.com/noticias/nota-informativa-en-relacion..."

sodaclean 1y ago
It's intentional- If people can't use the internet they're more likely to watch the "game." For once management might have learned something from employees- take a dive, cry foul.
embedding-shape 1y ago
At least you get some message about why. I'm on Vodafone and the only thing I saw was "Por causas ajenas a Vodafone, esta web no está disponible". Fucking censorship sucks, and seemingly people still see Spain as a modern democracy when shit like this happens in public and everyone knows about it, yet here we are. Because of football we can't browse the web when there are matches...
- balaz 1y ago
  Who sees Spain as a modern democracy? Only those who benefit from the rampant clientelism.
  - embedding-shape 1y ago
    Most people outside of Spain? Just as one example, "The Economist Democracy Index" lists Spain at spot 21, yet we have rampant government censorship, make that make sense.
    - balaz 1y ago
      Those lists are a joke. They show how much the priorities of the government align with those of the newspaper they are printed on.
      - embedding-shape 1y ago
        Right, another thing you can try (if you haven't) is traveling to any country except Spain and think what those people think of Spain, and you'll learn the same thing.

Spooky23 1y ago

Paul Tagliamonte sounds like a nice guy who has thought about these issues at length. He's reached the second level of DNS enlightenment: "There's no way it's DNS". Finality will arrive, and Paul will internalize the knowledge.

oliyoung 1y ago

Nope, the other times it's CORS

jongjong 1y ago
Though at least with CORS, once you actually get the damn thing working, it keeps working.

unilynx 1y ago

> but it is not the operational hazard it’s made out to be Until you flip that DNSSEC toggle

teddyh 1y ago

> a DNSSEC rollout bricking prod for hours He links to the Slack incident. But that problem wasn’t caused by a DNSSEC rollout; the problem was entirely caused by a completely botched attempt to back out of DNSSEC, by doing it the worst way possible.

tptacek 1y ago
What's your point?
- teddyh 1y ago
  Truth. Unlike some people, I find it important.
  - tptacek 1y ago
    What decision would people make differently knowing the extra detail you just provided?
    - teddyh 1y ago
      It wasn’t an “extra detail”, it was a correction. Because accuracy matters.

ZebusJesus 1y ago

Tell that to AWS East 1

sshine 1y ago

I had the CEO and CTO of our ccTLD registry give a guest lecture to my CS students, and one question came up regarding the AWS incident. Prior to the question, the CEO boasted a 100% uptime (not just five nines), and the CTO said “We’re basically 30 people maintaining a 1GB text file.” So the question was, “How come 30 people can have 100% uptime, and the biggest cloud with all of its expertise can’t? Sure, it was DNS, but are you even doing the same thing?” And the answer was, (paraphrasing) “No, what we do is simple. They use DNS to solve all sorts of distributed problems.” As did the CTO with all of these new record types embedding authentication. But running CoreDNS in a Kubernetes megacluster is not “maintaining a 1GB text file”.

hdgvhicv 1y ago
Maintaining uptime on complex systems is hard. That’s why the best systems have as little complexity as possible But that doesn’t help boost your resume or get a bonus.

inopinatus 1y ago

The full maxim I was taught being, “it’s either DNS or permissions”. The fatal design flaw for the Domain Name System was failure to learn from SCSI, viz. that it should always be possible to sacrifice a goat to whatever gods are necessary to receive a blessing of stability. It hardly remains to observe that animal sacrifice is non-normative for IETF standards-track documents and the consequences for distributed systems everywhere are plainly evident. Goats notwithstanding, I think it is splitting hairs to suggest that the phrase “it’s always DNS” is erroneously reductive, merely because it does not explicitly convey that an adjacent control-plane mechanism updating the records may also be implicated. I don’t believe this aphorism drives a misconception that DNS itself is an inherently unreliable design. We’re not laughing it off to the extent of terminating further investigation, root-cause analysis, or subsequent reliability and consistency improvement. More constructively, also observe that the industry standard joke book has another one covering us for this circumstance, viz. “There are only two hard problems in distributed systems: 2. Exactly-once delivery 1. Guaranteed order of processing 2. Exactly-once delivery”

throw0101a 1y ago
A SCSI bus always needed three terminations: one at either end of the cable, and a black rooster.
whatever1 1y ago
Why Computers engineers refuse to talk with manufacturing graybeards that operate critical systems at scale ? The design shit I am seeing would not pass at a chemical plant not even a preliminary review.
- 01HNNWZ0MV43FF 1y ago
  I don't know any manufacturing graybeards. Where could I meet some?
  - whatever1 1y ago
    Conferences! IEEE, AICHE, IMTS, Fabtech, Automate, Productronica

TacticalCoder 1y ago

[dead]

ricudis 1y ago

It's always DNS, except when it's BGP.

inlined 1y ago

Is this meant to be a defense of the DNS protocol? I’ve never assumed the meme was that the DNS protocol is flawed, but that these changes are particularly sensitive/dangerous. At Google we noticed the main cause of outages are config changes. Does that mean external config is dangerous? Of course not! But it does remind you to be vigilant

jiggawatts 1y ago

It’s DNS far too often in large part because Linux has the default behaviour of having a singular (“the”) name server. If you configure multiple, this is not the same as in Windows, MacOS, iOS, or even Android. It’s not a pair of redundant servers, it’s a sequential list of logically layered configurations, like an override. An outage of the primary will cause repeated timeouts until it comes back up. Contrast this with every other operating system that does not seek to emulate forever and ever the specific network setup of some computer lab at Berkeley in the 1970s, where failover is near instant (sub-second in Windows Vista and later) and persists, so that a small failure doesn’t become a big one. To compound things, thanks to response caching and the relatively good stability of typical DNS servers, this failure mode is rare enough that most admins have never encountered it and probably don’t even recognise the problem. Or worse, they’ll handwave it away saying things starting with “You can…” or “You should…”. Who’s this “You” person? It’s not me! It’s not most admins, none I’ve met are named You. None have changed this default in my experience. Not one. That would be a design decision requiring sign off, testing, rollout, etc… I’ve seen this take down corporations for a day. Defaults matter.

ranger207 1y ago

It's worth noting that the meme of "it was DNS", including the haiku[0], is from the old school sysadmin world, which has a lot more terrible DNS implementations than modern stuff (especially including Active Directory which has DNS attached to a massive complex system that does dozens of other things as well and because of which has its reliability suffer), so the meme is really a reflection of a harsher time. [0] the original source of the haiku: https://www.reddit.com/r/sysadmin/comments/4oj7pv/network_solutions_haiku/d4czk91 https://www.reddit.com/r/sysadmin/comments/4oj7pv/network_so...

phicoh 1y ago

This takes a rather narrow view at what is DNS. For DNS as a service to work, it has to be accessible and give the right answers. It doesn't matter why it is not accessible or why it doesn't give the right answers. If it doesn't, then the service is broken. DNS is in the unique position that it is relatively high up in the network stack, so lots of (network) failures affect DNS as a service. It is a big distributed database, which gives many possibilities for wrong data, it is used by almost all applications, so a failure of DNS as a service is highly noticeable. Finally, DNS has by nature (some what) centralized choke points. If you have a domain like company.com, then just about everything that company does has to go through the DNS servers for company.com. Any small failure there can have a huge effect. So DNS is a pretty exciting field to work in.