10 ms·

1.1.1.1 outage explanation

459 points by adamch 8y ago

walrus01 8y ago
Commendable honesty and level of detail in the public RFO.
frenchie4111 8y ago
Understandable outage. I switched back to cloudflare when it came back up, but this did prompt me to drop 8.8.8.8 in as a 3rd fallback.
gcommer 8y ago
This is a great write up. It's also why the DNS root servers have a policy of surviving DDoS through massively over-provisioned, multi-org, anycasted redundancy rather this sort of smart DDoS mitigation that drops traffic: DNS is so critical that any risk of dropping real traffic is unacceptable. (obviously, such a scale is impractical for 99% of services) A good takeaway from this outage for the average user would be to make sure that your fallback DNS resolvers are operated by totally separate providers. (eg, configure 1.1.1.1 with 8.8.8.8 as a fallback, rather than 1.1.1.1 and 1.0.0.1) (Edit: fixed cloudflare's secondary address)
- Dylan16807 8y ago
  (the cloudflare secondary is 1.0.0.1)
  - cmurf 8y ago
    Cloudfare makes it weirdly difficult to find this. 1.1.1.1 is plastered over many pages but not concomitant with the secondary.
    - bobf 8y ago
      That's not necessarily a bad thing, since it's probably better to use another source for a secondary if that's your primary.
      - tracker1 8y ago
        I'm using cloudflare first, and google as secondary in my router. For a while I was running my own, but it became less necessary when I stopped doing as much dev at home.
    - chdefrene 8y ago
      That is not my experience at all. By following the `install`-instructions on http://1.1.1.1 http://1.1.1.1, all available DNS addresses are listed for both ipv4 and ipv6.
      - Mononokay 8y ago
        FYI, https://1.1.1.1 https://1.1.1.1 works over SSL, so there's no need to format it http://1.1.1.1 http://1.1.1.1.
- berti 8y ago
  Or 9.9.9.9 if you're not as comfortable with Google services.
  - spondyl 8y ago
    For anyone wondering, 9.9.9.9 is https://www.quad9.net/ https://www.quad9.net/ who claim to not only resolve requests but also check them against IBM X-Force's threat intelligence database.
    - nerdponx 8y ago
      For the curious, they also have an extremely detailed privacy policy: https://www.quad9.net/policy/ https://www.quad9.net/policy/ Seems like a fair deal to me. I get free DNS service, and companies sponsoring this program get metrics on threats and general Internet usage. I'm a little skeptical of their claims that individuals can't be identified from their anonymized data. E.g. I probably only get one or two hits on my personal website every week, so it might not be hard for a malicious employee to deanonymize visitors to my site. Some highlights from the policy: Many nations classify IP addresses as Personally-Identifiable Information (PII), and we take a conservative approach in treating IP addresses as PII in all jurisdictions in which our systems reside. Our normal course of data management does not have any IP address information or other PII logged to disk or transmitted out of the location in which the query was received. We may aggregate certain counters to larger network block levels for statistical collection purposes, but those counters do not maintain specific IP address data nor is the format or model of data stored capable of being reverse-engineered to ascertain what specific IP addresses made what queries. There are exceptions to this storage model: In the event of events or observed behaviors which we deem malicious or anomalous, we may utilize more detailed logging to collect more specific IP address data in the process of normal network defense and mitigation. This collection and transmission off-site will be limited to IP addresses that we determine are involved in the event. ... We do not correlate or combine information from our logs with any personal information that you have provided Quad9 for other services, or with your specific IP address. ... Quad9 DNS Services generate and share high level anonymized aggregate statistics including threat metrics on threat type, geolocation, and if available, sector, as well as other vertical metrics including performance metrics on the Quad9 DNS Services (i.e. number of threats blocked, infrastructure uptime) when available with the Quad9 threat intelligence (TI) partners, academic researchers, or the public. Quad9 DNS Services share anonymized data on specific domains queried (records such as domain, timestamp, geolocation, number of hits, first seen, last seen) with its threat intelligence partners. Quad9 DNS Services also builds, stores, and may share certain DNS data streams which store high level information about domain resolved, query types, result codes, and timestamp. These streams do not contain IP address information of requestor and cannot be correlated to IP address or other PII. ... Quad9 does not track visitors over time and across third-party websites, and therefore does not respond to Do Not Track signaling.
- h1d 8y ago
  Not if you're not comfortable giving up so many of your internet activity to Google but as others have pointed out there are alternatives to spread.
- bluedino 8y ago
  I’m amazed at how many people abandoned other providers and blindly switched to 1.1.1.1 I can’t even use that address with the ISP Alestra in Mexico
  - toomuchtodo 8y ago
    The lack of operational knowledge is real, and there is no easy fix other than resilient defaults by a benevolent dictator (in this case, resolvers across netblock and provider demarcations) or spending the time to educate yourself first on the “why” then the “how”.
- crtasm 8y ago
  I read on the Pi-Hole forums that 'fallback' is a misleading term because clients don't work that way - they will happily spread requests between two functioning DNS servers. Can anyone confirm this or provide further insight?
  - mjcl 8y ago
    Different OSes handle it differently. Windows tries the primary, waits 1 second and then starts trying secondaries.
    - dharmab 8y ago
      glibc tries in order. musl (Alpine Linux) tries all in parallel and returns thebl first resolution (If any).
  - TravelTechGuy 8y ago
    Can confirm. If I have a secondary DNS specified in my router, other than my pi-hole, it becomes useless. My guess is the router is either measuring response time, and goes with the most efficient, or otherwise round-robining the requests. Either way, requests bypass the pi-hole in such quantities that it became useless. PS: someone here mentioned that this behavior is OS-dependent - nope, this happens on the router level, and all devices in my apartment suffer.
    - detaro 8y ago
      Then your devices are likely using your router as a DNS resolver, which in turn talks to your pihole and the external one. And thus it depends on your router's OS what it does.
    - jschwartzi 8y ago
      It depends on how your router's DHCP server is configured. If you configure your router to pass its own IP address out as the DNS server for the local subnet then the router's behavior dictates how DNS works. If your router is passing out an external DNS in the DHCP configuration, then you'll get OS-dependent behavior. My router uses a DNS resolver internally, and it will spread-cast to multiple DNS servers and use the quickest response it can get. It also caches using the TTL in the DNS response, and so it will serve up cached records transparently.
throwaway2048 8y ago
another decent alternative fallback with the same featureset is quad9 https://www.quad9.net/ https://www.quad9.net/
- kondro 8y ago
  The core feature of Cloudflare's DNS is getting you closer and faster resolves to their CDN (probably one of the largest at this point, barring the established enterprise encumbants) in a privacy-first way. Quad9's goal seems to be about threat-detection and prevention. It would be nice if you could have both simultaneously (and maybe you can), but at the moment both services are actually quite different.
  - grimgrin 8y ago
    Maybe check out what NS1 is doing with DNS https://ns1.com/ https://ns1.com/
  - bigiain 8y ago
    9.9.9.10 is "quad nine without the threat detection" from memory.
    - kondro 8y ago
      Yeah, but the reason why 1.1.1.1 is so fast for sites that use Cloudflare as DNS is because Cloudflare is the authoritative DNS for them. The only way you get that in a more generic sense is if a specialist DNS CDN provider started up that provided DNS services for all the existing CDNs (or they all agreed to some type of federated standard that let them share the same recursive multicast IP addresses for DNS resolution).
      - dx034 8y ago
        Also, Cloudflare has a huge amount of data centres by now, probably more than any other service. Even Google often underperforms them. Debatable if a few ms make a difference but it can for people living in remote areas where CF has a centre and the next 9.9.9.9/8.8.8.8 is 100ms away.
saghm 8y ago
> Our FRP framework allows us to express this in clear and readable code. For example, this is part of the code responsible for performing DNS attack mitigation: > > def action_gk_dns(...): > [...] > > if port != 53: > return None > > if whitelisted_ip.get(ip): > return None > > if ip not in ANYCAST_IPS: > return None > > [...] What does this code sample have to do with FRP? This code seems extremely trivial and doesn't give any real indication to me why you'd need a framework of any sort. It seems like they really want to emphasize that they use FRP, but this code just seems completely unrelated.
- deleted 8y ago
  [deleted]
- Chris911 8y ago
  From one of the linked presentations: https://github.com/cloudflare/gatelogic https://github.com/cloudflare/gatelogic
dingo_bat 8y ago
TL;DR: we should have used an IP that is not traditionally used for testing and internal stuff by everybody including Cisco.
- Dylan16807 8y ago
  Not even close. The system had a glitch because they were doing a major DNS resolver at all. It had nothing to do with the baggage that comes with 1.1.1.1 specifically.
  - isostatic 8y ago
    However a few days ago there was a bgp based outage on cloudfare DNs want there?
- SmellyGeekBoy 8y ago
  Sounds like it was truly a "DR" in your case.
- sp332 8y ago
  They were given the IP block by ARPA specifically with an agreement that they would analyze the junk traffic and report back on it. No one attempting this before has been able to keep up with the amount of noise pointed at that IP. Everyone involved knew exactly what they were getting into.
drawkbox 8y ago
Great information on the outage. Looks like another version 2 syndrome side effect [1]. > Today, in an effort to reclaim some technical debt, we deployed new code that introduced Gatebot to Provision API. > What we did not account for, and what Provision API didn’t know about, was that 1.1.1.0/24 and 1.0.0.0/24 are special IP ranges. Frankly speaking, almost every IP range is "special" for one reason or another, since our IP configuration is rather complex. But our recursive DNS resolver ranges are even more special: they are relatively new, and we're using them in a very unique way. Our hardcoded list of Cloudflare addresses contained a manual exception specifically for these ranges. > As you might be able to guess by now, we didn't implement this manual exception while we were doing the integration work. Remember, the whole idea of the fix was to remove the hardcoded gotchas! When porting legacy code it is not only important to understand the edge cases and technical debt built up over time, but to test more heavily in production because you never know if you got them all because some smart guy built them long ago and/or there are unknown hacks that were cornerstones of the system for better or worse. Phased and alpha/beta rollouts in an almost A/B testing way is good for replacement systems. Version 2 systems can also add new attack vectors or other single points of failure that aren't as know from legacy problems, the Provision API seems like it is a candidate for that. Over time the Version 2 system will be hardened just before it is EOL and replaced again to fix all the new problems that arise over time. Version 2's do innovate but they also shroud fixing old issues and pain points for new unknown problems. [1] https://en.wikipedia.org/wiki/Second-system_effect https://en.wikipedia.org/wiki/Second-system_effect
- teddyh 8y ago
  “Yes, I know, it’s just a simple function to display a window, but it has grown little hairs and stuff on it and nobody knows why. Well, I’ll tell you why: those are bug fixes. One of them fixes that bug that Nancy had when she tried to install the thing on a computer that didn’t have Internet Explorer. Another one fixes that bug that occurs in low memory conditions. Another one fixes that bug that occurred when the file is on a floppy disk and the user yanks out the disk in the middle. That LoadLibrary call is ugly but it makes the code work on old versions of Windows 95. Each of these bugs took weeks of real-world usage before they were found.” — Things You Should Never Do, Part I, (https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ https://www.joelonsoftware.com/2000/04/06/things-you-should-...)
cm2187 8y ago
Do they just use python as pseudo code or do they actually run their attack detection in python?
- usmannk 8y ago
  It is python. They linked a presentation[1] and a talk about the bot. [1] https://speakerdeck.com/majek04/gatelogic-somewhat-functional-reactive-framework-in-python https://speakerdeck.com/majek04/gatelogic-somewhat-functiona...
  - cm2187 8y ago
    It's interesting. I would have expected them to use rather something low latency/high performance like c++ or erlang given their scale and performance criticality.
    - heavenlyblue 8y ago
      This is something that only keeps track of user settings data and consequently configures the endpoints given that data. It has got nothing to do with packet filtering per se.
- askz 8y ago
  Any supplementary thoughts on the fact they are running their attack detection in python? What's the point?
  - Spare_account 8y ago
    There is a reply in a parallel chain (timestamped after your comment) which expands on the line of thought: https://news.ycombinator.com/item?id=17204069 https://news.ycombinator.com/item?id=17204069
decker 8y ago
I wonder if they are going to factor the availability into all the blog posts about how fast they are.
obeattie 8y ago
Major credit to Cloudflare for publishing a clear, honest, and detailed description of what happened. I wish more companies would do this. One thing I’d be interested to know more about is why it took 17 minutes to fix. While you can and always should strive to make them less likely, outages are inevitable, so how you respond is crucial. Here the outage was very obviously caused by a deployment that I’d assume was supervised by humans – why did it take 17 minutes to roll back?
- blackbrokkoli 8y ago
  I'm not an expert, but is 17 minutes for: - shit is not working - is this an attack? - no it's us - how? - that's how - let's go back - have to get supervisor - roll back huge thing really that long?
  - dx034 8y ago
    With ~150 data centres, roll back alone probably took 5-10 minutes. Don't think 17 minutes is that long.
    - Cthulhu_ 8y ago
      For simple PagerDuty alerts I already need 15 minutes to open the app / logs and figure out what's going on.
      - wool_gather 8y ago
        Good point, although the way it's described it sounds like the problem cropped up right after deploy. So they would have been watching it actively. But as said above, 17 minutes to notice, figure out what's going on, decide what to do, and propagate the resolution seems reasonable.
    - dfcowell 8y ago
      Not to mention selectively purging all of the rules created (I assume) at every edge server in their network that were blackholing all of the traffic to the resolver. There's probably a command for it, but all in all 17 minutes seems quite a respectable turnaround time.
Robin_Message 8y ago
What's interesting here is that the automatic cure (DDoS protection) was worse than the disease (even if there was an attack, blocking all access to the DNS servers is potentially worse than letting them get overloaded). I wonder if it would be possible to express the idea that if a block being applied drops traffic well below expected levels, it must be a mistake?
- ben509 8y ago
  Is it the block causing the traffic drop, or the DDOS though?
originalsimba 8y ago
This has been a comedy of errors. People have given me a lot of grief for being down on cloudflare over the past year but trouble sure does seem to follow them around doesn't it. They launch these DNS services to much hype and hoopla and 3 weeks later they're down. I'm not surprised at all. Why does anyone throw their faith behind these gigantic corporations with bad track records? Just because they're large? Because they have a lot of money? This trend towards foolishness is bizarre to me. Recognizing these people (cloudflares, googles, facebooks, so on and so forth) for what they are is a necessary trait for survival.
baby 8y ago
I switched to 1.1.1.1 when it was released and since I’ve had multiple issues with free wifis where they would fail to hijack my dns requests to allow me to login to their portal. I Assume this is a good thing but can someone explain to me why this is happening and what’s the state on improving these wifi portals? PS: sorry for hijacking the thread.
- syntheticcdo 8y ago
  Might be unrelated. Always try http://neverssl.com http://neverssl.com to trigger the captive login on public wifi.
- woofcat 8y ago
  Set your DNS to 1.0.0.1 and it might be a bit better. Still cloudflare's DNS resolver but used by less stupid Wi-Fi portals.
- acdha 8y ago
  Cisco shipped some hardware which incorrectly used 1.1.1.1 as an internal interface. The long-term fix will be those networks being correctly configured but that may not happen until that hardware is EOLed. In the meantime, have you tried adding other addresses such as 1.0.0.1, 8.8.8.8, or 9.9.9.9 to see whether the fallback works?
IronWolve 8y ago
Doesn't seem fixed, still can't resolve archive.is cloudflare is giving off cloudflare DNS web errors. https://i.imgur.com/APpQPTJ.png https://i.imgur.com/APpQPTJ.png
- nestramutat 8y ago
  I’ve never been able to resolve some of those domains, specifically archive.il. Not a fan, why I switched away.
- acdha 8y ago
  That's apparently due to the archive.is servers returning different results based on the requesting IP: https://community.cloudflare.com/t/archive-is-error-1001/18227/3 https://community.cloudflare.com/t/archive-is-error-1001/182...
- zouhair 8y ago
  Yup now I remember why I took it off.
8_hours_ago 8y ago
I wonder if the change was manually reverted after 17 minutes, or if Cloudflare has a system that watches for a spike in failures and automatically reverts the most recent change.
jonnismash 8y ago
>The next time we mitigate 1.1.1.1 traffic, we will make sure there is a legitimate attack hitting us. I fucking love these guys
known 8y ago
Error 1001 DNS resolution error when trying to access archive.li But works with 9.9.9.9
xtrimsky1234 8y ago
This downtime annoyed me at a critical time. Even though I think it's great they are transparent about it, I don't see myself using their DNS again.

459 points by adamch 8y ago

Commendable honesty and level of detail in the public RFO.

Understandable outage. I switched back to cloudflare when it came back up, but this did prompt me to drop 8.8.8.8 in as a 3rd fallback.

gcommer 8y ago

This is a great write up. It's also why the DNS root servers have a policy of surviving DDoS through massively over-provisioned, multi-org, anycasted redundancy rather this sort of smart DDoS mitigation that drops traffic: DNS is so critical that any risk of dropping real traffic is unacceptable. (obviously, such a scale is impractical for 99% of services) A good takeaway from this outage for the average user would be to make sure that your fallback DNS resolvers are operated by totally separate providers. (eg, configure 1.1.1.1 with 8.8.8.8 as a fallback, rather than 1.1.1.1 and 1.0.0.1) (Edit: fixed cloudflare's secondary address)

Dylan16807 8y ago
(the cloudflare secondary is 1.0.0.1)
- cmurf 8y ago
  Cloudfare makes it weirdly difficult to find this. 1.1.1.1 is plastered over many pages but not concomitant with the secondary.
  - bobf 8y ago
    That's not necessarily a bad thing, since it's probably better to use another source for a secondary if that's your primary.
    - tracker1 8y ago
      I'm using cloudflare first, and google as secondary in my router. For a while I was running my own, but it became less necessary when I stopped doing as much dev at home.
  - chdefrene 8y ago
    That is not my experience at all. By following the `install`-instructions on http://1.1.1.1 http://1.1.1.1, all available DNS addresses are listed for both ipv4 and ipv6.
    - Mononokay 8y ago
      FYI, https://1.1.1.1 https://1.1.1.1 works over SSL, so there's no need to format it http://1.1.1.1 http://1.1.1.1.
berti 8y ago
Or 9.9.9.9 if you're not as comfortable with Google services.
- spondyl 8y ago
  For anyone wondering, 9.9.9.9 is https://www.quad9.net/ https://www.quad9.net/ who claim to not only resolve requests but also check them against IBM X-Force's threat intelligence database.
  - nerdponx 8y ago
    For the curious, they also have an extremely detailed privacy policy: https://www.quad9.net/policy/ https://www.quad9.net/policy/ Seems like a fair deal to me. I get free DNS service, and companies sponsoring this program get metrics on threats and general Internet usage. I'm a little skeptical of their claims that individuals can't be identified from their anonymized data. E.g. I probably only get one or two hits on my personal website every week, so it might not be hard for a malicious employee to deanonymize visitors to my site. Some highlights from the policy: Many nations classify IP addresses as Personally-Identifiable Information (PII), and we take a conservative approach in treating IP addresses as PII in all jurisdictions in which our systems reside. Our normal course of data management does not have any IP address information or other PII logged to disk or transmitted out of the location in which the query was received. We may aggregate certain counters to larger network block levels for statistical collection purposes, but those counters do not maintain specific IP address data nor is the format or model of data stored capable of being reverse-engineered to ascertain what specific IP addresses made what queries. There are exceptions to this storage model: In the event of events or observed behaviors which we deem malicious or anomalous, we may utilize more detailed logging to collect more specific IP address data in the process of normal network defense and mitigation. This collection and transmission off-site will be limited to IP addresses that we determine are involved in the event. ... We do not correlate or combine information from our logs with any personal information that you have provided Quad9 for other services, or with your specific IP address. ... Quad9 DNS Services generate and share high level anonymized aggregate statistics including threat metrics on threat type, geolocation, and if available, sector, as well as other vertical metrics including performance metrics on the Quad9 DNS Services (i.e. number of threats blocked, infrastructure uptime) when available with the Quad9 threat intelligence (TI) partners, academic researchers, or the public. Quad9 DNS Services share anonymized data on specific domains queried (records such as domain, timestamp, geolocation, number of hits, first seen, last seen) with its threat intelligence partners. Quad9 DNS Services also builds, stores, and may share certain DNS data streams which store high level information about domain resolved, query types, result codes, and timestamp. These streams do not contain IP address information of requestor and cannot be correlated to IP address or other PII. ... Quad9 does not track visitors over time and across third-party websites, and therefore does not respond to Do Not Track signaling.
h1d 8y ago
Not if you're not comfortable giving up so many of your internet activity to Google but as others have pointed out there are alternatives to spread.
bluedino 8y ago
I’m amazed at how many people abandoned other providers and blindly switched to 1.1.1.1 I can’t even use that address with the ISP Alestra in Mexico
- toomuchtodo 8y ago
  The lack of operational knowledge is real, and there is no easy fix other than resilient defaults by a benevolent dictator (in this case, resolvers across netblock and provider demarcations) or spending the time to educate yourself first on the “why” then the “how”.
crtasm 8y ago
I read on the Pi-Hole forums that 'fallback' is a misleading term because clients don't work that way - they will happily spread requests between two functioning DNS servers. Can anyone confirm this or provide further insight?
- mjcl 8y ago
  Different OSes handle it differently. Windows tries the primary, waits 1 second and then starts trying secondaries.
  - dharmab 8y ago
    glibc tries in order. musl (Alpine Linux) tries all in parallel and returns thebl first resolution (If any).
- TravelTechGuy 8y ago
  Can confirm. If I have a secondary DNS specified in my router, other than my pi-hole, it becomes useless. My guess is the router is either measuring response time, and goes with the most efficient, or otherwise round-robining the requests. Either way, requests bypass the pi-hole in such quantities that it became useless. PS: someone here mentioned that this behavior is OS-dependent - nope, this happens on the router level, and all devices in my apartment suffer.
  - detaro 8y ago
    Then your devices are likely using your router as a DNS resolver, which in turn talks to your pihole and the external one. And thus it depends on your router's OS what it does.
  - jschwartzi 8y ago
    It depends on how your router's DHCP server is configured. If you configure your router to pass its own IP address out as the DNS server for the local subnet then the router's behavior dictates how DNS works. If your router is passing out an external DNS in the DHCP configuration, then you'll get OS-dependent behavior. My router uses a DNS resolver internally, and it will spread-cast to multiple DNS servers and use the quickest response it can get. It also caches using the TTL in the DNS response, and so it will serve up cached records transparently.

throwaway2048 8y ago

another decent alternative fallback with the same featureset is quad9 https://www.quad9.net/ https://www.quad9.net/

kondro 8y ago
The core feature of Cloudflare's DNS is getting you closer and faster resolves to their CDN (probably one of the largest at this point, barring the established enterprise encumbants) in a privacy-first way. Quad9's goal seems to be about threat-detection and prevention. It would be nice if you could have both simultaneously (and maybe you can), but at the moment both services are actually quite different.
- grimgrin 8y ago
  Maybe check out what NS1 is doing with DNS https://ns1.com/ https://ns1.com/
- bigiain 8y ago
  9.9.9.10 is "quad nine without the threat detection" from memory.
  - kondro 8y ago
    Yeah, but the reason why 1.1.1.1 is so fast for sites that use Cloudflare as DNS is because Cloudflare is the authoritative DNS for them. The only way you get that in a more generic sense is if a specialist DNS CDN provider started up that provided DNS services for all the existing CDNs (or they all agreed to some type of federated standard that let them share the same recursive multicast IP addresses for DNS resolution).
    - dx034 8y ago
      Also, Cloudflare has a huge amount of data centres by now, probably more than any other service. Even Google often underperforms them. Debatable if a few ms make a difference but it can for people living in remote areas where CF has a centre and the next 9.9.9.9/8.8.8.8 is 100ms away.

saghm 8y ago

> Our FRP framework allows us to express this in clear and readable code. For example, this is part of the code responsible for performing DNS attack mitigation: > > def action_gk_dns(...): > [...] > > if port != 53: > return None > > if whitelisted_ip.get(ip): > return None > > if ip not in ANYCAST_IPS: > return None > > [...] What does this code sample have to do with FRP? This code seems extremely trivial and doesn't give any real indication to me why you'd need a framework of any sort. It seems like they really want to emphasize that they use FRP, but this code just seems completely unrelated.

deleted 8y ago
[deleted]
Chris911 8y ago
From one of the linked presentations: https://github.com/cloudflare/gatelogic https://github.com/cloudflare/gatelogic

dingo_bat 8y ago

TL;DR: we should have used an IP that is not traditionally used for testing and internal stuff by everybody including Cisco.

Dylan16807 8y ago
Not even close. The system had a glitch because they were doing a major DNS resolver at all. It had nothing to do with the baggage that comes with 1.1.1.1 specifically.
- isostatic 8y ago
  However a few days ago there was a bgp based outage on cloudfare DNs want there?
SmellyGeekBoy 8y ago
Sounds like it was truly a "DR" in your case.
sp332 8y ago
They were given the IP block by ARPA specifically with an agreement that they would analyze the junk traffic and report back on it. No one attempting this before has been able to keep up with the amount of noise pointed at that IP. Everyone involved knew exactly what they were getting into.

drawkbox 8y ago

Great information on the outage. Looks like another version 2 syndrome side effect [1]. > Today, in an effort to reclaim some technical debt, we deployed new code that introduced Gatebot to Provision API. > What we did not account for, and what Provision API didn’t know about, was that 1.1.1.0/24 and 1.0.0.0/24 are special IP ranges. Frankly speaking, almost every IP range is "special" for one reason or another, since our IP configuration is rather complex. But our recursive DNS resolver ranges are even more special: they are relatively new, and we're using them in a very unique way. Our hardcoded list of Cloudflare addresses contained a manual exception specifically for these ranges. > As you might be able to guess by now, we didn't implement this manual exception while we were doing the integration work. Remember, the whole idea of the fix was to remove the hardcoded gotchas! When porting legacy code it is not only important to understand the edge cases and technical debt built up over time, but to test more heavily in production because you never know if you got them all because some smart guy built them long ago and/or there are unknown hacks that were cornerstones of the system for better or worse. Phased and alpha/beta rollouts in an almost A/B testing way is good for replacement systems. Version 2 systems can also add new attack vectors or other single points of failure that aren't as know from legacy problems, the Provision API seems like it is a candidate for that. Over time the Version 2 system will be hardened just before it is EOL and replaced again to fix all the new problems that arise over time. Version 2's do innovate but they also shroud fixing old issues and pain points for new unknown problems. [1] https://en.wikipedia.org/wiki/Second-system_effect https://en.wikipedia.org/wiki/Second-system_effect

teddyh 8y ago
“Yes, I know, it’s just a simple function to display a window, but it has grown little hairs and stuff on it and nobody knows why. Well, I’ll tell you why: those are bug fixes. One of them fixes that bug that Nancy had when she tried to install the thing on a computer that didn’t have Internet Explorer. Another one fixes that bug that occurs in low memory conditions. Another one fixes that bug that occurred when the file is on a floppy disk and the user yanks out the disk in the middle. That LoadLibrary call is ugly but it makes the code work on old versions of Windows 95. Each of these bugs took weeks of real-world usage before they were found.” — Things You Should Never Do, Part I, (https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ https://www.joelonsoftware.com/2000/04/06/things-you-should-...)

cm2187 8y ago

Do they just use python as pseudo code or do they actually run their attack detection in python?

usmannk 8y ago
It is python. They linked a presentation[1] and a talk about the bot. [1] https://speakerdeck.com/majek04/gatelogic-somewhat-functional-reactive-framework-in-python https://speakerdeck.com/majek04/gatelogic-somewhat-functiona...
- cm2187 8y ago
  It's interesting. I would have expected them to use rather something low latency/high performance like c++ or erlang given their scale and performance criticality.
  - heavenlyblue 8y ago
    This is something that only keeps track of user settings data and consequently configures the endpoints given that data. It has got nothing to do with packet filtering per se.
askz 8y ago
Any supplementary thoughts on the fact they are running their attack detection in python? What's the point?
- Spare_account 8y ago
  There is a reply in a parallel chain (timestamped after your comment) which expands on the line of thought: https://news.ycombinator.com/item?id=17204069 https://news.ycombinator.com/item?id=17204069

decker 8y ago

I wonder if they are going to factor the availability into all the blog posts about how fast they are.

obeattie 8y ago

Major credit to Cloudflare for publishing a clear, honest, and detailed description of what happened. I wish more companies would do this. One thing I’d be interested to know more about is why it took 17 minutes to fix. While you can and always should strive to make them less likely, outages are inevitable, so how you respond is crucial. Here the outage was very obviously caused by a deployment that I’d assume was supervised by humans – why did it take 17 minutes to roll back?

blackbrokkoli 8y ago
I'm not an expert, but is 17 minutes for: - shit is not working - is this an attack? - no it's us - how? - that's how - let's go back - have to get supervisor - roll back huge thing really that long?
- dx034 8y ago
  With ~150 data centres, roll back alone probably took 5-10 minutes. Don't think 17 minutes is that long.
  - Cthulhu_ 8y ago
    For simple PagerDuty alerts I already need 15 minutes to open the app / logs and figure out what's going on.
    - wool_gather 8y ago
      Good point, although the way it's described it sounds like the problem cropped up right after deploy. So they would have been watching it actively. But as said above, 17 minutes to notice, figure out what's going on, decide what to do, and propagate the resolution seems reasonable.
  - dfcowell 8y ago
    Not to mention selectively purging all of the rules created (I assume) at every edge server in their network that were blackholing all of the traffic to the resolver. There's probably a command for it, but all in all 17 minutes seems quite a respectable turnaround time.

Robin_Message 8y ago

What's interesting here is that the automatic cure (DDoS protection) was worse than the disease (even if there was an attack, blocking all access to the DNS servers is potentially worse than letting them get overloaded). I wonder if it would be possible to express the idea that if a block being applied drops traffic well below expected levels, it must be a mistake?

ben509 8y ago
Is it the block causing the traffic drop, or the DDOS though?

originalsimba 8y ago

This has been a comedy of errors. People have given me a lot of grief for being down on cloudflare over the past year but trouble sure does seem to follow them around doesn't it. They launch these DNS services to much hype and hoopla and 3 weeks later they're down. I'm not surprised at all. Why does anyone throw their faith behind these gigantic corporations with bad track records? Just because they're large? Because they have a lot of money? This trend towards foolishness is bizarre to me. Recognizing these people (cloudflares, googles, facebooks, so on and so forth) for what they are is a necessary trait for survival.

baby 8y ago

I switched to 1.1.1.1 when it was released and since I’ve had multiple issues with free wifis where they would fail to hijack my dns requests to allow me to login to their portal. I Assume this is a good thing but can someone explain to me why this is happening and what’s the state on improving these wifi portals? PS: sorry for hijacking the thread.

syntheticcdo 8y ago
Might be unrelated. Always try http://neverssl.com http://neverssl.com to trigger the captive login on public wifi.
woofcat 8y ago
Set your DNS to 1.0.0.1 and it might be a bit better. Still cloudflare's DNS resolver but used by less stupid Wi-Fi portals.
acdha 8y ago
Cisco shipped some hardware which incorrectly used 1.1.1.1 as an internal interface. The long-term fix will be those networks being correctly configured but that may not happen until that hardware is EOLed. In the meantime, have you tried adding other addresses such as 1.0.0.1, 8.8.8.8, or 9.9.9.9 to see whether the fallback works?

IronWolve 8y ago

Doesn't seem fixed, still can't resolve archive.is cloudflare is giving off cloudflare DNS web errors. https://i.imgur.com/APpQPTJ.png https://i.imgur.com/APpQPTJ.png

nestramutat 8y ago
I’ve never been able to resolve some of those domains, specifically archive.il. Not a fan, why I switched away.
acdha 8y ago
That's apparently due to the archive.is servers returning different results based on the requesting IP: https://community.cloudflare.com/t/archive-is-error-1001/18227/3 https://community.cloudflare.com/t/archive-is-error-1001/182...
zouhair 8y ago
Yup now I remember why I took it off.

8_hours_ago 8y ago

I wonder if the change was manually reverted after 17 minutes, or if Cloudflare has a system that watches for a spike in failures and automatically reverts the most recent change.

jonnismash 8y ago

>The next time we mitigate 1.1.1.1 traffic, we will make sure there is a legitimate attack hitting us. I fucking love these guys

known 8y ago

Error 1001 DNS resolution error when trying to access archive.li But works with 9.9.9.9

xtrimsky1234 8y ago

This downtime annoyed me at a critical time. Even though I think it's great they are transparent about it, I don't see myself using their DNS again.