13 ms·
AT&T explains why it blocked Cloudflare DNS: It was just an accident
- jamiek88 9y agoWell that thread of conspiracy theories, hot air and cries of censorship yesterday was instructive. These corporations have zero benefit of the doubt level of trust. A few voices to be fair pointed out 1.1.1.1 is often incorrectly used and issues with it becoming ‘live’ mostly likely caused the outage. Not judging this response btw, I can clearly see why ATT and other big telco wouldn’t get much leeway.
- amaccuish 9y agoI wasn't one of the conspiracy theorists but I do expect ISPs to get these things right. It's literally their job, they should be experts.
- supertrope 9y agoAT&T has long provided a sub-par Internet connection service. Their wireless router and DSL modem "U-Verse" gateway device lacks a true bridge mode, has a small NAT table, and you must use it because it contains a 802.1x certificate. You can use your own router with a rather involved hack: http://blog.0xpebbles.org/Bypassing-At-t-U-verse-hardware-NAT-table-limits http://blog.0xpebbles.org/Bypassing-At-t-U-verse-hardware-NA... They instructed customers (including small businesses) to get off 10.X.X.X, presumably because they were considering CG-NAT, although they have since done some IPv6 deployment. The original AT&T faded away with declining long distance revenue which they milked to the end until cellphones and Internet access became the main market. It took tax credits meant for "video dial tone" broadband deployment and used them to build their cellular network. Its ADSL deployment use PPPoE and ATM which can cause MTU mismatch. ATM itself is a telco mistake. European access oligopoly members wanted 32 bytes because that suited voice frame transport and they thought data transport would never sell. American counterparts wanted 64 bytes for better data delivery. They compromised on 53: 48 for payload and 5 for header. With newer "U-Verse" ASDL they've finally eliminated this layer of overhead. (But no more 3rd party modems). It decided to deploy VDSL instead of make the true upgrade to fiber. But they also cut costs by not building enough remote terminals so many customers have slow and unreliable links. Fast VDSL speeds (e.g. 100Mbps) aren't possible if customers are a mile away, or more. How much money have they spent on truck rolls diagnosing issues? Is it less than just building a more robust network to begin with? Since 2005 they've been complaining that customers are using too much Internet and they need to slice and dice the Internet with data caps, application specific throttling, etc. And get more tax credits and deregulation which they promise will be used for broadband investment.
- tptacek 9y agoI've had better service from U-Verse and, more recently, AT&T Fiber than I have from Comcast. But really, the point this thread is making isn't about whether you approve of all of AT&T's business decisions, but rather whether their backbone engineers can properly manage a network. I'm interested in the opinions of people who have managed default-free BGP peering before on whether AT&T does a good job, but I'm not so much interested in reading another dslreports thread about whether AT&T is a "good ISP".
- supertrope 9y agoThis technical glitch is in their CPE boxes not the backbone network.
- tptacek 9y agoGood to know! Add to the set of people who's AT&T network engineering opinions are interesting "anyone who has managed a very large scale distribution layer network". :)
- mjcl 9y agoMy experience with AT&T was that they were overly bureaucratic but outside of those rules anything goes. For example, they didn't ask for an LOA for BGP announcements of other people's IP space. When we were turning up a new Comcast circuit we accidentally leaked Comcast's customer routes (~5000) to AT&T and they were accepted.
- paulie_a 9y agoCalling their modem/routers sub-par is being far too generous. They are pieces of shit.
- AnIdiotOnTheNet 9y agoFrom experience I can tell you that this guy is correct. Or at least he was. I had U-Verse as my first internet service at an apartment years ago and, in addition to waiting a month for them to hook it up, the modem they sent me didn't work. The tech came out (weeks later, of course), confirmed it didn't work, and proceeded to swap it with literally every single spare he had in his van. We got lucky on the last one. While chatting he admitted they were the some of the worst equipment he's ever seen. Some of the branch locations at work have U-Verse as their only option. I don't have to deal with their day-to-day problems but the help desk tells me they're still pretty crap.
- some_account 9y agoExperts get things wrong constantly.
- quakenul 9y agoWhile it's reasonable to expect them to get it right, it is obviously going to go wrong from time to time, as with experts working any other field.
- rayiner 9y agoI was just gchatting my wife in Edge, and the message entry box wraps lines on the character level (breaking a single word over two lines). Something that nobody would ever want to do and nobody could possibly think is the right behavior. Google and Microsoft develop two of the only three relevant browser engines, and indeed have basically supplanted W3C in setting web standards. But a simple text-entry field on a Google site running in Microsoft's browser doesn't work properly. So is it really surprising that AT&T handled an obscure IP address the wrong way?
- sixothree 9y agoI am not surprised if/that google degrades the experience of their products on other browsers. I believe this has been topic of discussion a number of times on HN.
- ben_jones 9y agoI think Google is aware their products perform better on Chrome and does not devote resources to correct this, leading the user experiences on Edge or even firefox to be similar but not equal. I'd make the argument that the parent should not be down voted because Google, perhaps in a small way, allows degraded experiences on other platforms for one reason or another.
- sli 9y agoThere was a short time when Google made Hangouts Chrome-only, but faking the User-Agent would make it load. Turned out, yep, Hangouts worked totally fine in Firefox. They just refused to let me load it and that's it. I don't think you can do that anymore, though. But I haven't tried it in a while. A huge problem I see with the web is that it's is missing concessions for "advanced" users. I get that we're a smaller market, but come on.
- leesalminen 9y agoI tried to use Hangouts last week with the latest Firefox on a beefy machine. While it wasn’t explicitly blocked, it froze the whole browser after 30 seconds. Never got into the meeting.
- halbritt 9y agoYeah, no. ISPs don't really employ experts any more. The experts won't work for them. They haven't done anything innovative since the heady days of the dotcom boom.
- cbsmith 9y agoYou have an amusing belief about the correlation between expertise and the lack of errors/mistakes. Also, reading your comment, I couldn't help but think of this video: https://www.youtube.com/watch?v=BKorP55Aqvg https://www.youtube.com/watch?v=BKorP55Aqvg
- monochromatic 9y agoI’m an expert at my job. I still get things wrong sometimes.
- tptacek 9y agoInstructive in the sense that for any given technical controversy on HN, there will be a couple people that have some connection to the topic and can speak with some justified confidence about it, and several hundred people who read the title and, if we're lucky, the lede graf of the story and then just write stream of consciousness extrapolations --- also, instructive to note that HN does not, ceteris paribus, do a good job of upvoting the authoritative comments over the superficial riffing.
- berg01 9y ago> ceteris paribus I enjoy googling a new phrase as much as the next person... but really? From WP: Ceteris paribus or caeteris paribus is a Latin phrase meaning "other things equal". English translations of the phrase include "all other things being equal" or "other things held constant" or "all else unchanged".
- throwaway41441 9y ago> lede graf Journalistic jargon for "lead paragraph" https://underthecurve.github.io/jekyll/update/2016/12/29/hed-dek-led-graf.html https://underthecurve.github.io/jekyll/update/2016/12/29/hed... > ceteris paribus "all other things being equal" https://en.m.wikipedia.org/wiki/Ceteris_paribus https://en.m.wikipedia.org/wiki/Ceteris_paribus
- brownbat 9y ago> HN does not, ceteris paribus, do a good job of upvoting the authoritative comments over the superficial riffing. Top comment: https://news.ycombinator.com/item?id=16982493 https://news.ycombinator.com/item?id=16982493 Seems... fine? Our comments were useful enough for Ars to apparently base their entire reporting on them. (Ok, mixed feelings there...) Or, wait, are you saying Prince's reaction was the useful, authoritative one in this example? Most commenters thought he was making a bit of a leap. Is that what you're actually pushing back on?
- notatoad 9y agoThe thread might have been fun of conspiracy theories, but the top comment the whole day was the one giving At&t the benefit of the doubt. (Fwiw, I was personally on board with the conspiracy theorists.)
- dang 9y agoIt's at least gratifying to see that the top two root comments after the dust settled were ones that got it right. Though I suppose some of the upvotes may have come in after this information was available. (Edit: I checked and it doesn't look like it.)
- kbenson 9y ago> Though I suppose some of the upvotes may have come in after this information was available. (Edit: I checked and it doesn't look like it.) Oh, man. It would be so fun to run vote analysis on comments over time. I'm jealous now. :)
- phyller 9y agodang should do a Hacker News AMA
- sillysaurus3 9y agoThe risk far outweighs the reward.
- craftyguy 9y agoWhy would you trust what AT&T is saying publicly? This could have very well been an experiment to "see if anyone noticed", folks noticed, so here's the default "we didn't mean it, you can trust us" explanation. Major ISPs generally do not act in the best interest of their customers, especially when the action is allowing access to service that hurts their ability to collect more information about their users. They certainly do not deserve the benefit of the doubt.
- notyourwork 9y agoHere come the tin foil hats. Not everything is a conspiracy. I get it these companies suck.
- craftyguy 9y agoIt's easy to dismiss comments you don't like as "tinfoil hat"-like, but it's much harder to explain why they should be suddenly trusted. Care to try?
- fourthark 9y agoDon't trust, but also don't overestimate. Incompetence is always the likelier explanation.
- craftyguy 9y agoExcept AT&T has a long history of malice under the guise of incompetence. Just today: https://www.techdirt.com/articles/20170426/05094537241/att-unveils-fake-5g-network-hopes-youll-ignore-t-mobile-is-kicking-ass.shtml https://www.techdirt.com/articles/20170426/05094537241/att-u...
- fourthark 9y agoI'm sure you can find a better example than that. That's not malicious, it's just annoying marketing.
- deleted 9y ago[deleted]
- avree 9y agoAnyone have a link to the thread?
- giancarlostoro 9y agoWell it never made sense to me why block one DNS provider and not the rest? Ah well...
- dsl 9y agoIt is obvious Cloudflare has no interest in operating a real recursive DNS service. If they did, they would have provided additional backup IPs for users they knew would be impacted by these issues. Building a production service on what even APNIC refers to as "research space" was foolish. If this was Cloudflares only line of business, they never would have taken such a risk.
- Klathmon 9y agoBut it's a fairly low risk for both cloudflare and users. Just about all systems I've worked on allow multiple DNS IP addresses, so just set an alternate. Cloudflare doesn't make any money from this (aside from the advertising it's giving them), so it doesn't really harm them that it's broken due to 3rd party abuse/issues.
- eridius 9y agoCloudflare does offer a backup IP: 1.0.0.1. And 2 IPv6 addresses as well.
- 0x0 9y agoI think that would be counter-productive. One of the main goals is to do research on how the 1.1.1.1-like addresses work in the real world. Offering a nice and free DNS service on those addresses is almost just a side-effect, which obviously triggers real-world reactions and uproar when ISPs fail to route properly. If they had offered backup addresses, maybe ISP customers wouldn't complain as loudly, and the media / attention levels would have been lower.
- eastdakota 9y agoWe (Cloudflare and APNIC) knew exactly what we were getting in to. We knew this was the only way that we'd be able to reclaim the space to be usable. And, in about a month, we've gone from 1.1.1.1 being routable by only 92% of networks to 98.7% of networks — and climbing. Don't accept that you can't change the status quo.
- dogecoinbase 9y agoThe space is allocated for research. To what end do you imagine it will be "usable", and for whom?
- abarringer 9y agoAssigning a 1.1.1.1 to a local modem interface? Cox: that ain't noth'n here hold my beer" and watch a 10.x traceroute go seven hops out the door and resolve somewhere on their network.
- icedchai 9y agoThere is a big difference: 10.x.x.x is for private use. Cox is using it privately for device management. You can also use it privately and there will be no conflict (assuming you are not attempting to manage those same devices, which is probably a safe assumption 99.99% of the time.)
- pacificmint 9y agoTo be fair, there is a network range specifically allocated for carrier grade nat (100.64/10), sounds like that might have been better to use here.
- icedchai 9y agoExcept, they're not using it for NAT. They're using it for device management (mostly CPE, if I recall.)
- MertsA 9y agoThose IP addresses are just being assigned to their own networking equipment. Cox using the CGNAT network for that would be just as wrong as you using it. There's absolutely nothing wrong with Cox using private address space internally. If anything, there's something wrong with routing requests to private address space out your WAN in the first place. What Cox is doing is completely appropriate and what AT&T and Cisco are doing is squatting on someone else's IP assignments.
- half-kh-hacker 9y agoIIRC, 1.0.0.1 was also blocked by the firmware update.
- alex_young 9y agoOK, I'm not sure I buy this explanation. If they want to revert whatever happened to 1.0.0.1, and fix 1.1.1.1, I'm fine with that. Here's what I do know: * I am an AT&T fiber customer. * 1.1.1.1 did not resolve DNS for me any time I tested it. * 1.0.0.1 did resolve DNS when I first tested it. * Neither 1.1.1.1 or 1.0.0.1 work from the external interface of my AT&T router now. * Strangely, 1.1.1.1 responds to ICMP requests from the diagnostic interface of the router, but not from the external interface. * 1.0.0.1 does not respond to ICMP requests from the diagnostic interface of the router or from the external interface. It sure looks like something upstream from my router is blocking requests to 1.0.0.1, and that this is new behavior. Like I said earlier, I don't especially care what the reason is, as long as they do something to fix it.
- vermilingua 9y agoThe reason you are getting an ICMP respose from 1.1.1.1 is that your router is responding. This is consistent with the explanation. If you still want to doubt it, clear all DNS and ARP tables, ping 1.1.1.1, and check the ARP entry for it. It will match the MAC of your routers inward facing interface.
- alex_young 9y agoFrom the same diagnostic interface: traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 38 byte packets 1 1dot1dot1dot1.cloudflare-dns.com (1.1.1.1) 0.327 ms 0.156 ms 0.120 ms
- nitrogen 9y agoThe fact that the reverse DNS is correct doesn't mean that it's reaching the real deal. The fact that the ping time is fast enough that light could only have traveled 5 or 6 miles one way means it's almost certainly not the real deal.
- alex_young 9y agoYou're right. The fact that it's only one hop should have told me the same.
- deleted 9y ago[deleted]
- _bxg1 9y agoI could believe they tried to sneak it through for malicious purposes and wrote it off as a mistake when they were caught. I could also believe it was a genuine mistake. Either way, I'm glad there are communities like Hacker News to make a stink when stuff like this happens and get it into the spotlight.
- deleted 9y ago[deleted]
- johnvega 9y agoThey need to explain why it is an accident in detail, otherwise they are just being incompetent for disrupting large number of their valued customer's internet connections, and likely wasting their valuable time. They can't just say it is unintentional, they need to convince me that it is. Technical word salad will not cut it. It has to be clear to most people.
- drawkbox 9y agoLarge ISPs wanting to get into CDN revenues and DNS, just a little hiccup to start testing the waters... ISPs are mad most people use Google or Cloudflare DNS now instead of their own which facilitate their ad networks and future CDN properties. Large ISPs want that DNS endpoint and are aiming for CDNs as well, they are gonna eventually sell it as other CDNs/DNS can't be workable due to issues like this. Large ISPs stopped innovating for decades and are mad others kept innovating on top of the network. ISPs think they own the network and are taking back this property not by innovation, but by bribes to 'representatives' and by force. Now that net neutrality is demolished and ISPs can sell your private data, they have a reason to ruin DNS and CDNs to eventually own that for tracking and revenue streams. We made IMMENSE mistakes in 2017 with allowing ISPs to bribe their way into removing privacy protections and removing net neutrality. We gave them reasons to dismantle systems built ON TOP of the network so they can own those areas by controlling the network. 2017 was regressive for the internet markets built on our internet utilities. Rather than ISPs building their own companies to compete on top of the network, they want to use their network lever to barge their way in with bribes like the Kool-aid man through walls that shouldn't be breached. ISPs are in all out war mode fighting being a utility or commodity, rather than building competitive products or innovating on top of the network, making the network better/faster/fast they want to win by force and milking it, not by building products and improvements people want. ISPs are also gunning to help build our government firewall/filter that is for censorship and IP protection [1]. AT&T in particular has been a privileged insider to the surveillance state and filtering[2]. It is also an extension of FOSTA/SESTA censorship. Turns out, building the government filter for surveillance also gets you a super efficient ad network by peering into all private data. No way blocking Cloudflare was a mistake, it was ISPs kicking the tires of their 'innovations'. In 2017, we allowed emboldened network provider monopolies to get more emboldened and now that they won, they want to run all ad networks and will do so in unison with the surveillance state and filtering/censorship. [1] https://www.wired.com/2017/04/internet-censorship-is-advancing-under-trump/ https://www.wired.com/2017/04/internet-censorship-is-advanci... [2] https://en.wikipedia.org/wiki/Room_641A https://en.wikipedia.org/wiki/Room_641A
- PMan74 9y ago> most people use Google or Cloudflare DNS now Do they? I'd say most people would be troubled to find their DNS settings let alone have the confidence to change them. I'm open to correction on that, is there any data on Google /CF DNS usage?
- magoon 9y agoI believe this. I’ve had at&t business connections that black hole certain IPs and ranges because they (incorrectly) use them internally. What’s worse is their own techs spend a large amount of time troubleshooting because it’s not clear to them.
- louky 9y agoI've got AT&T fiber using a BGW210-700 with firmware 1.5.11 and 1.1.1.1 and 1.0.0.1 have been and still are blocked for me. I'll be keeping an eye on it to see if it changes.
- bradknowles 9y agoAT&T: Oh, gee. It sure would be a shame if something happened to that nice new DNS service of yours....