33 ms·

AT&T updates firmware to block access to 1.1.1.1

999 points by antoinefink 8y ago

sxates 8y ago
Does this just apply to setting the default DNS on the router, or are the blocking traffic to 1.1.1.1 from any device connected to it?
- ReverseCold 8y ago
  I'm on ATT right now and I can't go to https://1.1.1.1 https://1.1.1.1 right now. It works fine when I disable WiFi on my phone (Verizon).
  - city41 8y ago
    I have AT&T internet and also can't get to http://1.1.1.1 http://1.1.1.1. but I can on my phone using AT&T's cellular service. Apparently not all of AT&T dislikes CloudFlare.
  - sxates 8y ago
    I have AT&T Gigabit fiber and am able to access 1.1.1.1, but perhaps I just don't have the router update yet? I'm also not using the AT&T router's wifi, but a separate router behind it.
- alex_young 8y ago
  I just tried setting DNS to CloudFlare (primary and secondary + ipv6) on my downstream router behind AT&T fiber and I can't resolve any hosts. They are explicitly blocking CF DNS.
mabbo 8y ago
I wonder if anyone has considered some sort of legislation whereby internet service providers are not allowed to block or disrupt service to certain parts of the internet in order to promote their own business model.
- bb88 8y ago
  The argument I've made is that if they're blocking certain parts of the internet, then they shouldn't be allowed to call themselves an Internet Service Provider.
  - wtallis 8y ago
    It does seem like quite a few ISPs are little more than WWW providers with partial email functionality.
  - sjm-lbm 8y ago
    I've made this argument before (and it does make some sense), but I also doubt that enough people will understand this nuance for it to really matter.
    - DoofusOfDeath 8y ago
      A court might.
      - robomc 8y ago
        but they'd just call themselves a "networking communications service provider" or something, or call themselves nothing, and people will still just use them.
        dzhiurgis 8y ago
        You can call yourself whatever you want, but these are the regulations.
        theandrewbailey 8y ago
        In the US, the internet is hardly regulated at all. ISPs can legally get away with anything.
    - 8y ago
cabaalis 8y ago
What is the likelihood of obtaining net neutrality through the courts? I.E. Cloudflare sues -> judicial process -> decision that establishes a "right to access"?
- nv-vn 8y ago
  Likely 0% chance. The court cannot just go off and make up its own laws because it wants to, all it can do is decide how existing laws should be applied.
  - derekp7 8y ago
    It's true that courts can't make laws, but they have shown a lot of leeway in the past of creatively interpreting laws (i.e., using the Interstate Commerce clause to say a farmer can't raise a particular crop to feed to his own animals). Could existing laws that prevent monopolies from unfair business practices be applied here?
- mrpippy 8y ago
  Unlikely through the courts, although state utility regulators (i.e. the California Public Utilities Commission) might take interest. Democrat members of the FCC also might be interested.
taf2 8y ago
I’m on at&t lte and this is working just fine... is this a broad band provider thing?
- outworlder 8y ago
  Yes
- city41 8y ago
  I have AT&T LTE on my phone and AT&T DSL at home. I can't get on 1.1.1.1 via DSL, but can via LTE.
fastball 8y ago
That's so crazy, I actually experienced this today. I've been using 1.1.1.1, and today went to the library for a quick work break. I pulled out my laptop and tried to connect to the wifi, and it wasn't working. After a few minutes of troubleshooting, I tried deleting my custom DNS entry in my network settings and that did the trick. I guess the library uses AT&T routers.
- j605 8y ago
  No, they use that for captive portals or broadcasts.
  - icelancer 8y ago
    Exactly. This is why 1.1.1.1 won't work on Airplane WiFi either.
exabrial 8y ago
File FCC complaints! This sorry if thing will definitely get a response.
- twexler 8y ago
  No it won't. The FCC doesn't work for the the citizenry anymore, just for lobbyists.
  - exabrial 8y ago
    I find the sappy, defeatest, whiney attitude with the FCC useless. File them if you're affected. They're cataloged and can be used as evidence in the future. The current administration is certainly against regulation, but blocking a DNS provider is an escalation. More than likely, this block is due to incompetence. My guess is ATT was using the IP internally for some purpose and is now getting DDOS'd.
  - jessaustin 8y ago
    Anymore? Have they ever made a regulatory decision with citizens' interests in mind? The abortive never-implemented two-year dalliance with Title II doesn't count.
m-p-3 8y ago
I'm wondering what CloudFlare response will be to this.
kev009 8y ago
Knowing how bad most telco networks are operated, I blithely wonder if maybe they were using stuff in 1./8 as PNI or some other privileged internal net and are going through some oh shit moments. Hanlon's razor as lots of DNS services are available on not as vanity IP space, and there is no evidence of blockage.
jedberg 8y ago
My guess is this is just incompetence and not intentionally made to block CloudFlare. I have one of those routers, and I couldn't use 1.1.1.1 because it was routing to an internal interface on the router. I confirmed this with ping, I was getting microsecond response times from 1.1.1.1. Under the new firmware, 1.1.1.1 is just dead. So it's probably still connected to the local interface, and nothing is listening.
- alex_young 8y ago
  They started blocking 1.0.0.1 and CF ipv6 DNS too. This has to be intentional.
  - jedberg 8y ago
    Hmmm that's a fair point. But then why not also block 8.8.8.8?
    - stonemetal 8y ago
      Everyone has heard of google, attacking them would cause back lash. Non technical people haven't heard of cloudflare so it is a softer target.
    - sjwright 8y ago
      Probably because they know they would never be able to convince anyone that it was a technical bug and not malice. A surprising number of people seem to be convinced this was unintentional.
    - anti_cf 8y ago
      At least google doesn't block rarbg or thepiratebay. Good riddance to CF.
jaas 8y ago
I'd say there is a 98% chance this is a bug in some firmware and a 2% chance AT&T is intentionally trying to block Cloudflare DNS. I get why people are paranoid about ISPs blocking content and net neutrality, but let's not cry wolf prematurely. The technical details here strongly suggest a bug rather than intentional blocking of 1.1.1.1 DNS traffic.
- droopybuns 8y ago
  Can someone link to the firmware? It shouldn't be hard to binwalk this and figure out wtf is happening. Also- If this was intentional- I'm betting they'd filter it for the mobile network as well. This has got to be a fuck-up.
- bbayles 8y ago
  You're absolutely right about this. This is almost certainly just there to block people who mistakenly paste in an example configuration somewhere. Back in 2010 there were problems that came up when IANA started allocating out of 1.0.0.0/8 (e.g. [1]). Things that were once assumed to be unused started being used, leading to strange issues. Also, why on earth would AT&T block 1.1.1.1 and not Google DNS and OpenDNS? [1] https://bgpmon.net/issues-with-allocating-from-1-0-0-08/ https://bgpmon.net/issues-with-allocating-from-1-0-0-08/
  - vvanders 8y ago
    According to the thread the timing on this looks pretty bad since those DNS IPs were previously working on the earlier firmware.
    - adventured 8y ago
      How would it make sense to block it only on a small fraction of their entire network? It wouldn't accomplish anything.
  - chillingeffect 8y ago
    when 1.1.1.1 was first announced a few weeks ago, many people pointed out at the time that it was already blocked because so many people had effectively polluted it by over-using it for demo examples and testing traffic. CF announced they knew this and intended to do a project analyzing the data. Perhaps this done, whether conveniently or not, with the same intention. We'll see if they reverse it.
intrasight 8y ago
As a consumer, you are free to switch to a different provider. I'm not saying what they're doing is ok, but let's not neglect the opportunity to vote with our $$$.
- jedberg 8y ago
  Only if you're in the 42% minority of people who have access to more than one ISP.
- DINKDINK 8y ago
  You assume no state-granted ISP monopoly.
- zxcvhjkl 8y ago
  I wish it was true. Where I live it’s either Comcast or ATT. Comcast will happily block ports to “protect” me: https://www.xfinity.com/support/articles/list-of-blocked-ports https://www.xfinity.com/support/articles/list-of-blocked-por...
js2 8y ago
This is likely due to incompetence, not malice. FWIW, it’s possible to bypass AT&T’s router: https://github.com/jaysoffian/eap_proxy https://github.com/jaysoffian/eap_proxy That said, I tried 1.1.1.1 and found I had to switch back to Google DNS since Cloudflare intentionally doesn’t support EDNS Client Subnet which was causing my AppleTV’s to have trouble loading content.
- djsumdog 8y ago
  I've been meaning to try eap_proxy for a while. I've seen it mentioned several times. My ATT router doesn't get in my way enough to bother with it yet, but it still pisses me off they won't let me use a 10.x range at home. Also I've heard that their routers report your entire network topology back when they phone home.
  - deleted 8y ago
    [deleted]
- IshKebab 8y ago
  Can you not just put the router in bridge mode and use a different sane one? In the UK Virgin forces you to use their moderately shit modem/router, but even that lets you use bridge mode.
  - ben1040 8y ago
    The AT&T gateway doesn't offer a bridge mode. The best you get is a "passthrough" mode where a router behind it will get assigned the public-facing IP so it doesn't ultimately behave like double-NAT, but the gateway still maintains an internal NAT table for everything going across. You can't just use your own VDSL modem or plug your own router into the fiber ONT, as AT&T uses 802.1x auth and the key is burned into the gateway hardware.
- ReverseCold 8y ago
  I don't know much about networking, but I do have that router. Can you please explain what this does/why someone would want this?
cyanbane 8y ago
Do they block quad9? Although I trust AT&T about as far as I can throw them, this may just be a bad config/update.
_bxg1 8y ago
Jesus Christ. Fortunately I only have AT&T on mobile and it still works there, but I will ditch them in a heartbeat if that changes. At least in the cellular space there's still some consumer choice to be had.
dingo_bat 8y ago
Good. If cloud fare is allowed to block sites from their hosting service based on opinions, then att should be allowed to do the same. Also fuck cloud fare for choosing 1.1.1.1 when any network engineer worth his salt would have told them it's going to cause problems. There are things like conventions and traditions, you break them at your own peril.
- 24gttghh 8y ago
  >APNIC's research group held the IP addresses 1.1.1.1 and 1.0.0.1. While the addresses were valid, so many people had entered them into various random systems that they were continuously overwhelmed by a flood of garbage traffic. APNIC wanted to study this garbage traffic but any time they'd tried to announce the IPs, the flood would overwhelm any conventional network. >We talked to the APNIC team about how we wanted to create a privacy-first, extremely fast DNS system. They thought it was a laudable goal. We offered Cloudflare's network to receive and study the garbage traffic in exchange for being able to offer a DNS resolver on the memorable IPs. And, with that, 1.1.1.1 was born.[0] [0]https://blog.cloudflare.com/announcing-1111/ https://blog.cloudflare.com/announcing-1111/ It's not a reserved address like 192.168.0.0/16 or 10.0.0.0/8[1][2], nor is it one of the other reserved addresses for documentation or testing. So I think people using it before as test or LAN addresses are actually in the wrong here. This kind of "tradition" in networking is wrong. That's what things like RFC's are for. [1]https://tools.ietf.org/html/rfc5735 https://tools.ietf.org/html/rfc5735 [2]https://www.iana.org/assignments/iana-ipv4-special-registry/iana-ipv4-special-registry.xhtml https://www.iana.org/assignments/iana-ipv4-special-registry/...
  - dingo_bat 8y ago
    > It's not a reserved address I know. That's why I wrote "tradition" instead of the RFC numbers. Way to miss my point though.
    - 24gttghh 8y ago
      You seem to miss my point, in that Cloudflare specifically chose that IP in order to share research data with APNIC regarding people erroneously using 1.1.1.1 in the wild. Just because something is a tradition doesn't make it a right course of action.
johnvega 8y ago
If at&t does not provide any official explanation, what's your opinion on how people should respond. The first thing that came to mind for me is to switch over to Xfinity on my next contract cycle.
- zxcvhjkl 8y ago
  It’s not like Comcast is any better. They block ports: https://www.xfinity.com/support/articles/list-of-blocked-ports https://www.xfinity.com/support/articles/list-of-blocked-por...
- mmilano 8y ago
  Boycott all AT&T services we can if they are blocking. Mobile + DirecTV too.
- mr_spothawk 8y ago
  Breaking the contract is a reasonable option, maybe? At scale & among people who can afford to (ahem, HN) openly refuse I'd argue it could have more immediate impact. Frankly, NEVER paying the bill is an option, too. Downloading Netflix is sweet, maybe you can pool with your neighbor? that's another topic It's expensive to enforce payment. If you've never been in collections, it's an experience you might enjoy for sport. If you live in fear of not being able to get a cheap interest rate on a loan for some shit you don't need... well, maybe you'd better not take part in that type of protest.
thetwentyone 8y ago
FWIW as an ATT Fiber customer, I was not able to (and am still not able to) access 1.1.1.1. I tried just a couple days after Cloudflare announced the service, and requests timed out. I can access with a VPN, however.
tuna-piano 8y ago
Here is the original Cloudflare post on what 1.1.1.1 is [1]. For those who don't know, 1.1.1.1 is Cloudflare's privacy focused DNS service. That means that when you type in www.google.com, that URL can be sent to 1.1.1.1, and then 1.1.1.1 resolves that URL an IP address and send the IP back to the user. All user requests are then sent to the IP address, not the URL. Supposedly this is better than using the DNS server of ATT+Comcast, because ATT+Comcast want your browsing history while Cloudflare does not. What I don't understand is how this really helps user privacy much. If AT&T, Comcast, etc want to know your browsing habits, can't they still see the IP addresses you're browsing and figure out the URL from the IPs? I can't see that as too big an impediment, but maybe someone with more knowledge can share. [1] https://blog.cloudflare.com/announcing-1111/ https://blog.cloudflare.com/announcing-1111/
- ben174 8y ago
  Got a link to the original hacker news article by chance? I’d like to see the comments
  - jessaustin 8y ago
    Use the search at the bottom of this page: https://hn.algolia.com/?query=announcing%201111&type=story https://hn.algolia.com/?query=announcing%201111&type=story
  - lightbyte 8y ago
    https://news.ycombinator.com/item?id=16727869 https://news.ycombinator.com/item?id=16727869
- nebulous1 8y ago
  One point: the whole URL doesn't get sent to the DNS server, just the domain name. Regarding privacy, Cloudflare are at least saying they aren't spying on you. Your ISP may not even be saying this. Also, Cloudflare don't necessarily have access to your name and address, whereas your ISP does. Also, many different sites can be hosted on the same IP address, so merely tracking the IP addresses a client is connected to won't necessarily tell you what sites they're visiting.
walrus01 8y ago
There are an astonishing number of corporate end users also using "unused" chunks /8 sized of IP space internally. As if rfc1918 wasn't big enough.
Abishek_Muthian 8y ago
Anyone else facing such issues with their ISP for 1.1.1.1 ? Cloudflare DNS seems to be down for couple of major ISP's in India as well according to CF forums - [ACT] https://community.cloudflare.com/t/cloudfare-dns-blocked-with-act-isp-in-india/16916/10 https://community.cloudflare.com/t/cloudfare-dns-blocked-wit... [Airtel] https://community.cloudflare.com/t/cloudflare-dns-not-working-in-india-isp-airtel-may-have-blocked-it/16419 https://community.cloudflare.com/t/cloudflare-dns-not-workin...
- NoCFHere 8y ago
  I'm on comcast and can't seem to ping em.
mstaoru 8y ago
Shanghai. One of the largest Chinese data-centers with direct peering to all major national networks. I'm inside, testing a new colocation unit we just put there. Pinging 1.1.1.1 in 4.2ms, wow! Putting it in resolv.conf. Nothing works. WTF? Turns out they route 1.1.1.1 across the whole DC to one of their internal services "for engineers' convenience". Not gonna change. TIC.
aosmith 8y ago
Is this only DSL? I have ATT fiber, no problems here.
- thetwentyone 8y ago
  I experience this on ATT fiber. ping'ing 1.1.1.1 times out.
  - aosmith 8y ago
    Woof that sucks. I'm in the Bay Area, where are you?
- __david__ 8y ago
  I have their fiber, too. 1.1.1.1 doesn't work (and didn't work the day of the announcement), but 1.0.0.1 worked and continues to work.
arriu 8y ago
While far from perfect, for anyone looking for a temporary solution, run pi-hole on a remote server and have it use 1.1.1.1 as its DNS. You'll get the benefit of pi-hole blocking ads.
cottsak 8y ago
How is the ISP performing this remote update? Is it TR-069/CWMP or an open SSH port or something? Many routers will allow the user to disable TR-069 even while it's running. Often a hardware reset will also disable it and then the user can put the manufactures update on it and prevent the ISP from managing it in the future. If it's an open SSH port then we all have bigger problems.
- sean8102 8y ago
  AT&T's internet service requires you to use one of THEIR "gateways". Which is a combination modem and wireless router. When AT&T wants a new gateway they go to a company (mainly Arris now) and have them build a gateway that will only be for AT&T to deploy . AT&T completely controls the software/firmware on the device. There is no site you can go to and download a "manufacturer" firmware. Even if you could it wouldn't accept it because it wouldn't be signed by AT&T. And yes AT&T uses CWMP to remotely manage the gateway. That's how they can send firmware updates, customer service can retrieve signal stats, remotely reboot the gateway etc etc. And no they certainly do not put in a option on the gateway to disable CWMP or any of the remote management stuff they use. You can turn off the Wi-Fi on AT&T's gateway and run your own router behind the AT&T hardware. But since your router is behind the gateway everything still goes through it and AT&T still can do all the CWMP stuff to their gateway.
cottsak 8y ago
This problem has been around for a while and is pretty serious! https://www.routersecurity.org/ISProuters.php https://www.routersecurity.org/ISProuters.php
exabrial 8y ago
My parent company uses 1.1.1.1 as a captive portal address on the guest network. Easy to remember, but cloudflare probably needs to stand up some more conventional DNS ips.
- auscompgeek 8y ago
  Cloudflare already have IPv6.
- Klathmon 8y ago
  No your parent company needs to stop abusing that IP. Cloudflare is using a conventional IP, you are the one that isn't.
  - exabrial 8y ago
    I wasn't disagreeing...? They're using an IP that wasn't assigned by IANA.
    - fuzzy2 8y ago
      What exactly do you mean by “wasn’t assigned”? According to this article [1], 1/8 was reserved in 1981. Only from 2008 to 2010 was 1.1.1.0/24 ever truly unallocated. If, after 8 years, most providers still haven’t moved to either private networks or officially assigned networks, honestly – they suck. [1]: https://labs.ripe.net/Members/franz/content-pollution-18 https://labs.ripe.net/Members/franz/content-pollution-18
cottsak 8y ago
More solid advice regarding home internet/ISP routers: https://www.tomsguide.com/us/home-router-security,news-19245.html https://www.tomsguide.com/us/home-router-security,news-19245... Try to avoid the cheap bundled cable/fibre/DSL routers that ISPs "throw in" with their plans/packages. Disable the remote management/update/TR-069/CWMP/SSH/etc if you can. You don't wanna trust someone else to secure your home.
aosmith 8y ago
Is there any reason you couldn't just tunnel / proxy your DNS? I know that isn't an option for most people but I think that would solve the problem.
okket 8y ago
Sure this is intentional? The headline suggests so, otherwise "AT&T firmware update blocks access to 1.1.1.1" would be more accurate IMHO.
PinkMilkshake 8y ago
This is going to reveal my lack of networking knowledge but how does a company get an IP like 1.1.1.1? A bucket load of cash?
- y2kenny 8y ago
  Help from / collaboration with APNIC: https://blog.cloudflare.com/announcing-1111/ https://blog.cloudflare.com/announcing-1111/
  - utefan001 8y ago
    Geoff Huston is the chief scientist at APNIC. I highly recommend checking out some of his talks on yourtube.
- kchr 8y ago
  Busted!
deleted 8y ago
[deleted]
chrissnell 8y ago
Some folks use a Ubiquiti EdgeRouter and a user-space proxy to forward EAP (authentication) packets to the AT&T router but otherwise use the EdgeRouter to route LAN traffic out to the ONT (fiber to Ethernet translator) and the internet, thus bypassing the shitty AT&T router for most stuff. This would be sufficient to ensure that 1.1.1.1 is reachable. It's not a good solution for me, however, because I run PFSense, which is FreeBSD-based and lacks the PF_RING socket support to filter out those EAP packets. As far as I know, PFSense's PF packet filter cannot strain them out, either. Traditional libpcap is available on FreeBSD (slow) and netmap (fast), too. I looked into writing an EAP proxy in Go using a special netmap-enabled libpcap but it was way too much yak shaving and I eventually gave up. I should take another look, or maybe learn enough C to do it natively with netmap. My goal is native EAP proxy support for PFSense that can support filtering EAP out of a wirespeed gigabit fiber connection.
hamandcheese 8y ago
I really wish Cloudflare would have used a "normal" IP for their DNS service. That way there would be no confusion whatsoever as to whether this is malicious or a bug.
- skrause 8y ago
  1.1.1.1 is a normal IP.
  - regecks 8y ago
    > The Cloudflare-APNIC experiment uses two IPv4 address ranges, 1.1.1/24 and 1.0.0/24, which have been reserved for research use. Cloudflare's new DNS uses two addresses within those ranges, 1.1.1.1 and 1.0.0.1. They had acknowledged to themselves going into it that the IPs weren't "normal". They could have easily chosen a safer range if that was a priority.
    - slenk 8y ago
      1.1.1.1 is a normal IP as it was reserved for internet use. There are already IP ranges that are supposed to be used for internal use, and 1.1.1.1 is not one of them
      - plopz 8y ago
        This whole thing is like the .dev TLD debacle.
  - hamandcheese 8y ago
    1.1.1.1 is widely known to be a dumping ground of random traffic, as well as a common internal address for captive portals and whatnot. The entire rollout of 1.1.1.1 has been characterized by legacy bugs and misconfigurations of network hardware preventing its proper use. Regardless of what the standards say about 1.1.1.1, it was a poor choice if Cloudflare values widespread adoption.
justinzollars 8y ago
I'm going to ask for a partial refund every month if they are blocking parts of the internet.
- kchr 8y ago
  Better check the EULA first...
- robin_reala 8y ago
  They’re blocking 1 of 4,294,967,296 addresses, so I guess they’d argue that your bill should be reduced by 0.000000232%.
waldbeere 8y ago
hong-kong airport free Wifi 1.1.1.1 not works with this DNS
xtf 8y ago
From https://en.wikipedia.org/wiki/1.1.1.1#Criticism_and_problems https://en.wikipedia.org/wiki/1.1.1.1#Criticism_and_problems : Technological websites noted that by using 1.1.1.1 as the IP address for their service, Cloudflare created problems with existing setups. While 1.1.1.1 was not a reserved IP address, it was and is used by many existing routers (mostly those sold by Cisco Systems) and companies for hosting login pages to private networks, exit pages or other purposes, rendering the use of 1.1.1.1 as a manually configured DNS server impossible on those systems. Additionally, 1.1.1.1 is blocked on many networks and by multiple ISPs because the simplicity of the address means that it was previously often used for testing purposes and not legitimate use. These previous uses has lead to a huge influx of "garbage" data to Cloudflare's servers.
- fuzzy2 8y ago
  That’s intentional, from what I remember. All non-DNS traffic is analyzed for research purposes (not by Cloudflare though). A wake-up call for all those (ab)users of public address space is also desperately needed. All IPv4 addresses will soon be allocated. Failure to use only private address spaces will cause problems, very soon.
- rbanffy 8y ago
  What kind of demented person uses 1.1.1.1, a routable public address since 2010, for internal addresses. What's wrong with 10.0.0.0/8 or 192.168/16?
  - jdironman 8y ago
    I'm gonna guess they valued the aesthetics over the problems / conditions.
    - SmellyGeekBoy 8y ago
      You'd think in that case they'd have gone with 10.10.10.10. Silly people.
    - slenk 8y ago
sitepodmatt 8y ago
It shocks me that there are no AT&T network/sysadmins at the right level and department on this forum that don't cringe in shame and sort this out.
- gk1 8y ago
  I'm certain there are, but AT&T is a 250,000-person organization with a bureaucracy to match. Things take weeks to sort out, assuming the right person is pushing for it.
pedrosanta 8y ago
I would cancel any broadband contract of any ISP that did this when providing me a service. We need to stand up to these sort of things. (Disclaimer: I live in Europe though.)
- justherefortart 8y ago
  You wouldn't when you don't have a better alternative.
netsec_burn 8y ago
This isn't malice. AT&T has an internal IP they assigned to 1.1.1.1 because it was unused and they used it as an image caching proxy so it browsing the internet would feel faster on early phones. I've seen it when I was reverse engineering on Android a while back.
- masklinn 8y ago
  So it's not just malice but doubly so: they used an IP they didn't have the rights to and they're now blocking proper users of it.
  - jamespo 8y ago
    "Never attribute to malice that which is adequately explained by stupidity"
    - paulie_a 8y ago
      Except when it is AT&T. Then you can just assume it was more malicious than you even originally thought
      - jdironman 8y ago
        Equal parts malice and stupidity.
    - Aissen 8y ago
      It's stupidity, then malice as a cover-up.
      - rando444 8y ago
        No, that's not what malice means. Unless you're actually trying to say that AT&T has a grudge against Cloudflare and are only doing this to harm their company. This is something more like negligence or gross negligence.
mdip 8y ago
I'll go on record as saying I am an ardent hater of U-Verse and AT&T due to personal experience with their service and would like nothing more than for this to be a purposeful act that would result in backlash on that company... ... that said, I'm going to fall in the camp of stating that this is likely an unintentional bug. If they truly wanted to block 1.1.1.1 (and it's backup), doing so via firmware would seem to be the most difficult and unreliable way of doing so. The benefits of doing so are also limited: (a) If the motivation was to avoid losing the ability to spy on their customers via DNS requests, well ... they can still do that. Yes, Cloudflare supports encrypted DNS, but the half of one percent of folks who have this set up wouldn't be worth the effort[0]. (b) If there was some other reason to want customers using their DNS (i.e. redirection to advertising pages when lookup fails), they could simply do packet rewrites (of non-encrypted DNS lookups) to send them over to AT&Ts infrastructure -- the benefit of doing this is that it would be more likely to go unnoticed[1]. (c) There have been several other, far more popular and just as well publicized public DNS services that they haven't messed with -- why pick on a new entrant -- why not break 8.8.8.8 or OpenDNS? More likely is the explanation that 1.1.1.1 was being used as a defact-o 10.x.x.x address for other purposes. It had a few benefits -- it was far less likely to be used as an internal address for customers (being ... not a traditional non-routable address) and up until recently, it was unlikely to be used for legitimate services. Or ... it's something else. Firmware bugs are everywhere and having had their service and the particular brand of modem they're using, I'm not the least bit surprised. I had to root my modem to make my service work reliably[2]. Heck, I worked for a telecom for 17 years, and the first half of that, the guy who set our network up used 1-10.x.x.x as internal addresses. [0] It's not terribly difficult to do, but few take the effort. I've got an internal DNS server configured (for AD purposes) which forwards to another internal DNS server that makes all DNS requests out to cloudflare via encrypted DNS. It was a 5 minute change to my internal setup, a lot of which was the time it took to download the container, reboot the host for testing purposes and validation of everything. [1] It probably would have managed to be hidden an entire minute longer than this debacle. [2] On their DSL (re-labeled U-Verse despite it having nothing to do with their U-Verse TV/Internet -- it's the old DSL limited to 12Mb down if you're lucky), my modem would randomly display the "Internet is down" page for all requests despite everything being fine. I forgot, exactly, what I had to do to resolve it, but it required hitting their ping page to trigger a buffer overflow, allowing me to get console access and running some command. I also wanted to be able to ping the modem remotely (something they disable with no customer-facing option to correct) to correlate it with weather so as to prove to customer service (...and at least a little to myself) that this bizarre happenstance wasn't all in my head. My next-door neighbors also had this problem, so I suspected it was something in the wiring (expansion/contraction-like) up the street, but it was hard to track down where because all but two people on that street (including us) used those homes as summer vacation homes and were rarely there in the winter -- many didn't have service and those who did were unlikely to be around when the weather hit about 40 degrees, so AT&T wasn't getting reports of outages in enough frequency to do anything about it. Two years ago, they sent a truck, took everyone down and re-did a pole 8 houses down. Since then, the problem hasn't happened.
ryan-c 8y ago
Late to the party, but here's some traceroutes run from AT&T Gigapower with their router entirely bypassed via an 802.1x MitM: # traceroute 1.0.0.1 traceroute to 1.0.0.1 (1.0.0.1), 30 hops max, 60 byte packets 1 45-18-124-1.lightspeed.austtx.sbcglobal.net (45.18.124.1) 59.462 ms 61.348 ms 63.373 ms 2 71.149.77.208 (71.149.77.208) 1.304 ms 1.695 ms 1.957 ms 3 75.8.128.136 (75.8.128.136) 1.329 ms 1.682 ms 1.393 ms 4 12.83.68.145 (12.83.68.145) 2.673 ms 2.661 ms 2.648 ms 5 12.123.18.233 (12.123.18.233) 8.877 ms 12.753 ms 8.800 ms 6 192.205.36.206 (192.205.36.206) 6.663 ms 6.375 ms 6.680 ms 7 66.110.56.158 (66.110.56.158) 6.885 ms 6.725 ms 6.436 ms 8 1dot1dot1dot1.cloudflare-dns.com (1.0.0.1) 6.855 ms 6.557 ms 6.662 ms # traceroute 1.1.1.1 traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 60 byte packets 1 45-18-124-1.lightspeed.austtx.sbcglobal.net (45.18.124.1) 163.322 ms 163.927 ms 174.243 ms 2 71.149.77.208 (71.149.77.208) 1.346 ms 1.779 ms 2.035 ms 3 75.8.128.136 (75.8.128.136) 1.215 ms 1.214 ms 1.564 ms 4 12.83.68.137 (12.83.68.137) 1.495 ms 12.83.68.145 (12.83.68.145) 2.289 ms 12.83.68.137 (12.83.68.137) 2.283 ms 5 12.123.18.233 (12.123.18.233) 7.783 ms 11.766 ms 11.757 ms 6 192.205.36.206 (192.205.36.206) 6.163 ms 6.160 ms 6.202 ms 7 66.110.56.158 (66.110.56.158) 6.909 ms 6.931 ms 6.423 ms 8 1dot1dot1dot1.cloudflare-dns.com (1.1.1.1) 6.922 ms 6.492 ms 7.075 ms ; <<>> DiG 9.9.5-9+deb8u14-Debian <<>> cloudflare.com @1.1.1.1 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 15100 ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1536 ;; QUESTION SECTION: ;cloudflare.com. IN A ;; ANSWER SECTION: cloudflare.com. 53 IN A 198.41.214.162 cloudflare.com. 53 IN A 198.41.215.162 ;; Query time: 7 msec ;; SERVER: 1.1.1.1#53(1.1.1.1) ;; WHEN: Thu May 03 13:40:52 UTC 2018 ;; MSG SIZE rcvd: 75 ; <<>> DiG 9.9.5-9+deb8u14-Debian <<>> cloudflare.com @1.0.0.1 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 61685 ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1536 ;; QUESTION SECTION: ;cloudflare.com. IN A ;; ANSWER SECTION: cloudflare.com. 66 IN A 198.41.214.162 cloudflare.com. 66 IN A 198.41.215.162 ;; Query time: 7 msec ;; SERVER: 1.0.0.1#53(1.0.0.1) ;; WHEN: Thu May 03 13:40:39 UTC 2018 ;; MSG SIZE rcvd: 75 I'm not going to paste the output, but `curl https://1.1.1.1/` https://1.1.1.1/` works as well. Doesn't look like it's anything onn AT&T's internal network.
JumpCrisscross 8y ago
What about AT&T's wireless network?
jacksmith21006 8y ago
Is there anyone before Google that went after getting one of these marketing IPs? First time I saw it was 8.8.8.8. I personally had one had in my head from the 80s 128.252.120.1. bit it is obviously not a special one.
akshatkedia 8y ago
1.1.1.1 blocked in India too on BSNL connections.
AgentK20 8y ago
Cloudflare's CEO confirms: https://twitter.com/eastdakota/status/991718955021623296 https://twitter.com/eastdakota/status/991718955021623296
- noobermin 8y ago
  How is this not illegal?
  - awat 8y ago
    That’s what I want to know. I’ll save the soapbox speech and just leave it at isn’t this why monopoly laws exist?
    - nv-vn 8y ago
      The issue is that the cable companies have monopolies set in law already. There are numerous regulations designed to stop any new last-mile telecom companies from starting up, which literally guarantees a monopoly for the few companies that already exist in the vast majority of the US. As good as Net Neutrality sounds in theory, all we really need to do is drop the regulations and allow new players to enter the game and the market will fix it for itself.
      - deleted 8y ago
        [deleted]
    - komali2 8y ago
      Unenforced laws might as well not exist. See: Monopoly/duopoly of telecom in the USA, net neutrality protections, and the speed limit signs on 280.
      - mr_spothawk 8y ago
        > Unenforced laws might as well not exist. until they are selectively enforced.
  - 8y ago
techjuice 8y ago
I always thought it was strange to see the example loopback address listed as 1.1.1.1 or 1.xxx.xxx.xxx in many of tutorials and official network certification guides and why they did not use a private. This is more than likely why many users are having problems because they are being routed to a loopback address on their router or another router. Hopefully network admins and engineers will choose a non public ip space as their loopback address to resolve the problem.
- outworlder 8y ago
  Yes, they were not the sharpest crayons in the box. But CloudFlare has addressed this with lots of vendors. This update is a new one.
- codetrotter 8y ago
  Indeed. I wish most people used TEST-NET-1, TEST-NET-2 and TEST-NET-3 in documentation and training material. RFC 5735: > 192.0.2.0/24 - This block is assigned as "TEST-NET-1" for use in documentation and example code. It is often used in conjunction with domain names example.com or example.net in vendor and protocol documentation. As described in RFC5737, addresses within this block do not legitimately appear on the public Internet and can be used without any coordination with IANA or an Internet registry. > 198.51.100.0/24 - This block is assigned as "TEST-NET-2" for use in documentation and example code. It is often used in conjunction with domain names example.com or example.net in vendor and protocol documentation. As described in RFC5737, addresses within this block do not legitimately appear on the public Internet and can be used without any coordination with IANA or an Internet registry. > 203.0.113.0/24 - This block is assigned as "TEST-NET-3" for use in documentation and example code. It is often used in conjunction with domain names example.com or example.net in vendor and protocol documentation. As described in RFC5737, addresses within this block do not legitimately appear on the public Internet and can be used without any coordination with IANA or an Internet registry. https://tools.ietf.org/html/rfc5735 https://tools.ietf.org/html/rfc5735
  - deleted 8y ago
    [deleted]
bvinc 8y ago
How are they going to spy on your DNS traffic and sell it to advertisers after you secure it?
- ddtaylor 8y ago
  For most people who aren't configuring DNSec or TLS can't the ISP still see all of the plain-text domain names in port 53 traffic?
  - ZoF 8y ago
    Yes, they can; regardless of your resolver they can collect that if you're not using DNSec. Same goes for https handshakes leaking your target domain(otherwise SNI wouldn't work), so DNSSec alone is fairly pointless for regular web traffic obfuscation; and of course the IP is in each TCP frame regardless. It becomes more a matter of are they doing it yet(re:DNS monitoring in this manner); with enough people using third party resolvers(I'd argue google's public DNS already has enough usage to warrant it) they will be. Optimally you'd VPN at all times to a provider you trust or one you've setup yourself. What it all really boils down to though is that the populace simply can't be trusted(nor should they need to be) to make themselves acceptably secure from third party monitoring. We need to have much more discussion around data privacy and retention for ISP's. It's not a matter of if the data will be misused, it's truly a matter of when and it's not fair to the general public.
  - tptacek 8y ago
    DNSSEC doesn’t encrypt DNS traffic; it only signs it.
    - ddtaylor 8y ago
      Derp, good point.

999 points by antoinefink 8y ago

Does this just apply to setting the default DNS on the router, or are the blocking traffic to 1.1.1.1 from any device connected to it?

ReverseCold 8y ago
I'm on ATT right now and I can't go to https://1.1.1.1 https://1.1.1.1 right now. It works fine when I disable WiFi on my phone (Verizon).
- city41 8y ago
  I have AT&T internet and also can't get to http://1.1.1.1 http://1.1.1.1. but I can on my phone using AT&T's cellular service. Apparently not all of AT&T dislikes CloudFlare.
- sxates 8y ago
  I have AT&T Gigabit fiber and am able to access 1.1.1.1, but perhaps I just don't have the router update yet? I'm also not using the AT&T router's wifi, but a separate router behind it.
alex_young 8y ago
I just tried setting DNS to CloudFlare (primary and secondary + ipv6) on my downstream router behind AT&T fiber and I can't resolve any hosts. They are explicitly blocking CF DNS.

mabbo 8y ago

I wonder if anyone has considered some sort of legislation whereby internet service providers are not allowed to block or disrupt service to certain parts of the internet in order to promote their own business model.

bb88 8y ago
The argument I've made is that if they're blocking certain parts of the internet, then they shouldn't be allowed to call themselves an Internet Service Provider.
- wtallis 8y ago
  It does seem like quite a few ISPs are little more than WWW providers with partial email functionality.
- sjm-lbm 8y ago
  I've made this argument before (and it does make some sense), but I also doubt that enough people will understand this nuance for it to really matter.
  - DoofusOfDeath 8y ago
    A court might.
    - robomc 8y ago
      but they'd just call themselves a "networking communications service provider" or something, or call themselves nothing, and people will still just use them.
      - dzhiurgis 8y ago
        You can call yourself whatever you want, but these are the regulations.
        theandrewbailey 8y ago
        In the US, the internet is hardly regulated at all. ISPs can legally get away with anything.
  - 8y ago

cabaalis 8y ago

What is the likelihood of obtaining net neutrality through the courts? I.E. Cloudflare sues -> judicial process -> decision that establishes a "right to access"?

nv-vn 8y ago
Likely 0% chance. The court cannot just go off and make up its own laws because it wants to, all it can do is decide how existing laws should be applied.
- derekp7 8y ago
  It's true that courts can't make laws, but they have shown a lot of leeway in the past of creatively interpreting laws (i.e., using the Interstate Commerce clause to say a farmer can't raise a particular crop to feed to his own animals). Could existing laws that prevent monopolies from unfair business practices be applied here?
mrpippy 8y ago
Unlikely through the courts, although state utility regulators (i.e. the California Public Utilities Commission) might take interest. Democrat members of the FCC also might be interested.

taf2 8y ago

I’m on at&t lte and this is working just fine... is this a broad band provider thing?

outworlder 8y ago
Yes
city41 8y ago
I have AT&T LTE on my phone and AT&T DSL at home. I can't get on 1.1.1.1 via DSL, but can via LTE.

fastball 8y ago

That's so crazy, I actually experienced this today. I've been using 1.1.1.1, and today went to the library for a quick work break. I pulled out my laptop and tried to connect to the wifi, and it wasn't working. After a few minutes of troubleshooting, I tried deleting my custom DNS entry in my network settings and that did the trick. I guess the library uses AT&T routers.

j605 8y ago
No, they use that for captive portals or broadcasts.
- icelancer 8y ago
  Exactly. This is why 1.1.1.1 won't work on Airplane WiFi either.

exabrial 8y ago

File FCC complaints! This sorry if thing will definitely get a response.

twexler 8y ago
No it won't. The FCC doesn't work for the the citizenry anymore, just for lobbyists.
- exabrial 8y ago
  I find the sappy, defeatest, whiney attitude with the FCC useless. File them if you're affected. They're cataloged and can be used as evidence in the future. The current administration is certainly against regulation, but blocking a DNS provider is an escalation. More than likely, this block is due to incompetence. My guess is ATT was using the IP internally for some purpose and is now getting DDOS'd.
- jessaustin 8y ago
  Anymore? Have they ever made a regulatory decision with citizens' interests in mind? The abortive never-implemented two-year dalliance with Title II doesn't count.

m-p-3 8y ago

I'm wondering what CloudFlare response will be to this.

kev009 8y ago

Knowing how bad most telco networks are operated, I blithely wonder if maybe they were using stuff in 1./8 as PNI or some other privileged internal net and are going through some oh shit moments. Hanlon's razor as lots of DNS services are available on not as vanity IP space, and there is no evidence of blockage.

jedberg 8y ago

My guess is this is just incompetence and not intentionally made to block CloudFlare. I have one of those routers, and I couldn't use 1.1.1.1 because it was routing to an internal interface on the router. I confirmed this with ping, I was getting microsecond response times from 1.1.1.1. Under the new firmware, 1.1.1.1 is just dead. So it's probably still connected to the local interface, and nothing is listening.

alex_young 8y ago
They started blocking 1.0.0.1 and CF ipv6 DNS too. This has to be intentional.
- jedberg 8y ago
  Hmmm that's a fair point. But then why not also block 8.8.8.8?
  - stonemetal 8y ago
    Everyone has heard of google, attacking them would cause back lash. Non technical people haven't heard of cloudflare so it is a softer target.
  - sjwright 8y ago
    Probably because they know they would never be able to convince anyone that it was a technical bug and not malice. A surprising number of people seem to be convinced this was unintentional.
  - anti_cf 8y ago
    At least google doesn't block rarbg or thepiratebay. Good riddance to CF.

jaas 8y ago

I'd say there is a 98% chance this is a bug in some firmware and a 2% chance AT&T is intentionally trying to block Cloudflare DNS. I get why people are paranoid about ISPs blocking content and net neutrality, but let's not cry wolf prematurely. The technical details here strongly suggest a bug rather than intentional blocking of 1.1.1.1 DNS traffic.

droopybuns 8y ago
Can someone link to the firmware? It shouldn't be hard to binwalk this and figure out wtf is happening. Also- If this was intentional- I'm betting they'd filter it for the mobile network as well. This has got to be a fuck-up.
bbayles 8y ago
You're absolutely right about this. This is almost certainly just there to block people who mistakenly paste in an example configuration somewhere. Back in 2010 there were problems that came up when IANA started allocating out of 1.0.0.0/8 (e.g. [1]). Things that were once assumed to be unused started being used, leading to strange issues. Also, why on earth would AT&T block 1.1.1.1 and not Google DNS and OpenDNS? [1] https://bgpmon.net/issues-with-allocating-from-1-0-0-08/ https://bgpmon.net/issues-with-allocating-from-1-0-0-08/
- vvanders 8y ago
  According to the thread the timing on this looks pretty bad since those DNS IPs were previously working on the earlier firmware.
  - adventured 8y ago
    How would it make sense to block it only on a small fraction of their entire network? It wouldn't accomplish anything.
- chillingeffect 8y ago
  when 1.1.1.1 was first announced a few weeks ago, many people pointed out at the time that it was already blocked because so many people had effectively polluted it by over-using it for demo examples and testing traffic. CF announced they knew this and intended to do a project analyzing the data. Perhaps this done, whether conveniently or not, with the same intention. We'll see if they reverse it.

intrasight 8y ago

As a consumer, you are free to switch to a different provider. I'm not saying what they're doing is ok, but let's not neglect the opportunity to vote with our $$$.

jedberg 8y ago
Only if you're in the 42% minority of people who have access to more than one ISP.
DINKDINK 8y ago
You assume no state-granted ISP monopoly.
zxcvhjkl 8y ago
I wish it was true. Where I live it’s either Comcast or ATT. Comcast will happily block ports to “protect” me: https://www.xfinity.com/support/articles/list-of-blocked-ports https://www.xfinity.com/support/articles/list-of-blocked-por...

js2 8y ago

This is likely due to incompetence, not malice. FWIW, it’s possible to bypass AT&T’s router: https://github.com/jaysoffian/eap_proxy https://github.com/jaysoffian/eap_proxy That said, I tried 1.1.1.1 and found I had to switch back to Google DNS since Cloudflare intentionally doesn’t support EDNS Client Subnet which was causing my AppleTV’s to have trouble loading content.

djsumdog 8y ago
I've been meaning to try eap_proxy for a while. I've seen it mentioned several times. My ATT router doesn't get in my way enough to bother with it yet, but it still pisses me off they won't let me use a 10.x range at home. Also I've heard that their routers report your entire network topology back when they phone home.
- deleted 8y ago
  [deleted]
IshKebab 8y ago
Can you not just put the router in bridge mode and use a different sane one? In the UK Virgin forces you to use their moderately shit modem/router, but even that lets you use bridge mode.
- ben1040 8y ago
  The AT&T gateway doesn't offer a bridge mode. The best you get is a "passthrough" mode where a router behind it will get assigned the public-facing IP so it doesn't ultimately behave like double-NAT, but the gateway still maintains an internal NAT table for everything going across. You can't just use your own VDSL modem or plug your own router into the fiber ONT, as AT&T uses 802.1x auth and the key is burned into the gateway hardware.
ReverseCold 8y ago
I don't know much about networking, but I do have that router. Can you please explain what this does/why someone would want this?

cyanbane 8y ago

Do they block quad9? Although I trust AT&T about as far as I can throw them, this may just be a bad config/update.

_bxg1 8y ago

Jesus Christ. Fortunately I only have AT&T on mobile and it still works there, but I will ditch them in a heartbeat if that changes. At least in the cellular space there's still some consumer choice to be had.

dingo_bat 8y ago

Good. If cloud fare is allowed to block sites from their hosting service based on opinions, then att should be allowed to do the same. Also fuck cloud fare for choosing 1.1.1.1 when any network engineer worth his salt would have told them it's going to cause problems. There are things like conventions and traditions, you break them at your own peril.

24gttghh 8y ago
>APNIC's research group held the IP addresses 1.1.1.1 and 1.0.0.1. While the addresses were valid, so many people had entered them into various random systems that they were continuously overwhelmed by a flood of garbage traffic. APNIC wanted to study this garbage traffic but any time they'd tried to announce the IPs, the flood would overwhelm any conventional network. >We talked to the APNIC team about how we wanted to create a privacy-first, extremely fast DNS system. They thought it was a laudable goal. We offered Cloudflare's network to receive and study the garbage traffic in exchange for being able to offer a DNS resolver on the memorable IPs. And, with that, 1.1.1.1 was born.[0] [0]https://blog.cloudflare.com/announcing-1111/ https://blog.cloudflare.com/announcing-1111/ It's not a reserved address like 192.168.0.0/16 or 10.0.0.0/8[1][2], nor is it one of the other reserved addresses for documentation or testing. So I think people using it before as test or LAN addresses are actually in the wrong here. This kind of "tradition" in networking is wrong. That's what things like RFC's are for. [1]https://tools.ietf.org/html/rfc5735 https://tools.ietf.org/html/rfc5735 [2]https://www.iana.org/assignments/iana-ipv4-special-registry/iana-ipv4-special-registry.xhtml https://www.iana.org/assignments/iana-ipv4-special-registry/...
- dingo_bat 8y ago
  > It's not a reserved address I know. That's why I wrote "tradition" instead of the RFC numbers. Way to miss my point though.
  - 24gttghh 8y ago
    You seem to miss my point, in that Cloudflare specifically chose that IP in order to share research data with APNIC regarding people erroneously using 1.1.1.1 in the wild. Just because something is a tradition doesn't make it a right course of action.

johnvega 8y ago

If at&t does not provide any official explanation, what's your opinion on how people should respond. The first thing that came to mind for me is to switch over to Xfinity on my next contract cycle.

zxcvhjkl 8y ago
It’s not like Comcast is any better. They block ports: https://www.xfinity.com/support/articles/list-of-blocked-ports https://www.xfinity.com/support/articles/list-of-blocked-por...
mmilano 8y ago
Boycott all AT&T services we can if they are blocking. Mobile + DirecTV too.
mr_spothawk 8y ago
Breaking the contract is a reasonable option, maybe? At scale & among people who can afford to (ahem, HN) openly refuse I'd argue it could have more immediate impact. Frankly, NEVER paying the bill is an option, too. Downloading Netflix is sweet, maybe you can pool with your neighbor? that's another topic It's expensive to enforce payment. If you've never been in collections, it's an experience you might enjoy for sport. If you live in fear of not being able to get a cheap interest rate on a loan for some shit you don't need... well, maybe you'd better not take part in that type of protest.

thetwentyone 8y ago

FWIW as an ATT Fiber customer, I was not able to (and am still not able to) access 1.1.1.1. I tried just a couple days after Cloudflare announced the service, and requests timed out. I can access with a VPN, however.

tuna-piano 8y ago

Here is the original Cloudflare post on what 1.1.1.1 is [1]. For those who don't know, 1.1.1.1 is Cloudflare's privacy focused DNS service. That means that when you type in www.google.com, that URL can be sent to 1.1.1.1, and then 1.1.1.1 resolves that URL an IP address and send the IP back to the user. All user requests are then sent to the IP address, not the URL. Supposedly this is better than using the DNS server of ATT+Comcast, because ATT+Comcast want your browsing history while Cloudflare does not. What I don't understand is how this really helps user privacy much. If AT&T, Comcast, etc want to know your browsing habits, can't they still see the IP addresses you're browsing and figure out the URL from the IPs? I can't see that as too big an impediment, but maybe someone with more knowledge can share. [1] https://blog.cloudflare.com/announcing-1111/ https://blog.cloudflare.com/announcing-1111/

ben174 8y ago
Got a link to the original hacker news article by chance? I’d like to see the comments
- jessaustin 8y ago
  Use the search at the bottom of this page: https://hn.algolia.com/?query=announcing%201111&type=story https://hn.algolia.com/?query=announcing%201111&type=story
- lightbyte 8y ago
  https://news.ycombinator.com/item?id=16727869 https://news.ycombinator.com/item?id=16727869
nebulous1 8y ago
One point: the whole URL doesn't get sent to the DNS server, just the domain name. Regarding privacy, Cloudflare are at least saying they aren't spying on you. Your ISP may not even be saying this. Also, Cloudflare don't necessarily have access to your name and address, whereas your ISP does. Also, many different sites can be hosted on the same IP address, so merely tracking the IP addresses a client is connected to won't necessarily tell you what sites they're visiting.

walrus01 8y ago

There are an astonishing number of corporate end users also using "unused" chunks /8 sized of IP space internally. As if rfc1918 wasn't big enough.

Abishek_Muthian 8y ago

Anyone else facing such issues with their ISP for 1.1.1.1 ? Cloudflare DNS seems to be down for couple of major ISP's in India as well according to CF forums - [ACT] https://community.cloudflare.com/t/cloudfare-dns-blocked-with-act-isp-in-india/16916/10 https://community.cloudflare.com/t/cloudfare-dns-blocked-wit... [Airtel] https://community.cloudflare.com/t/cloudflare-dns-not-working-in-india-isp-airtel-may-have-blocked-it/16419 https://community.cloudflare.com/t/cloudflare-dns-not-workin...

NoCFHere 8y ago
I'm on comcast and can't seem to ping em.

mstaoru 8y ago

Shanghai. One of the largest Chinese data-centers with direct peering to all major national networks. I'm inside, testing a new colocation unit we just put there. Pinging 1.1.1.1 in 4.2ms, wow! Putting it in resolv.conf. Nothing works. WTF? Turns out they route 1.1.1.1 across the whole DC to one of their internal services "for engineers' convenience". Not gonna change. TIC.

aosmith 8y ago

Is this only DSL? I have ATT fiber, no problems here.

thetwentyone 8y ago
I experience this on ATT fiber. ping'ing 1.1.1.1 times out.
- aosmith 8y ago
  Woof that sucks. I'm in the Bay Area, where are you?
__david__ 8y ago
I have their fiber, too. 1.1.1.1 doesn't work (and didn't work the day of the announcement), but 1.0.0.1 worked and continues to work.

arriu 8y ago

While far from perfect, for anyone looking for a temporary solution, run pi-hole on a remote server and have it use 1.1.1.1 as its DNS. You'll get the benefit of pi-hole blocking ads.

cottsak 8y ago

How is the ISP performing this remote update? Is it TR-069/CWMP or an open SSH port or something? Many routers will allow the user to disable TR-069 even while it's running. Often a hardware reset will also disable it and then the user can put the manufactures update on it and prevent the ISP from managing it in the future. If it's an open SSH port then we all have bigger problems.

sean8102 8y ago
AT&T's internet service requires you to use one of THEIR "gateways". Which is a combination modem and wireless router. When AT&T wants a new gateway they go to a company (mainly Arris now) and have them build a gateway that will only be for AT&T to deploy . AT&T completely controls the software/firmware on the device. There is no site you can go to and download a "manufacturer" firmware. Even if you could it wouldn't accept it because it wouldn't be signed by AT&T. And yes AT&T uses CWMP to remotely manage the gateway. That's how they can send firmware updates, customer service can retrieve signal stats, remotely reboot the gateway etc etc. And no they certainly do not put in a option on the gateway to disable CWMP or any of the remote management stuff they use. You can turn off the Wi-Fi on AT&T's gateway and run your own router behind the AT&T hardware. But since your router is behind the gateway everything still goes through it and AT&T still can do all the CWMP stuff to their gateway.

cottsak 8y ago

This problem has been around for a while and is pretty serious! https://www.routersecurity.org/ISProuters.php https://www.routersecurity.org/ISProuters.php

exabrial 8y ago

My parent company uses 1.1.1.1 as a captive portal address on the guest network. Easy to remember, but cloudflare probably needs to stand up some more conventional DNS ips.

auscompgeek 8y ago
Cloudflare already have IPv6.
Klathmon 8y ago
No your parent company needs to stop abusing that IP. Cloudflare is using a conventional IP, you are the one that isn't.
- exabrial 8y ago
  I wasn't disagreeing...? They're using an IP that wasn't assigned by IANA.
  - fuzzy2 8y ago
    What exactly do you mean by “wasn’t assigned”? According to this article [1], 1/8 was reserved in 1981. Only from 2008 to 2010 was 1.1.1.0/24 ever truly unallocated. If, after 8 years, most providers still haven’t moved to either private networks or officially assigned networks, honestly – they suck. [1]: https://labs.ripe.net/Members/franz/content-pollution-18 https://labs.ripe.net/Members/franz/content-pollution-18

cottsak 8y ago

More solid advice regarding home internet/ISP routers: https://www.tomsguide.com/us/home-router-security,news-19245.html https://www.tomsguide.com/us/home-router-security,news-19245... Try to avoid the cheap bundled cable/fibre/DSL routers that ISPs "throw in" with their plans/packages. Disable the remote management/update/TR-069/CWMP/SSH/etc if you can. You don't wanna trust someone else to secure your home.

aosmith 8y ago

Is there any reason you couldn't just tunnel / proxy your DNS? I know that isn't an option for most people but I think that would solve the problem.

okket 8y ago

Sure this is intentional? The headline suggests so, otherwise "AT&T firmware update blocks access to 1.1.1.1" would be more accurate IMHO.

PinkMilkshake 8y ago

This is going to reveal my lack of networking knowledge but how does a company get an IP like 1.1.1.1? A bucket load of cash?

y2kenny 8y ago
Help from / collaboration with APNIC: https://blog.cloudflare.com/announcing-1111/ https://blog.cloudflare.com/announcing-1111/
- utefan001 8y ago
  Geoff Huston is the chief scientist at APNIC. I highly recommend checking out some of his talks on yourtube.
kchr 8y ago
Busted!

deleted 8y ago

[deleted]

chrissnell 8y ago

Some folks use a Ubiquiti EdgeRouter and a user-space proxy to forward EAP (authentication) packets to the AT&T router but otherwise use the EdgeRouter to route LAN traffic out to the ONT (fiber to Ethernet translator) and the internet, thus bypassing the shitty AT&T router for most stuff. This would be sufficient to ensure that 1.1.1.1 is reachable. It's not a good solution for me, however, because I run PFSense, which is FreeBSD-based and lacks the PF_RING socket support to filter out those EAP packets. As far as I know, PFSense's PF packet filter cannot strain them out, either. Traditional libpcap is available on FreeBSD (slow) and netmap (fast), too. I looked into writing an EAP proxy in Go using a special netmap-enabled libpcap but it was way too much yak shaving and I eventually gave up. I should take another look, or maybe learn enough C to do it natively with netmap. My goal is native EAP proxy support for PFSense that can support filtering EAP out of a wirespeed gigabit fiber connection.

hamandcheese 8y ago

I really wish Cloudflare would have used a "normal" IP for their DNS service. That way there would be no confusion whatsoever as to whether this is malicious or a bug.

skrause 8y ago
1.1.1.1 is a normal IP.
- regecks 8y ago
  > The Cloudflare-APNIC experiment uses two IPv4 address ranges, 1.1.1/24 and 1.0.0/24, which have been reserved for research use. Cloudflare's new DNS uses two addresses within those ranges, 1.1.1.1 and 1.0.0.1. They had acknowledged to themselves going into it that the IPs weren't "normal". They could have easily chosen a safer range if that was a priority.
  - slenk 8y ago
    1.1.1.1 is a normal IP as it was reserved for internet use. There are already IP ranges that are supposed to be used for internal use, and 1.1.1.1 is not one of them
    - plopz 8y ago
      This whole thing is like the .dev TLD debacle.
- hamandcheese 8y ago
  1.1.1.1 is widely known to be a dumping ground of random traffic, as well as a common internal address for captive portals and whatnot. The entire rollout of 1.1.1.1 has been characterized by legacy bugs and misconfigurations of network hardware preventing its proper use. Regardless of what the standards say about 1.1.1.1, it was a poor choice if Cloudflare values widespread adoption.

justinzollars 8y ago

I'm going to ask for a partial refund every month if they are blocking parts of the internet.

kchr 8y ago
Better check the EULA first...
robin_reala 8y ago
They’re blocking 1 of 4,294,967,296 addresses, so I guess they’d argue that your bill should be reduced by 0.000000232%.

waldbeere 8y ago

hong-kong airport free Wifi 1.1.1.1 not works with this DNS

xtf 8y ago

From https://en.wikipedia.org/wiki/1.1.1.1#Criticism_and_problems https://en.wikipedia.org/wiki/1.1.1.1#Criticism_and_problems : Technological websites noted that by using 1.1.1.1 as the IP address for their service, Cloudflare created problems with existing setups. While 1.1.1.1 was not a reserved IP address, it was and is used by many existing routers (mostly those sold by Cisco Systems) and companies for hosting login pages to private networks, exit pages or other purposes, rendering the use of 1.1.1.1 as a manually configured DNS server impossible on those systems. Additionally, 1.1.1.1 is blocked on many networks and by multiple ISPs because the simplicity of the address means that it was previously often used for testing purposes and not legitimate use. These previous uses has lead to a huge influx of "garbage" data to Cloudflare's servers.

fuzzy2 8y ago
That’s intentional, from what I remember. All non-DNS traffic is analyzed for research purposes (not by Cloudflare though). A wake-up call for all those (ab)users of public address space is also desperately needed. All IPv4 addresses will soon be allocated. Failure to use only private address spaces will cause problems, very soon.
rbanffy 8y ago
What kind of demented person uses 1.1.1.1, a routable public address since 2010, for internal addresses. What's wrong with 10.0.0.0/8 or 192.168/16?
- jdironman 8y ago
  I'm gonna guess they valued the aesthetics over the problems / conditions.
  - SmellyGeekBoy 8y ago
    You'd think in that case they'd have gone with 10.10.10.10. Silly people.
  - slenk 8y ago

sitepodmatt 8y ago

It shocks me that there are no AT&T network/sysadmins at the right level and department on this forum that don't cringe in shame and sort this out.

gk1 8y ago
I'm certain there are, but AT&T is a 250,000-person organization with a bureaucracy to match. Things take weeks to sort out, assuming the right person is pushing for it.

pedrosanta 8y ago

I would cancel any broadband contract of any ISP that did this when providing me a service. We need to stand up to these sort of things. (Disclaimer: I live in Europe though.)

justherefortart 8y ago
You wouldn't when you don't have a better alternative.

netsec_burn 8y ago

This isn't malice. AT&T has an internal IP they assigned to 1.1.1.1 because it was unused and they used it as an image caching proxy so it browsing the internet would feel faster on early phones. I've seen it when I was reverse engineering on Android a while back.

masklinn 8y ago
So it's not just malice but doubly so: they used an IP they didn't have the rights to and they're now blocking proper users of it.
- jamespo 8y ago
  "Never attribute to malice that which is adequately explained by stupidity"
  - paulie_a 8y ago
    Except when it is AT&T. Then you can just assume it was more malicious than you even originally thought
    - jdironman 8y ago
      Equal parts malice and stupidity.
  - Aissen 8y ago
    It's stupidity, then malice as a cover-up.
    - rando444 8y ago
      No, that's not what malice means. Unless you're actually trying to say that AT&T has a grudge against Cloudflare and are only doing this to harm their company. This is something more like negligence or gross negligence.

mdip 8y ago

I'll go on record as saying I am an ardent hater of U-Verse and AT&T due to personal experience with their service and would like nothing more than for this to be a purposeful act that would result in backlash on that company... ... that said, I'm going to fall in the camp of stating that this is likely an unintentional bug. If they truly wanted to block 1.1.1.1 (and it's backup), doing so via firmware would seem to be the most difficult and unreliable way of doing so. The benefits of doing so are also limited: (a) If the motivation was to avoid losing the ability to spy on their customers via DNS requests, well ... they can still do that. Yes, Cloudflare supports encrypted DNS, but the half of one percent of folks who have this set up wouldn't be worth the effort[0]. (b) If there was some other reason to want customers using their DNS (i.e. redirection to advertising pages when lookup fails), they could simply do packet rewrites (of non-encrypted DNS lookups) to send them over to AT&Ts infrastructure -- the benefit of doing this is that it would be more likely to go unnoticed[1]. (c) There have been several other, far more popular and just as well publicized public DNS services that they haven't messed with -- why pick on a new entrant -- why not break 8.8.8.8 or OpenDNS? More likely is the explanation that 1.1.1.1 was being used as a defact-o 10.x.x.x address for other purposes. It had a few benefits -- it was far less likely to be used as an internal address for customers (being ... not a traditional non-routable address) and up until recently, it was unlikely to be used for legitimate services. Or ... it's something else. Firmware bugs are everywhere and having had their service and the particular brand of modem they're using, I'm not the least bit surprised. I had to root my modem to make my service work reliably[2]. Heck, I worked for a telecom for 17 years, and the first half of that, the guy who set our network up used 1-10.x.x.x as internal addresses. [0] It's not terribly difficult to do, but few take the effort. I've got an internal DNS server configured (for AD purposes) which forwards to another internal DNS server that makes all DNS requests out to cloudflare via encrypted DNS. It was a 5 minute change to my internal setup, a lot of which was the time it took to download the container, reboot the host for testing purposes and validation of everything. [1] It probably would have managed to be hidden an entire minute longer than this debacle. [2] On their DSL (re-labeled U-Verse despite it having nothing to do with their U-Verse TV/Internet -- it's the old DSL limited to 12Mb down if you're lucky), my modem would randomly display the "Internet is down" page for all requests despite everything being fine. I forgot, exactly, what I had to do to resolve it, but it required hitting their ping page to trigger a buffer overflow, allowing me to get console access and running some command. I also wanted to be able to ping the modem remotely (something they disable with no customer-facing option to correct) to correlate it with weather so as to prove to customer service (...and at least a little to myself) that this bizarre happenstance wasn't all in my head. My next-door neighbors also had this problem, so I suspected it was something in the wiring (expansion/contraction-like) up the street, but it was hard to track down where because all but two people on that street (including us) used those homes as summer vacation homes and were rarely there in the winter -- many didn't have service and those who did were unlikely to be around when the weather hit about 40 degrees, so AT&T wasn't getting reports of outages in enough frequency to do anything about it. Two years ago, they sent a truck, took everyone down and re-did a pole 8 houses down. Since then, the problem hasn't happened.

ryan-c 8y ago

Late to the party, but here's some traceroutes run from AT&T Gigapower with their router entirely bypassed via an 802.1x MitM: # traceroute 1.0.0.1 traceroute to 1.0.0.1 (1.0.0.1), 30 hops max, 60 byte packets 1 45-18-124-1.lightspeed.austtx.sbcglobal.net (45.18.124.1) 59.462 ms 61.348 ms 63.373 ms 2 71.149.77.208 (71.149.77.208) 1.304 ms 1.695 ms 1.957 ms 3 75.8.128.136 (75.8.128.136) 1.329 ms 1.682 ms 1.393 ms 4 12.83.68.145 (12.83.68.145) 2.673 ms 2.661 ms 2.648 ms 5 12.123.18.233 (12.123.18.233) 8.877 ms 12.753 ms 8.800 ms 6 192.205.36.206 (192.205.36.206) 6.663 ms 6.375 ms 6.680 ms 7 66.110.56.158 (66.110.56.158) 6.885 ms 6.725 ms 6.436 ms 8 1dot1dot1dot1.cloudflare-dns.com (1.0.0.1) 6.855 ms 6.557 ms 6.662 ms # traceroute 1.1.1.1 traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 60 byte packets 1 45-18-124-1.lightspeed.austtx.sbcglobal.net (45.18.124.1) 163.322 ms 163.927 ms 174.243 ms 2 71.149.77.208 (71.149.77.208) 1.346 ms 1.779 ms 2.035 ms 3 75.8.128.136 (75.8.128.136) 1.215 ms 1.214 ms 1.564 ms 4 12.83.68.137 (12.83.68.137) 1.495 ms 12.83.68.145 (12.83.68.145) 2.289 ms 12.83.68.137 (12.83.68.137) 2.283 ms 5 12.123.18.233 (12.123.18.233) 7.783 ms 11.766 ms 11.757 ms 6 192.205.36.206 (192.205.36.206) 6.163 ms 6.160 ms 6.202 ms 7 66.110.56.158 (66.110.56.158) 6.909 ms 6.931 ms 6.423 ms 8 1dot1dot1dot1.cloudflare-dns.com (1.1.1.1) 6.922 ms 6.492 ms 7.075 ms ; <<>> DiG 9.9.5-9+deb8u14-Debian <<>> cloudflare.com @1.1.1.1 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 15100 ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1536 ;; QUESTION SECTION: ;cloudflare.com. IN A ;; ANSWER SECTION: cloudflare.com. 53 IN A 198.41.214.162 cloudflare.com. 53 IN A 198.41.215.162 ;; Query time: 7 msec ;; SERVER: 1.1.1.1#53(1.1.1.1) ;; WHEN: Thu May 03 13:40:52 UTC 2018 ;; MSG SIZE rcvd: 75 ; <<>> DiG 9.9.5-9+deb8u14-Debian <<>> cloudflare.com @1.0.0.1 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 61685 ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1536 ;; QUESTION SECTION: ;cloudflare.com. IN A ;; ANSWER SECTION: cloudflare.com. 66 IN A 198.41.214.162 cloudflare.com. 66 IN A 198.41.215.162 ;; Query time: 7 msec ;; SERVER: 1.0.0.1#53(1.0.0.1) ;; WHEN: Thu May 03 13:40:39 UTC 2018 ;; MSG SIZE rcvd: 75 I'm not going to paste the output, but `curl https://1.1.1.1/` https://1.1.1.1/` works as well. Doesn't look like it's anything onn AT&T's internal network.

JumpCrisscross 8y ago

What about AT&T's wireless network?

jacksmith21006 8y ago

Is there anyone before Google that went after getting one of these marketing IPs? First time I saw it was 8.8.8.8. I personally had one had in my head from the 80s 128.252.120.1. bit it is obviously not a special one.

akshatkedia 8y ago

1.1.1.1 blocked in India too on BSNL connections.

AgentK20 8y ago

Cloudflare's CEO confirms: https://twitter.com/eastdakota/status/991718955021623296 https://twitter.com/eastdakota/status/991718955021623296

noobermin 8y ago
How is this not illegal?
- awat 8y ago
  That’s what I want to know. I’ll save the soapbox speech and just leave it at isn’t this why monopoly laws exist?
  - nv-vn 8y ago
    The issue is that the cable companies have monopolies set in law already. There are numerous regulations designed to stop any new last-mile telecom companies from starting up, which literally guarantees a monopoly for the few companies that already exist in the vast majority of the US. As good as Net Neutrality sounds in theory, all we really need to do is drop the regulations and allow new players to enter the game and the market will fix it for itself.
    - deleted 8y ago
      [deleted]
  - komali2 8y ago
    Unenforced laws might as well not exist. See: Monopoly/duopoly of telecom in the USA, net neutrality protections, and the speed limit signs on 280.
    - mr_spothawk 8y ago
      > Unenforced laws might as well not exist. until they are selectively enforced.
- 8y ago

techjuice 8y ago

I always thought it was strange to see the example loopback address listed as 1.1.1.1 or 1.xxx.xxx.xxx in many of tutorials and official network certification guides and why they did not use a private. This is more than likely why many users are having problems because they are being routed to a loopback address on their router or another router. Hopefully network admins and engineers will choose a non public ip space as their loopback address to resolve the problem.

outworlder 8y ago
Yes, they were not the sharpest crayons in the box. But CloudFlare has addressed this with lots of vendors. This update is a new one.
codetrotter 8y ago
Indeed. I wish most people used TEST-NET-1, TEST-NET-2 and TEST-NET-3 in documentation and training material. RFC 5735: > 192.0.2.0/24 - This block is assigned as "TEST-NET-1" for use in documentation and example code. It is often used in conjunction with domain names example.com or example.net in vendor and protocol documentation. As described in RFC5737, addresses within this block do not legitimately appear on the public Internet and can be used without any coordination with IANA or an Internet registry. > 198.51.100.0/24 - This block is assigned as "TEST-NET-2" for use in documentation and example code. It is often used in conjunction with domain names example.com or example.net in vendor and protocol documentation. As described in RFC5737, addresses within this block do not legitimately appear on the public Internet and can be used without any coordination with IANA or an Internet registry. > 203.0.113.0/24 - This block is assigned as "TEST-NET-3" for use in documentation and example code. It is often used in conjunction with domain names example.com or example.net in vendor and protocol documentation. As described in RFC5737, addresses within this block do not legitimately appear on the public Internet and can be used without any coordination with IANA or an Internet registry. https://tools.ietf.org/html/rfc5735 https://tools.ietf.org/html/rfc5735
- deleted 8y ago
  [deleted]

bvinc 8y ago

How are they going to spy on your DNS traffic and sell it to advertisers after you secure it?

ddtaylor 8y ago
For most people who aren't configuring DNSec or TLS can't the ISP still see all of the plain-text domain names in port 53 traffic?
- ZoF 8y ago
  Yes, they can; regardless of your resolver they can collect that if you're not using DNSec. Same goes for https handshakes leaking your target domain(otherwise SNI wouldn't work), so DNSSec alone is fairly pointless for regular web traffic obfuscation; and of course the IP is in each TCP frame regardless. It becomes more a matter of are they doing it yet(re:DNS monitoring in this manner); with enough people using third party resolvers(I'd argue google's public DNS already has enough usage to warrant it) they will be. Optimally you'd VPN at all times to a provider you trust or one you've setup yourself. What it all really boils down to though is that the populace simply can't be trusted(nor should they need to be) to make themselves acceptably secure from third party monitoring. We need to have much more discussion around data privacy and retention for ISP's. It's not a matter of if the data will be misused, it's truly a matter of when and it's not fair to the general public.
- tptacek 8y ago
  DNSSEC doesn’t encrypt DNS traffic; it only signs it.
  - ddtaylor 8y ago
    Derp, good point.