4 ms·

What happened with your Internet last night, and what we are doing about it

41 points by hw 11y ago

taspeotis 11y ago
What a generous window of time... This $5 credit offer is being extended to our customers with active XFINITY Internet service in California, Washington State, and Tucson, Arizona who were affected by the issue with our DNS servers that was first reported on 6/1/15. The form above will be available until 6/11/15
- yeukhon 11y ago
  I agree... They should just add that credit automatically...
  - cpeterso 11y ago
    Seriously. They know which customers are in the affected areas because they have their billing addresses. Just credit them $5 on their next bill.
- Redoubts 11y ago
  Further: Please allow 4 to 6 weeks for the credit to be posted to your Comcast account.
  - zatkin 11y ago
    So, they refund you $5 in Comcast credit that will eventually come out of your next bill?
    - Yhippa 11y ago
      What if this incident caused you to switch to another ISP prior to said bill. Based on my experiences good luck getting that $5.
      - jedmeyers 11y ago
        "Switch to another ISP"? That's a funny joke.
    - deleted 11y ago
      [deleted]
- bby 11y ago
  Fuck Comcast. Opt-in trick to fuck over more people.
- reagency 11y ago
  Comcast exists to prove that government isn't always less efficient at providing service than the private sector.
  - baddox 11y ago
    The rule of thumb is that government tends to be less efficient than private markets. The residential ISP "market" isn't much of a market in lots of places. Comcast tends to operate as a monopoly or near-monopoly.
  - adventured 11y ago
    I'd have to wonder at the product we'd get if we swapped out Comcast and let the Federal Government take over their business. I think it would cost net twice as much (total expenditures out of the consumer pocket via taxation, the plans wouldn't cost more), be about as reliable, about as fast, lose lots of money and still be completely unaccountable to its customers. The system / infrastructure upgrades would be slow, bureaucratic, always vastly over-budget, and a constant funding fight in Congress. (I say this understanding fully that few advocate for a federal system, but rather local muni connectivity)
  - aggie 11y ago
    Comcast doesn't face typical market competition; it is uncommon for it's customers to have good alternatives. That's not a good representation of the efficiency and quality the private sector produces.
    - nmrm2 11y ago
      > That's not a good representation of the efficiency and quality the private sector produces. It's a good representation of the danger in the common quasi-religious belief that hasty privatization necessarily entails a marketplace with healthy competition.
    - sdenton4 11y ago
      Surely no true scotsman....
- rbritton 11y ago
  There's also this part too: This request form is for our residential customers only. I have a business account, and my connection was affected...
  - comcastluser 11y ago
    Business is same as residential, they just show up at your house first.
anigbrowl 11y ago
That's nice. I figured out what was going on (though I was very frustrated by that point) and switched my DNS servers to Google's public DNS servers (8.8.8.8 & 8.8.4.4) which worked fine. I figured that calling would be a waste of both my time and Comcast's - the people on the other end of the phone have no idea how long a fix is going to take other than 'as soon as possible'. The straightforward explanation and humble acceptance of responsibility were a refreshing surprise, however. A $5 credit is not that big a deal but it is good to see a firm putting its money where its mouth is instead of trying to deflect or minimize the disruption to customers.
- yalogin 11y ago
  So Google runs a public DNS service too? First time hearing this. I was annoyed with the constant DNS outages but did not do anything.
  - sharth 11y ago
    They have since 2009. https://developers.google.com/speed/public-dns/docs/using https://developers.google.com/speed/public-dns/docs/using
  - cmdrfred 11y ago
    8.8.8.8 man, what else do you ping to see if the internet is working?
    - mirimir 11y ago
      For nostalgia, sometimes 4.4.4.4 ;) Edit: Make that 4.2.2.2 (it's been a while).
    - girvo 11y ago
      www.something.com usually!
    - AnkhMorporkian 11y ago
      127.0.0.1, but I do get quite a few false positives.
thaumaturgy 11y ago
The trouble description seems a bit off. They had a backbone switch in San Jose fail (according to mtr), but it only affected DNS depending on your routing. In my case, 8.8.8.8 and 8.8.4.4 were both on the working side of the switch, but a lot of other sites and systems weren't -- even when accessing by IP. So it wasn't really a DNS issue. I couldn't ping Hurricane Electric servers, but Slashdot worked OK for example. The frustrating thing was that I was going through a Sacramento hop first but still couldn't reach systems in Reno -- I thought there was fiber all the way up the 80 corridor, but apparently most of my Comcast traffic has to go to San Jose before it can proceed East.
- LukeShu 11y ago
  but a lot of other sites and systems weren't -- even when accessing by IP. So it wasn't really a DNS issue. That's just an intermittent issue that Comcast customers deal with--I experienced that in Indiana a couple of months ago. I believe that it relates to them re-jiggering routers such that the appropriate default gateway for your modem changes, but because your modem is already up, it doesn't get notified of the change. So the modem doesn't send the message to the correct router, and the router it sends it to may or may not know how to get it to the destination.
  - thaumaturgy 11y ago
    I typically have logs streaming through terminal windows from several servers and multitask online -- I notice even brief transient network issues. FWIW what you're describing hasn't been a problem for me (though I don't doubt it is for lots of customers), and the San Jose outage was definitely out of the ordinary. Anyway, I wish Sacramento was a bigger network hub, is all.
sjg007 11y ago
It would be nice if these regulated monopolies would actually charge a decent flat price instead of promo rates that then jack up 6, 12 or 24 months later. This $5 is a different percentage of about 6 different price levels for Internet.
Aloha 11y ago
So either business class customers were not effected, or are not eligible - personally. I didn't notice a DNS outage, I run my own DNS locally.
kzahel 11y ago
My home comcast connection in SF started being extremely flaky last week and I switched over to 4.2.2.2 etc and haven't looked back.
- LukeShu 11y ago
  Level 3 isn't cool with non-customers using their DNS server. Google operates 8.8.8.8 and 8.8.4.4 that are for public use.
  - userbinator 11y ago
    Level3's VP disagrees: http://blog.level3.com/level-3-network/a-flawed-study-of-cdns-and-dns/ http://blog.level3.com/level-3-network/a-flawed-study-of-cdn... "Open DNS is provided by companies like OpenDNS, Google and Level 3. You can use it wherever you are on the Internet with no restrictions or authentication required."
    - LukeShu 11y ago
      Huh, that's interesting. As an organization, there's no public info on the availability of those servers. Last year they showed ads for failed DNS requests for non-customers. http://james.bertelson.me/blog/2014/01/level-3-are-now-hijacking-failed-dns-requests-for-ad-revenue-on-4-2-2-x/ http://james.bertelson.me/blog/2014/01/level-3-are-now-hijac...
dba7dba 11y ago
NOW it makes sense. I was writing up a tutorial which involves working with subdomains. I was QAing the tutorial by following the tutorial along but the DNS changes wouldn't work. I know it worked before but not this time. And meanwhile I was able to visit the websites that I usually visit. After seeing the Comcast letter, I'm concluding that the DNS changes weren't working because of Comcast's issue. I was still able to visit websites because my home DNS server had cached DNS names. Oh well.
stox 11y ago
If my recent experiences with them in regards to DNS is any indicator, their DNS infrastructure is a colossal joke. 4 days to get a reverse DNS entry done? Really? I wont even mention the typos in some of their SOA records.
hw 11y ago
I'm not sure if $5 adequately compensates for the productivity (and in some cases entertainment) loss encountered. I guess that equates to roughly a day's credit depending on what your plan is like. Plus I don't see why they require us to file a compensation claim, when they could just credit every account that was in the affected areas. I wonder if they would be compensating more if the Internet and TV went down during a big event like the NBA Finals yesterday.
- philipw 11y ago
  As a consumer service is there even an SLA!? I mean stuff goes wrong, if your home internet connection is so important then multi-home leased lines is what you need..
- pbreit 11y ago
  Step outside. Do some jumping jacks. Talk to your kids. You pay $150 for Comcast Internet? Obviously requiring an opt in cuts the payout by 99%.
  - FireBeyond 11y ago
    Technically I pay $224/mo for Comcast Internet, but apparently (home) business customers aren't eligible...
georgemcbay 11y ago
How generous of them to offer this for anyone who happens to learn about it in the limited time window they are offering it for... (/s) Also: $5 is half of the $10/month modem rental fee (that I assume is commonly paid by a lot of their customers who don't know any better) on a ~$70 modem (probably much cheaper at the scale they buy them at) that has probably been paid for in rental fees 5 times over already.
ntakasaki 11y ago
>We are also building a Web site that impacted customers can visit to receive their credit. We will update this post with a link to that site as soon as it is available and will share the link on Twitter through our customer support handle @comcastcares. If you need to use a social media account name that says "company name cares" then its obvious that most customers think you don't care.

41 points by hw 11y ago

What a generous window of time... This $5 credit offer is being extended to our customers with active XFINITY Internet service in California, Washington State, and Tucson, Arizona who were affected by the issue with our DNS servers that was first reported on 6/1/15. The form above will be available until 6/11/15

yeukhon 11y ago
I agree... They should just add that credit automatically...
- cpeterso 11y ago
  Seriously. They know which customers are in the affected areas because they have their billing addresses. Just credit them $5 on their next bill.
Redoubts 11y ago
Further: Please allow 4 to 6 weeks for the credit to be posted to your Comcast account.
- zatkin 11y ago
  So, they refund you $5 in Comcast credit that will eventually come out of your next bill?
  - Yhippa 11y ago
    What if this incident caused you to switch to another ISP prior to said bill. Based on my experiences good luck getting that $5.
    - jedmeyers 11y ago
      "Switch to another ISP"? That's a funny joke.
  - deleted 11y ago
    [deleted]
bby 11y ago
Fuck Comcast. Opt-in trick to fuck over more people.
reagency 11y ago
Comcast exists to prove that government isn't always less efficient at providing service than the private sector.
- baddox 11y ago
  The rule of thumb is that government tends to be less efficient than private markets. The residential ISP "market" isn't much of a market in lots of places. Comcast tends to operate as a monopoly or near-monopoly.
- adventured 11y ago
  I'd have to wonder at the product we'd get if we swapped out Comcast and let the Federal Government take over their business. I think it would cost net twice as much (total expenditures out of the consumer pocket via taxation, the plans wouldn't cost more), be about as reliable, about as fast, lose lots of money and still be completely unaccountable to its customers. The system / infrastructure upgrades would be slow, bureaucratic, always vastly over-budget, and a constant funding fight in Congress. (I say this understanding fully that few advocate for a federal system, but rather local muni connectivity)
- aggie 11y ago
  Comcast doesn't face typical market competition; it is uncommon for it's customers to have good alternatives. That's not a good representation of the efficiency and quality the private sector produces.
  - nmrm2 11y ago
    > That's not a good representation of the efficiency and quality the private sector produces. It's a good representation of the danger in the common quasi-religious belief that hasty privatization necessarily entails a marketplace with healthy competition.
  - sdenton4 11y ago
    Surely no true scotsman....
rbritton 11y ago
There's also this part too: This request form is for our residential customers only. I have a business account, and my connection was affected...
- comcastluser 11y ago
  Business is same as residential, they just show up at your house first.

anigbrowl 11y ago

That's nice. I figured out what was going on (though I was very frustrated by that point) and switched my DNS servers to Google's public DNS servers (8.8.8.8 & 8.8.4.4) which worked fine. I figured that calling would be a waste of both my time and Comcast's - the people on the other end of the phone have no idea how long a fix is going to take other than 'as soon as possible'. The straightforward explanation and humble acceptance of responsibility were a refreshing surprise, however. A $5 credit is not that big a deal but it is good to see a firm putting its money where its mouth is instead of trying to deflect or minimize the disruption to customers.

yalogin 11y ago
So Google runs a public DNS service too? First time hearing this. I was annoyed with the constant DNS outages but did not do anything.
- sharth 11y ago
  They have since 2009. https://developers.google.com/speed/public-dns/docs/using https://developers.google.com/speed/public-dns/docs/using
- cmdrfred 11y ago
  8.8.8.8 man, what else do you ping to see if the internet is working?
  - mirimir 11y ago
    For nostalgia, sometimes 4.4.4.4 ;) Edit: Make that 4.2.2.2 (it's been a while).
  - girvo 11y ago
    www.something.com usually!
  - AnkhMorporkian 11y ago
    127.0.0.1, but I do get quite a few false positives.

thaumaturgy 11y ago

The trouble description seems a bit off. They had a backbone switch in San Jose fail (according to mtr), but it only affected DNS depending on your routing. In my case, 8.8.8.8 and 8.8.4.4 were both on the working side of the switch, but a lot of other sites and systems weren't -- even when accessing by IP. So it wasn't really a DNS issue. I couldn't ping Hurricane Electric servers, but Slashdot worked OK for example. The frustrating thing was that I was going through a Sacramento hop first but still couldn't reach systems in Reno -- I thought there was fiber all the way up the 80 corridor, but apparently most of my Comcast traffic has to go to San Jose before it can proceed East.

LukeShu 11y ago
but a lot of other sites and systems weren't -- even when accessing by IP. So it wasn't really a DNS issue. That's just an intermittent issue that Comcast customers deal with--I experienced that in Indiana a couple of months ago. I believe that it relates to them re-jiggering routers such that the appropriate default gateway for your modem changes, but because your modem is already up, it doesn't get notified of the change. So the modem doesn't send the message to the correct router, and the router it sends it to may or may not know how to get it to the destination.
- thaumaturgy 11y ago
  I typically have logs streaming through terminal windows from several servers and multitask online -- I notice even brief transient network issues. FWIW what you're describing hasn't been a problem for me (though I don't doubt it is for lots of customers), and the San Jose outage was definitely out of the ordinary. Anyway, I wish Sacramento was a bigger network hub, is all.

sjg007 11y ago

It would be nice if these regulated monopolies would actually charge a decent flat price instead of promo rates that then jack up 6, 12 or 24 months later. This $5 is a different percentage of about 6 different price levels for Internet.

Aloha 11y ago

So either business class customers were not effected, or are not eligible - personally. I didn't notice a DNS outage, I run my own DNS locally.

kzahel 11y ago

My home comcast connection in SF started being extremely flaky last week and I switched over to 4.2.2.2 etc and haven't looked back.

LukeShu 11y ago
Level 3 isn't cool with non-customers using their DNS server. Google operates 8.8.8.8 and 8.8.4.4 that are for public use.
- userbinator 11y ago
  Level3's VP disagrees: http://blog.level3.com/level-3-network/a-flawed-study-of-cdns-and-dns/ http://blog.level3.com/level-3-network/a-flawed-study-of-cdn... "Open DNS is provided by companies like OpenDNS, Google and Level 3. You can use it wherever you are on the Internet with no restrictions or authentication required."
  - LukeShu 11y ago
    Huh, that's interesting. As an organization, there's no public info on the availability of those servers. Last year they showed ads for failed DNS requests for non-customers. http://james.bertelson.me/blog/2014/01/level-3-are-now-hijacking-failed-dns-requests-for-ad-revenue-on-4-2-2-x/ http://james.bertelson.me/blog/2014/01/level-3-are-now-hijac...

dba7dba 11y ago

NOW it makes sense. I was writing up a tutorial which involves working with subdomains. I was QAing the tutorial by following the tutorial along but the DNS changes wouldn't work. I know it worked before but not this time. And meanwhile I was able to visit the websites that I usually visit. After seeing the Comcast letter, I'm concluding that the DNS changes weren't working because of Comcast's issue. I was still able to visit websites because my home DNS server had cached DNS names. Oh well.

stox 11y ago

If my recent experiences with them in regards to DNS is any indicator, their DNS infrastructure is a colossal joke. 4 days to get a reverse DNS entry done? Really? I wont even mention the typos in some of their SOA records.

hw 11y ago

I'm not sure if $5 adequately compensates for the productivity (and in some cases entertainment) loss encountered. I guess that equates to roughly a day's credit depending on what your plan is like. Plus I don't see why they require us to file a compensation claim, when they could just credit every account that was in the affected areas. I wonder if they would be compensating more if the Internet and TV went down during a big event like the NBA Finals yesterday.

philipw 11y ago
As a consumer service is there even an SLA!? I mean stuff goes wrong, if your home internet connection is so important then multi-home leased lines is what you need..
pbreit 11y ago
Step outside. Do some jumping jacks. Talk to your kids. You pay $150 for Comcast Internet? Obviously requiring an opt in cuts the payout by 99%.
- FireBeyond 11y ago
  Technically I pay $224/mo for Comcast Internet, but apparently (home) business customers aren't eligible...

georgemcbay 11y ago

How generous of them to offer this for anyone who happens to learn about it in the limited time window they are offering it for... (/s) Also: $5 is half of the $10/month modem rental fee (that I assume is commonly paid by a lot of their customers who don't know any better) on a ~$70 modem (probably much cheaper at the scale they buy them at) that has probably been paid for in rental fees 5 times over already.

ntakasaki 11y ago

>We are also building a Web site that impacted customers can visit to receive their credit. We will update this post with a link to that site as soon as it is available and will share the link on Twitter through our customer support handle @comcastcares. If you need to use a social media account name that says "company name cares" then its obvious that most customers think you don't care.