9 ms·
I was visiting my SO's parents in Portland for the holidays when this happened. Their cable TV stopped working for at least two days and we all got an emergenc
by CydeWeys 8y ago
I was visiting my SO's parents in Portland for the holidays when this happened. Their cable TV stopped working for at least two days and we all got an emergency services text with the local emergency number because 911 wasn't going to work. (At least their Internet through CenturyLink remained working.)
This also happened to coincide with them receiving a notice of a $50/month increase on their bill from CenturyLink, as well as my SO giving them a Roku for the holidays and setting them up with streaming services, which they got lots of practice using in the two whole days the cable TV didn't work.
Guess who's cutting the cord and switching to Internet-only service from Verizon FIOS at a much cheaper rate now?
This is what happens when you treat IT like a cost center and don't provide the necessary funds to tackle technical debt and keep your services up and running: You get huge costly outages revealing your basic incompetence and customers fleeing to superior competitors forever.
- jandrese 8y agoHonestly, if you had FiOS as an option why would you stay with CenturyLink in the first place? Verizon is just as assholish, but at least they can deliver.
- waynesonfire 8y agocenturylink fiber is pretty great as well
- deleted 8y ago[deleted]
- loeg 8y agoYeah, my experience with Clink's fiber offerings has been a good one. It's definitely "Gigabit" — there are many fiber splitters in play, which means (a) your downstream traffic is broadcast to everyone on the same splitter network in plaintext and filtered at individual ISP-owned termination devices, much like cable networks, and (b) you definitely never reach 1000 Mbps down, but often can reach 900 Mbps up. Typical is 500-600 Down, 900 Up; the highest I've seen down on e.g. fast.com is 900 Mbps. And the price is extremely competitive with Comcast ($80/mo for "gigabit"), unlike their DSL offerings, which are complete crap. The major downside is that their IPv6 support is nonexistent and they have no plans for rolling it out.
- chocolatkey 8y agoHi, I'm on centurylink's fiber offering and have the same experience as you getting much faster upload than download (though download speed increases quite a bit during sleeping hours). Are you saying this is because the DL is congested during the day hours? Or is there some other technological block in play?
- loeg 8y ago> Are you saying this is because the DL is congested during the day hours? Yeah, it's congestion on the transmitter (upstream) end.
- pooppaint 8y agoYou sure it’s plaintext? GPON networks should be using AES128 to each ONT.
- loeg 8y agoNo, not sure. I wasn't aware of that, thanks. Do you know more about the key negotiation and encryption protocol? Wikipedia doesn't have even mention AES. Thanks.
- ransford 8y agoI've been getting perfectly adequate IPv6 via their 6rd gateway (on fiber, in Seattle). I wrote some nerd notes on using one's own wifi router and setting up IPv6: http://b.tra.in/2015/07/notes-on-centurylink-fiber.html http://b.tra.in/2015/07/notes-on-centurylink-fiber.html
- loeg 8y ago6rd is pretty awful, generally, and I couldn't get Centurylink's to work last time I tried. I appreciate the notes and will take a look, thank you. (One big problem I've experienced in the past is that any time you configure both IPv6 and IPv4, all programs prefer IPv6. But tunneled IPv6 has much worse performance than the native IPv4, so you really want applications to prefer IPv4 in a tunneled v6 environment.)
- sharms 8y agoHere in Denver we get full 1Gbps for $75/mo, every provider has outages and this was the first partial outage we experienced with the service (pages loaded just slowly most of the day)
- gonzo 8y agoIf only it was available in Golden... I could hit the CL POP with a baseball (less than a block away).
- joecool1029 8y agoDoesn't exist across 99% of their last mile footprint. Just one or two cities. I would guess the vast majority of their revenue comes from extorting old people and farmers with shitty overpriced ADSL that squirrels eat into and take down annually.
- justtopost 8y agoRural folk here. Biannually.
- Trisell 8y agoAt least in my area Century Link business is the only isp I can get at my house that doesnt come with a data cap. Both CenturyLink home and our cable provider have data caps that are enforced. And working from home along with Netflix and YouTube blow me through the 80gb limit every month. Causing me to have to raise my plan. And I’m not paying $200 a month for unlimited bandwidth.
- sethdj 8y agoYour cable options have an 80gb limit?! WTF!! Comcast has data caps (which is stupid) but it's 1TB and I think my 7 person household has only passed the halfway mark once, so it's meh. But 80GB?! That's insane! Destiny 2 is an 80GB download itself.
- AckSyn 8y agoThis is why amazes me. At my home we blow through on average 25 GB a day in gaming, streaming video, other downloads etc. limits like these are terrible. 80 a month? Wtf?!
- wlesieutre 8y ago25 GB a day? Are you streaming 4K?
- jsjohnst 8y agoI live alone and use 50-100 gb/day on days I’m home all day, so 25gb/day isn’t that much. I cut cable TV years ago and exclusively stream. I like having the TV on for background noise when home, even though I’m usually not actively watching it the whole time, and you’d be surprised how fast one can chew through data usage. I usually use around 1.5TB a month when you factor in offsite backups.
- Jach 8y agoI don't know if it's widespread but at least here in WA you can pay Comcast an extra $50/month and they'll remove the data cap.
- CydeWeys 8y agoI'd say the majority of consumers stick with their existing suppliers out of sheer inertia. It takes time, effort, and hassle to switch, and absent any meaningful change in status quo (like a prolonged outage or sharp price increase), most people will coast. Especially when they're not technically inclined, as the difference in service will be less meaningful or apparent to them.
- tomjakubowski 8y agoFiOS in some parts of the country (including, I think, Oregon) is now run by Frontier, who are nowhere near Verizon's competence.
- jka 8y agoTo be fair, this does seem like an extremely rare root cause, but yes the impact was awful, so it'll be worth noting how CenturyLink responds and what they'll do to try to prevent similar issues in future.
- throwaway98434 8y agoAt scale, rare errors will occur on a semi-regular basis. Multi-day outages tend to be indicative of a dysfunctional internal organization.
- jka 8y agoTotally fair - I don't mean to excuse the results. That's an interesting insight that perhaps it's possible to identify organizations who have scaled (or whose products have scaled) too quickly based on whether they can cope with outages.
- techrich 8y agoJust deciding they want to put it up 50 dollars a month is insane!
- cptskippy 8y ago> This is what happens when you treat IT like a cost center and don't provide the necessary funds to tackle technical debt and keep your services up and running: Especially now with cordcutting becoming the norm and traditional cable subscription number dropping, large providers are scrambling to find ways to extract profit. With the rise of streaming, peering arrangements are more lopsided than ever and ISPs are actually having to pay for their own internet access. Because of this we're going to start seeing a lot more data caps, neglecting basic network infrastructure, and other tactics to extract as much profit as possible from customers.
- mr_toad 8y ago> With the rise of streaming, peering arrangements are more lopsided than ever and ISPs are actually having to pay for their own internet access. Don’t most ISPs cache this sort of content closer to their customers? Netflix at least offers to install hardware that does this.
- scurvy 8y agoYouTube and Netflix cache boxes are important for smaller ISP's. Their cache performance can literally make or break the monthly P&L. That said, I haven't heard about other OTT services offering a caching appliance. And there are more and more OTT streaming services coming. It's great from a consumer perspective, but it's going to be challenging for smaller players.
- scurvy 8y ago> This is what happens when you treat IT like a cost center and don't provide the necessary funds to tackle technical debt< Cost-cutting didn't seem to have played a role in this outage. They had a bad linecard in their out of band network that was somehow impacting the control plane of their devices. You can talk the pros/cons of how they built out their national Out-Of-Band (OOB) network, but you can't blame this outage on cost cutting. You're simply bashing a large company that had a bad day. They might deserve your vitriol for plenty of other reasons in other cases, but you can't blame this outage on cost cutting.
- CydeWeys 8y agoIt seems to me that they clearly had some serious problems with their overall network architecture if a single bad card could bring everything down. If they'd invested more money into IT over the past years then they could've taken care of a lot of their technical debt proactively rather than reactively. Odds are decent they would've implemented bad packet filtering before the major outage instead of afterwards, as that seems like a plausible thing an IT/networking department would've gotten around to with more funding, and more and higher-skilled, higher-paid veteran staff who could pre-emptively identify and fix this issue through experience and testing. Any time something really bad happens that could have been prevented, it's reasonable to ask why it wasn't prevented. This wasn't some black swan event, just a routine hardware failure that they didn't have defenses against.
- scurvy 8y agoWho's to say that it's technical debt? Maybe they liked the way that their management network is designed? It's only debt if you realize it's bad and ignore it. To be perfectly honest, I don't know what's the best choice for running a management out-of-band network for a network provider of this size. Do you?
- wpietri 8y agoWe don't, and apparently they didn't either. But it was their job to do so. I think it's very reasonable to suspect underinvestment and technical debt here. Not only did they have a problem, they had a very hard time a) finding the cause of the problem, and b) mitigating the problem while they were looking for the cause. We can't know, of course. But I think it would be hard to argue that this is optimal, and that nobody at CenturyLink ever suggested it could be better if they spent a bit more money.