3 ms·

I'm having trouble finding any reference for the "five minutes per decade" downtime limit. That would be "6 9s" or 99.9999% uptime which is just crazy in today'

I'm having trouble finding any reference for the "five minutes per decade" downtime limit. That would be "6 9s" or 99.9999% uptime which is just crazy in today's world. Even today VOIP providers only claim 99.999%, though it seems like in practice many fail to even get close too that. I think I found the 756 page consent decree between Bell and the US you mentioned but I can't find a reference there either, though it is quite a massive doc with so-so OCR: https://www.google.com/books/edition/Consent_Decree_Program_of_the_Department/ovboRuQHqhAC?hl=en&bshm=ncc/1,ncc/1&gbpv=0 https://www.google.com/books/edition/Consent_Decree_Program_... Stepping back a bit, it seems like the FCC would be in charge of establishing rules like this but that consent decree I found (which might be the wrong one), is with the house Antitrust committee. (After my own Googling failed I also asked GPT-4 and it isn't aware of this either)

sosodev 3y ago
I think they got 5 nines mixed up with 6 nines. The 5 nines is 5 minutes per year. I asked ChatGPT and it claims they had a target of 5 nines which makes sense.
gumby 3y ago
Well as I noted in my comment the consent decree itself IIRC specified availability rather than uptime. However the 5 min figure comes from my customers like DSC (R.I.P), Ericsson and Nokia building POTS switches. These guys were deadly serious, like the folks who made spacecraft and medical devices, plus the Ericsson and Nokia folks were nice too. In DSC’s case they were so paranoid that they paid us a massive amount to maintain a special tool chain for just for them. It was frozen in time (no upgrades) and when they reported a bug and we sent them an updated tool chain they diffed the binaries and made sure that every delta was due to the big fix and nothing else (that the dev hadn’t snuck in some other patch for some reason)! They did some other headstands with their hardware and software, but in the end it didn’t save them. It’s cool that the consent decree is online. That arrangement with the Bell system was very clever, though it led to a lot of weird anomalies and distortions but I think it did end up with a better phone system than the PTT model. It’s also been a better model than what’s happened with the power and water utilities. The FCC back then was a better regulator for phone customers than the (now obsolete) ICC had been.
- didacticBuffalo 3y ago
  We kept a data closet in Manhattan in a sister building to 33 Thomas. Customers would ask for a BRP. We had N+2 for data level stuff and higher but some customers would get hung up on not having second location. We took the stance if the facility fails we all have bigger issues. We had one 18 minute issue in the ten years of use. It was the best facility we ever used.
  - gumby 3y ago
    I remember in the late 90s/early 00s a lot of cases when sites would go down (everybody self-hosted) due to backhoe events and the like. These companies weren't idiots: they had replication for their databases and leased redundant transmission service because they knew this kind of thing could happen. The problem is you'd buy transmission from two providers...whose fiber turned out to be in the same conduit, or even both had rented bandwidth on the same fiber. At least people are smarter these days, and with widespread cloud service there are fewer people who need to keep track of this stuff.
- kccqzy 3y ago
  They could be serious enough about it, and do everything right, and still have an outage. https://news.ycombinator.com/item?id=34665023 https://news.ycombinator.com/item?id=34665023
themerone 3y ago
When I interned a phone company I was surprised by the lack of hardware redundancy. The equipment was trusted to not fail in it's lifetime.

sosodev 3y ago

I think they got 5 nines mixed up with 6 nines. The 5 nines is 5 minutes per year. I asked ChatGPT and it claims they had a target of 5 nines which makes sense.

gumby 3y ago

Well as I noted in my comment the consent decree itself IIRC specified availability rather than uptime. However the 5 min figure comes from my customers like DSC (R.I.P), Ericsson and Nokia building POTS switches. These guys were deadly serious, like the folks who made spacecraft and medical devices, plus the Ericsson and Nokia folks were nice too. In DSC’s case they were so paranoid that they paid us a massive amount to maintain a special tool chain for just for them. It was frozen in time (no upgrades) and when they reported a bug and we sent them an updated tool chain they diffed the binaries and made sure that every delta was due to the big fix and nothing else (that the dev hadn’t snuck in some other patch for some reason)! They did some other headstands with their hardware and software, but in the end it didn’t save them. It’s cool that the consent decree is online. That arrangement with the Bell system was very clever, though it led to a lot of weird anomalies and distortions but I think it did end up with a better phone system than the PTT model. It’s also been a better model than what’s happened with the power and water utilities. The FCC back then was a better regulator for phone customers than the (now obsolete) ICC had been.

didacticBuffalo 3y ago
We kept a data closet in Manhattan in a sister building to 33 Thomas. Customers would ask for a BRP. We had N+2 for data level stuff and higher but some customers would get hung up on not having second location. We took the stance if the facility fails we all have bigger issues. We had one 18 minute issue in the ten years of use. It was the best facility we ever used.
- gumby 3y ago
  I remember in the late 90s/early 00s a lot of cases when sites would go down (everybody self-hosted) due to backhoe events and the like. These companies weren't idiots: they had replication for their databases and leased redundant transmission service because they knew this kind of thing could happen. The problem is you'd buy transmission from two providers...whose fiber turned out to be in the same conduit, or even both had rented bandwidth on the same fiber. At least people are smarter these days, and with widespread cloud service there are fewer people who need to keep track of this stuff.
kccqzy 3y ago
They could be serious enough about it, and do everything right, and still have an outage. https://news.ycombinator.com/item?id=34665023 https://news.ycombinator.com/item?id=34665023

themerone 3y ago

When I interned a phone company I was surprised by the lack of hardware redundancy. The equipment was trusted to not fail in it's lifetime.