4 ms·

> When it went to flip into the backup, we discovered that the backup fiber had a break Pretty grim that a life critical system wasn't designed to report that

by cube00 12d ago

> When it went to flip into the backup, we discovered that the backup fiber had a break Pretty grim that a life critical system wasn't designed to report that the backup fibre was unserviceable until they attempted to switch over to it. I wonder how long it was down? Days, weeks, months?

readthenotes1 12d ago
A lot of people don't check their backups until they need to restore. A lot of people are incompetent.
- thewebguyd 12d ago
  A backup without a restore test isn't a backup at all
  - bonoboTP 12d ago
    You don't always have a copy of your hardware to restore onto. And the test's entire purpose is that you're not yet sure whether your restore will truly work. So you can't just run a backup and restore on your true prod system, because you're not sure it won't wreck it. So you need extra money to have a second system onto which you try to restore. If you don't have a lot of money, you will want to actually use your disks for storage, not to put them into a second testing server. Of course I'm not talking about very professional companies with super critical data. Just simpler smaller scale places or consumers.
    - deleted 12d ago
      [deleted]
- andrewjf 12d ago
  Nobody wants backups as a feature. The feature is restore.
toast0 12d ago
Sometimes your multiple fiber paths end up in the same bundle, severed by the same backhoe. It's always a fun day when that becomes apparent.
- koolba 12d ago
  Put all your backups in one basket, and then pray that nobody crushes the basket.
  - oasisbob 12d ago
    This is a pretty classic network operator story: go to great lengths to provide for physically diverse paths, then not notice when your provider refactors and grooms them onto the same bundle.
- konfusinomicon 12d ago
  last time this happened in my area a local farmer was burying a cow and took out the whole towns connection
krashidov 12d ago
Could this be sabotage?
- kqgnkqgn 12d ago
  That seems more than merely plausible given current tensions and the location/timing connection with NYC and UN.
zoomablemind 12d ago
That's very uncharacteristic, especially given all those contingency requirements (backup policies, failovers, testing schedules etc.) imposed on corporations/companies in the wake of 9/11.
- pixl97 12d ago
  You actually have to ensure that someone isn't just checking boxes the tests passed and the tests actually passed. I worked for a regional ISP that had a major outage when the redundant fiber provided by the telephone company was cut in one place triggering a full loss of connectivity. It also caused a massive 911 outage for 200,000 people as it isolated the 911 center from the city core. Turns out the phone company didn't connect one side of the ring topology even though they certified they did. Needless to say lawsuits abounded.

by cube00 12d ago

readthenotes1 12d ago

A lot of people don't check their backups until they need to restore. A lot of people are incompetent.

thewebguyd 12d ago
A backup without a restore test isn't a backup at all
- bonoboTP 12d ago
  You don't always have a copy of your hardware to restore onto. And the test's entire purpose is that you're not yet sure whether your restore will truly work. So you can't just run a backup and restore on your true prod system, because you're not sure it won't wreck it. So you need extra money to have a second system onto which you try to restore. If you don't have a lot of money, you will want to actually use your disks for storage, not to put them into a second testing server. Of course I'm not talking about very professional companies with super critical data. Just simpler smaller scale places or consumers.
  - deleted 12d ago
    [deleted]
andrewjf 12d ago
Nobody wants backups as a feature. The feature is restore.

toast0 12d ago

Sometimes your multiple fiber paths end up in the same bundle, severed by the same backhoe. It's always a fun day when that becomes apparent.

koolba 12d ago
Put all your backups in one basket, and then pray that nobody crushes the basket.
- oasisbob 12d ago
  This is a pretty classic network operator story: go to great lengths to provide for physically diverse paths, then not notice when your provider refactors and grooms them onto the same bundle.
konfusinomicon 12d ago
last time this happened in my area a local farmer was burying a cow and took out the whole towns connection

krashidov 12d ago

Could this be sabotage?

kqgnkqgn 12d ago
That seems more than merely plausible given current tensions and the location/timing connection with NYC and UN.

zoomablemind 12d ago

That's very uncharacteristic, especially given all those contingency requirements (backup policies, failovers, testing schedules etc.) imposed on corporations/companies in the wake of 9/11.

pixl97 12d ago
You actually have to ensure that someone isn't just checking boxes the tests passed and the tests actually passed. I worked for a regional ISP that had a major outage when the redundant fiber provided by the telephone company was cut in one place triggering a full loss of connectivity. It also caused a massive 911 outage for 200,000 people as it isolated the 911 center from the city core. Turns out the phone company didn't connect one side of the ring topology even though they certified they did. Needless to say lawsuits abounded.