3 ms·

By reading the postmortem I got the feeling that it focuses primarily on providing a timeline of historical events without being future-looking. I did not find

by vladd 7y ago

By reading the postmortem I got the feeling that it focuses primarily on providing a timeline of historical events without being future-looking. I did not find any questions that would trigger a path of improvement for similar future events: - Why did the recovery take so long (up until Thursday morning)? - If the surge lasted for 20 seconds, why weren't the systems operational again 1 minute after the event? - Why is an on-site technician needed in order to re-activate the equipment? - Could we build a system that enables the reactivation remotely? There might be very well-rooted answers to the questions above (based on how real world electrical supplies work), but if the questions aren't asked, we won't end up exploring better solutions for the answers.

goodcanadian 7y ago
Why did the recovery take so long (up until Thursday morning)? Systems were up within an hour, but if you throw an hour's delay into a busy part of the rail system, you have knock on effects (trains aren't where they are supposed to be) that will basically last until the end of the service day. If the surge lasted for 20 seconds, why weren't the systems operational again 1 minute after the event? Why is an on-site technician needed in order to re-activate the equipment? Basically, the breakers were flipped by the surge. Resetting such things is best done manually on site to ensure there was no equipment damage that occurred because of the surge. Could we build a system that enables the reactivation remotely? Probably, but that strikes me as dangerous. The article explicitly talks about investigating what can be done to make their systems more robust to such power events. I think it is very future looking.
- gsnedders 7y ago
  > Systems were up within an hour, but if you throw an hour's delay into a busy part of the rail system, you have knock on effects (trains aren't where they are supposed to be) that will basically last until the end of the service day. And at a location where almost all lines are running at near capacity you can't "catch up" by getting more trains through the location once things are working again to get stock back in place, you can only get the regular timetable through it.
- vidarh 7y ago
  For those unaware, Croydon is basically a massive chokepoint, with trains coming in from London Bridge and London Victoria to the North and large part of the South coast to the South. If trains start piling up near either Victoria (and Clapham Junction) or London Bridge, it quickly causes trains to back up all the way to East Croydon, a 15 minute train journey away. Once it does, it blocks trains towards the other line, as East Croydon only has 6 platforms and no tracks bypassing it. Once that happens, trains can start backing up tens of miles further South, and will be out of place when things are up and running again. Victoria is a terminus, and is at times busy enough that trains gets 'stacked' two to a platform on the platforms serving Croydon, so turning trains back quickly becomes a fun exercise... (London Bridge also has lots of platforms that terminates, but seems to have fewer platform capacity issues after the overhaul) So even if the trains that are in position get moving again pretty quickly, untangling that and getting platform space for all of those trains in short order can compound short delays very substantially.

by vladd 7y ago

goodcanadian 7y ago

Why did the recovery take so long (up until Thursday morning)? Systems were up within an hour, but if you throw an hour's delay into a busy part of the rail system, you have knock on effects (trains aren't where they are supposed to be) that will basically last until the end of the service day. If the surge lasted for 20 seconds, why weren't the systems operational again 1 minute after the event? Why is an on-site technician needed in order to re-activate the equipment? Basically, the breakers were flipped by the surge. Resetting such things is best done manually on site to ensure there was no equipment damage that occurred because of the surge. Could we build a system that enables the reactivation remotely? Probably, but that strikes me as dangerous. The article explicitly talks about investigating what can be done to make their systems more robust to such power events. I think it is very future looking.

gsnedders 7y ago
> Systems were up within an hour, but if you throw an hour's delay into a busy part of the rail system, you have knock on effects (trains aren't where they are supposed to be) that will basically last until the end of the service day. And at a location where almost all lines are running at near capacity you can't "catch up" by getting more trains through the location once things are working again to get stock back in place, you can only get the regular timetable through it.
vidarh 7y ago
For those unaware, Croydon is basically a massive chokepoint, with trains coming in from London Bridge and London Victoria to the North and large part of the South coast to the South. If trains start piling up near either Victoria (and Clapham Junction) or London Bridge, it quickly causes trains to back up all the way to East Croydon, a 15 minute train journey away. Once it does, it blocks trains towards the other line, as East Croydon only has 6 platforms and no tracks bypassing it. Once that happens, trains can start backing up tens of miles further South, and will be out of place when things are up and running again. Victoria is a terminus, and is at times busy enough that trains gets 'stacked' two to a platform on the platforms serving Croydon, so turning trains back quickly becomes a fun exercise... (London Bridge also has lots of platforms that terminates, but seems to have fewer platform capacity issues after the overhaul) So even if the trains that are in position get moving again pretty quickly, untangling that and getting platform space for all of those trains in short order can compound short delays very substantially.