7 ms·

Croydon Shut Down

129 points by lelf 7y ago

rossmohax 7y ago
Acccording to [1] just 67% of all UK trains arrived on time (no more than a minute late), with 2.8% scheduled trains cancelled all togther. How does it fare against other EU countries? What is so different abount Japan trains, that they are just 0.7 minutes late on average? [1] https://dataportal.orr.gov.uk/media/1630/passenger-performance-2019-20-q2.pdf https://dataportal.orr.gov.uk/media/1630/passenger-performan...
- kiwijamo 7y ago
  67% for right time (i.e. <1min late) arrival seems to be pretty average. I’d be more interested in the % of arrival at destination within 5mins as that’s the standard used in many countries and thus easier to compare different systems.
  - gsnedders 7y ago
    In many parts of the GB network, once you get more than a minute or two late you end up with delays propagating throughout the network, because enough is timetabled with lines near capacity. You get up to about ~82% within 3 minutes and ~88% within 5 minutes. Beyond that you see little gains.
- rwmj 7y ago
  I can tell you from experience the trains in the UK are expensive and miserable. The common theory for why is because of the mess made of privatising British Rail in the 1990s. The government did it in a way which managed to create monopolies while at the same time splitting out responsibility between different companies in such a way that millions are drained away on legal costs while the companies argue with each other about who is responsible for each problem. There are many articles analysing the problems you can find online, here's one that turned up at the top of search: https://www.theguardian.com/uk-news/2018/jan/07/railways-privatisation-nationalisation https://www.theguardian.com/uk-news/2018/jan/07/railways-pri... Edit: Better article: https://www.citylab.com/transportation/2012/09/why-britains-railway-privatization-failed/3378/ https://www.citylab.com/transportation/2012/09/why-britains-...
  - deleted 7y ago
    [deleted]
  - NeedMoreTea 7y ago
    It's a well worn topic. Had they not achieved the weird corporate separation, such that track, train, station building and ticket can all be separate companies putting on a new service, or adjusting timetabling and even putting up new posters could have been achievable things. As is they spend months, and millions going nowhere. [Edit: Not forgetting that station hotels, ferries and hospitality were separated out to yet more companies. Just because.] Had they just gone back to the big 4 of 1921-1947, or created a "medium 8" with extra rights to run charters on the tracks we might have rail that works by now. Then there's the 50+ year story of line and infrastructure upgrades... Each time we update, we seem to update to the previous generation's capability.
    - simplesleeper 7y ago
      Britain runs 20% of Europe's rail journeys. It has the least accidents and deaths of passengers and staff. It is considered the most improved rail network year on year since 1997. It has some of the lowest fares if you book far in advance (and count the cost of subsidisation for foreign fares).
      - NeedMoreTea 7y ago
        How many decades did it take to electrify Europe's busiest line, the West Coast Main Line? Did it become modern? No. It used last generation signalling such that when we had 125mph HST they were limited to 80 and 100, when we bought back BR's APT as 150mph Pendolino they were limited to 1980s HST speeds as the tracks were electrified and upgraded to 125mph. In 2010. The "negotiations" between train operators wanting better rolling stock, railtrack (government), station upgrades and the inconvenient length of franchises made the whole process suitable for a Laurel and Hardy or Buster Keaton movie. Little wonder they're most improved, they gave themselves so much scope...
      - lentil_soup 7y ago
        Do you have any sources where I can read more about this? I find it surprising
- rtpg 7y ago
  I bet a big part of it in Japan is the sheer number of trains running drags down that average. If you count the Yamanote, with peak frequency of trains every 2 or 3 minutes, you’re gonna get a lot of stuff on time. Think about how rare power outages are nowadays. When infrastructure becomes important enough it becomes run pretty smoothly.
  - fishnchips 7y ago
    Rail !== tube. You’d be comparing Tokyo’s Yamanote to London’s Circle. I used both to commute at one point or another and found them equally robust. OTOH rail in the UK and Japan are very different stories indeed.
    - rtpg 7y ago
      I think it would depend on the stats used. Yamanote is still JR. The Joban line goes all the way to Sendai, if you really want it to. So if you got lateness stats on trains, it would likely take all the JR stuff and mix stuff up I agree with your sentiment, my comment is more about my theory that stats would not make the distinction (though it would likely exclude things like Tokyo Metro or the monorails).
- jacquesm 7y ago
  One part may well be that some of the rolling stock should probably be in a museum, not transporting passengers daily.
kiwijamo 7y ago
Nice to see a post mortem made public. Also fairly well written for laymen while still retaining detail. Most rail infrastructure operators from other countries don’t make this sort of info public (apart from PR type messages which typically don’t explain the issue at at all and are more focused on deflecting blame etc). Kudos to Network Rail.
- richardcrossley 7y ago
  Cynical me: The UK railway system has plenty of practice of delays and cancellations, it seems they have learned to at least provide some detail in "lay" form. Practical me: Having that information allows better decisions to be made. Rather than simply wasting my evening at a station other arrangements can be made, possibly having a meal in a local bar/restaurant rather than shivering on a station.
- paxswill 7y ago
  One of the things I've really liked about my local commuter rail (Virginia Railway Express [0]) is the level of detail and communication they have with the community/riders. Their RSS feed typically has a post if there's any delay over 10 minutes explaining the delay and how long it is expected to be. They'll also post when the GPS tracking for a train isn't working but is still on schedule. The after-incident reports are also really nice; this one [1] resulted in delays of over an hour and a half because of track slippage leaving a station. Since then VRE has been letting us (riders) know if they're reversing the engine placement to mitigate possible slippage, so we're able to see that they're actually implementing the changes they propose. Other disruptions are also communicated fairly clearly and promptly [2]. As a side note, I'm pleasantly surprised by the transparency with historical service levels [3] and passenger utilization [4]. Saying all this though, the level of service is low when compared to other countries' commuter rail systems. One of the challenges VRE works with is that it hasn't owned the rail lines it's running on; they're owned by a freight railroad and passenger rail is typically run at a lower priority than freight. There has been some recent movement though by the state [5] to significantly improve passenger rail transit, especially in the northern part of the state (almost certainly related to Amazon HQ2). 0: https://www.vre.org/ https://www.vre.org/ 1: https://content.govdelivery.com/accounts/VAVRE/bulletins/26af3e9 https://content.govdelivery.com/accounts/VAVRE/bulletins/26a... 2: https://content.govdelivery.com/accounts/VAVRE/bulletins/26fe6ae https://content.govdelivery.com/accounts/VAVRE/bulletins/26f... 3: https://www.vre.org/service/daily-performance/ https://www.vre.org/service/daily-performance/ 4: https://www.vre.org/service/rider/train-utilization-trends/ https://www.vre.org/service/rider/train-utilization-trends/ (not actually trends) 5: https://www.governor.virginia.gov/newsroom/all-releases/2019/december/headline-850120-en.html https://www.governor.virginia.gov/newsroom/all-releases/2019...
herpderperator 7y ago
> We are investigating whether fitting Uninterruptable Power Supplies to each signalling location would have prevented this failure, although it is also possible they would have had to shut down to protect themselves too, as happened at another location during the same event. This specific remark sounds rather amateurish. This is part of the reason UPSes were invented. A Double Conversion Online UPS would have been able to handle this situation just fine.
- wilhil 7y ago
  I worked for an ISP that was deploying switches nationwide... We got a lot of space in BT Telephone Exchanges and thought nothing more of it. I have never seen such dirty and bad power! Literally once a week we had to go to exchanges to reset fuses that had tripped. We then started deploying (rather entry level) UPS to all locations and the problem just disappeared completely... we then invested in more high end UPS systems, but, yeah - dirty power is a big problem and UPS help.
  - Aloha 7y ago
    I'm surprised CO gear was not running on -48v rather than AC power
    - viraptor 7y ago
      Is DC actually common these days? I haven't been to a physical datacenter in a decade or do, but I remember almost everything running on AC.
      - wilhil 7y ago
        I'm an IT and networking guy... This is one thing that got on my nerves completely - most equipment comes as AC or DC, and I'm never sure what to order, so, I usually go for the AC one as it's the safe choice
      - Aloha 7y ago
        It is in Telecom Applications.
nathell 7y ago
I don’t mind delays very much, even long ones, as long as there is information readily available about why the delay happens and how long it is likely to last. I have very positive experience with UK railways in this regard; this post is a prime example. Contrast with PKP (Polish rail), where it’s not uncommon for a train to stand in the middle of nowhere for half an hour, without so much as a single word of explanation from staff. It’s changing for the better, but slowly.
Aloha 7y ago
A 20 second overvolt period does seem exceedingly long - I'm curious to find the cause of that.
- ksec 7y ago
  Yes I thought power surge happens in milliseconds to a second range, may be few seconds at worst, first time I ever read a double digit seconds power surge. Luckily the system shutter itself down as intended.
Animats 7y ago
OK, that's what should happen for an overvoltage. The control systems tripped offline to protect themselves, all signals went to red, and all trains stopped. Electric railways have multiple different power systems - traction power, signaling power, and utility power, at least. The possibility of a short between systems means a conservative shutdown-on-overvoltage circuit breaker setup is appropriate.
- ec109685 7y ago
  The need to manually turn this system back on is non optimal. A control loop powered by a battery backup should have been able to reset the system once the voltage came back down.
  - Animats 7y ago
    This is a railroad signaling system. It's designed to fail safe, and that's not a buzzword. Any broken wire, go to red. Any broken relay, go to red. Any rail break, go to red. Loss of power, go to red. If you're getting power surges in signaling power, it's time to shut down, go to red, and not reset until someone has tested the system.
    - ec109685 7y ago
      The backup system can be programmed to do the same testing a human can to bootstrap the system. Another benefit is that they can simulate a surge in the system regularly and watch it recover itself versus a human running a checklist once every 10 years.
londons_explore 7y ago
Rail equipment is all ancient and highly redundant for both uptime and safety. Yet the actual system reliability and safety doesn't seem great. Signal outages happen nearly every day in the UK, and delay tens of thousands of passengers. Safety failures in signalling systems kill on average a few people per year. How about a modern system, built with JavaScript-de-jure, running over 4G and WiFi, consumer GPS, "Download train-control from the app store", running on tablets or the train drivers BYOD phone, etc. How would it work out for safety/reliability? I suspect counterintuitively it would work out much better. Simple cheap portable devices means a bunch of spares can be around always. "Phone battery dead? No worries, there's a tablet I can control the train from too". No WiFi? It also works with 4G or Bluetooth. Server down? It has a Peer connection backup. Hackers broke in? The system has safety logic built in to every device, and uses a consensus system so a few rogue systems can't crash trains. Power supply broken? We have 24 hours of battery, longer if I turn the brightness down.
- bschwindHN 7y ago
  > How about a modern system, built with JavaScript-de-jure No thank you.
  - WrtCdEvrydy 7y ago
    import { train } from 'lodash';
- thinkingemote 7y ago
  Oh my sweet summer silicon valley child! We should disrupt every safety equipment and process everywhere!
- pergadad 7y ago
  Better 1000 failures than one deadly accident. These systems are built in basic and redundant ways for a reason. See also https://news.ycombinator.com/item?id=12071240 https://news.ycombinator.com/item?id=12071240
- seanparsons 7y ago
bloak 7y ago
The article doesn't mention a date. Presumably the "Wednesday" referred to is 18 Dec 2019. On a previous occasion when rail equipment failed because of a fluctuation in supply voltage the electricity company was not at fault: the voltage had remained within specified limits. I find it a bit suspicious that the article doesn't mention what the change in voltage was this time, and what variation the specification allows. The penultimate paragraph is perhaps an implicit admission that the electricity supply was within its specification. I wonder if any of this is indirectly related to the official change in supply voltage, from 240 V to 230 V, in 1995. Nah, probably not; they can't have equipment that old, can they?
- Nextgrid 7y ago
  Did you get the voltages backwards? If the supply voltage was lowered wouldn't it mean that older equipment designed for 240V would have an even bigger safety margin when it comes to over voltage?
- makomk 7y ago
  It could be related to the official change in supply voltage, though not for the reason you're thinking. Although Britain's mains supply officially changed to 230V, in reality it's the same 240V as before and there's a wider tolerance band for voltages above the nominal 230V than below to make this work. So if Network Rail installed equipment that expected actual continental-style 230V, that could cause problems. It wouldn't be the first time that rail companies not paying attention to actual grid specs caused transport chaos this year. Back in August, a power failure caused load shedding that dropped a million houses off the grid but preserved power to the rail network as intended. However, one train company had bought Siemens trains that couldn't cope and immediately shut down when they saw a dip in frequency of the level required to cause automatic load shedding, and required an engineer visit to restart. That one set of trains effectively rendered all the measures taken to ensure the rail network retained power almost useless, as they turned into immobile blocks all over the network.
  - avianlyric 7y ago
    > rail companies not paying attention to actual grid specs I would give the rail companies a little slack. They did specify trains that should have handled the frequency change. But Siemens didn’t build the trains to spec. The shutdown was unexpected and entirely down to Siemens not building their power supplies the specs set out by the rail company.
amluto 7y ago
One odd omission from the article is why it took an hour to reset everything. Given that they have redundant power, I would imagine that they could remotely select which power supply to use or at least to remotely reset the breakers. Alternatively, a good enough automatic transfer switch should transfer the power supply rather than tripping if the primary supply goes out of acceptable parameters. (This goes to one of basics of high availability: failures, when they happen, should be short.)
- avianlyric 7y ago
  Most power systems are built with a mix or clever automated systems that can self recover, along with backup dumb systems that can’t. This is done to ensure that if your smart stuff fails, you don’t end up seriously damaging something. Unfortunately the dumb stuff pretty much always has to be reset manually. Usually because the dumb stuff tripping means you’ve encountered a scenario that no one anticipated, you really want someone to double check that nothing got damaged before the safety tripped. It’s a little like a database starting up and discovering that part of its Write-Ahead-Log is corrupt. You really want a human to go in there and see what happened, rather than just ignoring the damage, plowing on and loosing data.
- gsnedders 7y ago
  > One odd omission from the article is why it took an hour to reset everything. Given that they have redundant power, I would imagine that they could remotely select which power supply to use or at least to remotely reset the breakers. No remote control of them, had to get people to each breaker. (This was mentioned on Twitter closer to the time by NR.)
vladd 7y ago
By reading the postmortem I got the feeling that it focuses primarily on providing a timeline of historical events without being future-looking. I did not find any questions that would trigger a path of improvement for similar future events: - Why did the recovery take so long (up until Thursday morning)? - If the surge lasted for 20 seconds, why weren't the systems operational again 1 minute after the event? - Why is an on-site technician needed in order to re-activate the equipment? - Could we build a system that enables the reactivation remotely? There might be very well-rooted answers to the questions above (based on how real world electrical supplies work), but if the questions aren't asked, we won't end up exploring better solutions for the answers.
- goodcanadian 7y ago
  Why did the recovery take so long (up until Thursday morning)? Systems were up within an hour, but if you throw an hour's delay into a busy part of the rail system, you have knock on effects (trains aren't where they are supposed to be) that will basically last until the end of the service day. If the surge lasted for 20 seconds, why weren't the systems operational again 1 minute after the event? Why is an on-site technician needed in order to re-activate the equipment? Basically, the breakers were flipped by the surge. Resetting such things is best done manually on site to ensure there was no equipment damage that occurred because of the surge. Could we build a system that enables the reactivation remotely? Probably, but that strikes me as dangerous. The article explicitly talks about investigating what can be done to make their systems more robust to such power events. I think it is very future looking.
  - gsnedders 7y ago
    > Systems were up within an hour, but if you throw an hour's delay into a busy part of the rail system, you have knock on effects (trains aren't where they are supposed to be) that will basically last until the end of the service day. And at a location where almost all lines are running at near capacity you can't "catch up" by getting more trains through the location once things are working again to get stock back in place, you can only get the regular timetable through it.
  - 7y ago
jhiouoiurew 7y ago
> how we can improve their resilience so we can stop this from happening in future I'm not so sure about that. Just a few months ago a temporary power cut in London left some modern Siemens trains stuck for hours because they had shutdown do to irregular power and had to be manually rebooted. Apparently nobody thought to investigate if the signaling system is also vulnerable to irregular power.
makach 7y ago
It's not necessarily a failure when you fail. This PM exuberates professionalism with a clear and concise message! This is an excellent learning experience, for bystanders, travelers, and employees. Seems to me that this was handled in a good way. The issue was highly unusual and difficult to anticipate but now you know. Good luck mitigating this issue preparing for the future. You will now be better as a result of this failure.
szc 7y ago
In my opinion, the problem is in poor logical thinking in the design of the supply switchover systems. The article indicates there are multiple power feeds and power supplies (presumably to provide low voltage equipment power). For a single power feed all of the following results in the loss of the output, (1) no input voltage, (2) input over voltage (under voltage also???). However, in the article it implies that switching to one of the other redundant power feeds or power supplies only happens for (1). If the redundancy system had considered either (1) or (2) as a fault and to "find another supply, then shutdown this supply" then complete shutoff would have been avoided. Also, since the active power supply shut down and switchover also didn't happen, there must not be any monitoring of the output of the individual supplies to trigger a switchover. To me, this suggests there is also another single point of failure in the current design. I guess there is a market for smarter power failover systems.

129 points by lelf 7y ago

Acccording to [1] just 67% of all UK trains arrived on time (no more than a minute late), with 2.8% scheduled trains cancelled all togther. How does it fare against other EU countries? What is so different abount Japan trains, that they are just 0.7 minutes late on average? [1] https://dataportal.orr.gov.uk/media/1630/passenger-performance-2019-20-q2.pdf https://dataportal.orr.gov.uk/media/1630/passenger-performan...

kiwijamo 7y ago
67% for right time (i.e. <1min late) arrival seems to be pretty average. I’d be more interested in the % of arrival at destination within 5mins as that’s the standard used in many countries and thus easier to compare different systems.
- gsnedders 7y ago
  In many parts of the GB network, once you get more than a minute or two late you end up with delays propagating throughout the network, because enough is timetabled with lines near capacity. You get up to about ~82% within 3 minutes and ~88% within 5 minutes. Beyond that you see little gains.
rwmj 7y ago
I can tell you from experience the trains in the UK are expensive and miserable. The common theory for why is because of the mess made of privatising British Rail in the 1990s. The government did it in a way which managed to create monopolies while at the same time splitting out responsibility between different companies in such a way that millions are drained away on legal costs while the companies argue with each other about who is responsible for each problem. There are many articles analysing the problems you can find online, here's one that turned up at the top of search: https://www.theguardian.com/uk-news/2018/jan/07/railways-privatisation-nationalisation https://www.theguardian.com/uk-news/2018/jan/07/railways-pri... Edit: Better article: https://www.citylab.com/transportation/2012/09/why-britains-railway-privatization-failed/3378/ https://www.citylab.com/transportation/2012/09/why-britains-...
- deleted 7y ago
  [deleted]
- NeedMoreTea 7y ago
  It's a well worn topic. Had they not achieved the weird corporate separation, such that track, train, station building and ticket can all be separate companies putting on a new service, or adjusting timetabling and even putting up new posters could have been achievable things. As is they spend months, and millions going nowhere. [Edit: Not forgetting that station hotels, ferries and hospitality were separated out to yet more companies. Just because.] Had they just gone back to the big 4 of 1921-1947, or created a "medium 8" with extra rights to run charters on the tracks we might have rail that works by now. Then there's the 50+ year story of line and infrastructure upgrades... Each time we update, we seem to update to the previous generation's capability.
  - simplesleeper 7y ago
    Britain runs 20% of Europe's rail journeys. It has the least accidents and deaths of passengers and staff. It is considered the most improved rail network year on year since 1997. It has some of the lowest fares if you book far in advance (and count the cost of subsidisation for foreign fares).
    - NeedMoreTea 7y ago
      How many decades did it take to electrify Europe's busiest line, the West Coast Main Line? Did it become modern? No. It used last generation signalling such that when we had 125mph HST they were limited to 80 and 100, when we bought back BR's APT as 150mph Pendolino they were limited to 1980s HST speeds as the tracks were electrified and upgraded to 125mph. In 2010. The "negotiations" between train operators wanting better rolling stock, railtrack (government), station upgrades and the inconvenient length of franchises made the whole process suitable for a Laurel and Hardy or Buster Keaton movie. Little wonder they're most improved, they gave themselves so much scope...
    - lentil_soup 7y ago
      Do you have any sources where I can read more about this? I find it surprising
rtpg 7y ago
I bet a big part of it in Japan is the sheer number of trains running drags down that average. If you count the Yamanote, with peak frequency of trains every 2 or 3 minutes, you’re gonna get a lot of stuff on time. Think about how rare power outages are nowadays. When infrastructure becomes important enough it becomes run pretty smoothly.
- fishnchips 7y ago
  Rail !== tube. You’d be comparing Tokyo’s Yamanote to London’s Circle. I used both to commute at one point or another and found them equally robust. OTOH rail in the UK and Japan are very different stories indeed.
  - rtpg 7y ago
    I think it would depend on the stats used. Yamanote is still JR. The Joban line goes all the way to Sendai, if you really want it to. So if you got lateness stats on trains, it would likely take all the JR stuff and mix stuff up I agree with your sentiment, my comment is more about my theory that stats would not make the distinction (though it would likely exclude things like Tokyo Metro or the monorails).
jacquesm 7y ago
One part may well be that some of the rolling stock should probably be in a museum, not transporting passengers daily.

kiwijamo 7y ago

Nice to see a post mortem made public. Also fairly well written for laymen while still retaining detail. Most rail infrastructure operators from other countries don’t make this sort of info public (apart from PR type messages which typically don’t explain the issue at at all and are more focused on deflecting blame etc). Kudos to Network Rail.

richardcrossley 7y ago
Cynical me: The UK railway system has plenty of practice of delays and cancellations, it seems they have learned to at least provide some detail in "lay" form. Practical me: Having that information allows better decisions to be made. Rather than simply wasting my evening at a station other arrangements can be made, possibly having a meal in a local bar/restaurant rather than shivering on a station.
paxswill 7y ago
One of the things I've really liked about my local commuter rail (Virginia Railway Express [0]) is the level of detail and communication they have with the community/riders. Their RSS feed typically has a post if there's any delay over 10 minutes explaining the delay and how long it is expected to be. They'll also post when the GPS tracking for a train isn't working but is still on schedule. The after-incident reports are also really nice; this one [1] resulted in delays of over an hour and a half because of track slippage leaving a station. Since then VRE has been letting us (riders) know if they're reversing the engine placement to mitigate possible slippage, so we're able to see that they're actually implementing the changes they propose. Other disruptions are also communicated fairly clearly and promptly [2]. As a side note, I'm pleasantly surprised by the transparency with historical service levels [3] and passenger utilization [4]. Saying all this though, the level of service is low when compared to other countries' commuter rail systems. One of the challenges VRE works with is that it hasn't owned the rail lines it's running on; they're owned by a freight railroad and passenger rail is typically run at a lower priority than freight. There has been some recent movement though by the state [5] to significantly improve passenger rail transit, especially in the northern part of the state (almost certainly related to Amazon HQ2). 0: https://www.vre.org/ https://www.vre.org/ 1: https://content.govdelivery.com/accounts/VAVRE/bulletins/26af3e9 https://content.govdelivery.com/accounts/VAVRE/bulletins/26a... 2: https://content.govdelivery.com/accounts/VAVRE/bulletins/26fe6ae https://content.govdelivery.com/accounts/VAVRE/bulletins/26f... 3: https://www.vre.org/service/daily-performance/ https://www.vre.org/service/daily-performance/ 4: https://www.vre.org/service/rider/train-utilization-trends/ https://www.vre.org/service/rider/train-utilization-trends/ (not actually trends) 5: https://www.governor.virginia.gov/newsroom/all-releases/2019/december/headline-850120-en.html https://www.governor.virginia.gov/newsroom/all-releases/2019...

herpderperator 7y ago

> We are investigating whether fitting Uninterruptable Power Supplies to each signalling location would have prevented this failure, although it is also possible they would have had to shut down to protect themselves too, as happened at another location during the same event. This specific remark sounds rather amateurish. This is part of the reason UPSes were invented. A Double Conversion Online UPS would have been able to handle this situation just fine.

wilhil 7y ago
I worked for an ISP that was deploying switches nationwide... We got a lot of space in BT Telephone Exchanges and thought nothing more of it. I have never seen such dirty and bad power! Literally once a week we had to go to exchanges to reset fuses that had tripped. We then started deploying (rather entry level) UPS to all locations and the problem just disappeared completely... we then invested in more high end UPS systems, but, yeah - dirty power is a big problem and UPS help.
- Aloha 7y ago
  I'm surprised CO gear was not running on -48v rather than AC power
  - viraptor 7y ago
    Is DC actually common these days? I haven't been to a physical datacenter in a decade or do, but I remember almost everything running on AC.
    - wilhil 7y ago
      I'm an IT and networking guy... This is one thing that got on my nerves completely - most equipment comes as AC or DC, and I'm never sure what to order, so, I usually go for the AC one as it's the safe choice
    - Aloha 7y ago
      It is in Telecom Applications.

nathell 7y ago

I don’t mind delays very much, even long ones, as long as there is information readily available about why the delay happens and how long it is likely to last. I have very positive experience with UK railways in this regard; this post is a prime example. Contrast with PKP (Polish rail), where it’s not uncommon for a train to stand in the middle of nowhere for half an hour, without so much as a single word of explanation from staff. It’s changing for the better, but slowly.

Aloha 7y ago

A 20 second overvolt period does seem exceedingly long - I'm curious to find the cause of that.

ksec 7y ago
Yes I thought power surge happens in milliseconds to a second range, may be few seconds at worst, first time I ever read a double digit seconds power surge. Luckily the system shutter itself down as intended.

Animats 7y ago

OK, that's what should happen for an overvoltage. The control systems tripped offline to protect themselves, all signals went to red, and all trains stopped. Electric railways have multiple different power systems - traction power, signaling power, and utility power, at least. The possibility of a short between systems means a conservative shutdown-on-overvoltage circuit breaker setup is appropriate.

ec109685 7y ago
The need to manually turn this system back on is non optimal. A control loop powered by a battery backup should have been able to reset the system once the voltage came back down.
- Animats 7y ago
  This is a railroad signaling system. It's designed to fail safe, and that's not a buzzword. Any broken wire, go to red. Any broken relay, go to red. Any rail break, go to red. Loss of power, go to red. If you're getting power surges in signaling power, it's time to shut down, go to red, and not reset until someone has tested the system.
  - ec109685 7y ago
    The backup system can be programmed to do the same testing a human can to bootstrap the system. Another benefit is that they can simulate a surge in the system regularly and watch it recover itself versus a human running a checklist once every 10 years.

londons_explore 7y ago

Rail equipment is all ancient and highly redundant for both uptime and safety. Yet the actual system reliability and safety doesn't seem great. Signal outages happen nearly every day in the UK, and delay tens of thousands of passengers. Safety failures in signalling systems kill on average a few people per year. How about a modern system, built with JavaScript-de-jure, running over 4G and WiFi, consumer GPS, "Download train-control from the app store", running on tablets or the train drivers BYOD phone, etc. How would it work out for safety/reliability? I suspect counterintuitively it would work out much better. Simple cheap portable devices means a bunch of spares can be around always. "Phone battery dead? No worries, there's a tablet I can control the train from too". No WiFi? It also works with 4G or Bluetooth. Server down? It has a Peer connection backup. Hackers broke in? The system has safety logic built in to every device, and uses a consensus system so a few rogue systems can't crash trains. Power supply broken? We have 24 hours of battery, longer if I turn the brightness down.

bschwindHN 7y ago
> How about a modern system, built with JavaScript-de-jure No thank you.
- WrtCdEvrydy 7y ago
  import { train } from 'lodash';
thinkingemote 7y ago
Oh my sweet summer silicon valley child! We should disrupt every safety equipment and process everywhere!
pergadad 7y ago
Better 1000 failures than one deadly accident. These systems are built in basic and redundant ways for a reason. See also https://news.ycombinator.com/item?id=12071240 https://news.ycombinator.com/item?id=12071240
seanparsons 7y ago

bloak 7y ago

The article doesn't mention a date. Presumably the "Wednesday" referred to is 18 Dec 2019. On a previous occasion when rail equipment failed because of a fluctuation in supply voltage the electricity company was not at fault: the voltage had remained within specified limits. I find it a bit suspicious that the article doesn't mention what the change in voltage was this time, and what variation the specification allows. The penultimate paragraph is perhaps an implicit admission that the electricity supply was within its specification. I wonder if any of this is indirectly related to the official change in supply voltage, from 240 V to 230 V, in 1995. Nah, probably not; they can't have equipment that old, can they?

Nextgrid 7y ago
Did you get the voltages backwards? If the supply voltage was lowered wouldn't it mean that older equipment designed for 240V would have an even bigger safety margin when it comes to over voltage?
makomk 7y ago
It could be related to the official change in supply voltage, though not for the reason you're thinking. Although Britain's mains supply officially changed to 230V, in reality it's the same 240V as before and there's a wider tolerance band for voltages above the nominal 230V than below to make this work. So if Network Rail installed equipment that expected actual continental-style 230V, that could cause problems. It wouldn't be the first time that rail companies not paying attention to actual grid specs caused transport chaos this year. Back in August, a power failure caused load shedding that dropped a million houses off the grid but preserved power to the rail network as intended. However, one train company had bought Siemens trains that couldn't cope and immediately shut down when they saw a dip in frequency of the level required to cause automatic load shedding, and required an engineer visit to restart. That one set of trains effectively rendered all the measures taken to ensure the rail network retained power almost useless, as they turned into immobile blocks all over the network.
- avianlyric 7y ago
  > rail companies not paying attention to actual grid specs I would give the rail companies a little slack. They did specify trains that should have handled the frequency change. But Siemens didn’t build the trains to spec. The shutdown was unexpected and entirely down to Siemens not building their power supplies the specs set out by the rail company.

amluto 7y ago

One odd omission from the article is why it took an hour to reset everything. Given that they have redundant power, I would imagine that they could remotely select which power supply to use or at least to remotely reset the breakers. Alternatively, a good enough automatic transfer switch should transfer the power supply rather than tripping if the primary supply goes out of acceptable parameters. (This goes to one of basics of high availability: failures, when they happen, should be short.)

avianlyric 7y ago
Most power systems are built with a mix or clever automated systems that can self recover, along with backup dumb systems that can’t. This is done to ensure that if your smart stuff fails, you don’t end up seriously damaging something. Unfortunately the dumb stuff pretty much always has to be reset manually. Usually because the dumb stuff tripping means you’ve encountered a scenario that no one anticipated, you really want someone to double check that nothing got damaged before the safety tripped. It’s a little like a database starting up and discovering that part of its Write-Ahead-Log is corrupt. You really want a human to go in there and see what happened, rather than just ignoring the damage, plowing on and loosing data.
gsnedders 7y ago
> One odd omission from the article is why it took an hour to reset everything. Given that they have redundant power, I would imagine that they could remotely select which power supply to use or at least to remotely reset the breakers. No remote control of them, had to get people to each breaker. (This was mentioned on Twitter closer to the time by NR.)

vladd 7y ago

By reading the postmortem I got the feeling that it focuses primarily on providing a timeline of historical events without being future-looking. I did not find any questions that would trigger a path of improvement for similar future events: - Why did the recovery take so long (up until Thursday morning)? - If the surge lasted for 20 seconds, why weren't the systems operational again 1 minute after the event? - Why is an on-site technician needed in order to re-activate the equipment? - Could we build a system that enables the reactivation remotely? There might be very well-rooted answers to the questions above (based on how real world electrical supplies work), but if the questions aren't asked, we won't end up exploring better solutions for the answers.

goodcanadian 7y ago
Why did the recovery take so long (up until Thursday morning)? Systems were up within an hour, but if you throw an hour's delay into a busy part of the rail system, you have knock on effects (trains aren't where they are supposed to be) that will basically last until the end of the service day. If the surge lasted for 20 seconds, why weren't the systems operational again 1 minute after the event? Why is an on-site technician needed in order to re-activate the equipment? Basically, the breakers were flipped by the surge. Resetting such things is best done manually on site to ensure there was no equipment damage that occurred because of the surge. Could we build a system that enables the reactivation remotely? Probably, but that strikes me as dangerous. The article explicitly talks about investigating what can be done to make their systems more robust to such power events. I think it is very future looking.
- gsnedders 7y ago
  > Systems were up within an hour, but if you throw an hour's delay into a busy part of the rail system, you have knock on effects (trains aren't where they are supposed to be) that will basically last until the end of the service day. And at a location where almost all lines are running at near capacity you can't "catch up" by getting more trains through the location once things are working again to get stock back in place, you can only get the regular timetable through it.
- 7y ago

jhiouoiurew 7y ago

> how we can improve their resilience so we can stop this from happening in future I'm not so sure about that. Just a few months ago a temporary power cut in London left some modern Siemens trains stuck for hours because they had shutdown do to irregular power and had to be manually rebooted. Apparently nobody thought to investigate if the signaling system is also vulnerable to irregular power.

makach 7y ago

It's not necessarily a failure when you fail. This PM exuberates professionalism with a clear and concise message! This is an excellent learning experience, for bystanders, travelers, and employees. Seems to me that this was handled in a good way. The issue was highly unusual and difficult to anticipate but now you know. Good luck mitigating this issue preparing for the future. You will now be better as a result of this failure.

szc 7y ago

In my opinion, the problem is in poor logical thinking in the design of the supply switchover systems. The article indicates there are multiple power feeds and power supplies (presumably to provide low voltage equipment power). For a single power feed all of the following results in the loss of the output, (1) no input voltage, (2) input over voltage (under voltage also???). However, in the article it implies that switching to one of the other redundant power feeds or power supplies only happens for (1). If the redundancy system had considered either (1) or (2) as a fault and to "find another supply, then shutdown this supply" then complete shutoff would have been avoided. Also, since the active power supply shut down and switchover also didn't happen, there must not be any monitoring of the output of the individual supplies to trigger a switchover. To me, this suggests there is also another single point of failure in the current design. I guess there is a market for smarter power failover systems.