6 ms·

A lot of the headline grabbing stuff wasn't the grid's fault though: * the railways didn't have traction power disconnected -- but one type of (new) train saw

by pm215 7y ago

A lot of the headline grabbing stuff wasn't the grid's fault though: * the railways didn't have traction power disconnected -- but one type of (new) train saw the within-specification frequency drop, shut down, and in many cases needed a fitter to go out and restart it. Everything else behind a dead train was stuck. I imagine the train company are discussing this with their supplier... * Ipswich hospital had a power outage, but that was because their own systems disconnected from the grid when they saw the frequency drop, and then they had a problem with one of their generators. *One regional airport seems to have been accidentally not on the list of key sites not to disconnect -- sounds like an admin snafu, and has now been fixed. None of these seem like "bad management" -- they are a collection of relatively minor, entirely unrelated, problems of the sort you're likely to run into if you have to activate an emergency procedure you haven't needed to use in a decade.

mytailorisrich 7y ago
I did mention the railways and indeed the importance of making sure the whole system can cope, which means what you list in addition to the grid in isolation. > None of these seem like "bad management" They do seem like bad management because to me it looks like these issues were not taken seriously. > problems of the sort you're likely to run into if you have to activate an emergency procedure you haven't needed to use in a decade. That's bad management right there if an organisation's emergency procedure has been collecting dust for a decade.
- DanBC 7y ago
  > They do seem like bad management. But not bad management by the Grid.
  - mytailorisrich 7y ago
    Including by the grid because key infrastructures did lose power. But in general a worrying aspect of what happened is that there seems to have been oversights all over the place (grid, hospitals, railways, etc).
    - DanBC 7y ago
      How many 9s do you want from the grid? There are lots of reasons why hospitals lose power, which is why they all have backup generators. Ipswich's backup generator had a faulty battery. This is nothing to do with the grid. This is entirely the fault of the hospital trust. Most of the hospital's backups worked. https://www.ipswichstar.co.uk/news/ipswich-hospital-blackout-caused-by-faulty-battery-1-6219161 https://www.ipswichstar.co.uk/news/ipswich-hospital-blackout... > A spokeswoman said: "Our initial investigation has shown that at 4.52pm on Friday, August 9, we lost our electricity supply until 5pm. We have 11 generators at Ipswich Hospital and all of them worked immediately. > "What we have found is that the battery which automatically operates one of the switch over systems from mains power to generator power was faulty despite being within its recommended life. Here's the trust document talking about the new installation when it was put in. http://www.ipswichhospital.nhs.uk/aboutourhospital/Documents/Publication%20Scheme/Annual%20Report%202015-16.pdf http://www.ipswichhospital.nhs.uk/aboutourhospital/Documents... > Electrical High Voltage Infrastructure was a critically important project to the Trust in providing a resilient high voltage electrical network to the hospital. Through a range of electrical projects, including the replacement and upgrading of outpatients generator ‘C’, the maternity generator project and the upgrading of the high voltage switchgear for the forthcoming biofuels project, this represented a total overall investment in excess of £2m. Virtually all the Trust’s electrical switchgear has now been improved and modernised. Along with the new backup generators the hospital is protected for years to come and a major significant risk to hospital has been dealt with.
      - mytailorisrich 7y ago
        It's clear that there were issues at hospitals (as expected cynical me thinks, knowing British healthcare...) but the fact that key infrastructure lost power does points to issues at the grid level as well (because sites that lost power did so based on grid procedures).
        Zenst 7y ago
        When they cut the power to an area, they can't selectively keep isolated locations active unless they are isolated in that they operate upon their own spur. That is why Hospitals would be deemed critical and important not to cut out, they are handled by onsight failover power supply in the form of backup generators. In this case, it was like asking somebody to be able to catch a ball, they say they are ready and yet fail to catch that ball when you thru it to them. Is that the fault of the person who throws the ball or the person that was not prepared to catch the ball when they said they could. In this instance it falls into the infortunate situation, and from what I can tell, you can't lay blame onto anybody or party for the failure and it is just one of those things. Sure leasons will be learned, but from the grid procedure in this hospital case, there will be no change I can see that would be learned. However the Hospital will be learning from this and may have more frequent testing and checks of their failover, may make changes. So if blame is to be leveraged upon anybody - you could say it would be from the party that had to make changes. But in this instance, would that be fair. Hence I would call this unfortunate and whilst lessons will be learned, you can not lay blame out of those lessons all the time. However - if it happens again and those same mistakes transpire - then, maybe you have some grounds to lay some blame.
- mellosouls 7y ago
  Reading this exchange as a third party, both sides make fair points, but one side is being excessively downvoted, which is either unfortunate or possibly a regrettable sign of emotional over intellectual engagement in the discussion somewhere.
  - mytailorisrich 7y ago
    It is rather odd, indeed, but usual. Their often seems to be an "incorrect" opinion. It seems that one of the commenters work for the National Grid or related so might be a bit partial...
baxtr 7y ago
Sounds like a good chance to test the systems
pedrocr 7y ago
In a way these events become a blessing in disguise because of those things. We've had to deal with similar crisis situations in our supply chain. It really helps to test your procedures and operations fixing stuff along the way. The team also gains experience and becomes much more prepared for worse events should they come.
- toyg 7y ago
  I would go as far as suggesting that these things should be routinely tested. In the UK most public buildings test fire alarms once a week; a programmed plan of localised, managed blackouts, to be carried out once every few years, would probably help in the long run - particularly as the grid is rapidly changing as described at the end of the post.
  - pedrocr 7y ago
    Definitely. We use seasonal demand peaks to learn how to cope with extremes. But if we didn't have those plus strikes I'd definitely be planning a few test runs.
  - mytailorisrich 7y ago
    Of course they should be routinely tested. This is not a new idea and is an established best practice. This is why I am very surprised when I read that hospitals generators failing to take over is just "one of those things". It's like the 'old' saying that you don't have backups until you test recovery from backups...
    - andygates 7y ago
      We didn't hear about all the other hospitals that easily switched over to generators, just the one that failed. There's room for investigation there, but it's on the site, not the system.
topspin 7y ago
I read your list and see a bunch of 'technical debt.' Trains put into operation without the ability to survive a power transient without intervention. Life critical backup power system failures. The set of critical sites is not being maintained properly. In my mind these failures are are not entirely unrelated; reliable power is taken for granted and the testing and maintenance necessary to cope with power outages is neglected.
- mytailorisrich 7y ago
  This is not technical debt. These are signs of oversights or potential poor management. Indeed that's what neglecting these procedures is. This is why a comprehensive investigation is needed to ensure the details are known and all lessons learned or, unfortunately, re-learned.
  - walshemj 7y ago
    Its shockly poor procurement - those new trains are not fit for purpose.
    - growse 7y ago
      Do you mean that they don't meet the requirements of the tender spec ("not fit for purpose") or that the spec in the tender process was deficient ("shockingly poor procurement")?
      - deleted 7y ago
        [deleted]
      - walshemj 7y ago
        Both the older electrostars where far better
- growse 7y ago
  > Trains put into operation without the ability to survive a power transient without intervention. The trains were built to a environmental tolerance spec, and shutting down when that spec is exceeded seems like a reasonable thing to do. Having an expert test that all the safety systems are still working properly after they've been exposed to conditions they weren't designed for also seems prudent. Do you think people should pay higher ticket prices / more tax to fund trains that can tolerate a wider range of input frequencies, given how rarely these sorts of events happen?
walshemj 7y ago
Ah I wondered why the ST Pancras line was in such a mess hours later

by pm215 7y ago

mytailorisrich 7y ago

I did mention the railways and indeed the importance of making sure the whole system can cope, which means what you list in addition to the grid in isolation. > None of these seem like "bad management" They do seem like bad management because to me it looks like these issues were not taken seriously. > problems of the sort you're likely to run into if you have to activate an emergency procedure you haven't needed to use in a decade. That's bad management right there if an organisation's emergency procedure has been collecting dust for a decade.

DanBC 7y ago
> They do seem like bad management. But not bad management by the Grid.
- mytailorisrich 7y ago
  Including by the grid because key infrastructures did lose power. But in general a worrying aspect of what happened is that there seems to have been oversights all over the place (grid, hospitals, railways, etc).
  - DanBC 7y ago
    How many 9s do you want from the grid? There are lots of reasons why hospitals lose power, which is why they all have backup generators. Ipswich's backup generator had a faulty battery. This is nothing to do with the grid. This is entirely the fault of the hospital trust. Most of the hospital's backups worked. https://www.ipswichstar.co.uk/news/ipswich-hospital-blackout-caused-by-faulty-battery-1-6219161 https://www.ipswichstar.co.uk/news/ipswich-hospital-blackout... > A spokeswoman said: "Our initial investigation has shown that at 4.52pm on Friday, August 9, we lost our electricity supply until 5pm. We have 11 generators at Ipswich Hospital and all of them worked immediately. > "What we have found is that the battery which automatically operates one of the switch over systems from mains power to generator power was faulty despite being within its recommended life. Here's the trust document talking about the new installation when it was put in. http://www.ipswichhospital.nhs.uk/aboutourhospital/Documents/Publication%20Scheme/Annual%20Report%202015-16.pdf http://www.ipswichhospital.nhs.uk/aboutourhospital/Documents... > Electrical High Voltage Infrastructure was a critically important project to the Trust in providing a resilient high voltage electrical network to the hospital. Through a range of electrical projects, including the replacement and upgrading of outpatients generator ‘C’, the maternity generator project and the upgrading of the high voltage switchgear for the forthcoming biofuels project, this represented a total overall investment in excess of £2m. Virtually all the Trust’s electrical switchgear has now been improved and modernised. Along with the new backup generators the hospital is protected for years to come and a major significant risk to hospital has been dealt with.
    - mytailorisrich 7y ago
      It's clear that there were issues at hospitals (as expected cynical me thinks, knowing British healthcare...) but the fact that key infrastructure lost power does points to issues at the grid level as well (because sites that lost power did so based on grid procedures).
      - Zenst 7y ago
        When they cut the power to an area, they can't selectively keep isolated locations active unless they are isolated in that they operate upon their own spur. That is why Hospitals would be deemed critical and important not to cut out, they are handled by onsight failover power supply in the form of backup generators. In this case, it was like asking somebody to be able to catch a ball, they say they are ready and yet fail to catch that ball when you thru it to them. Is that the fault of the person who throws the ball or the person that was not prepared to catch the ball when they said they could. In this instance it falls into the infortunate situation, and from what I can tell, you can't lay blame onto anybody or party for the failure and it is just one of those things. Sure leasons will be learned, but from the grid procedure in this hospital case, there will be no change I can see that would be learned. However the Hospital will be learning from this and may have more frequent testing and checks of their failover, may make changes. So if blame is to be leveraged upon anybody - you could say it would be from the party that had to make changes. But in this instance, would that be fair. Hence I would call this unfortunate and whilst lessons will be learned, you can not lay blame out of those lessons all the time. However - if it happens again and those same mistakes transpire - then, maybe you have some grounds to lay some blame.
mellosouls 7y ago
Reading this exchange as a third party, both sides make fair points, but one side is being excessively downvoted, which is either unfortunate or possibly a regrettable sign of emotional over intellectual engagement in the discussion somewhere.
- mytailorisrich 7y ago
  It is rather odd, indeed, but usual. Their often seems to be an "incorrect" opinion. It seems that one of the commenters work for the National Grid or related so might be a bit partial...

baxtr 7y ago

Sounds like a good chance to test the systems

pedrocr 7y ago

In a way these events become a blessing in disguise because of those things. We've had to deal with similar crisis situations in our supply chain. It really helps to test your procedures and operations fixing stuff along the way. The team also gains experience and becomes much more prepared for worse events should they come.

toyg 7y ago
I would go as far as suggesting that these things should be routinely tested. In the UK most public buildings test fire alarms once a week; a programmed plan of localised, managed blackouts, to be carried out once every few years, would probably help in the long run - particularly as the grid is rapidly changing as described at the end of the post.
- pedrocr 7y ago
  Definitely. We use seasonal demand peaks to learn how to cope with extremes. But if we didn't have those plus strikes I'd definitely be planning a few test runs.
- mytailorisrich 7y ago
  Of course they should be routinely tested. This is not a new idea and is an established best practice. This is why I am very surprised when I read that hospitals generators failing to take over is just "one of those things". It's like the 'old' saying that you don't have backups until you test recovery from backups...
  - andygates 7y ago
    We didn't hear about all the other hospitals that easily switched over to generators, just the one that failed. There's room for investigation there, but it's on the site, not the system.

topspin 7y ago

I read your list and see a bunch of 'technical debt.' Trains put into operation without the ability to survive a power transient without intervention. Life critical backup power system failures. The set of critical sites is not being maintained properly. In my mind these failures are are not entirely unrelated; reliable power is taken for granted and the testing and maintenance necessary to cope with power outages is neglected.

mytailorisrich 7y ago
This is not technical debt. These are signs of oversights or potential poor management. Indeed that's what neglecting these procedures is. This is why a comprehensive investigation is needed to ensure the details are known and all lessons learned or, unfortunately, re-learned.
- walshemj 7y ago
  Its shockly poor procurement - those new trains are not fit for purpose.
  - growse 7y ago
    Do you mean that they don't meet the requirements of the tender spec ("not fit for purpose") or that the spec in the tender process was deficient ("shockingly poor procurement")?
    - deleted 7y ago
      [deleted]
    - walshemj 7y ago
      Both the older electrostars where far better
growse 7y ago
> Trains put into operation without the ability to survive a power transient without intervention. The trains were built to a environmental tolerance spec, and shutting down when that spec is exceeded seems like a reasonable thing to do. Having an expert test that all the safety systems are still working properly after they've been exposed to conditions they weren't designed for also seems prudent. Do you think people should pay higher ticket prices / more tax to fund trains that can tolerate a wider range of input frequencies, given how rarely these sorts of events happen?

walshemj 7y ago

Ah I wondered why the ST Pancras line was in such a mess hours later