4 ms·
> It was discovered that there was a change made to the alarm text during this AGC update that caused the failure. Prior to the update, when the PII value excee
by segfaultbuserr 7y ago
> It was discovered that there was a change made to the alarm text during this AGC update that caused the failure. Prior to the update, when the PII value exceeded +/-999 MW, the value in the alarm text defaulted to “xxxx,” preventing dispatchers from having an accurate indication of the amount of PII change. During this deployment, the two MW value fields in the PII alarm text were modified from i4 to i5 (4 digit integers to 5 digit integers) to allow for an additional digit.
> The original alarm text array had 79 characters which almost hit the max character limit of 80, and the change from i4 to i5 resulted in the alarm text increasing by 2 characters to 81 characters in length. Writing an 81 character text string into an 80 character fixed length array resulted in a run-time abort of the task.
I bet nobody expected that modifying a string can crash the entire control system due to a (caught) buffer overflow. Apparently the reason it went undetected was that the modification was not tested on a demo system first, since it was believed to be trivial.
> Lesson Learned: No matter how small the change is, validate changes in a test environment first. This requires using a thorough test script for the changes.
- 7thaccount 7y agoGenerally, SCADA, AGC, and any EMS software updates should go through multiple lower environments before getting into a production environment. Even critical patches would spend a little bit if time in a lower environment prior to production.
- deleted 7y ago[deleted]
- generatorguy 7y agoAre there lower AGC environments ? The document concluded they need to test more but I imagine there is just one AGC per system operator. BC hydro, fortis, BPA, PGE? AGC just balances power flows so imports/exports over interconnections between power authorities flow as scheduled right?
- 7thaccount 7y agoYeah it depends on the system operator, but I know some do for sure. In a test environment you might still receive the same input SCADA data, but could have entirely new calculations running. It's just that those calculations aren't sent anywhere.
- BunsanSpace 7y agoIf my SCADA work is anything, it goes on a demo/staging area for a week then is deployed, in 90% of cases. It's rare for an org (and usually only very large heavily regulated government orgs) do full testing on new builds.
- csense 7y agoI would assume controlling GW of power generation would be a very large, heavily regulated org...
- 7thaccount 7y agoIs your stuff for a system operator or single generator?
- _bxg1 7y agoSoftware was a mistake.