3 ms·

Chaos Engineering in Healthcare: Getting Started with Sensitive Workloads

40 points by kiyanwang 6y ago

monkeydreams 6y ago
I'm not seeing anything particularly insightful in regards to Chaos Engineering with Healthcare. None of the unique challenges or strengths in the Healthcare domain are covered. Few industries have the baked-in resilience to systems failures that the Healthcare industry possess. EMR down? Move to paper for arrivals. MPI offline? Here are some pre-allocated IDs and paper forms to fill out. Healthcare providers have a different measure for criticality. Merely slow, merely down - these can increase patient or financial risk, but they are generally manageable in the frame of reference of an outage. There are always manual workflows for these. Corruption of a handful of patient records due to an interpretation of an obscure standard? That is a major risk and will unleash real chaos.
- devin 6y ago
  Agreed. What I was more interested in hearing about was some sort of approach to deidentification of PHI in lower environments and synthetic traffic consisting of same.
- pc86 6y ago
  I've worked at a private healthcare company for the past 2 years, and worked at a public healthcare company for several years earlier in my career. Coming from e-commerce, consulting, and logistics where downtime === lose revenue, it's refreshing how many times I've heard "X is down but it's not affecting patient care so is not an emergency."
OliverJones 6y ago
Cerner and other electronic medical records systems have, as a huge component of their operation, a scheme for ingesting HL7 data streams (and emitting them to other systems). It's likely the reference to ingestion in the article refers to that. Chaos-monkeying with those data streams is flat out a good idea when building robust HL7 IT systems. HL7 encodes a series of changes. For example, you might get records like this. at 2020-10-06T14:33:00 patient 144616 moved from Floor 6 Rm 603 Bed 2 to OR3 at 2020-10-06T15:47:00 patient 144616 moved from OR3 to Recovery 3 at 2020-10-06T18:20:00 patient 144616 moved from Recovery 3 to Floor 6 Rm 603 Bed 2 It's all state changes. Reconstructing the current state of affairs requires knowing which state was the latest for interesting object (patient, usually, but also maybe OR or other location in my trivial example). These HL7 records don't necessarily show up in chronological order either. Trouble is, preexisting chaos monkeys (bugs) in systems like these sometimes say to themselves "oops somebody lost sync, I better resync." Resyncing is done by resending a backlog of many days' worth of HL7 records to the unsuspecting consumer of those records. So anything that can be done, by Cerner or any other HL7-using system, continually to test the oddball edge cases involving loss of sync is a good idea. And, oh, yeah, everything in those records is protected health information. And often they're sent over ordinary TCP streams without encryption. Health care IT relies on firewalls big time. I don't think the linked article is written for most HN readers; it's written for Cerner CIOs and other in-the-trenches health care IT workers.

40 points by kiyanwang 6y ago

I'm not seeing anything particularly insightful in regards to Chaos Engineering with Healthcare. None of the unique challenges or strengths in the Healthcare domain are covered. Few industries have the baked-in resilience to systems failures that the Healthcare industry possess. EMR down? Move to paper for arrivals. MPI offline? Here are some pre-allocated IDs and paper forms to fill out. Healthcare providers have a different measure for criticality. Merely slow, merely down - these can increase patient or financial risk, but they are generally manageable in the frame of reference of an outage. There are always manual workflows for these. Corruption of a handful of patient records due to an interpretation of an obscure standard? That is a major risk and will unleash real chaos.

devin 6y ago
Agreed. What I was more interested in hearing about was some sort of approach to deidentification of PHI in lower environments and synthetic traffic consisting of same.
pc86 6y ago
I've worked at a private healthcare company for the past 2 years, and worked at a public healthcare company for several years earlier in my career. Coming from e-commerce, consulting, and logistics where downtime === lose revenue, it's refreshing how many times I've heard "X is down but it's not affecting patient care so is not an emergency."

OliverJones 6y ago

Cerner and other electronic medical records systems have, as a huge component of their operation, a scheme for ingesting HL7 data streams (and emitting them to other systems). It's likely the reference to ingestion in the article refers to that. Chaos-monkeying with those data streams is flat out a good idea when building robust HL7 IT systems. HL7 encodes a series of changes. For example, you might get records like this. at 2020-10-06T14:33:00 patient 144616 moved from Floor 6 Rm 603 Bed 2 to OR3 at 2020-10-06T15:47:00 patient 144616 moved from OR3 to Recovery 3 at 2020-10-06T18:20:00 patient 144616 moved from Recovery 3 to Floor 6 Rm 603 Bed 2 It's all state changes. Reconstructing the current state of affairs requires knowing which state was the latest for interesting object (patient, usually, but also maybe OR or other location in my trivial example). These HL7 records don't necessarily show up in chronological order either. Trouble is, preexisting chaos monkeys (bugs) in systems like these sometimes say to themselves "oops somebody lost sync, I better resync." Resyncing is done by resending a backlog of many days' worth of HL7 records to the unsuspecting consumer of those records. So anything that can be done, by Cerner or any other HL7-using system, continually to test the oddball edge cases involving loss of sync is a good idea. And, oh, yeah, everything in those records is protected health information. And often they're sent over ordinary TCP streams without encryption. Health care IT relies on firewalls big time. I don't think the linked article is written for most HN readers; it's written for Cerner CIOs and other in-the-trenches health care IT workers.