3 ms·

> There's a lot of people who know exactly what's going on. Who or where are these people? (For any industry / system.) In my experience, people understand th

by bmitc 4y ago

> There's a lot of people who know exactly what's going on. Who or where are these people? (For any industry / system.) In my experience, people understand things at a local level but quickly lose understanding at a system level. And I think there is a subtle difference from figuring out what went wrong once something did go wrong from understanding things prior to such an occurrence. Humans are pretty good at figuring out things after the fact, but we have a pretty poor track record of systematic understanding and preventing failures.

yourapostasy 4y ago
> Who or where are these people? (For any industry / system.) If you hear someone on a huge incident conference call leading the capturing of new information, threading their way from one subsystem to another (most of which are outside the person's usual silo) based upon what they see, paging in teams as necessary to get the answers they need, giving a blow-by-blow account of the troubleshooting picture they're currently painting, with reasoning behind what they think might be happening, how to test that, what possible next steps might be, instantly discarding anything that doesn't move everyone closer to resolution, crafting ad hoc "good enough" tooling on the spot as needed to get enough information to make a decision but not enough to know details to the last degree, freely admitting if they're on a subsystem they know nothing about yet able to quickly grasp the essentials in minutes from domain experts to work with those domain experts to pull just the information they need...you've found one of those people. Extremely rare to find in one person, usually you see 2-3 people in such incidents splitting such a dynamic. They don't know "exactly what's going on". But they sufficiently grasp how systems work to approximate that outcome close enough for most business purposes.
eep_social 4y ago
These people are SREs (source: am one) and there is a reason everyone is trying to hire a team of them. It’s rare to find much teaching on complex systems in school so the hiring pipeline is fraught with peril and low tier companies understand the buzz but not the job. In my experience SRE is mostly people wander in and realize they’re in the right place, rather than setting out to be good at the required skills like managing complex systems and large scale incident response.

by bmitc 4y ago

yourapostasy 4y ago

> Who or where are these people? (For any industry / system.) If you hear someone on a huge incident conference call leading the capturing of new information, threading their way from one subsystem to another (most of which are outside the person's usual silo) based upon what they see, paging in teams as necessary to get the answers they need, giving a blow-by-blow account of the troubleshooting picture they're currently painting, with reasoning behind what they think might be happening, how to test that, what possible next steps might be, instantly discarding anything that doesn't move everyone closer to resolution, crafting ad hoc "good enough" tooling on the spot as needed to get enough information to make a decision but not enough to know details to the last degree, freely admitting if they're on a subsystem they know nothing about yet able to quickly grasp the essentials in minutes from domain experts to work with those domain experts to pull just the information they need...you've found one of those people. Extremely rare to find in one person, usually you see 2-3 people in such incidents splitting such a dynamic. They don't know "exactly what's going on". But they sufficiently grasp how systems work to approximate that outcome close enough for most business purposes.

eep_social 4y ago

These people are SREs (source: am one) and there is a reason everyone is trying to hire a team of them. It’s rare to find much teaching on complex systems in school so the hiring pipeline is fraught with peril and low tier companies understand the buzz but not the job. In my experience SRE is mostly people wander in and realize they’re in the right place, rather than setting out to be good at the required skills like managing complex systems and large scale incident response.