4 ms·

I suppose they saw changes in the numbers, but they were still hitting their marks, so they didn't worry too much. I watched the livestream, and honestly, I di

I suppose they saw changes in the numbers, but they were still hitting their marks, so they didn't worry too much. I watched the livestream, and honestly, I didn't even know something bad had happened. Maybe they weren't glued to the video but rather the numbers on their screens? Although, historically it seems that NASA control teams pride themselves on their ability to stay levelheaded during bad situations. Are there a lot of ex-NASA employees at SpaceX?

ChuckMcM 14y ago
Its a lot of data to comprehend. Doing operations on a large cluster architecture has a similar issue which is that there are many things that can go wrong and are 'ok' because the system is designed to deal with it seamlessly. But I'm a big fan of tools that let you have a red/yellow/green indication of nominal/warn/fail for each sub system. Looking at such a display you can comprehend the entire system is 'ok' (or not).
- caw 14y ago
  I manage a lot of complex data where many things could result an undesired state. Traffic lights don't always illustrate these conditions well enough, or let you diagnose the problem quickly. Take the somewhat simple example of rebuilding a drive array. All drives = green; Within RAID tolerance = yellow Degraded = red What about the status of the hotspare? Is yellow rebuilding? What if there's no more hot spares, but you're within RAID tolerance? I'm not knocking traffic lights (sometimes they're good), but it's worth spending time trying to figure out your data displays to give you the whole picture. It takes a while, but makes monitoring so much easier.
  - ChuckMcM 14y ago
    Actually large format data displays are great for calling out unexpected transitions. I had a chance to look at the control panel of a nuclear reactor for an aircraft carrier once, and what I learned was that the panel had annunciator lights that would blink on an 'unexpected' change. So for example steam pressure would by static in one of two states (cold shutdown, operation) and smoothly changing when state changing, so the light would start blinking if either the steam pressure changed when the reactor wasn't changing state, or if it stopped changing in the correct way when the reactor was changing state. Looking across a sea of lights, if you saw one that was blinking you would go over to it, read off the legend (that would tell you the conditions under which that light would blink) and then take action. Once action was in process you pushed the button to 'acknowledge' the anomaly. (which also set new parameters for when it would start blinking again) All in all a very cool system. I've been sorely tempted to hack something similar for our search and crawling clusters. Sadly I don't think its practical to have a mockup of the Enterprise-D warp core which pulses in response to query rate, although that would be very very cool :-).

ChuckMcM 14y ago

Its a lot of data to comprehend. Doing operations on a large cluster architecture has a similar issue which is that there are many things that can go wrong and are 'ok' because the system is designed to deal with it seamlessly. But I'm a big fan of tools that let you have a red/yellow/green indication of nominal/warn/fail for each sub system. Looking at such a display you can comprehend the entire system is 'ok' (or not).

caw 14y ago
I manage a lot of complex data where many things could result an undesired state. Traffic lights don't always illustrate these conditions well enough, or let you diagnose the problem quickly. Take the somewhat simple example of rebuilding a drive array. All drives = green; Within RAID tolerance = yellow Degraded = red What about the status of the hotspare? Is yellow rebuilding? What if there's no more hot spares, but you're within RAID tolerance? I'm not knocking traffic lights (sometimes they're good), but it's worth spending time trying to figure out your data displays to give you the whole picture. It takes a while, but makes monitoring so much easier.
- ChuckMcM 14y ago
  Actually large format data displays are great for calling out unexpected transitions. I had a chance to look at the control panel of a nuclear reactor for an aircraft carrier once, and what I learned was that the panel had annunciator lights that would blink on an 'unexpected' change. So for example steam pressure would by static in one of two states (cold shutdown, operation) and smoothly changing when state changing, so the light would start blinking if either the steam pressure changed when the reactor wasn't changing state, or if it stopped changing in the correct way when the reactor was changing state. Looking across a sea of lights, if you saw one that was blinking you would go over to it, read off the legend (that would tell you the conditions under which that light would blink) and then take action. Once action was in process you pushed the button to 'acknowledge' the anomaly. (which also set new parameters for when it would start blinking again) All in all a very cool system. I've been sorely tempted to hack something similar for our search and crawling clusters. Sadly I don't think its practical to have a mockup of the Enterprise-D warp core which pulses in response to query rate, although that would be very very cool :-).