4 ms·

I wouldn't want to be the one debugging this. Talk about pressure to deliver and difficult constraints!

stevewepay 11y ago
On the contrary, this is where you can really prove that you are worth your salt. There is no better arena to prove yourself than a real-live production-down situation.
- Aldo_MX 11y ago
  I beg to differ, a situation with a lot of factors you can't control is rarely a way to prove that you're worth your salt, It's the way you communicate your advancements what makes a difference.
  - iancarroll 11y ago
    Being "worth your salt" definitely includes situations out of your control, because you need to be able to handle issues that probably aren't in your control. You're only useful if you can help with the current issues...
- JshWright 11y ago
  This is a pretty common sentiment among firefighters/paramedics. You don't wish for something bad to happen to someone, but you absolutely want to be there if it does.
- Sharlin 11y ago
  "Proving oneself" is not a thing everyone values.
- madaxe_again 11y ago
  That assumes you're proving yourself to a rational engineer - rather more likely one would be attempting to prove oneself to a non-tech manager, and what happens there is that typically the fixer is so deeply associated with the problem that they end up being seen as the problem. I mean, how many times have you fixed a downed server through no fault of your own to then have your head blown off by a client?
- kabouseng 11y ago
  It's a shame really, as the engineers fighting the crisis' gets a lot of attention, but very often those crisis' are caused by themselves. It is the engineers who's projects run smoothly who is ultimately worth more, as they can predict and prevent problems before they become a crisis, but get no recognition for it.
  - digi_owl 11y ago
    And may even get the boot, because the beancounters wonder what said engineer is really doing for the paycheck... I guess it can be seen as some variant of the Lucas Critique. https://en.wikipedia.org/wiki/Lucas_critique https://en.wikipedia.org/wiki/Lucas_critique
  - carlisle_ 11y ago
    And on the other hand there are things even a number of engineers working together can't predict. Everybody makes mistakes, and having people who can work well under pressure is important regardless of whose fault caused the problem to begin with.
  - ende 11y ago
    Yes, this. It reminds me of goalies in hockey, and how people are in awe when a goalie makes some ridiculous save when in fact the goalie would have never had to have made such a save if they hadn't been out of position in the first place. The best goalies are pretty boring to watch.
    - JohnBooty 11y ago
      Yes! The Phillies used to have a fan-favorite outfielder who played hard and often made spectacular catches - but he was actually a pretty bad outfielder; the reason he made spectacular catches is because he turned routine plays into adventures.
      - speeder 11y ago
        I remembered now how some people considered the US goalie one of the best world cup players... The thing is, he was considered one of the best world cup players because the US defense was so bad, but so bad, that without him US would have ended the cup losing all games outright.
- verandaguy 11y ago
  It might be a shame when your real-live production-down situation is a multimillion dollar space probe nine years into a mission to capture the first high-resolution imagery as well as collect mounds of scientific data about one of the (ex-)planets in our solar system we know the least about, right after discovering surface anomalies including four suspicious dark spots on one of the body's faces. This isn't about proving you're worth your salt. It's about "let's get this thing working, ASAP, because who knows when we'll have another chance, since there's not exactly a plan B."
mdellabitta 11y ago
I think I figured it out. Someone put their book on the F8 key during a restart.
raverbashing 11y ago
Very likely it's cosmic rays causing a memory error
wkcamp 11y ago
Does anyone know which language(s) the NASA engineers write the spacecraft's software in?
- themodelplumber 11y ago
  Pretty interesting maybe-answer to that, from some searching: http://interviewly.com/i/nasas-new-horizons-oct-2014-reddit http://interviewly.com/i/nasas-new-horizons-oct-2014-reddit > Which language is used for programming New Horizon's flight software? How long is code? How do you make sure that it's gonna work? We don't do the coding for the flight software, that's SciOps. However here's what I know, each set of spacecraft commands is put up as a "command load", which has a name that's the year plus the day of year. So the 15188 load runs on July 7, and has commands for nine days. Each load has a multi week coding and vetting process and is simulated on the ground, on a system called "NHOPS", pronounced "nops". -AZ I emailed some folk, here's an answer to #3, from Jillian, one of the members of our SciOps team. 3: We have a program called Statesim that checks to make sure constraints are not violated. For instance the Alice aperture door shouldn't be open within 20 degrees of the Sun. We also have something called NHOps which is a software EM of New Horizons and the commands are executed on it and data is downloaded and reviewed by the instruments. We also had a rehearsal in 2013 for the Pluto Closest Approach load. -AZ Okay, here's an answer from Helen Hart at Johns Hopkins University Applied Physics Lab (AZ): We do not program New Horizons flight software. We store commands, packaged into macros, in the portion of C&DH memory designated for command storage, and execute the macros via a Time Tag. See #1. How are we sure that it¹s gonna work? That¹s a really, really big question, involving multiple layers, multiple platforms, and a lengthy, intensive Load Build and Review process. For more details on the process, talk to the Science Sequencers. The command load is simulated in SEQGEN, which is also where it gets built. The Seqgen Modeling File contains hundreds of checks for problems with commands vs. the state of the spacecraft/instrument. The Seqgen modeling file also contains the Mission Operations Playback Data Volume models. We have a SOFTWARE SIMULATOR called stateSim which models the response of the spacecraft and payload to the command sequence. stateSim generates the file that Mission Operations Command Sequencer turns into the Command Sequence Timeline - the excel spreadsheet that is distributed as part of the load review process. We have HARDWARE SIMULATORS, called NHOPS-1 and NHOPS-2, which contains HARDWARE modules for the onboard processors, including C&DH-1, C&DH-2, SSR-2, SSR-2, G&C-1, G&C-2, P-Alice, LORRI, PEPSSI, Ralph, REX-1, REX-2, SDC, and SWAP. The NHOPS and stateSim simulators have different weaknesses; for example: a: NHOPS gets all the details of slews correct; stateSim gets the slew start time and duration correct, but does not track the exact path of the track because that part of the CG&C cannot be modeled in software. b: stateSim correctly predicts C&DH Thermal Control, but NHOPS does not predict this at all c: SSR usage: stateSim does not model SSR usage. NHOPS does, but gets the wrong answer for anything that gets Compressed or Packetized. SEQGEN is tied directly to commanded data downlinks, and cannot be manipulated to produce interim data volumes; it correctly models downlink data volume TO THE ACCURACY OF the Compression Ratios specified by the Science Sequencers (lately, DataTrack). -- Helen
patcon 11y ago
Yeah, it's bad enough when there's latency and my SSH session is delayed several seconds... 3 billion miles is 4h at light speed. Well hey, at least they're probably not using SSH ;)

On the contrary, this is where you can really prove that you are worth your salt. There is no better arena to prove yourself than a real-live production-down situation.

Aldo_MX 11y ago
I beg to differ, a situation with a lot of factors you can't control is rarely a way to prove that you're worth your salt, It's the way you communicate your advancements what makes a difference.
- iancarroll 11y ago
  Being "worth your salt" definitely includes situations out of your control, because you need to be able to handle issues that probably aren't in your control. You're only useful if you can help with the current issues...
JshWright 11y ago
This is a pretty common sentiment among firefighters/paramedics. You don't wish for something bad to happen to someone, but you absolutely want to be there if it does.
Sharlin 11y ago
"Proving oneself" is not a thing everyone values.
madaxe_again 11y ago
That assumes you're proving yourself to a rational engineer - rather more likely one would be attempting to prove oneself to a non-tech manager, and what happens there is that typically the fixer is so deeply associated with the problem that they end up being seen as the problem. I mean, how many times have you fixed a downed server through no fault of your own to then have your head blown off by a client?
kabouseng 11y ago
It's a shame really, as the engineers fighting the crisis' gets a lot of attention, but very often those crisis' are caused by themselves. It is the engineers who's projects run smoothly who is ultimately worth more, as they can predict and prevent problems before they become a crisis, but get no recognition for it.
- digi_owl 11y ago
  And may even get the boot, because the beancounters wonder what said engineer is really doing for the paycheck... I guess it can be seen as some variant of the Lucas Critique. https://en.wikipedia.org/wiki/Lucas_critique https://en.wikipedia.org/wiki/Lucas_critique
- carlisle_ 11y ago
  And on the other hand there are things even a number of engineers working together can't predict. Everybody makes mistakes, and having people who can work well under pressure is important regardless of whose fault caused the problem to begin with.
- ende 11y ago
  Yes, this. It reminds me of goalies in hockey, and how people are in awe when a goalie makes some ridiculous save when in fact the goalie would have never had to have made such a save if they hadn't been out of position in the first place. The best goalies are pretty boring to watch.
  - JohnBooty 11y ago
    Yes! The Phillies used to have a fan-favorite outfielder who played hard and often made spectacular catches - but he was actually a pretty bad outfielder; the reason he made spectacular catches is because he turned routine plays into adventures.
    - speeder 11y ago
      I remembered now how some people considered the US goalie one of the best world cup players... The thing is, he was considered one of the best world cup players because the US defense was so bad, but so bad, that without him US would have ended the cup losing all games outright.
verandaguy 11y ago
It might be a shame when your real-live production-down situation is a multimillion dollar space probe nine years into a mission to capture the first high-resolution imagery as well as collect mounds of scientific data about one of the (ex-)planets in our solar system we know the least about, right after discovering surface anomalies including four suspicious dark spots on one of the body's faces. This isn't about proving you're worth your salt. It's about "let's get this thing working, ASAP, because who knows when we'll have another chance, since there's not exactly a plan B."

mdellabitta 11y ago

I think I figured it out. Someone put their book on the F8 key during a restart.

raverbashing 11y ago

Very likely it's cosmic rays causing a memory error

wkcamp 11y ago

Does anyone know which language(s) the NASA engineers write the spacecraft's software in?

themodelplumber 11y ago
Pretty interesting maybe-answer to that, from some searching: http://interviewly.com/i/nasas-new-horizons-oct-2014-reddit http://interviewly.com/i/nasas-new-horizons-oct-2014-reddit > Which language is used for programming New Horizon's flight software? How long is code? How do you make sure that it's gonna work? We don't do the coding for the flight software, that's SciOps. However here's what I know, each set of spacecraft commands is put up as a "command load", which has a name that's the year plus the day of year. So the 15188 load runs on July 7, and has commands for nine days. Each load has a multi week coding and vetting process and is simulated on the ground, on a system called "NHOPS", pronounced "nops". -AZ I emailed some folk, here's an answer to #3, from Jillian, one of the members of our SciOps team. 3: We have a program called Statesim that checks to make sure constraints are not violated. For instance the Alice aperture door shouldn't be open within 20 degrees of the Sun. We also have something called NHOps which is a software EM of New Horizons and the commands are executed on it and data is downloaded and reviewed by the instruments. We also had a rehearsal in 2013 for the Pluto Closest Approach load. -AZ Okay, here's an answer from Helen Hart at Johns Hopkins University Applied Physics Lab (AZ): We do not program New Horizons flight software. We store commands, packaged into macros, in the portion of C&DH memory designated for command storage, and execute the macros via a Time Tag. See #1. How are we sure that it¹s gonna work? That¹s a really, really big question, involving multiple layers, multiple platforms, and a lengthy, intensive Load Build and Review process. For more details on the process, talk to the Science Sequencers. The command load is simulated in SEQGEN, which is also where it gets built. The Seqgen Modeling File contains hundreds of checks for problems with commands vs. the state of the spacecraft/instrument. The Seqgen modeling file also contains the Mission Operations Playback Data Volume models. We have a SOFTWARE SIMULATOR called stateSim which models the response of the spacecraft and payload to the command sequence. stateSim generates the file that Mission Operations Command Sequencer turns into the Command Sequence Timeline - the excel spreadsheet that is distributed as part of the load review process. We have HARDWARE SIMULATORS, called NHOPS-1 and NHOPS-2, which contains HARDWARE modules for the onboard processors, including C&DH-1, C&DH-2, SSR-2, SSR-2, G&C-1, G&C-2, P-Alice, LORRI, PEPSSI, Ralph, REX-1, REX-2, SDC, and SWAP. The NHOPS and stateSim simulators have different weaknesses; for example: a: NHOPS gets all the details of slews correct; stateSim gets the slew start time and duration correct, but does not track the exact path of the track because that part of the CG&C cannot be modeled in software. b: stateSim correctly predicts C&DH Thermal Control, but NHOPS does not predict this at all c: SSR usage: stateSim does not model SSR usage. NHOPS does, but gets the wrong answer for anything that gets Compressed or Packetized. SEQGEN is tied directly to commanded data downlinks, and cannot be manipulated to produce interim data volumes; it correctly models downlink data volume TO THE ACCURACY OF the Compression Ratios specified by the Science Sequencers (lately, DataTrack). -- Helen

patcon 11y ago

Yeah, it's bad enough when there's latency and my SSH session is delayed several seconds... 3 billion miles is 4h at light speed. Well hey, at least they're probably not using SSH ;)