5 ms·

These folks that design and run these experiments must have nerves of steel. You know that feeling when you have a file server that is half way around the worl

These folks that design and run these experiments must have nerves of steel. You know that feeling when you have a file server that is half way around the world and all the sudden it's gone unresponsive? Then you have to start a recovery process with on-site people, etc? The dread? The uncertainty? Now think of a principle scientist that quite possibly has devoted their entire life's academic work to one of these experiments, and something like this happens. From concept to funding to launch to all the billions of dollars to all the math and luck and other stuff that must go right. I don't know if I could do it. And then it comes down to one damn tiny pin or something and all could be lost. It would be like spending 20 years configuring a server just right in a specific manner, deploying it, and it's taken out for ever because of a typo in code somewhere - and it's all gone. Poof. Nightmare fuel.

amelius 3y ago
Everyone who ships hardware products has this problem. You don't want to recall thousands of items because you forgot to add a capacitor in your circuit. Software engineers are just lucky ...
- CoastalCoder 3y ago
  Many are. Some of us write very high-stakes software.
  - doctor_eval 3y ago
    And there are a lot more parts to a software system.
    - aetherspawn 3y ago
      I disagree, I do both, and a modern ECU hardware can be extremely complex .. more so than its software. This is because the hardware needs to meet a stack of conflicting specifications and standards for EMC, robustness, etc. Things need to work at a wide range of temperatures -40degC to 150degC. Optimising for one standard or parameter causes all your other calibration to go out of whack. Adding a protection element to the circuit can itself fail and be the reason that the circuit fails. At times it can be like the Jenga tower ready to topple over. Modern standards like ISO 26262 make it even more complicated because now we have to calculate and add up the failure probabilities of every single resistor and capacitor to find the whole-circuit failure probability, and in some applications we would need to prove less than 10 FIT (10 per trillion hours). Usually this can’t be reached without adding redundant components, and then maintaining the document trail to explain why they exist and can't be removed. And suppose you need to change a component in the design .. you have to find something to do with the 10,000 units of the old component that you bought a year ago before the design was finished because it was on back order for 100 weeks.
    - ChrisMarshallNY 3y ago
      I have had feet in both worlds. I started as an EE, and have since moved to pure software. One of the cool things about writing drivers and firmware, is you can do real-world damage. I’ve blown up $40,000 RF scanners (in 1980s dollars), because I accidentally reversed the position of a switch. Also, if you think hardware systems are simpler than software, I invite you to open up a modern DSLR (not mirrorless) camera. I worked for a major camera manufacturer for many years, and saw some crazy complex stuff. But nothing I ever did can hold a candle to the OP.
      - eichin 3y ago
        I think senior engineers should always have good "knowledge gained is proportional to equipment damaged" stories, they're great for helping juniors through their own "learning experiences". (At my current startup, the one about me and the founder/CTO figuring out how to use an API by experimenting on our only robot arm, resulting in slamming a $30k arm into a concrete wall (mmm, polar coordinates) is great for getting across "you'll be more careful next time, because look, really, there will be a next time!")
      - doctor_eval 3y ago
        I’m not here to start any kind of software or hardware war since both are amazing fields that I respect and enjoy learning about. So rather than take this thread away from the amazing job done by the Juice team, I’ll just say great job and how amazing it all is.
foobarbecue 3y ago
Indeed. I'm a rover driver & arm operator on Curiosity and there have been a few sandy patches where things were a little ... touch and go. I often think: what if I made a bad call, and my drive ended the mission? Even with a mission that's been operating for more than 10 years, it would be awful: there are hundreds of people involved, including scientists who have been planning investigations for the rover to conduct along our journey over the next few years. I would feel terrible if it was my fault that we never got to their research site of interest. But then I remember: I'm not an airline pilot. I'm not a surgeon. Yes, my decisions matter, but this is not the highest-stakes job out there, not by a long shot.
- hectormalot 3y ago
  You’re driving a ‘car’ on Mars! Does it actually include touch and go like driving with the driver as decision maker? I expected that every move would be scrutinized by a team of specialists?
  - foobarbecue 3y ago
    In a nutshell, one rover planner (that's what drivers / arm operators are called) writes the sequence of commands, a second rover planner checks it and then walks through it line-by-line out loud with another ~2 rover planners watching. Then the second rover planner walks through the whole thing again with many other teams listening in and asking questions. The first rover planner also consults with a surface properties scientist during development of the drive command sequence. So, if things go wrong it's never really any single person's fault. But still, that first rover planner drafting the initial version of the command sequence bears a fair amount of responsibility.
    - foobarbecue 3y ago
      Just wanted to add that if you're a US taxpayer, that's your car we're driving. We really do try to take good care of it :-)
      - 3y ago

amelius 3y ago

Everyone who ships hardware products has this problem. You don't want to recall thousands of items because you forgot to add a capacitor in your circuit. Software engineers are just lucky ...

CoastalCoder 3y ago
Many are. Some of us write very high-stakes software.
- doctor_eval 3y ago
  And there are a lot more parts to a software system.
  - aetherspawn 3y ago
    I disagree, I do both, and a modern ECU hardware can be extremely complex .. more so than its software. This is because the hardware needs to meet a stack of conflicting specifications and standards for EMC, robustness, etc. Things need to work at a wide range of temperatures -40degC to 150degC. Optimising for one standard or parameter causes all your other calibration to go out of whack. Adding a protection element to the circuit can itself fail and be the reason that the circuit fails. At times it can be like the Jenga tower ready to topple over. Modern standards like ISO 26262 make it even more complicated because now we have to calculate and add up the failure probabilities of every single resistor and capacitor to find the whole-circuit failure probability, and in some applications we would need to prove less than 10 FIT (10 per trillion hours). Usually this can’t be reached without adding redundant components, and then maintaining the document trail to explain why they exist and can't be removed. And suppose you need to change a component in the design .. you have to find something to do with the 10,000 units of the old component that you bought a year ago before the design was finished because it was on back order for 100 weeks.
  - ChrisMarshallNY 3y ago
    I have had feet in both worlds. I started as an EE, and have since moved to pure software. One of the cool things about writing drivers and firmware, is you can do real-world damage. I’ve blown up $40,000 RF scanners (in 1980s dollars), because I accidentally reversed the position of a switch. Also, if you think hardware systems are simpler than software, I invite you to open up a modern DSLR (not mirrorless) camera. I worked for a major camera manufacturer for many years, and saw some crazy complex stuff. But nothing I ever did can hold a candle to the OP.
    - eichin 3y ago
      I think senior engineers should always have good "knowledge gained is proportional to equipment damaged" stories, they're great for helping juniors through their own "learning experiences". (At my current startup, the one about me and the founder/CTO figuring out how to use an API by experimenting on our only robot arm, resulting in slamming a $30k arm into a concrete wall (mmm, polar coordinates) is great for getting across "you'll be more careful next time, because look, really, there will be a next time!")
    - doctor_eval 3y ago
      I’m not here to start any kind of software or hardware war since both are amazing fields that I respect and enjoy learning about. So rather than take this thread away from the amazing job done by the Juice team, I’ll just say great job and how amazing it all is.

foobarbecue 3y ago

Indeed. I'm a rover driver & arm operator on Curiosity and there have been a few sandy patches where things were a little ... touch and go. I often think: what if I made a bad call, and my drive ended the mission? Even with a mission that's been operating for more than 10 years, it would be awful: there are hundreds of people involved, including scientists who have been planning investigations for the rover to conduct along our journey over the next few years. I would feel terrible if it was my fault that we never got to their research site of interest. But then I remember: I'm not an airline pilot. I'm not a surgeon. Yes, my decisions matter, but this is not the highest-stakes job out there, not by a long shot.

hectormalot 3y ago
You’re driving a ‘car’ on Mars! Does it actually include touch and go like driving with the driver as decision maker? I expected that every move would be scrutinized by a team of specialists?
- foobarbecue 3y ago
  In a nutshell, one rover planner (that's what drivers / arm operators are called) writes the sequence of commands, a second rover planner checks it and then walks through it line-by-line out loud with another ~2 rover planners watching. Then the second rover planner walks through the whole thing again with many other teams listening in and asking questions. The first rover planner also consults with a surface properties scientist during development of the drive command sequence. So, if things go wrong it's never really any single person's fault. But still, that first rover planner drafting the initial version of the command sequence bears a fair amount of responsibility.
  - foobarbecue 3y ago
    Just wanted to add that if you're a US taxpayer, that's your car we're driving. We really do try to take good care of it :-)
    - 3y ago