5 ms·

This appears to be a brilliant case of reliability engineering. I'm always super impressed with these sorts of tales; given how NASAs budget always seems to be

by dayjah 10y ago

This appears to be a brilliant case of reliability engineering. I'm always super impressed with these sorts of tales; given how NASAs budget always seems to be concern seeing them deliver solutions which have just enough wiggle room in them to deliver the intended results without going too far into gold plating their solutions really leaves the software infrastructure engineer in me in awe. Even more so when you consider the time spans involved in this; I personally believe we've a ton to learn in the software space from practices like this.

jeremiep 10y ago
I think we also have a ton to learn in the software project management space. Agile for example would be quickly laughed off and dismissed. Our field is currently full of cargo-cult practices, studies and best practices without any empirical research, java schools, a strong focus on "awesome" and easy to get started instead of useful and simple; we're still a pop culture for the most part.
- mayukh 10y ago
  Solution dictates the construction process. No point in over engineering a consumer facing app when flexibility dictates who wins in the market place. Not endorsing agile or any other practice in particular.
- saiya-jin 10y ago
  agile has its place, in some form, in some places. problem is people treat it like panacea. completely agree about the pop culture analogy especially with devs with say < 5-7 years experience, a good one :)
- SomeStupidPoint 10y ago
  What's wrong with Agile? I mean, I wouldn't design a rocket, bridge, or certain kinds of software that way... But talking about most web front-ends or feature services?
- KaiP 10y ago
  Use the right tool for the job. Risk profiles between web app software and rover software are totally different.
  - throwaway729 10y ago
    It's harder to sympathize with "it's ok to write shit code if failure doesn't mean a literal explosion or being literally stranded on Mars" after a completely preventable identity theft...
    - jeremiep 10y ago
      Its especially hard when that mentality doesn't account for future feature requests or maintenance. The original developer happily moved on to botch another project while you're the one fixing up the mess. Also, how come software is either a "web app" or a "rover". There's a huge spectrum in between. Not everyone writes CRUDs; even if these types of applications are usually the worst to maintain because they are developed so carelessly with little regard for simplicity.
- komali2 10y ago
  To be fair, NASA software engineers don't exactly act like your average agile start-up team,because once the "product" is out, patching is really feasible. Whereas for your average webapp, it's more strategic to iterate based on feedback.
  - Bartweiss 10y ago
    Yep. NASA can't release patches (hardware, least) and switching to "version 2" costs a solid $800,000,000. I feel confident that if the development costs of reworking a webapp were measured in years and millions, we'd see a pretty different approach to building them.
  - komali2 10y ago
    Whoops too late to edit. Should be "patching isn't really feasible".
    - greglindahl 10y ago
      Except they DO send out patches. Just not very often.
      - komali2 10y ago
        Well, my point was it's a bit more trouble to patch something on another planet that it is to `heroku push.`
    - karthikb 10y ago
      And yet, NASA frequently updates code on launched vehicles. In fact because of schedule and orbital mechanics, Spirit and Opportunity were launched before final flight code was ready, and an OTA was performed while the vehicles where in flight: http://spacenews.com/jan-15-2004-spirit-rover-begins-amazing-visit-mars/ http://spacenews.com/jan-15-2004-spirit-rover-begins-amazing...
Cerium 10y ago
Same thoughts here, they know how every part in their system can fail.
deleted 10y ago
[deleted]
Bartweiss 10y ago
There's a fantastic story relating to this in Roving Mars (about Spirit and Opportunity). The final design had both lander-departure ramps and the now-iconic foldout solar panels, but it was initially rejected as "gold plated" on budget and weight arguments. The only reason it eventually came to pass was the realization that while the foldout solar panels were overkill, their removal would be worse than anticipated and bring the mission under minimum lifespan. Steve Squyers talks about how he was told this as "terrible news", but realized that for him it was wonderful. There was no way NASA would accept luxuries, but if the 'budget' design wouldn't work they were willing to move up a notch. So the incredible longevity of Spirit and Opportunity is largely a function of a low-cost design that broke down too late to salvage. It almost killed the whole project, but got record results once things went through.
iamacynic 10y ago
lesson #1 is stop piping curl output to bash as part of your software installation procedure. that probably won't work on mars.
verulito 10y ago
I actually worked in reliability, albeit only for one summer and on the precursors to Curiosity. I was more on the electronics side of things though. Um, AMA?

by dayjah 10y ago

jeremiep 10y ago

I think we also have a ton to learn in the software project management space. Agile for example would be quickly laughed off and dismissed. Our field is currently full of cargo-cult practices, studies and best practices without any empirical research, java schools, a strong focus on "awesome" and easy to get started instead of useful and simple; we're still a pop culture for the most part.

mayukh 10y ago
Solution dictates the construction process. No point in over engineering a consumer facing app when flexibility dictates who wins in the market place. Not endorsing agile or any other practice in particular.
saiya-jin 10y ago
agile has its place, in some form, in some places. problem is people treat it like panacea. completely agree about the pop culture analogy especially with devs with say < 5-7 years experience, a good one :)
SomeStupidPoint 10y ago
What's wrong with Agile? I mean, I wouldn't design a rocket, bridge, or certain kinds of software that way... But talking about most web front-ends or feature services?
KaiP 10y ago
Use the right tool for the job. Risk profiles between web app software and rover software are totally different.
- throwaway729 10y ago
  It's harder to sympathize with "it's ok to write shit code if failure doesn't mean a literal explosion or being literally stranded on Mars" after a completely preventable identity theft...
  - jeremiep 10y ago
    Its especially hard when that mentality doesn't account for future feature requests or maintenance. The original developer happily moved on to botch another project while you're the one fixing up the mess. Also, how come software is either a "web app" or a "rover". There's a huge spectrum in between. Not everyone writes CRUDs; even if these types of applications are usually the worst to maintain because they are developed so carelessly with little regard for simplicity.
komali2 10y ago
To be fair, NASA software engineers don't exactly act like your average agile start-up team,because once the "product" is out, patching is really feasible. Whereas for your average webapp, it's more strategic to iterate based on feedback.
- Bartweiss 10y ago
  Yep. NASA can't release patches (hardware, least) and switching to "version 2" costs a solid $800,000,000. I feel confident that if the development costs of reworking a webapp were measured in years and millions, we'd see a pretty different approach to building them.
- komali2 10y ago
  Whoops too late to edit. Should be "patching isn't really feasible".
  - greglindahl 10y ago
    Except they DO send out patches. Just not very often.
    - komali2 10y ago
      Well, my point was it's a bit more trouble to patch something on another planet that it is to `heroku push.`
  - karthikb 10y ago
    And yet, NASA frequently updates code on launched vehicles. In fact because of schedule and orbital mechanics, Spirit and Opportunity were launched before final flight code was ready, and an OTA was performed while the vehicles where in flight: http://spacenews.com/jan-15-2004-spirit-rover-begins-amazing-visit-mars/ http://spacenews.com/jan-15-2004-spirit-rover-begins-amazing...

Cerium 10y ago

Same thoughts here, they know how every part in their system can fail.

deleted 10y ago

[deleted]

Bartweiss 10y ago

There's a fantastic story relating to this in Roving Mars (about Spirit and Opportunity). The final design had both lander-departure ramps and the now-iconic foldout solar panels, but it was initially rejected as "gold plated" on budget and weight arguments. The only reason it eventually came to pass was the realization that while the foldout solar panels were overkill, their removal would be worse than anticipated and bring the mission under minimum lifespan. Steve Squyers talks about how he was told this as "terrible news", but realized that for him it was wonderful. There was no way NASA would accept luxuries, but if the 'budget' design wouldn't work they were willing to move up a notch. So the incredible longevity of Spirit and Opportunity is largely a function of a low-cost design that broke down too late to salvage. It almost killed the whole project, but got record results once things went through.

iamacynic 10y ago

lesson #1 is stop piping curl output to bash as part of your software installation procedure. that probably won't work on mars.

verulito 10y ago

I actually worked in reliability, albeit only for one summer and on the precursors to Curiosity. I was more on the electronics side of things though. Um, AMA?