8 ms·

Mars rover Curiosity’s software upgraded

78 points by aritraghosh007 13y ago

sillysaurus2 13y ago
It's so cool that we can stream software updates across 34 million miles.
- rzimmerman 13y ago
  It's also cool that the update was sent partly through the MRO spacecraft, then forwarded down to the surface.
aroch 13y ago
I get nervous sometimes when flashing ROMs on my phone...And those were downloaded over a (comparatively) 0-latency connection with high throughput. Fact that sending a firmware to Mars is apparently a normal thing now notwithstanding, I find this incredible.
- bad_alloc 13y ago
  The rover has got two computers though and during upgrades only one is updated. If something goes wrong, the non-updated one takes over and they can restore the last working state. Phones are easy to brick since they lack a backup system.
  - nknighthb 13y ago
    The ease with which many consumer devices can be bricked by firmware updates is really ridiculous, though. You don't need full hardware redundancy to virtually eliminate them, just a redundant OS partition and a little sane engineering. At that point, any bricking is almost certain to be the result of an actual hardware failure, which is its own problem.
    - jotm 13y ago
      I haven't seen an easily brickable device for half a decade now... Android smartphones and tablets are very hard to brick, post-2008 laptops and motherboards have excellent BIOS recovery, what else is there? I bricked an external RAID enclosure by flashing the wrong firmware, but that POS was running on a SiliconImage chip that for some reason allowed flashing a completely different firmware to it...
      - nknighthb 13y ago
        Funny, I just saw a friend's Android phone get bricked by an update about two years ago. It was in a state I probably could have recovered it from if its bootloader were unlocked, but it was beyond reach of Joe User. I've also bricked or seen bricked routers, external HDDs, and TVs in the last five years. Many TVs, actually, if you include pre-release units. As far as I can tell, most of them rely entirely on testing to avoid bricking devices. They make zero provision for the inevitable case where something goes wrong. Power failures during updates remain a frequent cause of bricking. Your "exception" is quite a common case when firmware updates are performed manually. Those failures have, of course, dropped dramatically since devices started being able to self-update, but they're far from the only such failures out there. The "some reason" depends on the product and manufacturer, but it's usually a combination of laziness and penny-pinching. Somebody didn't want to do the work, or somebody else didn't want to pay for it to be done. I've seen the processes that lead to these situations up-close. There are a lot of engineers out there who don't think about failure modes, and a lot of managers who dismiss the engineers who do as paranoid and/or troublemakers.
        nickonline 13y ago
        > it was beyond reach of Joe User. I would argue that this is not bricking, if it's beyond joe user that's just a job for experts. Bricking to me is totally gone with no hope of recovery.
        aroch 13y ago
        There's outside of what the Average Joe can do and then there's needing a JTAG
- rzimmerman 13y ago
  The first attempt at this upgrade did have some issues: http://www.jpl.nasa.gov/news/news.php?release=2013-325 http://www.jpl.nasa.gov/news/news.php?release=2013-325 The software reset and booted back into the older version. There are some really good safeguards against bricking anything, but it's still scary. Especially with the data latencies involved - in this case the team had several hours between knowing "something's wrong" and getting more data.
atsaloli 13y ago
A lot of work has gone into making that software especially robust, including some custom tooling for facilitating peer review. Dr. Holzmann (formerly of Bell Labs, birthplace of Unix) heads JPL's Laboratory for Reliable Software to lead this work. If you'd like to learn more, I written about the LARS methodology after hearing Dr. Holzmann at USENIX Hot Topics in Dependable Software: http://www.verticalsysadmin.com/making_robust_software/ http://www.verticalsysadmin.com/making_robust_software/
- noir_lord 13y ago
  Fascinating, this type of programming has always interested me (as I was a desktop developer who moved to the web) but not something I think I'd want to do (the stress alone must be horrific when you work on any mission critical/safety critical software).
- noselasd 13y ago
  Though what you describe covers only the code. The design is just as big a part of the picture and is just as much involved in the verification, revising, reviewing and testing.
  - atsaloli 13y ago
    Excellent point, thank you.
- mturmon 13y ago
  And here is some more on Gerard's setup: http://gerard.holzmann.usesthis.com http://gerard.holzmann.usesthis.com
- cpeterso 13y ago
  Thanks for your article. It was very interesting to read about the development of such "serious" software. :)
  - atsaloli 13y ago
    You are very welcome!
johnw 13y ago
Also interesting is the amount of damage being done to the wheels as it drives around up there. Some good pictures of that here[1]. Apparently the wheels are designed to leave a mark so they can keep track of the rovers location by looking at the tracks[2]. The tracks spell out JPL in morse code. [1] http://news.discovery.com/space/martian-wear-and-tear-curiositys-wheel-damage-photos-131220.htm http://news.discovery.com/space/martian-wear-and-tear-curios... [2] http://www.planetary.org/blogs/emily-lakdawalla/2013/10022101-theres-a-hole-in-the-wheel-dear-liza.html http://www.planetary.org/blogs/emily-lakdawalla/2013/1002210...
- nknighthb 13y ago
  Considering the hole is non-structural and it's more than halfway through the original planned mission duration, those wheels look fine to me! I wouldn't mind seeing the progression, though, since they say it's accelerated recently on the rough terrain.
- ams6110 13y ago
  "It appears to be correlated with driving over rougher terrain." Who'd have thought....
  - hnriot 13y ago
    When I read the article that also popped out for me. Unbelievably obvious. Obviously somethings are different in the Martian environment, but the laws of physics didn't just look the other way when they saw the rover coming.
- ChuckMcM 13y ago
  Thanks for the links, I figured that the lack of compliance in the "tire" section of the wheels would be an issue but that is a bit more damage than I would expect. I am guessing that under some driving conditions you might get up to 50% of the rover weight on a wheel and if that was while the wheel was rolling over a rock, the tendency would be to cut into the aluminium. So one wonders why not titanium outer rims? or a compliance system that would allow the wheel to deform until enough weight had been distributed across enough surface area to keep it under the 'minimal damage' threshold.
fabriceleal 13y ago
Possibly cool idea: build a Curiosity emulator/simulator. Build the o.s. for it and/or user-space (does any of those concepts apply? probably not) software for it. Does this already exist? This looks like it. http://sourceforge.net/projects/marsroversim/ http://sourceforge.net/projects/marsroversim/
- cypher543 13y ago
  Curiosity definitely has an OS (VxWorks). But I asked one of the developers about releasing some code after Curiosity landed and was told that they couldn't. So I doubt an accurate Curiosity emulator will ever be possible unless JPL releases their code 45 years from now like NASA did with Apollo 11's guidance system. (https://code.google.com/p/virtualagc/ https://code.google.com/p/virtualagc/)
  - raverbashing 13y ago
    They usually have terrestrial duplicates where tests occur and bug troubleshooting is done
  - rzimmerman 13y ago
    I believe that the software technically falls under certain US export restrictions which is probably one reason it isn't made public.
    - jbuzbee 13y ago
      US export restrictions? Can't export it much further than Mars. ;-)
deleted 13y ago
[deleted]
waqasx 13y ago
A day later... "Adobe Acrobat Reader needs to update"

It's so cool that we can stream software updates across 34 million miles.

rzimmerman 13y ago
It's also cool that the update was sent partly through the MRO spacecraft, then forwarded down to the surface.

aroch 13y ago

I get nervous sometimes when flashing ROMs on my phone...And those were downloaded over a (comparatively) 0-latency connection with high throughput. Fact that sending a firmware to Mars is apparently a normal thing now notwithstanding, I find this incredible.

bad_alloc 13y ago
The rover has got two computers though and during upgrades only one is updated. If something goes wrong, the non-updated one takes over and they can restore the last working state. Phones are easy to brick since they lack a backup system.
- nknighthb 13y ago
  The ease with which many consumer devices can be bricked by firmware updates is really ridiculous, though. You don't need full hardware redundancy to virtually eliminate them, just a redundant OS partition and a little sane engineering. At that point, any bricking is almost certain to be the result of an actual hardware failure, which is its own problem.
  - jotm 13y ago
    I haven't seen an easily brickable device for half a decade now... Android smartphones and tablets are very hard to brick, post-2008 laptops and motherboards have excellent BIOS recovery, what else is there? I bricked an external RAID enclosure by flashing the wrong firmware, but that POS was running on a SiliconImage chip that for some reason allowed flashing a completely different firmware to it...
    - nknighthb 13y ago
      Funny, I just saw a friend's Android phone get bricked by an update about two years ago. It was in a state I probably could have recovered it from if its bootloader were unlocked, but it was beyond reach of Joe User. I've also bricked or seen bricked routers, external HDDs, and TVs in the last five years. Many TVs, actually, if you include pre-release units. As far as I can tell, most of them rely entirely on testing to avoid bricking devices. They make zero provision for the inevitable case where something goes wrong. Power failures during updates remain a frequent cause of bricking. Your "exception" is quite a common case when firmware updates are performed manually. Those failures have, of course, dropped dramatically since devices started being able to self-update, but they're far from the only such failures out there. The "some reason" depends on the product and manufacturer, but it's usually a combination of laziness and penny-pinching. Somebody didn't want to do the work, or somebody else didn't want to pay for it to be done. I've seen the processes that lead to these situations up-close. There are a lot of engineers out there who don't think about failure modes, and a lot of managers who dismiss the engineers who do as paranoid and/or troublemakers.
      - nickonline 13y ago
        > it was beyond reach of Joe User. I would argue that this is not bricking, if it's beyond joe user that's just a job for experts. Bricking to me is totally gone with no hope of recovery.
        aroch 13y ago
        There's outside of what the Average Joe can do and then there's needing a JTAG
rzimmerman 13y ago
The first attempt at this upgrade did have some issues: http://www.jpl.nasa.gov/news/news.php?release=2013-325 http://www.jpl.nasa.gov/news/news.php?release=2013-325 The software reset and booted back into the older version. There are some really good safeguards against bricking anything, but it's still scary. Especially with the data latencies involved - in this case the team had several hours between knowing "something's wrong" and getting more data.

atsaloli 13y ago

A lot of work has gone into making that software especially robust, including some custom tooling for facilitating peer review. Dr. Holzmann (formerly of Bell Labs, birthplace of Unix) heads JPL's Laboratory for Reliable Software to lead this work. If you'd like to learn more, I written about the LARS methodology after hearing Dr. Holzmann at USENIX Hot Topics in Dependable Software: http://www.verticalsysadmin.com/making_robust_software/ http://www.verticalsysadmin.com/making_robust_software/

noir_lord 13y ago
Fascinating, this type of programming has always interested me (as I was a desktop developer who moved to the web) but not something I think I'd want to do (the stress alone must be horrific when you work on any mission critical/safety critical software).
noselasd 13y ago
Though what you describe covers only the code. The design is just as big a part of the picture and is just as much involved in the verification, revising, reviewing and testing.
- atsaloli 13y ago
  Excellent point, thank you.
mturmon 13y ago
And here is some more on Gerard's setup: http://gerard.holzmann.usesthis.com http://gerard.holzmann.usesthis.com
cpeterso 13y ago
Thanks for your article. It was very interesting to read about the development of such "serious" software. :)
- atsaloli 13y ago
  You are very welcome!

johnw 13y ago

Also interesting is the amount of damage being done to the wheels as it drives around up there. Some good pictures of that here[1]. Apparently the wheels are designed to leave a mark so they can keep track of the rovers location by looking at the tracks[2]. The tracks spell out JPL in morse code. [1] http://news.discovery.com/space/martian-wear-and-tear-curiositys-wheel-damage-photos-131220.htm http://news.discovery.com/space/martian-wear-and-tear-curios... [2] http://www.planetary.org/blogs/emily-lakdawalla/2013/10022101-theres-a-hole-in-the-wheel-dear-liza.html http://www.planetary.org/blogs/emily-lakdawalla/2013/1002210...

nknighthb 13y ago
Considering the hole is non-structural and it's more than halfway through the original planned mission duration, those wheels look fine to me! I wouldn't mind seeing the progression, though, since they say it's accelerated recently on the rough terrain.
ams6110 13y ago
"It appears to be correlated with driving over rougher terrain." Who'd have thought....
- hnriot 13y ago
  When I read the article that also popped out for me. Unbelievably obvious. Obviously somethings are different in the Martian environment, but the laws of physics didn't just look the other way when they saw the rover coming.
ChuckMcM 13y ago
Thanks for the links, I figured that the lack of compliance in the "tire" section of the wheels would be an issue but that is a bit more damage than I would expect. I am guessing that under some driving conditions you might get up to 50% of the rover weight on a wheel and if that was while the wheel was rolling over a rock, the tendency would be to cut into the aluminium. So one wonders why not titanium outer rims? or a compliance system that would allow the wheel to deform until enough weight had been distributed across enough surface area to keep it under the 'minimal damage' threshold.

fabriceleal 13y ago

Possibly cool idea: build a Curiosity emulator/simulator. Build the o.s. for it and/or user-space (does any of those concepts apply? probably not) software for it. Does this already exist? This looks like it. http://sourceforge.net/projects/marsroversim/ http://sourceforge.net/projects/marsroversim/

cypher543 13y ago
Curiosity definitely has an OS (VxWorks). But I asked one of the developers about releasing some code after Curiosity landed and was told that they couldn't. So I doubt an accurate Curiosity emulator will ever be possible unless JPL releases their code 45 years from now like NASA did with Apollo 11's guidance system. (https://code.google.com/p/virtualagc/ https://code.google.com/p/virtualagc/)
- raverbashing 13y ago
  They usually have terrestrial duplicates where tests occur and bug troubleshooting is done
- rzimmerman 13y ago
  I believe that the software technically falls under certain US export restrictions which is probably one reason it isn't made public.
  - jbuzbee 13y ago
    US export restrictions? Can't export it much further than Mars. ;-)

deleted 13y ago

[deleted]

waqasx 13y ago

A day later... "Adobe Acrobat Reader needs to update"