13 ms·
Computing glitch may have doomed Mars lander
- dzdt 10y agoIs software easier or harder than hardware? Somehow people seem to assume software is easier, but then there are at least as many critical failures blamed on software as on hardware. Maybe there are different curves: software is easy to get to 90% working but hard to get from there to 99.9%. Hardware is hard to get to 90% working but easy to get from there to 99.9%.
- nickpeterson 10y agoI don't know much in the way of electrical engineering, but I think the complexity possible is fairly close, but its likely easier to judge a hardware design as being sound since it ultimately has to tie back to reality. I think the complexity of software is that it's almost completely abstraction. So the question reminds me of someone asking, "What's harder, physics or philosophy?"
- SixSigma 10y ago> the lander's computer thought it was ... > led the craft to believe it was... Don't anthropomorphize computers, they hate that. As an armchair space explorer, I would have imagined such a crucial part of the mission had serious redundancy.
- onion2k 10y agoI would have imagined such a crucial part of the mission had serious redundancy Redundancy adds either weight if you're computing things on two systems at once, or time if you're recomputing using different code. I'd guess that working out the balance between risk of failure, weight and time is a very complicated problem.
- pixl97 10y agoAlso redundancy doesn't solve a lot of potential problems. What if your redundant systems disagree? Which one is wrong?
- madaxe_again 10y agoIt probably did, and whatever went wrong went wrong across the redundant systems, which is why they're thinking it's a bug. It could be something straightforward that presented an unexpected input and made it behave in an equally unexpected fashion. I'd imagine they're currently replaying the sensor data to an instance on earth and seeing how it responds.
- dzdt 10y agoI think the anthropomorphizing in this case is fine. Someone familiar with programming easily takes "computer thought it was on the ground" as shorthand for "computer began executing subroutines designed to activate when the craft was on the ground." The anthropomorphized version is more concise for someone familiar with and more understandable for someone unfamiliar with the inner details of computer systems.
- maggit 10y agoPerhaps it is also a matter of how you fix it. Generally software bugs can be fixed by patching over the Internet, making them less scary, whereas hardware bugs are set in stone. (Stone is mostly silicon, right?) Of course, this is a bit different for software we ship to other planets :P
- logfromblammo 10y agoBy mass, stone is mostly oxygen. Silicon is in second place.
- user5994461 10y agoFrom my experience working across both industries: Software is harder than hardware (generally speaking). The root cause is that hardware has clear purpose and is limited by the real world. An aircraft from 2015 or from 2000 does the same thing: Get you from Point A to Point B, same world, same constraints, same goal. A software that does something today. It doesn't give any clue about what it will be or do in 2-5-10 years. IMO: Software projects are way more ambitious, more complex and evolving faster than hardware. What is expected from software is NOT expected (nor achievable) in hardware at all, that's one of the first thing I noticed. --- Nonetheless, hardware can run software onboard. So you get hardware and software dev projects, both running along. That get some quirks of both world. IMO, the hardware [and real world] limitations will quickly limit the software.
- tremon 10y agoIs software easier or harder than hardware? That's hard to say in general. With current practices, I'd say software is harder to get right, but that is mainly due to a failure to spec correctly. Hardware is engineered with very tight controls: dimensions, shapes, and properties are all produced according to a specification that doesn't just specify its nominal values, but also the acceptable tolerance for any deviations. No software that I know of is engineered to the same level of precision as hardware. The lack of specification-guided tooling (and validation for 3rd-party components) is what puts software development not even in the same league as hardware development.
- scrollaway 10y ago> No software that I know of is engineered to the same level of precision as hardware. Are you counting NASA-produced software? I'm not familiar with ESA's software pipeline but I imagine it's similar.
- tremon 10y agoSadly, I'm not. I truly wish I could say that NASA-produced software is within the range of "software that I know of", but it's not.
- jack1243star 10y agoComputing Needs Time [1], which focuses on the handling of time, is a nice read on this topic from the software point of view. [1] http://digitalassets.lib.berkeley.edu/techreports/ucb/text/EECS-2009-30.pdf http://digitalassets.lib.berkeley.edu/techreports/ucb/text/E...
- henrik_w 10y agoOn that subject, I like this quote: "Hardware eventually fails. Software eventually works." -- Michael Hartung
- tezza 10y agoWhat makes you assume the root cause was the software ? Perhaps the hardware altimeter was sending errors or the gyroscope was out ? The comet lander Philae bounced and was lost because the hardware harpoons failed to secure it.
- AtheistOfFail 10y agoThe issue is an ongoing battle where hardware functions get emulated in software to save costs which just makes the software more complex and error-prone.
- 3chelon 10y agoReminds me of the old joke. If it was a hardware failure, the hardware guys will spend months building models and testing them for stresses and strains and running simulations to find the point of failure. The software engineers will just ask if they can run it again, with breakpoints.
- tjxxgkxjt 10y ago>Is software easier or harder than hardware? Neither. Rather, it's the total lack of design accountability procedures in place for software. When a mechanical or electrical engineer messes up, there is a legal framework in place to hold the involved parties responsible for their actions. Depending on how bad the negligence was, this can include jail time. This motivates those designing the hardware aspect of a product to check and recheck their work many times over, and as a result you rarely read about hardware failures with this level of frequency. In contrast, when a software system fails, everyone seems to just throw their hands up in the air and say how this is par for the course. There is currently no legal PE certification available for software, so we have seen disproportionately many catastrophic software failures.
- planteen 10y agoIn my experience, hardware is much more difficult to debug. But my hardware experience is limited to FPGAs, never a full ASIC team. There are nightmarish bugs in hardware associated with things like clock domain crossings, bad timing constraints, and signal integrity. Give me software bugs over that any day!
- LordWinstanley 10y ago>>>...Then thrusters, designed to decelerate the craft for 30 seconds until it was metres off the ground, engaged for only around 3 seconds before they were commanded to switch off, because the lander's computer thought it was on the ground... >>...The lander even switched on its suite of instruments, ready to record Mars’s weather and electrical field... It's a shame. But, reading that, I can't help imagining the lander in the role of the Blue Whale from Hitchhiker's Guide to the Galaxy: "I wonder what this big flat thing rushing toward me is? I think I'll call it 'The Ground'"
- mhandley 10y agoGiven the history of ESA landers, the more appropriate quote might be "Oh no, not again!"
- zymhan 10y agoIt really puts into perspective what good job NASA does at what many call a "boring" mission.
- xenobioticants 10y agoConsidering NASA has a budget thats 3-4x that of ESA, I'd be a little less quick to judge if I were you.
- privong 10y ago> Considering NASA has a budget thats 3-4x that of ESA, I'd be a little less quick to judge if I were you. I don't think GP was judging ESA. I think they were merely pointing out how impressive NASA's success rate has been lately.
- xenobioticants 10y agoTruth, its especially commendable that most of their projects keep soldiering on well past EOL.
- farhanhubble 10y agoI hope they didn't mix up metric and imperial units this time again. Or is it the Agile specter that haunts software made in Europe?
- crdoconnor 10y agoI'm wondering if they used formal verification and how they tested the software.
- dignati 10y agoGiven that it's an European project I'd be surprised if it was a metric vs imperial unit mixup.
- dogma1138 10y agoUK says hi.
- mtgx 10y agoAnd soon "bye."
- mattlondon 10y agoFor what it is worth - apart from road signs and speeds - I've not really seen imperial measurements used for much any more in the UK. I would be amazed (and saddened) if someone had started a new, large-scale, serious, rigorous, engineering project in the past couple of decades (i.e. since the 90s) and used imperial units (I am sure there are lots of small-scale/personal/etc stuff done in imperial though - I dont count interplanetary space flight as small-scale!) Imperial is pretty much dead in the UK (or maybe just London?) apart from roads and conversational/casual usage where its often easier to say the imperial equivalent than the metric (e.g. "pint of beer" is easier than saying "568 millilitres of beer", "about a foot" is easier than saying "about 30 centimetres" just because of fewer syllables if nothing else).
- hliyan 10y agoI was about to suggest n-version programming redundancy, when it occurred to me: aren't we starting to reach a point where deep learning systems are good enough for unmanned spacecraft? How we would generate the learning data would be a question though.
- hvidgaard 10y agoYou wastely overestimate the capabilities of the equipment we send into space. It is nowhere near powerful enough to power any meaningful deep learning system in an online fashion. Offline has either to high latency, or a problem getting data to seed the initial learning (and even then we have a problem actually running the network).
- lorenzhs 10y agoWell than and there just isn't a whole lot of training data to play with. It's doubtful that a simulator could ever model actual landing situations well enough that the whole thing will a) work on Mars and b) not overfit weird behaviour in the simulator. Faulty ground sensing has resulted in the loss of more than one mission, and deep learning isn't going to fix any of that. Oh, and s/wastely/vastly/ ;)
- axonic 10y agothruster_test(3000); // todo: add powered descent code
- user5994461 10y ago> ESA is keen to stress that overall, the ExoMars mission can be seen as a triumph: Schiaparelli sent back test data from the majority of its descent, and its sister craft — the Trace Gas Orbiter — successfully manoeuvred into Martian orbit. How to turn a failure into a success: Just call it a success.
- VeejayRampay 10y agoOr maybe a successful landing was only a bonus and the whole thing can actually be viewed as a success because it will provide with tons of data and metrics for the engineers to poke at in the future, which will in turn make future missions more interesting and smooth. No need to be cynical at all cost.
- soneca 10y agoWell, even in the small quote you mention, it has arguments on why it is a success. So they definitely not "just called it a success". They explained why they think that.
- TeMPOraL 10y agoIt's the Kerbal way - when your rescue mission for a rescue mission for a rescue mission fails, and you now have 10 astronauts stranded on the Mün, just call it "establishing a colony" and be done with it.
- mrkgnao 10y ago> Mün That's almost exactly how Scott Manley says it.
- skoczymroczny 10y agoThey should have used Haskell.
- joonoro 10y agoDamn, beat me to it.
- SonicSoul 10y agothen they'd have 2 problems.
- planteen 10y agoI assume this is a joke? Is Haskell suitable for hard real-time systems with things like lazy evaluation and a GC?
- tome 10y agoI think it's a joke. Haskell is manifestly not suitable for writing a Mars lander. A verified DSL written in Haskell on the other hand ... http://ivorylang.org/ivory-introduction.html http://ivorylang.org/ivory-introduction.html
- riskable 10y agoIt seems no matter the problem there's always someone trying to blame IT.
- Pigo 10y agoI know this is off-topic, but reading about it's mission has me wondering why (as far as I know) no agencies are sending devices that could return more sophisticated visuals. Even something simple like the 360 YouTube videos would be beyond words, and everyone could experience it. I've really enjoyed the panoramic images. I just wish they'd focus more on trying to get people there virtually since it's a fraction of the effort of getting them there physically, and it could be shared with everyone. Is there some kind of barrier to sending that much data through space? Or should the focus always be on looking for signs of life?
- danpalmer 10y agoSatellites still use CPUs with speeds in the hundreds of MHz, because creating technology hardened enough to withstand travelling through space, or being used on another planet, is a very difficult challenge, and we just aren't at the point where we can make an Intel i7 (for example) work in space. This isn't even accounting for heat issues, and power issues. Because of this, my guess is that the camera technology that we can safely get to Mars is significantly worse than what we expect in our day-to-day lives.
- wwggggoi 10y agook you answered my point...but is there not another way to shield consumer electronics ? wouldn't it be cheaper to find a good shielding system (magnetic ?) than to develop expensive custom hardware ?
- adekok 10y agoEver try upgrading the CPU on an iPhone? It's like that. These projects started out 10-15 years ago. So upgrading them mid-stream is almost impossible. They're stuck with the hardware available when they chose that part. Re-designing for new hardware means changing everything.
- Pigo 10y agoI never considered that. I suppose that does give me some hope that we'll see future projects with today's hardware eventually.
- CoryG89 10y agoEveryone has a plan, until they get hit by a planet.
- xeromal 10y ago-Mars Tyson
- int0x80 10y agothis days, almost every failure can be related in some way to a "computing glitch".
- JoeAltmaier 10y agoThis was my first thought actually. Faulty ground sensing has doomed many a craft.
- fabriceleal 10y agoReminds me of that robocop scene "You call this a glitch???"