9 ms·
Software error doomed Japanese Hitomi spacecraft
- microcolonel 10y agoThis is just depressing. How was this not tested? This is complete sign reversal of a control output; you'd think it would show up immediately.
- asimuvPR 10y agoPerhaps it's one of those "only happens in real use" corner case. I had a stupid little robot shred itself to parts during live use (controlled environment no humans) due to a bug. Learn and move on I guess.
- johansch 10y ago(I have absolutely no insight into the software development practices of JAXA and their subcontractors, so I apologize if this is insensitive or uniformed.) Is this another sign of how bad japanese hardware-oriented companies and organizations are at doing software? Like the organisational software crisis at Toyota? Or was it a fluke?
- microcolonel 10y agoI hate to feel that way; but yeah most software I see come out of Japan scares me. Outside of legally-critical software, such as automotive control systems: it doesn't seem that there is much focus on quality. I suspect it has to do with the scale of integrators in the country being much smaller, and lacking in terms of B2B collaboration. There are some jewels here and there, though. It's probably just a matter of culture. Maybe they're also somewhat isolated from the English and Chinese language leads in software development. I don't think Japanese English language education is very effective. Their popular courses likely don't leave an individual comfortable with technical reading in English.
- sitkack 10y agoSoftware isn't valued as much in Japan, and the pay isn't that high.
- johansch 10y agoNintendo seems to care quite a lot (I worked at Opera when the two companies built the Wii browser together. Wow. They were incredibly brutally perfectionist, in a painful but somehow admirable way. I've never seen a company doing so many iterations of some particular detail before that.) But yeah, I agree, whenever a large Japanese company that is not Nintendo does software (at least for consumers), you can pretty much expect a disaster. (I know there's a small group of startups that do good stuff - I am not talking about them.)
- protomyth 10y agoI think there is an interview where Steve Jobs said as much in regards to Sony and why they didn't own the market before the iPod.
- johansch 10y agoMaybe the reason we think of asian hardware companies unable to build good software is tainted by the fact that one big american hardware company (Apple) managed to do so (build good software, I mean). I mean, which other large american hardware-centric companies have managed the transition to software well?
- protomyth 10y agoI don't disagree, but I would imagine most base their view on the titans of each country. Apple and Sony got a lot easier to compare in the 2000's and Sony really did have bad software (friggin e-reader). Steve Jobs talked about Sony a lot in interviews. I would bet if we talk South Korean products, Samsung would be the archetype. Microsoft for their part makes nice hardware with pretty good software to back it up. I haven't used a PlayStation in a long time so I don't know the current state, and I would guess Nest is not helping the US side of the equation. Apple isn't exactly covering itself in glory in the last couple of iOS releases (never mind the "California" series of OS X releases).
- sitkack 10y agoBrother wrote the software.
- martin_henk 10y agoSource?
- johansch 10y agoSo the rate of downvotes seem to be one per ten minutes since the time of posting. Why?
- jlgaddis 10y agoI didn't downvote your original comment (just saw it and this one at the same time), but perhaps it has something to do with your blanket categorization of all Japanese software?
- johansch 10y agoWell, if you read my comment properly, you'll realize that what I actually said is that large japanese hardware companies tend to be really bad at software. Nintendo is pretty much the one exception.
- PhantomGremlin 10y agolarge japanese hardware companies tend to be really bad at software That's probably true. But I think your statement is unnecessarily focused on Japan. The following is also true: large hardware companies tend to be really bad at software
- dwc 10y agoI would call it a fluke. Two things to keep in mind: * Don't compare JAXA (or anyone else) to NASA at this point. We (the US) have a ton of experience, some of it bought the hard way. * This isn't Japan's typical performance. They have a nice string of very good space science missions. And don't forget things like the Mars Climate Orbiter[1], where ground based commands sent it through the atmosphere due to a unit mix up. 1. https://en.wikipedia.org/wiki/Mars_Climate_Orbiter https://en.wikipedia.org/wiki/Mars_Climate_Orbiter
- gammarator 10y agoA recent NASA science satellite had its magnetic torque bars wired in backwards by the contractor, so it's not like this is a thing of the past in the US, either. Thankfully they saw the angular momentum building and shut it down before it spun out of control. Space is hard, and in many cases you have only one chance to get it right.
- pg_is_a_butt 10y agoyou're right... it was never tested. ever. they just sent up of bunch of bolted together junk and prayed to japanese jesus that it did something. you should be asking which physical component failed and how it failed. software is no good if the CPU is half melted with a hole through it.
- planteen 10y agoTesting a big spacecraft is hard. With a cube sat, you can put it on an air beating and make sure something isn't catastrophically wrong, like the reaction wheels are backwards. You can't do that on a big spacecraft. So instead, there's lots of testing with flight software and a simulator. But what if someone made a mistake interpreting the wheel direction specification to the mechanical body frame, for example?
- unchocked 10y agoIt seems like the core error was in the inertial measurement unit: it would be a common cause between the reaction wheel failures and the failure of the despin burn.
- sitkack 10y agoSeems like the kind of thing one would have multiple of, along with voting, it should also contain a kalman filter. The software correcting the rotation should have been run in a tighter feedback loop so that it would stop making the problem worse. Lots of these subsystems can be tested in a pure software simulator. Esp when it comes to faults.
- sitkack 10y agoThinking about it further, the simulator should consume the logs from the existing system and map those logs from existing simulation runs, use a form of compressed sensing and pattern matching to figure out what is occurring just from the logs and previous simulation runs.
- tlb 10y agoI'm surprised that they had to design custom inertial stabilization, considering how many times it's been done successfully before. Was it NIH mentality? Or did it have some requirement for more precise stabilization than other space telescopes?
- bayesian_horse 10y agoAt a guess, I'd say that most satellites have different requirements due to the way their weight is distributed and where the inertial stabilization is done. As far as I understand it, Satellite design is all about cramming the most amount of features in as small and light a package as possible. That would mean a lot of tight coupling and a hard time standardizing anything across different types of satellites.
- JshWright 10y agoWhile that's true for science payloads, when it comes to commercial satellites, there are a handful of 'busses' that form the core of the vast majority of satellites. https://en.wikipedia.org/wiki/Satellite_bus https://en.wikipedia.org/wiki/Satellite_bus
- rtkwe 10y agoCommercial payloads are a lot of the time. Science payload have a harder time doing that because the instruments they're designed to carry come in weird shapes and can't really be chopped up and rearranged to fit into the standard satellite bus.
- dwc 10y agoThis is the key passage: The spacecraft then automatically switched into a safe mode and, at about 4:10 a.m., fired thrusters to try to stop the rotation. But because the wrong command had been uploaded, the firing caused the spacecraft to accelerate further. (The improper command had been uploaded to the satellite weeks earlier without proper testing; JAXA says that it is investigating what happened.) Going into safe mode is a thing. It happens with NASA stuff, ESA stuff, whatever. The spacecraft failed to stabilize and went into safe mode, and that's proper. Whatever glitch in the systems, this would have saved it and allowed for recovery. But the uploaded command to shed rotational velocity was wrong. This is what caused the loss of the spacecraft. I'm sure there will be a pretty heavy postmortem on how this happened.
- quadrature 10y agoI would not want to be in that PR.
- azernik 10y agoThe more striking thing to me is the long failure cascade before that, mostly hardware failures: the star tracker that glitched over the South Atlantic Anomaly, the gyro that reported a nonexistent spin, and the reaction wheel that didn't spin down properly.
- ww520 10y agoIsn't there a parallel simulator on the ground to try out the commands to see the effect before hand?
- rtkwe 10y ago> The improper command had been uploaded to the satellite weeks earlier without proper testing All the available testing programs in the world doesn't matter if the tests aren't run.
- kinai 10y agosomeone committed Harakiri, I guarantee it
- ZoF 10y agoTrigger warning: There is text below this text. Heh, I laughed. Interesting to see where HN draws the line on 'offensive'
- kinai 10y agoNobody likes the truth, especially not when presented with some humor.
- dang 10y agoWe detached this subthread from https://news.ycombinator.com/item?id=11603023 https://news.ycombinator.com/item?id=11603023 and marked it off-topic.
- ZoF 10y agoAgreed! Political discussion regarding any topic is definitely off-topic. Lets go ahead and flag this as off-topic! Great job Dan! Tangentially related(or maybe directly related): would you find it off-topic if I were to openly support Donald Trump? That said(directly related to the above) If you do indeed think that nationalism is a bad thing can you please let us HN reader know why? As a White Male you're saying that Harikari is bad? Please defend your stance.
- ZoF 10y agoConsider substantiating your down-votes friends? I have nothing to be ashamed about regarding my above represented personal opinion. Can anyone liberally inclined explain why a BLM activist can assualt a Trunmp supporter but the opposite would be met with derision? If someone attacks you(illegally) can you defend yourself or not? Does the race change the answer? We're all human.
- jupiter90000 10y agoIt seems very strange that the 3 missions that would deploy an X-ray calorimeter have all failed. I feel sad for the scientists who have been hoping to get a working one into space for what sounds like over 16 years now (based on the article).
- chinathrow 10y agoAre you asking yourself whether theses losses were by chance or by conspiracy to fail?
- jupiter90000 10y agoI don't know the expected success rate for these missions, it just seemed odd that each time this particular instrument has been launched, an issue occurred to render it unusable. Maybe it is par for the course, but it does raise suspicions (at least in my mind).
- bayesian_horse 10y agoMaybe it's something like the Martian-Anti-Lander-Defense System?
- jupiter90000 10y agoI was thinking more like parties interested in sabotaging the line of research due to personal vendettas against the scientists who would gain from this success, or those who have ideological reasons to block what may be discovered using the technology. I realize this may be far-fetched conjecture, but I'm just sharing my thoughts.
- xioxox 10y agoSorry, but that's crazy. I was previously involved in this mission. The whole community was really excited to see the first results from the soft X-ray calorimeter. Furthermore, there was going to be an open call for proposals for observations, for any groups to apply to.
- dewiz 10y agoInteresting, I never heard about the south atlantic anomaly. https://en.m.wikipedia.org/wiki/South_Atlantic_Anomaly https://en.m.wikipedia.org/wiki/South_Atlantic_Anomaly
- lerax 10y agoSad.