13 ms·

Software Update Destroys $286M Japanese Satellite

156 points by stevekemp 11y ago

walrus01 11y ago
Or, "how not to drop a satellite on the floor": https://www.spaceflightnow.com/news/n0410/04noaanreport/ https://www.spaceflightnow.com/news/n0410/04noaanreport/
- B1FF_PSUVM 11y ago
  Glance through ... "The MIB found such violations were routinely practiced." ... what? Backtrack, backtrack ... "the NOAA N-PRIME Mishap Investigation Board (MIB)" ... ah, OK. (Men in Black movie, if that was before your time. Where the checkout line tabloid rags - clickbait before its time - were the research reports. ;-)
  - walrus01 11y ago
    and definitely not an SNMP MIB
  - krapp 11y ago
    > "the NOAA N-PRIME Mishap Investigation Board (MIB)" ... ah, OK. That's just what they want you to think.
- Terr_ 11y ago
  The wonderful thing about software is that a large number of our "checklists" can be automated. Unfortunately, the checklist of checklists to automate is still manual :p
  - walrus01 11y ago
    "We've put our automated checklist for site deployment on the internal mediawiki page at https://site.goes.here https://site.goes.here , problem is that Bob was trying to rack the 85 pound, 4U server containing the hypervisor that runs the wiki instance, and he dropped it on the floor from a 6' height"
- sandworm101 11y ago
  Honestly, this was the best place for this to happen. Better it tip onto the floor than something fail on lift-off or while in orbit. A sat in the garage is worth 1/4 of one in orbit.
  - walrus01 11y ago
    Probably wouldn't have happened during launch, though, as the mechanism attaching it to the rotational base would've been replaced with a special pyro platform rigged with explosive bolts to separate the satellite from the final carrier stage. Hopefully the people who rig the explosive bolts are a bit more thorough with their checklist. https://psemc.com/products/cartridge-actuated-devices-cads/explosive-bolts-sep-bolts/ https://psemc.com/products/cartridge-actuated-devices-cads/e...
- hcrisp 11y ago
  Thanks for the link. Earlier, I imagined this while watching the James Webb Space Telescope rollover [0]. Now I know it can really happen. [0] https://youtu.be/PhGfgREoBj4 https://youtu.be/PhGfgREoBj4
pmarreck 11y ago
Perhaps it should have been powered by Erlang (or Elixir): https://www.youtube.com/watch?v=96UzSHyp0F8 https://www.youtube.com/watch?v=96UzSHyp0F8
- boxy310 11y ago
  To people downvoting this: Cybernetics as a discipline is the study of systems governance ("kybernetes" being the Greek word for "governor" or "steersman") [1]. A solid understanding of self-reinforcing positive feedback loops like a bad spin reading leading to more spin thrusting is the exact kind of situation that should be planned for in self-adaptive systems not immediately available to direct control. [1] https://en.wikipedia.org/wiki/Cybernetics https://en.wikipedia.org/wiki/Cybernetics
- deleted 11y ago
  [deleted]
- curiousgal 11y ago
  That's really awesome
  - pmarreck 11y ago
    May not be 100% relevant but I agree, that video is awesome (and so is Erlang/Elixir)
mizzao 11y ago
This seems to follow a general pattern in today's devices: expensive hardware ruined by crappy software. Why is it that good software quality is generally an afterthought in so many systems?
- pnathan 11y ago
  Because the hardware folks are king in those shops, not leading to proper respect and care for software. It has to be a yin/yang situation, not a pyramidal one.
- onion2k 11y ago
  There's a lot of software out there. Every car, every phone, every plane, computer, TV, washing machine, factory robot, etc - every thing works pretty much flawlessly on a day to day basis. Occasionally things need a reboot or a patch but that's all. The number of times software fails in our daily lives is pretty low considering we interact with it so often. I'd contend that software quality is actually staggeringly high. It could be higher, but there's the law of diminishing returns and all that.
  - ssivark 11y ago
    That is not satisfactory. Some instances of software are more crucial than others eg. software running money transfers must have less tolerance for bugs than a game I play on my phone. The question is not whether software works most of the time, but how badly things go wrong when mistakes happen. Here's an analogy: A weather forecaster can predict "no hurricane" every single day and have a near-perfect success rate. Needless to say, that's next to useless. (False positives and false negatives have wildly different costs in this context.)
    - onion2k 11y ago
      At a guess I'd say that the probability of a fault in a piece of software does approximately match the cost of it failing. A phone game is very likely to have more bugs than a banking system. My point is that both applications actually work really well. In my experience premium phones games crash maybe once in every thousand runs. Bank money transfer software crashes perhaps once in every few billion runs (that's a total guess but we'd hear about it if it was higher and there are a lot of bank transfers every day). I think that's quite good.
smegel 11y ago
I wonder if a physics simulator would have predicted this outcome, and if in fact they have such a simulator for testing both the hardware and software together.
- pavel_lishin 11y ago
  It wasn't a matter of them not knowing what would happen if the satellite was stressed, it was a case of bad data that kept reporting that the satellite was spinning. The software then tried to correct the spin, which resulted in an actual spin, in the opposite direction, that kept accelerating since it was under the impression that the corrections weren't working.
  - fixermark 11y ago
    I'd love to see a full post-mortem. It smells like something was very off in either the hardware design or software configuration, but not knowing their architecture, it's very hard to say with any certainty what could have been improved. A couple of questions I have: - other systems I know of that care deeply about their attitude have multiple redundant sensors in place to "vote" on a consensus output in case one or more of them fails. Was that the case in this hardware design? If not, why not? If yes, how did the collective answer end up a constant error? - did they have other sensors (such as a strain gauge) that could have been integrated into the model to spot-check this kind of failure mode? A rule like "If the satellite 'feels' like it's tearing itself apart, stop accelerating" could perhaps have been useful (on the other hand, it'd leave the craft vulnerable to other known failure modes, such as "thruster stuck in the on position and must be countered by another thruster to keep the craft stable," which almost killed one of the U.S. manned missions).
    - walrus01 11y ago
      Something as simple as software to prevent extended firings of a thruster for any reason would have worked. In a LEO satellite it's constantly being exposed to night/day cycles and isn't in danger of draining the batteries in safe mode, no matter what orientation it is. LEO satellites have low-bandwidth TT&C (tracking telemetry and control) omnidirectional antennas and radio systems in the L and S bands that don't particularly care about the orientation of the satellite. Code as simple as "if thruster tries to fire for greater than period of time, call exception, place satellite in safe mode" would have worked. Using ground based TT&C systems it's possible to manually reorient a satellite in safe mode, or query what its star tracker sees.
ams6110 11y ago
A certain well-known electric car maker (maybe all of them, though) can push software updates to systems that have lives in their control not to mention costly machines. A sobering reminder of the need for very careful testing and control over this sort of thing.
- mmaunder 11y ago
  Which certainly makes vulnerabilities more interesting.
  - askafriend 11y ago
    "Oh you discovered a life threatening vulnerability that affects every car in the deployed fleet of electric vehicles? Thanks, here's $5k for your hard work as part of our bounty program."
    - beeboop 11y ago
      I know this is a joke, but I am unsure if you're pointing it directly at Tesla. To clarify, Tesla does not have a bug bounty program that I was able to find for their car software.
      - askafriend 11y ago
        Haha no, I was just poking fun at high profile bug bounty programs like Facebook/Google's which routinely offer pennies for very very serious vulnerabilities. I'm not in a position to discuss bug bounty ethics or judge what the right thing to do is given the market circumstances, I'm only in a position to make an opportune joke. But you gotta admit...it is amusing seeing very critical vulnerabilities like ubiquitous user login being reported by researchers whom get 5k or 10k for doing the right thing. These holes could instantly cripple the trust of a company's userbase and yet the people reporting them get pennies.
        lobotryas 11y ago
carapace 11y ago
Problems like this (and Nest thermostats bricking during winter) could be mitigated in the design phase by a thorough understanding of Cybernetics.
- fixermark 11y ago
  Interesting thought. Could you expand on it? What aspect of cybernetics could have improved this situation, for example?
  - carapace 11y ago
    Briefly, Information Theory is getting real world phenomena to behave like symbols, while Cybernetics is getting symbols to behave like real world phenomena. If you want to solve a math problem, then a computer plus a proper algorithm will suffice. If you want to design a system that interacts with its environment and can achieve goals while maintaining homoeostasis, the name for that is Cybernetics. "Introduction to Cybernetics" by Ashby has been made freely available in PDF form by his estate. A great and noble service for which I commend them. http://pespmc1.vub.ac.be/ASHBBOOK.html http://pespmc1.vub.ac.be/ASHBBOOK.html
- joezydeco 11y ago
  If by "Cybernetics" you mean "Ancient thermopile-driven furnace control systems", then yes.
wyattjoh 11y ago
For some reason I was expecting this to be related to a Windows 10 update.. Glad it wasn't.
Klasiaster 11y ago
The point is that you can not build a 100% perfect system, there will be always some mistakes even with a theoreme proven code base they appear on other places, just their number is reducable with much efforts.
r721 11y ago
Previous discussion: https://news.ycombinator.com/item?id=11602536 https://news.ycombinator.com/item?id=11602536
fixermark 11y ago
I'd be interested to see a full post-mortem on this. What would they change about their process to avoid this failure-mode in the future?
michael_storm 11y ago
Question for someone who knows more about satellites than me: > In satellites, the STT typically gets a good fix and sends the data to the IRU. The IRU uses the data to set its current reading and to measure how far it drifted since the last update. After calculating the drift it uses drift adjustments to compensate for the future drift. Clearly if the compensation calculation is wrong the future readings are going to be wrong. This appears to have played a role since the ACS attempted to correct a rotation that didn’t exist. The erroneous configuration information led the ACS to aggravate, not correct, the rotation. Does this mean that error will compound if either the attitude or compensation are calculated or performed incorrectly? If so, is there a way to reduce that compounding, perhaps by making them more independent systems? Or am I reading too much into a summary? (And, you know, space is hard.)
- walrus01 11y ago
  There are satellite that use the STT system almost entirely independently of all other systems. Particularly ones that need to remain in a certain orientation for 100% of their service life, such as geostationary telecom and weather satellites that orbit along the equator and are always aimed towards the visible hemisphere of earth. On those, the directional hemisphere and spot beam antennas are fixed in place (or can only move a few degrees motion at best, such as Ku band spot beam antennas), relying on the body orientation of the satellite to service a certain area of the visible hemisphere. Satellites are designed to go into "safe mode" if certain fault protection events happen, or the multiply redundant control systems/onboard computers don't agree with each other. Safe mode usually means shutting down all nonessential electrical loads and trying to orient themselves so that solar panels receive the greatest amount of charge, while listening for command and control data on their omni (L and S band) TT&C antennas. With this event it sounds like something REALLY went wrong since not only did the satellite try to correct a nonexistant wrong orientation via its reaction wheels (reaction wheels are not nearly as powerful in real life as they are in kerbal space program), it then decided to start expending propellant and spun itself up to such a RPM of revolutions that it tore off its own solar panels, and anything else that would be vulnerable to high centrifugal G forces. Automated code that expends propellant is usually checked much more carefully than this, since the amount of propellant is fixed and non renewable, usually the primary constraint on the total service life of the satellite. Most satellites run out of stationkeeping/orientation propellant (or propellant for ion engine delta-V changes) long before their multiply redundant solar/charge controller/battery/computer control systems fail.
ourcat 11y ago
"Are you sure you want to upgrade your satellite app?" Yes... boom...
- adrianlmm 11y ago
  I hope you get modded down, we are not in Slashdot.
  - ourcat 11y ago
    Sorry. I was not aware that there was such a low tolerance for a tiny bit of lightheartedness here. I hope you lighten up. Peace.
samwestdev 11y ago
Guys always read the change log before upgrading
- askafriend 11y ago
  You mean changelogs like these? https://twitter.com/cirbif/status/728114363563839488 https://twitter.com/cirbif/status/728114363563839488
- aaron695 11y ago
  I find this useless advice. I didn't write or are aware of much of the code I'm part of. Or remember some of the code I did write. Or even understand the ramifications of what the change log means. All assuming it's a well written change log. You need external process to ensure upgrades don't break things. Not a 'read the change log' policy so someone can blame you post breakage cause then it'll be easy to find issues in the change log.
- deleted 11y ago
  [deleted]
walrus01 11y ago
I know this was a science and LEO (low earth orbit) satellite, but here's something to consider when thinking about the difficulties of engineering hardware & software to work together in a satellite. Geostationary telecom (and weather, and SIGINT, etc) satellites: In the entire history of manned spaceflight no human has ever visited geostationary orbit. Once placed in a geostationary transfer orbit (about 450 x 36,000 km) and then onwards to final geostationary orbit, nobody will ever see a satellite again. The largest ones weigh 6000 kilograms and are the size of greyhound buses. They're out there right now operating with multiply-redundant everything and cost $150 million to build and launch. It is highly unlikely that any time in the next 25 years any human will ever visit one in person or touch one. When a satellite is encapsulated in its fairing/shroud for launch that is the last time anyone will ever see it until its ultimate end of life in hopefully 15 years. Every one of its control systems needs to be so thoroughly debugged and multiply redundant that it can operate out there with absolutely zero chance of repair or parts replacement.
- colemickens 11y ago
  Presumably these satellites are transmitting data back to Earth? Is that channel not bidirectional? If those satellites can be remote reprogrammed, are they really much different than satellites in geosynchronous orbit? Or are you suggesting that satellites in geosync orbit could be manually updated via EVA? I would think that would still be extremely cost-prohibitive (not to mention a huge risk to human life for a weather satellite! Has that ever actually happened?
  - walrus01 11y ago
    I'm saying sort of the opposite - there have been very, very rare cases where satellites in LEO were visited by humans and repaired or retrieved, but it's extremely uneconomical (Hubble, LDEF, etc). Actually cheaper to build and launch a new one. But at least it is technologically possible. Whereas nobody has ever gone to geostationary orbit. There have been proposals for ion engined orbital tugs to grab onto old, out-of-stationkeeping-propellant satellites that still have good electronics, for the purpose of extending their life, or moving them to a new orbital position. But nothing has actually flown.
kibwen 11y ago
What altitude was the satellite at when it broke up? Will the debris pose a problem for other satellites, or was it low enough that the pieces will reenter the atmosphere quickly?
- Nicholas_C 11y ago
  Is satellite debris an issue? Or does the vastness of the area in which satellites orbit take care of that?
  - dcposch 11y ago
    Yes, it is a serious issue https://en.wikipedia.org/wiki/Space_debris https://en.wikipedia.org/wiki/Space_debris https://en.wikipedia.org/wiki/Kessler_syndrome https://en.wikipedia.org/wiki/Kessler_syndrome
  - DarkTree 11y ago
    Very much a problem. [https://en.wikipedia.org/wiki/Kessler_syndrome https://en.wikipedia.org/wiki/Kessler_syndrome]
deleted 11y ago
[deleted]
nurblieh 11y ago
Ironically, the secondary payload that launched with Hitomi was a micro-sat which monitors space debris. https://www.frontier.phys.nagoya-u.ac.jp/en/chubusat/chubusat_satellite3.html https://www.frontier.phys.nagoya-u.ac.jp/en/chubusat/chubusa...
2PetitsVerres 11y ago
I'm don't completely agree with the fact that it's called a "software update". Reading different article about it, from what I understand, there was an initial error (software error probably triggered by an hardware error probably due to an upset due to radiation), but this error is not very important. A series of event from this error triggered the safe mode (that's expected), and there was the critical problem. They had updated parameter in the software describing the torque generated by each thruster (or the center of mass position, or the tensor of inertia, or parameters based on all this) These parameter are software parameters, but updating them is not updating the software. It's software data, not software code. Of course this does not change the fact that it is a critical error, but it's not exactly software update (IHMO). It's a configuration error. It's strange that they didn't see that in a simulator before updating them, but it's possible that they may have used the same value in the part simulating the software and the part simulating the thrusters themselves. (note: I'm working in the satellite on-board software/attitude control domain, but not for JAXA, in Europe. Anyway at my current position, I must both test this kind of code, and the parameter used in the code. And checking the parameters is much more difficult, because you must be sure that everyone agree on everything. This includes a lot of basic stuff, but it's a pain in the ass ;-) ) This pdf from JAXA is probably the initial source of all articles. http://global.jaxa.jp/press/2016/04/files/20160428_hitomi.pdf http://global.jaxa.jp/press/2016/04/files/20160428_hitomi.pd... I found it interesting to read, if you know how it works. I would of course prefer to have more detains. I always want more details. But it's for the press...
verytrivial 11y ago
Hindsight is 20:20 of course, backseat driver etc. but if the on-board systems detected a bad rotation, then started a burn to correct it, presumably it could have been possible to detect that the burn was not "helping" during the burn? And halted the burn? The thrusters aren't usually that powerful, so the erroneous death-spin probably took a fair while to spin up. Even if the sensors were wrong, if you're a computer trying to get variable 'X' into a range, and you apply control 'Y', but 'X' moves further and further away, let go of the controls and ask ask an adult help! (Easier said than done, I know. I'm amazed space engineering works as often as it does! Super hard stuff.)
- lloeki 11y ago
  Of course it was using a feedback loop, but GIGO applies: The STT and IRU disagreed on the attitude of the satellite. In this case the IRU takes priority, but its data apparently was wrong, reporting a rotation rate of 20 degrees per hour, which was not occurring. Starting from there it would have no way of knowing about the true value of 'X', so the feedback loop was fed with wrong data and just kept taking decisions† that made things worse, especially given that: The satellite configuration information uploaded earlier was wrong and the reaction wheels made the spin worse. [...] the ACS attempted to correct a rotation that didn’t exist. The erroneous configuration information led the ACS to aggravate, not correct, the rotation. † Even without misconfiguration, stopping an object from spinning in a vacuum isn't as direct and linear as accelerating/braking in a car, requiring precise coordination of multiple fixed thrusters and/or reaction wheels.
piyush_soni 11y ago
And you ask why I am afraid to install that "flash player update". :)

156 points by stevekemp 11y ago

Or, "how not to drop a satellite on the floor": https://www.spaceflightnow.com/news/n0410/04noaanreport/ https://www.spaceflightnow.com/news/n0410/04noaanreport/

B1FF_PSUVM 11y ago
Glance through ... "The MIB found such violations were routinely practiced." ... what? Backtrack, backtrack ... "the NOAA N-PRIME Mishap Investigation Board (MIB)" ... ah, OK. (Men in Black movie, if that was before your time. Where the checkout line tabloid rags - clickbait before its time - were the research reports. ;-)
- walrus01 11y ago
  and definitely not an SNMP MIB
- krapp 11y ago
  > "the NOAA N-PRIME Mishap Investigation Board (MIB)" ... ah, OK. That's just what they want you to think.
Terr_ 11y ago
The wonderful thing about software is that a large number of our "checklists" can be automated. Unfortunately, the checklist of checklists to automate is still manual :p
- walrus01 11y ago
  "We've put our automated checklist for site deployment on the internal mediawiki page at https://site.goes.here https://site.goes.here , problem is that Bob was trying to rack the 85 pound, 4U server containing the hypervisor that runs the wiki instance, and he dropped it on the floor from a 6' height"
sandworm101 11y ago
Honestly, this was the best place for this to happen. Better it tip onto the floor than something fail on lift-off or while in orbit. A sat in the garage is worth 1/4 of one in orbit.
- walrus01 11y ago
  Probably wouldn't have happened during launch, though, as the mechanism attaching it to the rotational base would've been replaced with a special pyro platform rigged with explosive bolts to separate the satellite from the final carrier stage. Hopefully the people who rig the explosive bolts are a bit more thorough with their checklist. https://psemc.com/products/cartridge-actuated-devices-cads/explosive-bolts-sep-bolts/ https://psemc.com/products/cartridge-actuated-devices-cads/e...
hcrisp 11y ago
Thanks for the link. Earlier, I imagined this while watching the James Webb Space Telescope rollover [0]. Now I know it can really happen. [0] https://youtu.be/PhGfgREoBj4 https://youtu.be/PhGfgREoBj4

pmarreck 11y ago

Perhaps it should have been powered by Erlang (or Elixir): https://www.youtube.com/watch?v=96UzSHyp0F8 https://www.youtube.com/watch?v=96UzSHyp0F8

boxy310 11y ago
To people downvoting this: Cybernetics as a discipline is the study of systems governance ("kybernetes" being the Greek word for "governor" or "steersman") [1]. A solid understanding of self-reinforcing positive feedback loops like a bad spin reading leading to more spin thrusting is the exact kind of situation that should be planned for in self-adaptive systems not immediately available to direct control. [1] https://en.wikipedia.org/wiki/Cybernetics https://en.wikipedia.org/wiki/Cybernetics
deleted 11y ago
[deleted]
curiousgal 11y ago
That's really awesome
- pmarreck 11y ago
  May not be 100% relevant but I agree, that video is awesome (and so is Erlang/Elixir)

mizzao 11y ago

This seems to follow a general pattern in today's devices: expensive hardware ruined by crappy software. Why is it that good software quality is generally an afterthought in so many systems?

pnathan 11y ago
Because the hardware folks are king in those shops, not leading to proper respect and care for software. It has to be a yin/yang situation, not a pyramidal one.
onion2k 11y ago
There's a lot of software out there. Every car, every phone, every plane, computer, TV, washing machine, factory robot, etc - every thing works pretty much flawlessly on a day to day basis. Occasionally things need a reboot or a patch but that's all. The number of times software fails in our daily lives is pretty low considering we interact with it so often. I'd contend that software quality is actually staggeringly high. It could be higher, but there's the law of diminishing returns and all that.
- ssivark 11y ago
  That is not satisfactory. Some instances of software are more crucial than others eg. software running money transfers must have less tolerance for bugs than a game I play on my phone. The question is not whether software works most of the time, but how badly things go wrong when mistakes happen. Here's an analogy: A weather forecaster can predict "no hurricane" every single day and have a near-perfect success rate. Needless to say, that's next to useless. (False positives and false negatives have wildly different costs in this context.)
  - onion2k 11y ago
    At a guess I'd say that the probability of a fault in a piece of software does approximately match the cost of it failing. A phone game is very likely to have more bugs than a banking system. My point is that both applications actually work really well. In my experience premium phones games crash maybe once in every thousand runs. Bank money transfer software crashes perhaps once in every few billion runs (that's a total guess but we'd hear about it if it was higher and there are a lot of bank transfers every day). I think that's quite good.

smegel 11y ago

I wonder if a physics simulator would have predicted this outcome, and if in fact they have such a simulator for testing both the hardware and software together.

pavel_lishin 11y ago
It wasn't a matter of them not knowing what would happen if the satellite was stressed, it was a case of bad data that kept reporting that the satellite was spinning. The software then tried to correct the spin, which resulted in an actual spin, in the opposite direction, that kept accelerating since it was under the impression that the corrections weren't working.
- fixermark 11y ago
  I'd love to see a full post-mortem. It smells like something was very off in either the hardware design or software configuration, but not knowing their architecture, it's very hard to say with any certainty what could have been improved. A couple of questions I have: - other systems I know of that care deeply about their attitude have multiple redundant sensors in place to "vote" on a consensus output in case one or more of them fails. Was that the case in this hardware design? If not, why not? If yes, how did the collective answer end up a constant error? - did they have other sensors (such as a strain gauge) that could have been integrated into the model to spot-check this kind of failure mode? A rule like "If the satellite 'feels' like it's tearing itself apart, stop accelerating" could perhaps have been useful (on the other hand, it'd leave the craft vulnerable to other known failure modes, such as "thruster stuck in the on position and must be countered by another thruster to keep the craft stable," which almost killed one of the U.S. manned missions).
  - walrus01 11y ago
    Something as simple as software to prevent extended firings of a thruster for any reason would have worked. In a LEO satellite it's constantly being exposed to night/day cycles and isn't in danger of draining the batteries in safe mode, no matter what orientation it is. LEO satellites have low-bandwidth TT&C (tracking telemetry and control) omnidirectional antennas and radio systems in the L and S bands that don't particularly care about the orientation of the satellite. Code as simple as "if thruster tries to fire for greater than period of time, call exception, place satellite in safe mode" would have worked. Using ground based TT&C systems it's possible to manually reorient a satellite in safe mode, or query what its star tracker sees.

ams6110 11y ago

A certain well-known electric car maker (maybe all of them, though) can push software updates to systems that have lives in their control not to mention costly machines. A sobering reminder of the need for very careful testing and control over this sort of thing.

mmaunder 11y ago
Which certainly makes vulnerabilities more interesting.
- askafriend 11y ago
  "Oh you discovered a life threatening vulnerability that affects every car in the deployed fleet of electric vehicles? Thanks, here's $5k for your hard work as part of our bounty program."
  - beeboop 11y ago
    I know this is a joke, but I am unsure if you're pointing it directly at Tesla. To clarify, Tesla does not have a bug bounty program that I was able to find for their car software.
    - askafriend 11y ago
      Haha no, I was just poking fun at high profile bug bounty programs like Facebook/Google's which routinely offer pennies for very very serious vulnerabilities. I'm not in a position to discuss bug bounty ethics or judge what the right thing to do is given the market circumstances, I'm only in a position to make an opportune joke. But you gotta admit...it is amusing seeing very critical vulnerabilities like ubiquitous user login being reported by researchers whom get 5k or 10k for doing the right thing. These holes could instantly cripple the trust of a company's userbase and yet the people reporting them get pennies.
      - lobotryas 11y ago

carapace 11y ago

Problems like this (and Nest thermostats bricking during winter) could be mitigated in the design phase by a thorough understanding of Cybernetics.

fixermark 11y ago
Interesting thought. Could you expand on it? What aspect of cybernetics could have improved this situation, for example?
- carapace 11y ago
  Briefly, Information Theory is getting real world phenomena to behave like symbols, while Cybernetics is getting symbols to behave like real world phenomena. If you want to solve a math problem, then a computer plus a proper algorithm will suffice. If you want to design a system that interacts with its environment and can achieve goals while maintaining homoeostasis, the name for that is Cybernetics. "Introduction to Cybernetics" by Ashby has been made freely available in PDF form by his estate. A great and noble service for which I commend them. http://pespmc1.vub.ac.be/ASHBBOOK.html http://pespmc1.vub.ac.be/ASHBBOOK.html
joezydeco 11y ago
If by "Cybernetics" you mean "Ancient thermopile-driven furnace control systems", then yes.

wyattjoh 11y ago

For some reason I was expecting this to be related to a Windows 10 update.. Glad it wasn't.

Klasiaster 11y ago

The point is that you can not build a 100% perfect system, there will be always some mistakes even with a theoreme proven code base they appear on other places, just their number is reducable with much efforts.

r721 11y ago

Previous discussion: https://news.ycombinator.com/item?id=11602536 https://news.ycombinator.com/item?id=11602536

fixermark 11y ago

I'd be interested to see a full post-mortem on this. What would they change about their process to avoid this failure-mode in the future?

michael_storm 11y ago

Question for someone who knows more about satellites than me: > In satellites, the STT typically gets a good fix and sends the data to the IRU. The IRU uses the data to set its current reading and to measure how far it drifted since the last update. After calculating the drift it uses drift adjustments to compensate for the future drift. Clearly if the compensation calculation is wrong the future readings are going to be wrong. This appears to have played a role since the ACS attempted to correct a rotation that didn’t exist. The erroneous configuration information led the ACS to aggravate, not correct, the rotation. Does this mean that error will compound if either the attitude or compensation are calculated or performed incorrectly? If so, is there a way to reduce that compounding, perhaps by making them more independent systems? Or am I reading too much into a summary? (And, you know, space is hard.)

walrus01 11y ago
There are satellite that use the STT system almost entirely independently of all other systems. Particularly ones that need to remain in a certain orientation for 100% of their service life, such as geostationary telecom and weather satellites that orbit along the equator and are always aimed towards the visible hemisphere of earth. On those, the directional hemisphere and spot beam antennas are fixed in place (or can only move a few degrees motion at best, such as Ku band spot beam antennas), relying on the body orientation of the satellite to service a certain area of the visible hemisphere. Satellites are designed to go into "safe mode" if certain fault protection events happen, or the multiply redundant control systems/onboard computers don't agree with each other. Safe mode usually means shutting down all nonessential electrical loads and trying to orient themselves so that solar panels receive the greatest amount of charge, while listening for command and control data on their omni (L and S band) TT&C antennas. With this event it sounds like something REALLY went wrong since not only did the satellite try to correct a nonexistant wrong orientation via its reaction wheels (reaction wheels are not nearly as powerful in real life as they are in kerbal space program), it then decided to start expending propellant and spun itself up to such a RPM of revolutions that it tore off its own solar panels, and anything else that would be vulnerable to high centrifugal G forces. Automated code that expends propellant is usually checked much more carefully than this, since the amount of propellant is fixed and non renewable, usually the primary constraint on the total service life of the satellite. Most satellites run out of stationkeeping/orientation propellant (or propellant for ion engine delta-V changes) long before their multiply redundant solar/charge controller/battery/computer control systems fail.

ourcat 11y ago

"Are you sure you want to upgrade your satellite app?" Yes... boom...

adrianlmm 11y ago
I hope you get modded down, we are not in Slashdot.
- ourcat 11y ago
  Sorry. I was not aware that there was such a low tolerance for a tiny bit of lightheartedness here. I hope you lighten up. Peace.

samwestdev 11y ago

Guys always read the change log before upgrading

askafriend 11y ago
You mean changelogs like these? https://twitter.com/cirbif/status/728114363563839488 https://twitter.com/cirbif/status/728114363563839488
aaron695 11y ago
I find this useless advice. I didn't write or are aware of much of the code I'm part of. Or remember some of the code I did write. Or even understand the ramifications of what the change log means. All assuming it's a well written change log. You need external process to ensure upgrades don't break things. Not a 'read the change log' policy so someone can blame you post breakage cause then it'll be easy to find issues in the change log.
deleted 11y ago
[deleted]

walrus01 11y ago

I know this was a science and LEO (low earth orbit) satellite, but here's something to consider when thinking about the difficulties of engineering hardware & software to work together in a satellite. Geostationary telecom (and weather, and SIGINT, etc) satellites: In the entire history of manned spaceflight no human has ever visited geostationary orbit. Once placed in a geostationary transfer orbit (about 450 x 36,000 km) and then onwards to final geostationary orbit, nobody will ever see a satellite again. The largest ones weigh 6000 kilograms and are the size of greyhound buses. They're out there right now operating with multiply-redundant everything and cost $150 million to build and launch. It is highly unlikely that any time in the next 25 years any human will ever visit one in person or touch one. When a satellite is encapsulated in its fairing/shroud for launch that is the last time anyone will ever see it until its ultimate end of life in hopefully 15 years. Every one of its control systems needs to be so thoroughly debugged and multiply redundant that it can operate out there with absolutely zero chance of repair or parts replacement.

colemickens 11y ago
Presumably these satellites are transmitting data back to Earth? Is that channel not bidirectional? If those satellites can be remote reprogrammed, are they really much different than satellites in geosynchronous orbit? Or are you suggesting that satellites in geosync orbit could be manually updated via EVA? I would think that would still be extremely cost-prohibitive (not to mention a huge risk to human life for a weather satellite! Has that ever actually happened?
- walrus01 11y ago
  I'm saying sort of the opposite - there have been very, very rare cases where satellites in LEO were visited by humans and repaired or retrieved, but it's extremely uneconomical (Hubble, LDEF, etc). Actually cheaper to build and launch a new one. But at least it is technologically possible. Whereas nobody has ever gone to geostationary orbit. There have been proposals for ion engined orbital tugs to grab onto old, out-of-stationkeeping-propellant satellites that still have good electronics, for the purpose of extending their life, or moving them to a new orbital position. But nothing has actually flown.

kibwen 11y ago

What altitude was the satellite at when it broke up? Will the debris pose a problem for other satellites, or was it low enough that the pieces will reenter the atmosphere quickly?

Nicholas_C 11y ago
Is satellite debris an issue? Or does the vastness of the area in which satellites orbit take care of that?
- dcposch 11y ago
  Yes, it is a serious issue https://en.wikipedia.org/wiki/Space_debris https://en.wikipedia.org/wiki/Space_debris https://en.wikipedia.org/wiki/Kessler_syndrome https://en.wikipedia.org/wiki/Kessler_syndrome
- DarkTree 11y ago
  Very much a problem. [https://en.wikipedia.org/wiki/Kessler_syndrome https://en.wikipedia.org/wiki/Kessler_syndrome]

deleted 11y ago

[deleted]

nurblieh 11y ago

Ironically, the secondary payload that launched with Hitomi was a micro-sat which monitors space debris. https://www.frontier.phys.nagoya-u.ac.jp/en/chubusat/chubusat_satellite3.html https://www.frontier.phys.nagoya-u.ac.jp/en/chubusat/chubusa...

2PetitsVerres 11y ago

I'm don't completely agree with the fact that it's called a "software update". Reading different article about it, from what I understand, there was an initial error (software error probably triggered by an hardware error probably due to an upset due to radiation), but this error is not very important. A series of event from this error triggered the safe mode (that's expected), and there was the critical problem. They had updated parameter in the software describing the torque generated by each thruster (or the center of mass position, or the tensor of inertia, or parameters based on all this) These parameter are software parameters, but updating them is not updating the software. It's software data, not software code. Of course this does not change the fact that it is a critical error, but it's not exactly software update (IHMO). It's a configuration error. It's strange that they didn't see that in a simulator before updating them, but it's possible that they may have used the same value in the part simulating the software and the part simulating the thrusters themselves. (note: I'm working in the satellite on-board software/attitude control domain, but not for JAXA, in Europe. Anyway at my current position, I must both test this kind of code, and the parameter used in the code. And checking the parameters is much more difficult, because you must be sure that everyone agree on everything. This includes a lot of basic stuff, but it's a pain in the ass ;-) ) This pdf from JAXA is probably the initial source of all articles. http://global.jaxa.jp/press/2016/04/files/20160428_hitomi.pdf http://global.jaxa.jp/press/2016/04/files/20160428_hitomi.pd... I found it interesting to read, if you know how it works. I would of course prefer to have more detains. I always want more details. But it's for the press...

verytrivial 11y ago

Hindsight is 20:20 of course, backseat driver etc. but if the on-board systems detected a bad rotation, then started a burn to correct it, presumably it could have been possible to detect that the burn was not "helping" during the burn? And halted the burn? The thrusters aren't usually that powerful, so the erroneous death-spin probably took a fair while to spin up. Even if the sensors were wrong, if you're a computer trying to get variable 'X' into a range, and you apply control 'Y', but 'X' moves further and further away, let go of the controls and ask ask an adult help! (Easier said than done, I know. I'm amazed space engineering works as often as it does! Super hard stuff.)

lloeki 11y ago
Of course it was using a feedback loop, but GIGO applies: The STT and IRU disagreed on the attitude of the satellite. In this case the IRU takes priority, but its data apparently was wrong, reporting a rotation rate of 20 degrees per hour, which was not occurring. Starting from there it would have no way of knowing about the true value of 'X', so the feedback loop was fed with wrong data and just kept taking decisions† that made things worse, especially given that: The satellite configuration information uploaded earlier was wrong and the reaction wheels made the spin worse. [...] the ACS attempted to correct a rotation that didn’t exist. The erroneous configuration information led the ACS to aggravate, not correct, the rotation. † Even without misconfiguration, stopping an object from spinning in a vacuum isn't as direct and linear as accelerating/braking in a car, requiring precise coordination of multiple fixed thrusters and/or reaction wheels.

piyush_soni 11y ago

And you ask why I am afraid to install that "flash player update". :)