5 ms·

"Cause still unknown after several thousand engineering-hours of review. Now parsing data with a hex editor to recover final milliseconds." 4:09 AM - 29 Jun 20

• colomon 11y ago
  Ugh, now imagining trying to debug a really sneaky bug that costs $100 million every time it triggers...
  • kabdib 11y ago
    I recommend Henry S.F. Cooper's book _The Morning Star_, which is about debugging hardware failures and OS race conditions on a spacecraft orbiting Venus. (There are similar stories about debugging stuff on the various Martian rovers, but Cooper's book is a very good treatment...).
    • twistedpair 11y ago
      It's actually "The Evening Star" Grab it used for $0.75 http://product.half.ebay.com/The-Evening-Star-Venus-Observed-by-Henry-S-Jr-Cooper-1993-Hardcover/145419 http://product.half.ebay.com/The-Evening-Star-Venus-Observed...
    • alphadevx 11y ago
      I think you mean The Evening Star, marvelous book: http://amzn.to/1Hq8TcX http://amzn.to/1Hq8TcX
    • kabdib 11y ago
      I stand corrected. Written in haste. Cooper is a wonderful writer, I've got all of his books :-)
  • drzaiusapelord 11y ago
    Stupid question perhaps, but don't we have the ability to 100% simulate the rocket physics and the code it runs on in a simulation environment? Or are they just punching out code, shoving it onto rocket controllers, and testing in the real world? I'm curious as to the QA system used here. I imagine with proper simulation this should have been catchable. I wonder if SpaceX's low cost approach means cutting certain corners and situations like these where catchable issues make it into the wild because of the difficulty of rocketry in general with the added difficulty of cheap spaceflight tacked on. I really hope they didn't just find themselves in a STS-51-L moment where it'll take months to truly iron out the root issues. Thank goodness there was no loss of life and SpaceX's stack isn't man rated yet.
    • nkoren 11y ago
      Rocket physics are actually relatively trivial and, yes, can be fully simulated (and are). Rocket plumbing is always the hard part. Perhaps the threads on some connector got stripped as it was screwed together. Perhaps there was a manufacturing flaw or unexpected bit of corrosion in a particularly vital bolt. Perhaps a bit of contamination in a fuel line caused something catalytic to happen. Perhaps a bit of excess H20 condensation in a LOX valve caused an ice dam to form and the supply line to over-pressurise. These are the sort of problems which bring rockets down: really damned complicated plumbing problems.
      • drzaiusapelord 11y ago
        I guess my point is, can we do a full simulation of every screw, material, plumbing, liquid dynamics, weather dynamics, etc and augment those with known fail scenarios and other fuzzy data to build out a real world KSP that predicts fails reliably? We should understand how things like corrosion and condensation work on a rocket engine. Considering the low cost of incredible amounts of CPU power, granular level simulation is possible on a certain level today if someone wanted to create it. We certainly see this kind of thing with stealth technology, where we can simulate every permutation of near every radar photon hitting the various surfaces of planes with various materials, scenarios, temperatures, etc. I imagine this level of simulation might not be entirely feasible yet. Maybe for the lack of trying or budget. In a growth industry or one powered by both commercial and technical pressures, it may be difficult to sit down and build something like this out. From a more practical point of view, it may make sense to just let things explode than spend years running expensive simulations instead of building things, launching, and collecting paychecks.
• rwhitman 11y ago
  After watching the press conference yesterday I got the sense that Musk's tweets aren't particularly helpful for SpaceX press relations
  • nkoren 11y ago
    The thing is -- despite all the "Iron Man" winks and glamour of being a billionaire who wants to colonise Mars, etc. -- Elon is truly, deep down, a nerd's nerd. Which means that his respect for the PR department is probably roughly proportional to how much duct tape they've wrapped around his head.
    • rwhitman 11y ago
      The PR effort in this case was being lead by Gwynne Shotwell who is the President & COO. He was apparently tweeting information that his COO wasn't prepared to comment on yet. That doesn't send a good message about leadership confidence to people in his organization regardless of how much of a "nerd's nerd" he's trying to present himself as
  • onewaystreet 11y ago
    SpaceX was criticized after its early failures for not being transparent enough, so Musk is probably still reacting to that.
• anotherangrydev 11y ago
  TBH, that hex editor thing sounds like bs.
  • timsally 11y ago
    Can you be more specific? If you have a partial piece of data (say because the data was streaming and a rocket exploded), the parser for that data is unlikely to work out of the box. Hence the use of a hex editor.
    • anotherangrydev 11y ago
      Yeah, but why not write your own temporary parser or whatever. They (should) have top-class software engineers in there, why they don't just make a quick and dirty script to sort out whatever they're looking for? IDK, that tweet felt like when you're watching a movie and then the 'hacker' comes around and you see a lot of props on the screen that are designed to excite 'geeks'. Like, right now, a lot of guys should be like: "OMG OMG! They are looking at the hex code directly! Damn, that is so looooooow-level, they call it rocket science for a reason!" Anyway, hopefully they find the problem; whether with hex editors or not SpaceX is really doing cool stuff.
      • ygra 11y ago
        Some hex editors allow you to actually define an ad-hoc data format to parse, e.g. http://www.x-ways.net/winhex/templates.html http://www.x-ways.net/winhex/templates.html – it doesn't have to be ed(1) for hex data. Besides, it could just have been a choice of words meant to convey that they have a bunch of incomplete/corrupted data that needs manual interpretation.
      • mikeash 11y ago
        How are you going to figure out how to write that quick and dirty script without looking at the raw data in a hex editor to figure out how it's been corrupted? Far from BS, pointing a hex editor at your data dump sounds like the very first thing you'd do when you get some data that your software can't immediately read.