11 ms·

My $500M Mars rover mistake

• javajosh 3y ago
  Another failure is from the person who used the multimeter to complete a circuit - and then didn't even leave a note on the multimeter. That person could reasonably have anticipated this error mode, and taken steps to prevent it.
• genter 3y ago
  So If I understand it correctly, the electrical connector was genderless? If so, that's relatively rare (the only ones I can think of are Anderson Powerpole, which I don't think are rated for interplanetary vehicles) and extremely stupid. Edit: I suppose he could've been using alligator leads.
  • Syonyk 3y ago
    You can configure an Anderson connector pair in a way that it won't reverse. I use them quite frequently, and have a local (though admittedly undocumented...) standard for what orientations mean what voltage. It's not bulletproof, but it does make me think when things don't line up.
  • m-ee 3y ago
    All the major suppliers make hermaphroditic connectors, but like you I’ve only ever seen the powerpoles in person.
  • spfzero 3y ago
    This was a test fixture so probably banana leads.
    • hiddencost 3y ago
      I color code _everything_.
  • myself248 3y ago
    A breakout box usually has both male and female sides, along with the banana or whatever breakout in the middle, so it can be plugged into either side of the circuit, or both at once and watch signals while the system operates. It's not genderless, it's a pass-through. Making an only-faces-the-motor breakout, and a separate only-faces-the-driver breakout, might've been prudent, presuming that they used unique and consistent connectors, for instance a single gender always on the motors. But that's quite an assumption and I can imagine a ton of reasons it might not apply.
• johnbatin 3y ago
  It's interesting story, but not a catastrophic failure.
• lmm 3y ago
  Why would you ever allow people to work 12-hour days on something so important? Grad student labour is cheap, surely trying to have one person do the work of two is a false economy.
  • stickfigure 3y ago
    On your next hospital visit for life-saving care, I'm sure you will be comforted to know that nurses (in the US) typically work 12 hour shifts and they're on their feet the whole time.
    • darrylb42 3y ago
      And doctors doing 24h shifts. At least in the NICU my kids were in.
    • ajmurmann 3y ago
      This is true, but IMO doesn't refute the point, it just makes me concerned about care quality. Were there any studies that showed that the long, grueling shifts are actually better or is it simply this way because it's always been that way and change would be hard and expensive, and because "my grandma walked up hill to school both ways, so young people can too"?
      • flashback2199 3y ago
        Who said it's gruelling? In the case of the rover, it's not year round, it's when they are preparing for a launch. Also, you're on HN, so surely you have heard of flow.
    • Aurornis 3y ago
      Nurses work 3x12 shifts to reduce the number of times patient care is handed off. If everyone worked 8 hour shifts you’d have 3 handoffs per day and a minimum of 3 different people caring for the patient. With 12 hour shifts you have only 2 handoffs per day and can have 2 people trade off on patients if their schedules line up. I’m sure it varies by location, but my nurse friends only work 3x12, giving them 4 days off per week. Working 12 hour shifts is much more acceptable when you have more days off than days spent working. They’re virtually unavailable on days they work, but then they’re off traveling or having fun for 4 days, some times more if they combine their days off back to back. My close nurse friend routinely takes week long vacations without actually taking any time off at all.
• Scubabear68 3y ago
  With only two weeks left to go, why was a multimeter completing the telemetry circuit?
  • Titan2189 3y ago
    So that it could be quickly and easily unplugged
  • kragen 3y ago
    maybe it was measuring the current rather than the voltage as he thought, which would require putting it in series with the circuit
    • spfzero 3y ago
      That stood out to me as well when I read the article.
      • CamperBob2 3y ago
        The multimeter in the photo has the probes plugged into the volts/ohms jack, though. Maybe it was a different meter. I figured this was going to be a story about trying to measure voltage with the meter set up on the 10A current range.
        • kragen 3y ago
          yeah i don't think he took a photo of the multimeter to use in the article before unplugging it. he didn't yet know it was important
  • elteto 3y ago
    Why would you not use a multimeter two weeks before launch? The two things are orthogonal to each other.
  • deleted 3y ago
    [deleted]
• pomian 3y ago
  What a great write up. The tension preceding launch must have been tremendous, and lack of sleep adding to the possibility of such a small yet critical error. Good lesson for all of us.
• jes5199 3y ago
  this sure seems like a symptom of a much larger organizational failure that could mistake "it's not plugged in" for "it's broken"
• avar 3y ago
  Are the electronics in these rovers really so bespoke that they don't have multiple copies of each electronic component warehoused on-site? I'd expect that the rover body itself would be bespoke this late in the process, although a parallel test vehicle would be useful, do they have that?). But in case someone fried the rover's electronics I'd think tearing it apart and replacing them while maintaining the chassis should be doable in 2 weeks, but what do I know?
  • el_toast 3y ago
    Typically your best bet is going with a provider that already produces whatever item, isn't bespoke to that particular program, and has an assembly line of sorts already set up. However the other problem is they are often built in low numbers to begin with so getting hold of a flight qualified unit will most likely be an issue as well. Also everything is tightly packed together which usually makes replacing something involve messing with a bunch of other items.
  • elteto 3y ago
    Of course they are bespoke. There are no COTS Mars rovers for sale.
    • r2_pilot 3y ago
      To be fair; there may be COTS Mars rovers for sale (possibly even cheap), they're just not certified or engineered specifically for Mars. Even an RC car might last long enough for a mission if it was free to get there.
      • amluto 3y ago
        Sometimes I wonder whether something like the SpaceX approach would work for these sorts of mission: develop a way to cheaply and reproducibly build the mission hardware, then iterate on it until it works.
  • deleted 3y ago
    [deleted]
• soorya3 3y ago
  I like this - “Let your scars serve you; they are an invaluable learning experience and investment in your capability and resilience.” I have had couple of scars of mine. I feel like sometimes you become risk-averse and When you are launching new things you will face the fear of failure.
• graysonpike 3y ago
  This clickbait title is misleading.
  • paulddraper 3y ago
    "My almost $500M mistake" doesn't have the same ring.
    • defanor 3y ago
      "My mistake that seemed to potentially cost up to $500M, but then it turned out fine" would be more accurate. While a more appropriate title would be something akin to "learning from mistakes" or "a Mars rover testing scare".
• irjustin 3y ago
  I'm reminded of the phrase - if your intern deleted the production database you don't have a bad intern; you have a bad process. Whether this was a process problem or a human one we don't really get to judge since we do expect more from a FTE. I'll just say putting myself into his shoes made me tear up as I read the dread and pangs of pain upon realizing what happened - then to have life again after the failure of the ray of hope. That weight, I've never had a project that so many people depended on. All heroes in my book.
  • pyrolistical 3y ago
    You should never blame the individual for organizational failures like this. I see two process issues: 1. The plug was allowed to be connected backwards. Either this should be impossible or this hazard should be identified and more than one human should verify orientation 2. In use tools like multimeters should never be disconnected. At worst you get problems like this at best you annoy whoever was using it Blaming individuals only gets them fired and weakens the entire organization. You just fired the one person who learned an expensive lesson. The only time when an individual should be blamed is when they intended harm, at which case the law could kick in
    • idlewords 3y ago
      You can't apply process thinking here, where the scenario is custom testing a unique probe, and you don't know what other constraints are in play (for example, the reason for the plug design). If NASA were sending these things to Mars by the dozen, then you can start to formalize things like test procedures and look for places mistakes can happen. But in this scenario, you're just disempowering your staff by not letting them choose the most effective and low-risk way to do one-off, highly specialized testing work.
      • hughes 3y ago
        Aerospace-grade connectors are specifically designed to support multiple keyings that prevent this kind of thing. It's definitely a problem preventable by careful design if the interface supports making this kind of mistake.
• dclowd9901 3y ago
  Ohhhh I hate things that you “just have to not screw up.” A fiddly manual process with so many possible ways to screw something up is almost guaranteed to see a catastrophic failure. If this really manual fiddly process was really the only way they could test the motors, I’d say that’s a big failure on the design engineer’s part.
  • spfzero 3y ago
    Test requirements often arise after system design is complete. Engineers need time to examine the system, theorize failure modes, and design the tests. Also time to test the system, find some unexpected failure mode, and then design yet more tests.
    • dclowd9901 3y ago
      Would it be so hard to add a diagnostic interface? That’s all we’re talking about here. It seems like you’re making the problem more difficult than it is. And they’d had to have known ahead of time if they wanted to test voltage curves of motors. Your answer is good in the general case, but for the anecdote, the design was clearly bad.
  • dylan604 3y ago
    Or it just needs a budget that can dramatically increase and/or have a deadline that can continue to be pushed back to allow for corrections. As exhibit A, I'd like to present the james webb space telescope.
• mywacaday 3y ago
  Any test that could cause a fatal destructive error should be risk assesed and a suitable protocol approved with four eyes approval on a final checklist before going hot on the electrics. The issue here is poor project governance not human error.
  • nomel 3y ago
    Yeah, the borrowed multimeter really hammers how silly this all was. You don't touch other peoples lab equipment unless the other ends of the wires are hanging free. A finger pointing to a meter needs to be followed up with a clear confirmation that the wires can be disconnected, and special care isn't needed in the process. If I need something that's connected, I always ask the person to disconnect it for me. Definitely a process/culture problem.
• bostondev1 3y ago
  How is this a $500 million mistake? It seems the issues didn’t cost $500 million.
  • paulddraper 3y ago
    It was feared to be a $500M mistake.
    • deleted 3y ago
      [deleted]
  • chikenf00t 3y ago
    I guess the idea is that it potentially could have cost $500 million therefore it was a $500 million mistake. It's not exactly accurate but it does help contextualize the gravity of their mistake.
    • magicalhippo 3y ago
      Reminds me of my favorite George Carlin bit[1]: Here's one they just made up: "near miss". When two planes almost collide, they call it a near miss. It's a near hit. A collision is a near miss. [1]: https://www.quotes.net/mquote/35854 https://www.quotes.net/mquote/35854
  • spartanatreyu 3y ago
    It's not "mistake" singular, it's actually mistakes. The first mistake is the $500 million rover is fried. The second mistake is believing the first mistake. (or put another way, the first mistake cost $500 million, the second mistake which they didn't realise at the time, saved $500 million) But you can't explain the second mistake without first explaining the first mistake, hence the title.
    • deleted 3y ago
      [deleted]
• SoftTalker 3y ago
  Reminds me of a quote attributed to Thomas J Watson: Recently, I was asked if I was going to fire an employee who made a mistake that cost the company $600,000. No, I replied, I just spent $600,000 training him. Why would I want somebody to hire his experience?
  • huytersd 3y ago
    Well be prepared to spend $600k more next month for the next “training” session.
    • superb_dev 3y ago
      The point is that they won't make the same mistake again
      • huytersd 3y ago
        I guess the idea is that if the problem was incompetency they will find some other way to mess up as opposed to a genuine mistake made by a competent person.
        • user_7832 3y ago
          > I guess the idea is that if the problem was incompetency they will find some other way to mess up as opposed to a genuine mistake made by a competent person. Are you suggesting that a competent person never messes up/makes a mistake? The most fundamental part of life is learning from mistakes, and today even AI is starting to do this. Mistakes and evolution are what _make us_ human and living.
          • huytersd 3y ago
            Competent people make fewer mistakes because they’re careful and… competent.
            • 3y ago
• dgunay 3y ago
  One thing about long aerospace missions like this with huge lead times that always gets me - you can spend years of your life working on a mission, only for it all to fail with potentially years until you can try again. This is a refreshingly humanizing article, but is also one written from the perspective of a survivor. Imagine if the rover were actually lost. I asked the question "what would you do if the mission failed after all of this work? How could you cope?" to the folks at (now bankrupt) Masten Aerospace during a job interview, and maybe it was a bad time to ask such a question, but I didn't get the sense they knew either. "The best thing we can do is learn from failure," one of them told me. An excellent thing to do, but not exactly what I asked. This to me stands out as the defining personal risk of caring about your job and working in aerospace. Get too invested, and you may literally see your life's work go up in flames.
  • Teever 3y ago
    > "The best thing we can do is learn from failure," one of them told me. I would argue that if we don't charge the process to prevent this kind of catastrophic failure mode then we really haven't learned from the failure.
  • typpo 3y ago
    > you may literally see your life's work go up in flames. Incidentally, this happened to Lewicki a few years later when Planetary Resources' first satellite blew up on an Antares rocket: https://www.geekwire.com/2014/rocket-carrying-planetary-resources-satellite-shortly-liftoff/ https://www.geekwire.com/2014/rocket-carrying-planetary-reso...
    • dgunay 3y ago
      Did they have a narrow launch window they couldn't afford to miss? I'm not talking about missions where you eat a big monetary loss on the launchpad and try again, I mean missions which rely on planetary alignments that may not happen again for years, or even the rest of your life, such as Voyager. Or even just missions where you launch successfully, but then after months (or years) of flight time the spacecraft is lost.
• deleted 3y ago
  [deleted]
• DoingIsLearning 3y ago
  Off-topic but I never realized how much kapton tape is used to put these things together, until I saw these internal 'guts' photos.
  • cuSetanta 3y ago
    As a spacecraft integration engineer I can confirm that most spacecraft are about 80% kapton tape.
• jay-barronville 3y ago
  Really well written story. As a software engineer, I have a couple stories like this from earlier in my career that still haunt me to this very day. Here’s a short version of one of them: Like 10 years ago, I was doing consulting work for a client. We worked together for months to build a new version of their web service. On launch day, I was asked to do the deployment. The development and deployment process they had in place was awful and nothing like what we have today—just about every aspect of the process was manual. Anyway, everything was going well. I wrote a few scripts and SQL queries to automate the parts I could. They gave me the production credentials for when I’m ready to deploy. I decided to run what you could call my migration script one last time just to be sure I’m ready. The very moment after I hit the Enter key, I realized I had made a mistake: I had just updated the script with the production credentials just before I made the decision to do another test run. The errors started piling and their service was unresponsive. I was 100% sure I had just wiped their database and I was losing it internally. What saved me was that one of their guys had just a couple hours earlier completed a backup of their database in anticipation of the launch; in the end, they lost a tiny bit of data but most of it was recovered via the backup. Ever since then, “careful” is an extreme understatement when it comes to how I interact with database systems—and production systems in general. Never again.
  • alberth 3y ago
    Hope you bought that guy a beer. Great story, thanks for sharing.
  • cdogl 3y ago
    Your excellent story compelled me to share another: We rarely interact directly with production databases as we have an event sourced architecture. When we do, we run a shell script which tunnels through a bastion host to give us direct access to the database in our production environment, and exposes the standard environment variables to configure a Postgres client. Our test suites drop and recreate our tables, or truncate them, as part of the test run. One day, a lead developer ran “make test” after he’d been doing some exploratory work in the prod database as part of a bug fix. The test code respected the environment variables and connected to prod instead of docker. Immediately, our tests dropped and recreated the production tables for that database a few dozen times.
• didntknowya 3y ago
  $500M is a lot to waste in private money, but unfortunately in taxpayer money it's pittance and usually swept under the carpet
• atonalfreerider 3y ago
  Reminds me of the NOAA-N Prime satellite that fell over, because there weren't enough bolts holding it to the test stand. The roots cause, and someone correct me if this is not accurate, was that the x-ray tested bolts to hold it down were so expensive, that they had been "borrowed" to use on another project, and not returned, so that when the time came to flip the satellite into a horizontal position, it fell to the floor. Repairs cost $135M. https://en.m.wikipedia.org/wiki/NOAA-19 https://en.m.wikipedia.org/wiki/NOAA-19
  • euroderf 3y ago
    And so when "Check for bolts" is added to flip procedure errata/addendum, is light sarcasm called for ?
• maestroia 3y ago
  $500M? Pocket change... THE LITTLE VAX THAT COULD https://userpages.umbc.edu/~rostamia/misc/vax.html https://userpages.umbc.edu/~rostamia/misc/vax.html
  • gmfawcett 3y ago
    I am as old as the hills. How have I never heard this story before? Thank you.
• anon-3988 3y ago
  Bravo to the author. I think if I had made the same mistake, and if the disaster is permanent, I would probably be scarred for life and never recover from it.
  • wolverine876 3y ago
    The author is scarred for life, reliving it in certain circumstances. They use the scar as a tool to sharpen their performance.
• initplus 3y ago
  Interesting to see that the worry could have been avoided if they had lined up their timelines better in the first place. If they'd compared the timestamp on the test readout to the last timestamp from the telemetry system, they'd have seen that the telemetry failed BEFORE the test was executed. Partially caused by using imprecise language "we seem to have lost all spacecraft telemetry just a bit ago" rather than an accurate timestamp. A cautionary lesson in properly checking how exactly events are connected during an incident. Easy to look at two separate signals and assume they must be causal in a particular direction, when in reality it is the other way around.
  • deleted 3y ago
    [deleted]
• chriscjcj 3y ago
  I work in TV. During my first job at a small market station 30 years ago, I was training to be a tape operator for a newscast. All the tapes for the show were in a giant stack. There were four playback VTRs. My job was to load each tape and cue it to a 1-second preroll. When a tape played and it was time to eject that tape, it was _very_ easy to lose your place and hit the eject button on the VTR that was currently being played on the air instead of the one that they just finished with. The fella who was training me did something very annoying, but it was effective: every time I went to hit the eject button, he would make a loud cautionary sound by sucking air through his closed teeth as if to tell me I was about to make a terrible mistake. I would hesitate, double check and triple check to make sure it was the right VTR, and then I would eject the tape. He made that sound every single time my finger went for the eject button. It really got on my nerves, but it was a very good way to condition me to be cautious. Our station had a policy: the first time you eject a tape on the air got you a day off without pay; the second time put you on probation; the third time was dismissal. I had several co-workers lose their jobs and wreck the newscast due to their chronic carelessness. Thanks to my annoying trainer, I learned to check, check again, and check again. I never ejected a tape on the air. It certainly would not have been a half-billion dollar mistake if I had, but at that point in my career it would have felt like it to me.
  • ksaj 3y ago
    That explains the old blooper reels that were popular on TV in the early 80's, where the reporter would be talking about something, and get video of something completely bonkers in the background instead.
    • maxerickson 3y ago
      Rolling the wrong tape still happens frequently enough on modern live broadcasts.
• curiousObject 3y ago
  The story is compellingly written, but I thought it was also confusing. It sounds as if this team made several mistakes, not just one mistake. It's also not clear if the result of these mistakes was that there might be real damage to the spacecraft, or if the result was just wasted time and hours of confusion about why the spacecraft wouldn't start up. The first mistake is they didn't realize that the multimeter was not only measuring, but it was also completing the circuit. That sounds like a real bad idea. But if it was totally necessary to arrange it like that, then that multimeter should never have been touched. That's not just one guy's error. It's at least two guys at fault, along with whoever is managing them, and whoever is in charge of the system that allows it. The second mistake is with the break-out-box. They think he misdirected power wrongly into the spacecraft. Then they jump to the conclusion that has generated a power surge which has damaged the spacecraft, because it won't start up. But they're not sure where the power surge went and what might be damaged. Anyhow they're wrong. The reason the spacecraft won't start up is just because he took the multimeter out of the circuit before the accident. I'm still sort of confused about what happened or if they ever really figured out what happened. He said "Weeks of analyses followed on the RAT-Revolve motor H-bridge channel leading to detailed discussions of possible thin-film demetallization". Does this mean that they decided that the misdirected power surge might have flowed into the RAT-Revolve motor H-bridge channel and damaged that?
  • maxerickson 3y ago
    It started up fine. The multimeter was connecting up the telemetry, so they weren't getting any information from it until they restored that circuit. The power absolutely did feed into that circuit, they were trying to decide if it would have damaged it (but a motor driver is going to be able to handle power coming from the motor, so they decided that it probably didn't damage it).
    • curiousObject 3y ago
      > It started up fine. Thanks. I understand that better now. The spacecraft did start up, but it seemed as if it could be badly damaged because they were not receiving any telemetry data
• deleted 3y ago
  [deleted]
• aunwick 3y ago
  Congratulations. 102 circuits would have taught this. But some short sighted bean counters merged EE & CE so you didn't get the opportunity. I suggest you ask for a tuition refund as they failed to educate you as they promised.
• aunwick 3y ago
  So, some short sighted bean counters combined the EE & CE degrees and you are the result? Ask for a tuition refund!
• vsgherzi 3y ago
  Likely my favorite article ever; amazing writing.
• goku12 3y ago
  If there is the possibility to make a mistake, somebody will certainly make it. You expect all the humans involved to be competent. But relying on that competence is a mistake. The emotional stress of dealing with such enormous responsibilities, the often long work hours and the long list of procedures will make any competent professional to inadvertently slip up at some point. In case of electrical connectors, the connectors are often grouped together in such a way as to avoid making wrong connections. Connectors with different sizes, keying, gender, etc are chosen to make this happen. This precaution is taken at design time. JPL is extremely experienced in these matters. There is probably something else left unsaid, that led to this mistake being possible. Meanwhile, motor controllers using H-bridge is something that's never boring. I once saw a motor control fail so spectacularly that we were scratching our heads for days afterwards. As always, a failure is never due to a single cause (due to careful design and redundancies). It's a chain of seemingly innocuous events with a disastrous final outcome. But the chain was so mind-bending that we had to write it down just to remember how it happened. Recently, I was watching the Chernobyl nuclear disaster and I got reminded of this failure. Our failure was nowhere near as disastrous - but the initial mistakes, the control system instability, the human intervention and the ultimate failure propagation were very similar in nature. Needless to say, it sent us back to the drawing board for a complete redesign. The robustness of the final design taught me the same lesson - failures are something you take advantage of.
• mk_stjames 3y ago
  I am a Mechanical/Aerospace engineer.... I wish my scariest stories 'only' involved a potential bricking of a main computer on an unmanned $500M rover. No... I was the senior safety-crit signoff on things carrying human lives. I had to look over pictures of parts broken from a crash and have the potential feeling of 'what-if that's my calculation gone wrong'. My joint that slipped. My inappropriate test procedure involving accelerated fatigue life prediction, or stress corrosion cracking. My rushing of putting parts into production processes that didn't catch something before it went out the door. It's interesting to read people's failure stories from similar fields but, to me, the ones that people so openly write about and get shared here on HN always come across as... well, workplace induced PTSD is not a competition. It's just therapy bills for some of us more than for others.
  • pants2 3y ago
    And the engineers designing safety mechanisms for nuclear weapons probably think you have it easy.
    • H8crilA 3y ago
      [flagged]
    • rgmerk 3y ago
      Yep, that's fair! But my understanding is the default behaviour of most nuclear weapons (other than Hiroshima-style ones) is "blows itself to pieces without detonating the nuclear part", rather than "vapourises everyone within a mile". Everything needs to go right for a nuclear weapon to actually blow up with a significant yield.
      • King-Aaron 3y ago
        Yeah, luckily they seem to have a pretty well sorted failure mode.
  • Terr_ 3y ago
    That reminds me of a fiction-quote by one of my favorite authors, where a welding-instructor has just finished sharing a (somewhat literal) post-mortem anecdote of falsified safety inspections. > He gathered his breath. “This is the most important thing I will ever say to you. The human mind is the ultimate testing device. You can take all the notes you want on the technical data, anything you forget you can look up again, but this must be engraved on your hearts in letters of fire. > “There is nothing, nothing, nothing more important to me in the men and women I train than their absolute personal integrity. Whether you function as welders or inspectors, the laws of physics are implacable lie-detectors. You may fool men. You will never fool the metal. That’s all.” > He let his breath out, and regained his good humor, looking around. The quaddie students were taking it with proper seriousness, good, no class cut-ups making sick jokes in the back row. In fact, they were looking rather shocked, staring at him with terrified awe. -- Falling Free by Lois McMaster Bujold
• kobieps 3y ago
  I once legit blew a $2K FPGA in postgraduate lab by fatfingering. Horrible memory.
• tamimio 3y ago
  And I thought I had a big mistake when I crashed an expensive drone.. nice read!
• nosrepa 3y ago
  At least it was just a Fluke.
• eightnoteight 3y ago
  I felt like the real lesson is a missing flight recorder system, if each action and behaviour is properly logged, it would have been pretty clear that the telemetry was lost before chris even began testing the RAT motor
• accrual 3y ago
  It's a beautiful story. As a space fan, especially of the interplanetary type, this story was riveting to me as it unveiled details of the spacecraft testing I've never imagined. I do far less important testing in my day to day but I was able to draw some similarities with the author. Many have posted of their failures here so I suppose I could share a couple of mine. - Pushing gigabytes of records into a Prod table only to realize the primary key was off by a digit, rendering the data useless for a go-live. It had to be deleted by the database admins and reloaded, which took precious hours. I forget why, but an update wasn't feasible. - A perfect storm of systems issues that lead to all servers in the pool becoming unavailable, causing an entire critical system to go dark. We got it back up within minutes, but harrowing nonetheless. - Realizing hours before a go-live that a key data element was missing, prompting a client who was now in a code freeze to make a change (they were quite upset). Pretty sure I got an unfavorable review from that, but haven't made the same mistake since.
• frenchie14 3y ago
  I worked at JPL for two years in college and helped with flight hardware testing a few times (probably in the same clean room this story took place in, albeit several years later). I can definitely see how a mistake like this could get made. A few stories I remember hearing from those days: 1. Bending pins from trying to insert a connector incorrectly 2. Running a full day of testing but forgetting to set up data recording 3. Accidentally leaving a screwdriver next to the hardware inside a thermal vac chamber in an overnight test Fun times!
• donatj 3y ago
  In my company people aren’t allowed to change org level GitHub settings without a second person watching over them, but NASA let a 28 year old kid run electricity into spacecraft without oversight? Really seems like anything which when done improperly could cause millions of dollars in damage, there should be a second person reviewing your setup first.
  • stets 3y ago
    The average age of engineers running Apollo 11 was supposedly 28 https://www.popularmechanics.com/space/a4288/4318625/?utm_source=reddit.com#:~:text=%22There%27s%20a%20guy%20standing%20by%20the%20console%20with%20a%20huge%20piece%20of%20paper.%20That%27s%20me.%20I%20got%20the%20signatures%20of%20everybody%20in%20that%20room%20and%20in%20the%20back%20room.%20Every%20time%20I%20did%20that%20I%20would%20ask%20them%20their%20age.%20Well%2C%20I%20sat%20down%20and%20ran%20it%20out.%20The%20average%20age%20the%20night%20we%20had%20splashdown%20was%2028.%22 https://www.popularmechanics.com/space/a4288/4318625/?utm_so...
• NanoYohaneTSU 3y ago
  It's crazy at how much better boomers had it. If this had happened in 2010 then he would have been fired after that.
• refactor_master 3y ago
  From reading the developer anecdotes in here I think it’s worth mentioning that if just one person can bring down the whole enterprise, a hacker only needs a point of entry to do the same. For our databases we have separate credentials, compartmentalized access and disallowed “dangerous” commands. This now seems like obviousness, but we only got this years in. Thankfully, no (major) incidents have occurred to this date.
• throwawayosiu1 3y ago
  I'll add my story here for posterity: My first job out of university, I was working for a content marketing startup who's tech stack involved PHP and PerconaDB (MySQL). I was relatively inexperienced with PHP but had the false confidence of a new grad (didn't get a job for 6 months after graduating - so I was desperate to impress). I was tasked with updating some feature flags that would turn on a new feature for all clients, except for those that explicitly wanted it off. These flags were stored in the database as integers (specifically values 4 and 5) in an array (as a string). I decided to use the PHP function (array_reverse)[https://www.php.net/manual/en/function.array-reverse.php https://www.php.net/manual/en/function.array-reverse.php] to achieve the necessary goal. However, what I didn't know (and didn't read up on the documentation) is that, without the 2nd argument, it only reversed the values not the keys. This corrupted the database with the exact opposite of what was needed (somehow this went through QA just fine). I found out about this hours later (used to commute ~3 hrs each way) and by that time, the senior leadership was involved (small startup). It was an easy fix - just a reverse script - but it highlighted many issues (QA, DB Backups not working etc.) I distinctly remember (and appreciate) that the lead architect came up to me the next day and told me that it was rite of passage of working with PHP - a mistake that he too had made early in his career. I ended up being fired (grew as an engineer and was better off for it) but in that moment and weeks after it, it definitely demoralized me.
  • notRobot 3y ago
    They fired you for that array_reverse mistake?
    • throwawayosiu1 3y ago
      Yep, I was put on PIP with impossible success criteria (no issues raised in PRs by senior engineers and no issues in code deployed to production - even if it was reviewed by senior engineers & QA) and fired (for failing that criteria) in 2 weeks. I worked there for ~8 months in total.
      • fullspectrumdev 3y ago
• cjbprime 3y ago
  > And I still remember the shock when Project Manager Pete delivered the decision and the follow-on news: ‘These tests will continue. And Chris will continue to lead them as we have paid for his education. He’s the last person on Earth who would make this mistake again.’ I wonder whether Pete had followed this 1989 general aviation/accident analysis story: > When he returned to the airfield Bob Hoover walked over to the man who had nearly caused his death and, according to the California Fullerton News-Tribune, said: "There isn’t a man alive who hasn’t made a mistake. But I’m positive you’ll never make this mistake again. That’s why I want to make sure that you’re the only one to refuel my plane tomorrow. I won’t let anyone else on the field touch it." -- https://www.squawkpoint.com/2014/01/criticism/ https://www.squawkpoint.com/2014/01/criticism/ (The incident above led to the creation and eventual mandated use of a new safety nozzle for refueling, which seems like a better long-term solution than having the people who've nearly killed you nearby to fuel your plane indefinitely: https://en.wikipedia.org/wiki/Bob_Hoover#Hoover_nozzle_and_Hoover_ring https://en.wikipedia.org/wiki/Bob_Hoover#Hoover_nozzle_and_H...)
• nagonago 3y ago
  Unrelated to the content of the article, but is the cover image AI generated? Looks kind of like DALL-E style output to me but it's hard to be sure. (Also, I know I'm breaking the rule "Please don't pick the most provocative thing in an article or post to complain about in the thread." My defense is this is less of a complaint and more just plain curiosity!)
  • criddell 3y ago
    Definitely is.
• cheese_van 3y ago
  I cost my company about 5 times my yearly salary once, long ago. I sampled an enormous amount of seismic data at 2ms instead of the proper 4ms. This was back when we rented IBM our mainframes for a pretty penny. The job ran for the entire weekend and Monday morning I was summoned by management, informed of my error, and asked, "You won't ever do that again will you?" and sent back to work. Knowing that you are allowed to fail, but are very much expected/required to learn from your failure, makes for rather a good employee, in my experience.
• fritzo 3y ago
  I used to work in a rural manufacturing plant. I once totaled my car on the way to town. Our shop foreman was also a volunteer firefighter and was the first one on the scene. I survived with a bloody nose and shattered confidence. The next day at work we needed some parts from town. I didn't have a car anymore, but that same shop foreman lent me his suped up pickup truck. I was totally confused. He wisely said: "Today is the safest day to let you drive my truck, cause I know you'll be extra careful"
  • erhaetherth 3y ago
    You'll be extra careful but I don't know about extra safe. I had a few close calls while driving and it made me a less confident driver which I don't think did anyone any favors.
• jeffrallen 3y ago
  Ernie is the hero in this story. As Fred Rogers said, "look for the helpers".
• drra 3y ago
  This really resonates with my experience. Working at a major airline, I was the one who would pick the most difficult and risky projects. One was a quick implementation of a new payment provider for their website. That website sold millions of euros worth of tickets every day. Seconds after deployment, it turned out that I had failed to recognize the differences between the test and live environments as one of the crucial variables was blank in production. I could have expected this if I had spent more time preparing and reading documentation. Sales died completely, and my heart sank. After a lengthy rollback procedure that resulted in a few hours without sales, a massive surge of angry customers, and a loss of several million euros, I approached the CEO of the company. I still remember catching him in an elevator. I explained that this incident was all my fault and I had failed to properly analyse the environment. I assured him that I was ready to bear full consequences, including being fired. He burst into laughter and said something like this: "Why would I want to get rid of you? You made a mistake that you'll never do again. You are better at your job than you were yesterday!" This experience was formative to me on many levels including true leadership. I successfully completed many high risk projects since than.
  • bboygravity 3y ago
    You worried about that? I'm a frequent flyer and I got a feeling that most airline ticket booking pages are broken in some way more than half the time. Maybe not often broken to the point that they're blank, but definitely broken to the point that booking a ticket isn't possible (I prefer blank, so that I don't waste like 30 minutes on not being able to book a ticket). Also most of the internet seems often broken. Oh hello Nike webshop errors upon payment (on Black Friday) for which helpdesk's solution is: just use the App.
    • thfuran 3y ago
      What always got me is that, for at least the first several years, Google couldn't get their store page to handle the load when they were releasing a new phone and it'd be crapping out for days.
      • 3y ago
• smsm42 3y ago
  I am a bit disappointed that the name of the author is not Howard Wolowitz.
• blauditore 3y ago
  The fact this can so easily happen shows a lack of safety mechanisms. All the software-related stories in this comment section go in a similar direction: They could have been prevented by simple safety nets. If you accidentally wipe a production database, then it was likely too easy to do so. Don't blame humans for occasional mistakes, it won't stop them from happening.
  • danw1979 3y ago
    The idea that one can learn to not make errors is toxic. To err is human. Sure, you can get more reliable at something, but everyone - even the most experienced - will fry the rover at some point.
• bambax 3y ago
  Great write-up. My two favorite quotes: > I had learned from countless experiences in this and other projects that bad news doesn’t get better with age That's so true! We tend to sit on bad news and hope that somehow time will blunt them; but if anything the opposite happens. And > I still remember the shock when Project Manager Pete delivered the decision and the follow-on news: ‘These tests will continue. And Chris [the author] will continue to lead them as we have paid for his education. He’s the last person on Earth who would make this mistake again.’ We sometimes think people who made one mistake will make another one, and it's better to go with the person who doesn't make mistakes. But that's not the correct approach. People who don't make mistakes are often people who don't do anything.
• anjel 3y ago
  An even worse engineering horror story: https://faroutmagazine.co.uk/the-steely-dan-song-that-was-almost-erased-from-history/ https://faroutmagazine.co.uk/the-steely-dan-song-that-was-al... (more sad trombone and far less "relieved engineer, happy ending.")
  • paulluuk 3y ago
    That's an interesting read! Not really sure whether this counts as "engineering" though (accidentally taping over a track), nor would I consider this "worse" than potentially destroying 500M worth of advanced equipment.
• onetimeuse92304 3y ago
  Decades ago I worked as a leader of a small team of sysadmins. We worked around the clock maintaining server room and critical applications for an average size bank. One of our responsibilities was to execute nightly checklists to run various processes, do backups at correct times, etc. These processes would be things like running calculations on loans, verifying results, etc. We had a huge ream of checklists to accomplish this and we were supposed to follow them religiously. We had two very similar applications, one our core and another a core from another bank we bought with the same application but older version and slightly different config. Consequently, we had two tracks of checklists with very similar steps. One of those steps was to change the accounting date in the system. The application was a terminal app. We would telnet to the server, log in, then we would execute commands in the menu driven app. To change the date we would have to go to a special menu for super dangerous applications. It required the user to log in again. Our core system, required logging in, selecting that we want to advance accounting date by one day, entering admin password again, pressing enter, then waiting for about 4 hours while the process ran. Then the process would exit back to the menu where the highlighted option would be to advance the day. Our legacy system required logging in, selecting the option to advance accounting date, entering admin password, pressing enter, then waiting for two hours after which a popup showed to ask a stupid question where we would just always press enter, then wait for another hour until it exits to the menu. We quickly figured out, that we can just press the enter key twice on our legacy system. The second enter press would just leave there in keyboard buffer and dismiss the popup. This was very useful for us, as this was the only operation that interrupted what would be the only time during night where we could have a kebab... One night I made a mistake, and I pressed the enter twice... on the wrong system. When I figured out that I did it I realised the process would exit to the menu and then should ask for the admin password. But, unfortunately, the application had a bug (or a feature). Once it exited to the menu, it came back in but for some reason it remembered that the admin password has already been entered and started advancing the accounting date again without asking for the password. Unfortunately, the date was December 24. For entire December 24, the entire bank was unable to process any operations while we were restoring from last good backup (before day close) and then redoing eod operations. Then on December 25 as a penalty, I had to sit for entire day with accounting department observing how they manually entered all of the operations that would normally happen automatically on Dec 24th. One extra key pressed.
• electrical_blu 3y ago
  It's a nice story with nice message. And it's pretty normal that mistakes happen, especially under pressure and within long shifts. The "mistake" itself is understandable: what is shocking to me it the multimeter thing. Learning that a mythical "NASA guy" in charge of really serious stuff cannot realize that the multimeter is measuring current, thus it's part of the circuit, and removing it will switch off something.
  • deleted 3y ago
    [deleted]
• fransje26 3y ago
  > I was into my unofficial second shift having already logged 12 hours that Wednesday. Long workdays are a nominal scenario for the assembly and test phase. Although the time pressure coming with the upcoming deadline is understandable, perhaps the bigger lessons here is that when you are possibly sleep-deprived, and have already pulled too long a shift, you are bound to make avoidable mistakes. And that is the last thing you want on a $500M mission with at limited flight window.
  • HenriTEL 3y ago
    Yep. Beyond the technical issue this story shows a people management issue.
• nojvek 3y ago
  I’m glad they didn’t fire him. I’m a firm believer that despite all the short comings of US, what makes it great is there are millions of engineers and scientists working to push the frontier of what is possible, and trillions of dollars in economy to fund that into reality. NASA is truly an inspiration. And also the private aerospace industry - SpaceX, ULA, Boeing, Lockheed Martin, Blue Origin, Planet Labs etc. No other country has that.
• belter 3y ago
  I thought this was about this story. Probably more interesting. "The First Bug on Mars: OS Scheduling, Priority Inversion, and the Mars Pathfinder" - https://kwahome.medium.com/the-first-bug-on-mars-os-scheduling-priority-inversion-and-the-mars-pathfinder-53586a631525 https://kwahome.medium.com/the-first-bug-on-mars-os-scheduli...
• squirrel23 3y ago
  He’s the last person on Earth who would make this mistake again. That hit HAAARD
• ge96 3y ago
  I worked in a video startup thing and watching the app fail after people sunk like 30 mins into using it... Oh man that was cringe. The videos were recorded because they were legal/financial related and also autotranscripted for text search. I would watch them to trace how the problem occured (I know there are many ways to log). I was the only developer so it was my fault ha. The app had so many parts and I just used an E2E test to make sure everything generally worked. It's cool Chrome has a fake video feed (spinning green circle/beeping).
• user3939382 3y ago
  When I saw the headline I thought it may have been a write up from this: https://www.latimes.com/archives/la-xpm-1999-oct-01-mn-17288-story.html https://www.latimes.com/archives/la-xpm-1999-oct-01-mn-17288... > NASA lost its $125-million Mars Climate Orbiter because spacecraft engineers failed to convert from English to metric measurements
• HeavyStorm 3y ago
  It's a interesting story, but the author may be overselling it. It's not a _failure_ story, nor was it a 500M mistake. I get that it was really stressful and the mistake could have cost him the job, but it didn't; it also didn't cost NASA anything other than a few hours of work (which, during testing, I would guess it's expected). When I'm asked to share failures, I'm usually not thinking about "that one time when I almost screwed up but everything was fine", instead, I'm thinking of when I actually did damage to the business and had to fix it somehow.
  • felideon 3y ago
    “My quasi-$500M Mars rover mistake” just doesn’t have quite the same ring to it. But, to your point, it is still a failure story that _could_ have lead to a much worse outcome than it did. The fact that it didn’t was mostly due to luck.
• atemerev 3y ago
  Ctrl-F, "The Martian" "Phrase not found" But... how? This is exactly the same story that happened in the book. Since Spirit is older is the book, perhaps this was the real-life inspiration.
• mosburger 3y ago
  Hey neat - I once destroyed a $5000 prototype disk drive (I was a Firmware engineer at Quantum in the late 1990s at my first post-college job), also with back-EMF (it flowed from the braking spindle motor into a power chip and melted the circuit board). I felt terrible and that was multiple orders of magnitude cheaper. :)
• JoeAltmaier 3y ago
  My friend reports his phone started dropping contacts methodically, until only one remained: a person he didn't know. Googling him, found it was a junior dev at a FAANG. Oops. An hour later the contacts repopulated, and all was well. But had to be a white-knuckle time for that poor shmuck.
• knallfrosch 3y ago
  That one time I deleted the files on the CEO's laptop and did the backup afterwards - wrong order. Oops. Never heard anything about it though, so he didn't fully entrust all his data to the intern. Wise decision.
• i-use-nixos-btw 3y ago
  I agree that the person who made such a mistake will be the person who never makes that mistake again. That's why firing someone who has slipped up (in a technical way) and is clearly mortified is typically a bad move. However, I don't agree that this is the "real" lesson. Given the costs at play and the risk presented, the lesson is that if you have components that are tested with a big surge of power, give them custom test connectors that are incompatible with components that are liable to go up in smoke. That's the lesson. This isn't a little breadboard project they're dealing with, it's a vast project built by countless people in a government agency that has a reputation for formal procedures that are the source of great time, expense, and in some cases ridicule. The "trust the 28 year old with the $500m robot that can go boom if they slip up" logic seems very peculiar.
  • crazygringo 3y ago
    That was my first thought as well. On the other hand, it's hard to make these kinds of judgment calls when you're talking about a one-off piece of equipment that's only going to go through this particular testing cycle a single time. In computing, there are a lot of similar "one-off" operations -- something you to do to the prod database or router config a single time as part of an upgrade or migration. Sometimes building a safeguard is more effort than just paying attention in the first place. And while we don't always perfectly pay attention, we also don't always perfectly build safeguards, and wind up making similar mistakes because we're trusting the faulty safeguard. In circumstances like the one in the story, the best approach might almost be the hardware equivalent of pair programming -- the author should have had a partner solely responsible for verifying everything he did was correct. (Not just an assistant like Mary who's helping, where they're splitting responsibilities -- no, somebody whose sole job is to follow along and verify.)
    • Retric 3y ago
      “One off” is never just a one off it’s always part of a class of activity such as server migrations etc. Just paying attention guarantees eventual failure when repeated enough times. This may be acceptable, but it comes down to managing risks. If failure means the company dies then taking a 1 in 10,000 risk to save 3 hours of work probably isn’t worth it. If failure means an extra 100 ours of work and 10k in lost revenue then sure take that 1 in 10,000 risk it’s a reasonable trade off.
• dimal 3y ago
  > failure is not an option — it comes pre-installed Love this. Whenever people say “failure is not an option”, I get the sense that they don’t really understand how the universe works. It’s like saying “entropy is not an option”. Uh...
• lbriner 3y ago
  There is a principle here that I can't see mentioned and that is how easy it is to discard something as a cause of a problem because it seems so minor, so routine (i.e. removing the "spare" multimeter) and therefore getting blinkered on what is going on. There are more than a few times where I am scratching my head as to "how could my change have possibly broken this" only to remember a couple of hours later that I had made another change somewhere, or rebooted, or changed a config file temporarily. I guess it just says that we all need to log everything we do, including removing spare multi-meters, so that by looking over the list we can remember these things.
• slicktux 3y ago
  Trust but verify!!
• flyingbarron 3y ago
  Fantastic story! One of the fascinating things about this are the parallels to Apollo 12 - Lightning strike / Power surge lead to a total loss of telemetry which raised the specter of losing the entire mission - and a similar resolution when telemetry was restored! I've written a bit about it myself - https://flyingbarron.medium.com/lightning-strikes-92482387cab8 https://flyingbarron.medium.com/lightning-strikes-92482387ca...
• Fourier864 3y ago
  When I was 20 and in college, I used to work at a Barnes and Noble. When I was working the registers one day, I apparently forgot to put a $300 gift card in a lady's bag at checkout. The store manager got a complaint about it like a month later, and she tracked down the gift card number from the receipt. It wound up getting loaded with even more money a few days later, and given to someone else. Anyway, the whole incident got me fired. And to this day, I always check that I put gift cards into bags (on the rare occasion it comes up in my job as a software engineer for NASA missions).
• mohas 3y ago
  So let me share mine, it goes like hey can you correct this issue directly on the database of 5 years of corporate financial data? and in midst of testing and looking for the issue I wrote a delete with incomplete where clause that would delete lots of data, fortunately it was stopped because of a constraint violation, I still remember the adrenaline rush when I was processing what just happened
• p_plno 3y ago
  I'm really at a loss of words. There is only one lesson from this incident: nobody is supposed to touch a 500M piece of equipment after 12 hours of work. Period. The author is not getting it, and nobody in the comments. The world is run by idiots and it shows.
  • HenriTEL 3y ago
    It seems like there was a lot of pressure to meet the deadline and everybody overworked in this environment. A perfect example of normalization of deviance. It's a shame that there still exists many workplaces like that. It's our duty to be aware of what's happening around us and tell the guy - Hey, what the fuck are you doing here? Get home, that's no critical emergency.
• niepiekm 3y ago
  Thanks for sharing your experience. It reminded me of another one I'd read a while back by Andrew Latham: https://www.linkedin.com/pulse/what-i-learned-from-my-biggest-mistake-f1-engineer-andrew-latham/ https://www.linkedin.com/pulse/what-i-learned-from-my-bigges...