7 ms·

Vulnerable transistors threaten to upend Europa Clipper mission

177 points by cratermoon 2y ago

gomoboo 2y ago
They found out about the failing transistors via colleagues at a conference. Have any of you learned of something of this magnitude in the same way? It got me thinking that I need to interact with my fellow devs more often.
- ChrisMarshallNY 2y ago
  This is the "status meeting paradox." Personally, I despise status meetings. 99% of them are worthless fluff. But, every now and then, you get something like this. I think that highly usable dashboards are a good way to deal with this. It's possible that AI could be a big help, here.
  - brookst 2y ago
    Also the hallway conversation thing. Most of the time it’s small talk and minor social interaction, every now and then it’s critical out of band information that would not have shown up in normal processes.
    - causal 2y ago
      Perusing Slack has become this for me
      - ghaff 2y ago
        Probably inevitable these days given hallway conversations are going to be a pretty random thing. Of course, assumes someone needs to think something is important enough to put in chat and doesn't mind putting it out in public. (Ignore $XYZ project that other group is doing. It's got all sort of problems.)
        brookst 2y ago
        Yeah, the ephemeral, no-record nature of verbal hallway conversations makes them more candid.
    - usernamed7 2y ago
      To me it's a matter of fostering serendipity. and a bit ironic that research has shown conferences to be a great place for serendipity to take place, as that's what happened here. I experienced this kind of situation, where only by chance conversation was a crisis averted, very much at my last FT. So much that I'm working on a startup for fostering serendipitous communication for remote teams, like private notes from coworkers left on stackoverflow questions (or anything on the web)
  - marcosdumay 2y ago
    Your highly usable dashboard will get filled with 99% of worthless fluff just because it's there and somebody feels the need to always say something. Have you even been in one of those meetings that just won't finish despite everything being done? Making it written doesn't solve the problem. Instead, it makes it worse.
    - ChrisMarshallNY 2y ago
      Good point. AI it is, then...
    - dylan604 2y ago
      That worthless fluff makes the dashboard look impressive when you put it up on large NOC type screens when you walk people through on tours though
    - lanstin 2y ago
      "We are at the point where everything that could be said has been said, but not everyone has said it."
    - carlmr 2y ago
      On point. This is also why good CI/CD automatically alerts users of major issues. It's just not a thing humans are good at to pay attention to a long stream of mostly boring information. Computers are good at this though. Now the only question is how you can automate the spec comparison such that issues with the spec and the parts used can be automatically compared. And that starts with a computer readable spec that is updated by the manufacturer.
- jetrink 2y ago
  I can't believe the manufacturer didn't alert them and they had to hear it from another customer. Surely the manufacturer wouldn't want to be named as the reason that a spacecraft orbiting Jupiter went dark due to their faulty components.
  - Sanzig 2y ago
    The article mentions that the defense sector discovered the issue. Rad hard defense electronics have more stringent TID (total ionizing dose) requirement than space, due to a need to survive in nuclear war scenarios. Space usually caps out at 100 krad, with some very stringent environments needing up to 300 krad. Defense can go all the way up to 1 MRad in some cases. My guess is the parts failed TID at the more stringent levels, and Infineon didn't follow up with NASA or their contractor because they assumed that NASA was okay with the lower rad tolerance levels typical of space. Usually that would be the case, but Europa Clipper is special because it's going to an extremely harsh radiation environment. The big question for me is: did the Europa Clipper program order a lower TID and try to upscreen, or did they order the high TID part? If it's the former, it's on NASA. If it's the latter, that's extremely concerning because Infineon should know that nobody orders expensive high TID parts for funsies, and they should have followed up with all customers as soon as they confirmed there was an issue. Just assuming NASA over-specified a part is absurd. The rad hard electronics market is small, everyone knows each other. Trust is king. Finally, I'm not sure if it's the part in question, but it looks like Infineon discontinued their 1 MRad MOSFETs in 2020, citing low order volumes: https://irf.com/product-info/hi-rel/alerts/fv5-d-21-0004.pdf https://irf.com/product-info/hi-rel/alerts/fv5-d-21-0004.pdf. In the light of this reporting, I have to wonder if there was more to it than that?
    - JumpCrisscross 2y ago
      > and Infineon didn't follow up with NASA or their contractor because they assumed that NASA was okay with the lower rad tolerance levels typical of space It's more likely that Infineon's folks talking to NASA were equally clueless about this change.
      - 2y ago
- Wololooo 2y ago
  This is much more common than one would think. This is a classic thing with Industry, they qualify a process that is working and shows good performance, but this process needs to be changed for reason XYZ, often because it is maybe a bit too expensive or doesn't align with the rest of their processes. The small change in the process wasn't that small and takes a little while to be identified because by the time you catch it you might be further down the line and this would be caught by a QA process and not a QC process, that might have deemed at that point not necessary because you had no reason to fault the part. The second part is that some things are rated and verified but not tested extensively, since you might have prototype you might misdiagnose a failure of a component for a behaviour of your prototype, when in fact you had a deeper problem, but timelines with the added fact that so far you didn't think about that problem because it shouldn't have been a problem can catch you really off guard. This is usually where people testing the same thing in an exotic environment can ring alarm bells for others and that often happens at conferences... People often under estimate how much you can get bitten in the back by such little details that become huge details. Depending on the electronics and where the MOSFETS are, I would be them I would probably trash the electronics, take the spare that they had, validate components that get in and rebuild a control box and re-integrate it, provided that this is doable. It's expensive but provided that you have no choice that gives you a backup system that you can test code on before pushing it on the actual probe and might help for problem solving by being able to do measurements and test on the actual setup... Provided that they have the time and resources. Otherwise I wouldn't YOLO it given the fact that it might just straight up not work at the moment you need it the most and a little delay is better than nothing and they can spend the time re-checking part of the design that might also be weaker... But heh, who am I but a random guy on the internet...
  - interroboink 2y ago
    Another example along these lines: scurvy! [1] They found it was cured by lemon juice, but they didn't understand the details. Over years, they switched to lime juice (less vitamin C), put it in copper pipes (leaches vitamin C). But ships were faster so there was more fresh food available, masking the problem. Then scurvy starts mysteriously popping up again 100 years after it was first "cured." Hard to keep track of the effects of all the details in the face of various co-dependent things changing simultaneously. Recipe for surprises. [1] https://www.bluesci.co.uk/posts/forgotten-knowledge https://www.bluesci.co.uk/posts/forgotten-knowledge
- Kenji 2y ago
  Yes, all the time. This is normal. Big news of problems travel through back channels. Nobody is gonna announce their big fuckup for the world to see, unless compelled by law, and even then most won't do it. We had to sign NDAs to find out about severe Intel processor silicon bugs. Obviously you're not gonna read that info on Hackernews afterwards.
- kibwen 2y ago
  This is what forums like this one are for. Ordinary news isn't going to have more than a passing mention of the xz hack, or log4j, or meltdown, or heartbleed. Find (or start) a private group chat for technologists you know to share news like this.
- ISL 2y ago
  Yep. Chatting with other practitioners is a powerful way to learn how things actually work. There are tons of things that "everyone" knows that are not well documented, and therefore unavailable to people outside the network. This is a more-consequential example of the things you can learn by chatting with others; it is an extreme example of, "Hey, are you guys using components from Widget Inc.? Their datasheets are good, but sometimes we get a bad batch." Those little things can save you a ton of time. In this case, it may have prevented mission-failure. Part of the blame falls to NASA, too. If the outcome is your responsibility, then open-loop trust of a vendor for a known failure-mode may not be acceptable. Integration rad-hard testing may be requisite. In the spacecraft environment, qualifying components is very difficult -- there's a good chance that NASA has these MOSFETs on an approved list because they've worked well before and have had few (or known) faults. They're probably not on that list anymore.
- mihaaly 2y ago
  Or maybe those making a living on selling a product claiming certain parameters should raise their voice when those parameters are not met, regardless if that product is used for space travel or turn on and off a ketle for f's sake.
ChrisMarshallNY 2y ago
Well, I'm glad they didn't find out, by the board crapping out, around Jupiter.
- sva_ 2y ago
  > The transistors cannot simply be replaced. Clipper’s aluminum-zinc electronics vault, meant to provide a measure of radiation resistance, was sealed in October 2023. Barring an indication that the faulty MOSFETs will cause catastrophic failure, the agency will likely seek to continue with the launch—although backup windows are available the next 2 years.
  - ChrisMarshallNY 2y ago
    Yeah, but if they figure out that the transistors are problematic, at least, they can do something about it, even if it does delay the launch. They may also use this as a spur to wargame failure mitigation strategies, so they'll be ready, if they do go belly-up.
- DannyBee 2y ago
  Right. Like for all the issues here, it's still better to find this out now.
- m4rtink 2y ago
  This happened already at least once with a couple Soviet Mars probes - they found a batch of transistors they used were faulty and deteriorate much too quickly in space - after the probes vere already launched. IIRC some did not even reach Mars while others failed soon after orbit insertion due to various sub-systems failing.
albumen 2y ago
>[Infineon] has already corrected the mistake, but Infineon did not report the flaw to NASA because the company did not know what the transistors would be used for, Fitzpatrick said. “They did not realize it was going to affect us.” Infineon did not respond to a request for comment. Not exactly responsible disclosure! NASA buys rad-hard transistors, and Infineon "didn't know what they'd be used for"?
- nine_k 2y ago
  I bet NASA buys rad-hardened electronics by a truckload, and buys from distributors, not Infineon directly. But it's a reasonable idea to notify all potential large consumers that are likely to have bought your specialty product; these are not numerous, and the impact may be large (as in this case).
  - bangaladore 2y ago
    That's certainly possible, but the distributer should have notified NASA. Maybe that email is sitting in someone's inbox.
    - DannyBee 2y ago
      Distributors will notify you of recalls, but no distributor of electronics i've worked with notifies you of erratum (and it would be really annoying if they did, honestly :P)
      - bangaladore 2y ago
        This reads like it should have been a recall. But that's hard to tell unless I knew exactly the specifics of the issue. Did NASA assume these were rated higher then they were? Did Infineon make a mistake in documentation, or did they straight up not test them or test them incorrectly.
        chmod775 2y ago
hoseja 2y ago
I am eagerly awaiting probe manufacturers learning anything at all from Ingenuity.
- bloopernova 2y ago
  Are you referring to off the shelf components used for the helicopter? https://en.wikipedia.org/wiki/Ingenuity_%28helicopter%29?wprov=sfla1 https://en.wikipedia.org/wiki/Ingenuity_%28helicopter%29?wpr... (in case anyone wanted to refresh their memory)
- sdmike1 2y ago
  The Ingenuity is a really interesting project with some important lessons for spaceflight with unhardened CotS parts. However, I would argue that it and the Europa Clipper are two very different designs for two very different environments. While mars is an elevated radiation environment when compared with earth, the Jovian radiation belts are on a whole other level, particles up to 1-2000 MeV are fairly common. To put that into context, a medical radiation beam therapy deals with 2-300 MeV on the absolute highest end. To get into the 1-2000 MeV range you generally are talking about energies found in the low end of particle accelerators. Ingenuity mostly had to worry about Total Lifetime Dose (TLD), one example of a TLD issue is dopant migration induced by high-energy heavy ion collisions which can change the on voltage of a transistor. At high energies you can have single events with enough energy to cause fatal latch-ups. For instance modern rad-hard FPGAs start encountering major issues around 60-70 MeV. Furthermore, these parts are power MOSFETs which control power for whole subsystems so their reliability is critical to the operation of the spacecraft. In addition, the biggest issue here is not just that there were issues that were addressed and fixed, it's that Infineon didn't issue an errata to the datasheet or inform NASA of the issue. As a result there are now transistors littered throughout the spacecraft which don't meet the radiation needs. This is going to require reworking the boards, re-validation of the subsystem, and re-integration of the subsystem into the spacecraft. This all comes at a non-trivial impact to budget and timelines which is to say nothing about what this does to the launch window the project was trying to hit for gravity assist / proximity. I hope you find this informative! :) EDITS: Spelling and an "is"
twh270 2y ago
From the article: "Infineon did not report the flaw to NASA because the company did not know what the transistors would be used for, Fitzpatrick said." They might not have "known", but come on, you're selling radiation-hardened chips to NASA. You can sure make an educated guess that they might be used for a probe. I'm guessing there's a clause missing in the contract that says Infineon must disclose all known problems to NASA regardless of how the chips will be used. Regardless, there are some people at NASA to whom 'Infineon' is now a curse word.
- laurencei 2y ago
  "They might not have "known", but come on, you're selling radiation-hardened chips to NASA. " But do people ever actually "invoice NASA" for components. It was probably one of 100 different sub contractors building the actual circuits to NASA specifications, i.e. it was lower in the chain rather than NASA itself. (Doesnt excuse the non-disclosure to those subcontractors)
  - 0xffff2 2y ago
    >But do people ever actually "invoice NASA" for components Yes, absolutely they do. I'm not a part of this mission, but I'm currently working on another NASA spacecraft mission. I don't know the percentages off hand, but a substantial portion of our spacecraft is built in house with parts purchased directly by NASA from the manufacturer. Regardless, there are lines of communication to subcontractors. The mere fact that they found out about this at a conference is significant evidence that Infineon didn't notify who they should have.
    - deleted 2y ago
      [deleted]
    - hvs 2y ago
      Off-topic, but when components are sourced directly from the manufacturer do you have to buy in bulk? I figured you didn't just go on Mouser or DigiKey, but I would think manufacturers don't like dealing in small amounts.
indoordin0saur 2y ago
Is more shielding not the obvious answer? A thin sheet of lead around the sensitive parts should do the trick. (Note: I'm not a physicist and have no idea what I'm talking about in this domain)
- klodolph 2y ago
  1. You’d need more than a thin sheet of lead. The radiation in space can be very energetic. It can easily penetrate several cm of shielding and if it is absorbed, you get secondary radiation. 2. Even a thin sheet of lead may be too heavy.
- JumpCrisscross 2y ago
  > Is more shielding not the obvious answer? A thin sheet of lead around the sensitive parts should do the trick Lead "is effective at stopping gamma rays and x-rays" [1]. Jupiter's radiation comes from "trapped particles [that] are about ten times more energetic than the ones from the equivalent radiation belts of Earth" and "several orders of magnitude more abundant" [2]. When those encounter lead they cause bremsstrahlung radiation [3], a sort of subatomic shrapnel that can be more dangerous than the original radiation. Lead is also heavy, which means not only increasing the mass of the spacecraft, but its balance and thus propulsion profile. That might mean upgrading and moving thrusters and propellant tanks--in effect, a complete redesign. (It's a good question that doesn't deserve to be downvoted.) [1] https://en.wikipedia.org/wiki/Lead_shielding https://en.wikipedia.org/wiki/Lead_shielding [2] https://www.spenvis.oma.be/help/background/planetary/traprad_jup.html#TRPP https://www.spenvis.oma.be/help/background/planetary/traprad... [3] https://en.wikipedia.org/wiki/Bremsstrahlung https://en.wikipedia.org/wiki/Bremsstrahlung
  - basementcat 2y ago
    Europa Clipper electronics are contained in a 9.2 mm thick aluminum-zinc vault. https://europa.nasa.gov/resources/342/electronics-vault/ https://europa.nasa.gov/resources/342/electronics-vault/
    - 2y ago
gene-h 2y ago
Europa Clipper also used a new approach for designing spacecraft. It's NASA's first major spacecraft designed with Model Based Systems Engineering(MBSE)[0]. Using diagrams in SysML to keep track of power use and interfaces is supposedly better than using spreadsheets [0]https://ses.gsfc.nasa.gov/ses_data_2021/210728_Bayer.pdf https://ses.gsfc.nasa.gov/ses_data_2021/210728_Bayer.pdf
- PaulGaspardo 2y ago
  Oh, I used to work on this :) For keeping track of power use and interfaces specifically it turns out doing it all with SysML diagrams wasn't so great. Aside from all the pointless futzing around with boxes and arrows the model eventually became so huge the authoring software could barely handle just opening it up. So it must have been shortly after these slides when all the power use tracking was shifted to a custom tool with a more tabular user interface that we were already using for tracking electrical interfaces (slide 15) with version control in git.
  - crocal 2y ago
    Unrelated question: how did you manage tabular data in git? It’s always a struggle to diff and merge changes.
    - baq 2y ago
      not OP but the usual applies - data is not actually stored as a table in git, tables are an UI thing. git would store standard issue json, xml or whatever custom git-friendly format is used by the tool.
    - PaulGaspardo 2y ago
      Yeah, like 'baq said the data wasn't stored in a tabular form, it was actually XML. So sometimes you could just look at the textual diff and it would make perfect sense, although it wasn't expected users would work with XML at the source level. There was also a semantic object-level diff we got for "free" by virtue of building on top of the Eclipse Modeling Framework. It was integrated into the Eclipse git UI and could help resolve merge conflicts without having to touch the XML directly, but merge conflicts were still annoying to deal with so generally engineers coordinated with each other to not touch the same part of the model at the same time. Normally for review though I think users tended to compare reports generated from the model rather than trying to diff the source model files directly. There was a sort of automated build process that took care of that once you pushed your branch to Github.
metiscus 2y ago
Sounds like Infineon may owe someone a new satellite soon. At least if it can be shown that they sent NASA bad parts and didn't notify them in time to prevent this failure.
rkagerer 2y ago
Id like to know how this turns out - ie. hiw they decide to mitigate. Where's the easiest / best place to check in a couple months for a followup?
chrisjj 2y ago
> Infineon did not report the flaw to NASA because the company did not know what the transistors would be used for, Either the parts were in spec or they weren't. Which is it?
- jjk166 2y ago
  That's not how specs work. When the requirements for a part are specified, it is based on assumptions that may or may not hold true. For example, if an issue tends to be all or nothing, then testing a small percentage of a lot should reasonably be expected to catch an issue. So you might specify that 1% of these transistors be tested and so long as that 1% passes the rest are considered good. If let's say there's a process change and lots become more variable, the confidence with which you can say the others are good based on that 1% testing goes down, but you are still testing to the same standard that you were before, which is what the specification calls for. The issue gets even more thorny when issues are conditional. For example a part might meet the voltage specification, the temperature specification, and the radiation specification individually, but when you put that same part simultaneously in a high voltage, low temperature, and high radiation environment it doesn't perform as well. Or perhaps one component used downstream of a particular other component has an effect. Perhaps the most basic example is oversized but in tolerance shaft meets undersized but in tolerance hole.
  - DoctorOetker 2y ago
    >For example a part might meet the voltage specification, the temperature specification, and the radiation specification individually, but when you put that same part simultaneously in a high voltage, low temperature, and high radiation environment it doesn't perform as well. I am not disagreeing, but at some point humanity should switch to actual probabilistic device models instead of vague datasheets. Imagine every datasheet has a .sample() function and you get a randomized SPICE model as if it came from the manufacturing line, you can draw a 100 and plot the properties. Want to measure the dynamic range of some ADC? A-weighted or not? Instead of specifying values highlighting figures of merit, each such figure of merit corresponds with an explicit SPICE circuit that measures that figure of merit on one and the same generator for random SPICE models with that specific part designation. If a brand tries to fool its customers by insinuating a desirably high or low value by changing the test method, its immediately clear. A user may specify his own test circuit, or reuse the test circuit from the interactive datasheet from a competitor etc in order to make apples to apples comparisons for different DUT's.
elzbardico 2y ago
1. Sell bunch of radiation-hardened parts to NASA. 2. Find out the parts you sold to NASA don't meet the specs. 3. Don't tell NASA, because NASA didn't tell you what those parts would be used for. This is criminally incompetent on the part of Infineon. WTF, NASA could use those transistors for a fancy inteliggent toilet FWIW, it doesn't matter, NASA doesn't have to tell you how they are going to use those parts. They bought parts based on a fucking SPECIFICATION, and if the parts you sold them don't meet the specs, you communicate immediatelly with the customer offering a replacement for free. Really, someone should be jailed for that.
- hvs 2y ago
  Jailing people for mistakes is a good way to ensure that mistakes are kept hidden.
  - elzbardico 2y ago
    There are mistakes, and there is negligence.
jjk166 2y ago
> Infineon did not report the flaw to NASA because the company did not know what the transistors would be used for, People are reading this as Infineon didn't know that the parts were going into a probe when it's far more likely they meant they didn't know how the transistors are being used in that probe, which might have a large effect on whether or not the problem will affect them.
fergbrain 2y ago
Did Infineon not issue a GIDEP Notice for these parts? If not, does that mean that maybe NASA is using them outside of their designed spec? (See also: https://nodis3.gsfc.nasa.gov/displayDir.cfm?Internal_ID=N_PR_8735_001D_&page_name=Chapter3 https://nodis3.gsfc.nasa.gov/displayDir.cfm?Internal_ID=N_PR...)
brcmthrowaway 2y ago
Germans and space industry, what could go wrong
- Space_rat66 2y ago
  the Mosfets are actually from IR HiRel which is an American legacy space company
fsckboy 2y ago
DAE when i read a sentence like this > Engineers at NASA’s Jet Propulsion Laboratory (JPL), which leads the development of Clipper, discovered the problem in May after talking with colleagues about a classified satellite at a conference. ...I immediately think "that's what they want us to think" if classification of projects means anything, the true intent of the passing of dis information is something dat we can only guess at. fnord.
omneity 2y ago
So can we say the transistors of Clipper are ... clipping? More seriously, I'd be interested to learn what failed in the quality assurance process, as NASA & its suppliers have a legendary reputation in these topics. RCA will be enlightening.
rsynnott 2y ago
My favourite bit about this project: > In June 2022, project scientist Robert Pappalardo revealed that mission planners for Europa Clipper were considering disposing of the probe by crashing it into the surface of Ganymede for Europan protection purposes, in case an extended mission was not approved early. He noted that an impact would help the ESA's JUICE mission collect more information about Ganymede's surface chemistry What about Ganymedian protection, eh? The Ganymedians, should they exist, are going to be furious. Also it is weird that the outcome of the EJSM divorce (originally there was going to be a joint NASA-ESA mission to the Jupiter icy moons; Europa Clipper and JUICE are a result of the breakup), is that America explores Europa and Europe explores Ganymede, as the other way around would be less confusing.
- amelius 2y ago
  If the Ganymedians got angry and attacked us, that would be a great scientific advancement.
LeifCarrotson 2y ago
> The transistors cannot simply be replaced. Clipper’s aluminum-zinc electronics vault, meant to provide a measure of radiation resistance, was sealed in October 2023. Surely they can be replaced! Humans (in cleanrooms) can put their hands inches away from them. Here's a picture, admittedly from 2022, before it was sealed: https://europa.nasa.gov/resources/342/electronics-vault/ https://europa.nasa.gov/resources/342/electronics-vault/ It's not the same as saying they've failed and are in orbit around Jupiter and so can't be replaced. They simply need to open and re-seal the vault or build and seal a new electronics vault before October.
- nebalee 2y ago
  Here's a photo of the vault as it was about to be sealed in October last year: https://www.nasa.gov/missions/europa-clipper/how-nasa-is-protecting-europa-clipper-from-space-radiation/ https://www.nasa.gov/missions/europa-clipper/how-nasa-is-pro... In the meantime NASA built a whole spacecraft around it and the craft went through numerous tests. If they'd have to undo and redo all of that, they may even miss the 2025 backup launch date. I'm sure NASA is very eager to seek other ways to deal with this problem.
  - rob74 2y ago
    Ok, that sounds a bit more convincing... I'm not the OP, but I was just trying to imagine why a "sealed vault" would work better against cosmic radiation than a more prosaic screwed shut container.
  - deleted 2y ago
    [deleted]
  - fbdab103 2y ago
    On one hand, it must be baller to work on assembling spacecraft. On the other, wearing those gowns all day must be terrible.
- 2o3jr2o34j 2y ago
  Do you understand that testing and certifying everything takes months and is a significant portion of the cost of the mission? And plenty of these components and closures were designed for single use, so they would have to be replaced. This is not a desktop PC.

177 points by cratermoon 2y ago

They found out about the failing transistors via colleagues at a conference. Have any of you learned of something of this magnitude in the same way? It got me thinking that I need to interact with my fellow devs more often.

ChrisMarshallNY 2y ago
This is the "status meeting paradox." Personally, I despise status meetings. 99% of them are worthless fluff. But, every now and then, you get something like this. I think that highly usable dashboards are a good way to deal with this. It's possible that AI could be a big help, here.
- brookst 2y ago
  Also the hallway conversation thing. Most of the time it’s small talk and minor social interaction, every now and then it’s critical out of band information that would not have shown up in normal processes.
  - causal 2y ago
    Perusing Slack has become this for me
    - ghaff 2y ago
      Probably inevitable these days given hallway conversations are going to be a pretty random thing. Of course, assumes someone needs to think something is important enough to put in chat and doesn't mind putting it out in public. (Ignore $XYZ project that other group is doing. It's got all sort of problems.)
      - brookst 2y ago
        Yeah, the ephemeral, no-record nature of verbal hallway conversations makes them more candid.
  - usernamed7 2y ago
    To me it's a matter of fostering serendipity. and a bit ironic that research has shown conferences to be a great place for serendipity to take place, as that's what happened here. I experienced this kind of situation, where only by chance conversation was a crisis averted, very much at my last FT. So much that I'm working on a startup for fostering serendipitous communication for remote teams, like private notes from coworkers left on stackoverflow questions (or anything on the web)
- marcosdumay 2y ago
  Your highly usable dashboard will get filled with 99% of worthless fluff just because it's there and somebody feels the need to always say something. Have you even been in one of those meetings that just won't finish despite everything being done? Making it written doesn't solve the problem. Instead, it makes it worse.
  - ChrisMarshallNY 2y ago
    Good point. AI it is, then...
  - dylan604 2y ago
    That worthless fluff makes the dashboard look impressive when you put it up on large NOC type screens when you walk people through on tours though
  - lanstin 2y ago
    "We are at the point where everything that could be said has been said, but not everyone has said it."
  - carlmr 2y ago
    On point. This is also why good CI/CD automatically alerts users of major issues. It's just not a thing humans are good at to pay attention to a long stream of mostly boring information. Computers are good at this though. Now the only question is how you can automate the spec comparison such that issues with the spec and the parts used can be automatically compared. And that starts with a computer readable spec that is updated by the manufacturer.
jetrink 2y ago
I can't believe the manufacturer didn't alert them and they had to hear it from another customer. Surely the manufacturer wouldn't want to be named as the reason that a spacecraft orbiting Jupiter went dark due to their faulty components.
- Sanzig 2y ago
  The article mentions that the defense sector discovered the issue. Rad hard defense electronics have more stringent TID (total ionizing dose) requirement than space, due to a need to survive in nuclear war scenarios. Space usually caps out at 100 krad, with some very stringent environments needing up to 300 krad. Defense can go all the way up to 1 MRad in some cases. My guess is the parts failed TID at the more stringent levels, and Infineon didn't follow up with NASA or their contractor because they assumed that NASA was okay with the lower rad tolerance levels typical of space. Usually that would be the case, but Europa Clipper is special because it's going to an extremely harsh radiation environment. The big question for me is: did the Europa Clipper program order a lower TID and try to upscreen, or did they order the high TID part? If it's the former, it's on NASA. If it's the latter, that's extremely concerning because Infineon should know that nobody orders expensive high TID parts for funsies, and they should have followed up with all customers as soon as they confirmed there was an issue. Just assuming NASA over-specified a part is absurd. The rad hard electronics market is small, everyone knows each other. Trust is king. Finally, I'm not sure if it's the part in question, but it looks like Infineon discontinued their 1 MRad MOSFETs in 2020, citing low order volumes: https://irf.com/product-info/hi-rel/alerts/fv5-d-21-0004.pdf https://irf.com/product-info/hi-rel/alerts/fv5-d-21-0004.pdf. In the light of this reporting, I have to wonder if there was more to it than that?
  - JumpCrisscross 2y ago
    > and Infineon didn't follow up with NASA or their contractor because they assumed that NASA was okay with the lower rad tolerance levels typical of space It's more likely that Infineon's folks talking to NASA were equally clueless about this change.
    - 2y ago
Wololooo 2y ago
This is much more common than one would think. This is a classic thing with Industry, they qualify a process that is working and shows good performance, but this process needs to be changed for reason XYZ, often because it is maybe a bit too expensive or doesn't align with the rest of their processes. The small change in the process wasn't that small and takes a little while to be identified because by the time you catch it you might be further down the line and this would be caught by a QA process and not a QC process, that might have deemed at that point not necessary because you had no reason to fault the part. The second part is that some things are rated and verified but not tested extensively, since you might have prototype you might misdiagnose a failure of a component for a behaviour of your prototype, when in fact you had a deeper problem, but timelines with the added fact that so far you didn't think about that problem because it shouldn't have been a problem can catch you really off guard. This is usually where people testing the same thing in an exotic environment can ring alarm bells for others and that often happens at conferences... People often under estimate how much you can get bitten in the back by such little details that become huge details. Depending on the electronics and where the MOSFETS are, I would be them I would probably trash the electronics, take the spare that they had, validate components that get in and rebuild a control box and re-integrate it, provided that this is doable. It's expensive but provided that you have no choice that gives you a backup system that you can test code on before pushing it on the actual probe and might help for problem solving by being able to do measurements and test on the actual setup... Provided that they have the time and resources. Otherwise I wouldn't YOLO it given the fact that it might just straight up not work at the moment you need it the most and a little delay is better than nothing and they can spend the time re-checking part of the design that might also be weaker... But heh, who am I but a random guy on the internet...
- interroboink 2y ago
  Another example along these lines: scurvy! [1] They found it was cured by lemon juice, but they didn't understand the details. Over years, they switched to lime juice (less vitamin C), put it in copper pipes (leaches vitamin C). But ships were faster so there was more fresh food available, masking the problem. Then scurvy starts mysteriously popping up again 100 years after it was first "cured." Hard to keep track of the effects of all the details in the face of various co-dependent things changing simultaneously. Recipe for surprises. [1] https://www.bluesci.co.uk/posts/forgotten-knowledge https://www.bluesci.co.uk/posts/forgotten-knowledge
Kenji 2y ago
Yes, all the time. This is normal. Big news of problems travel through back channels. Nobody is gonna announce their big fuckup for the world to see, unless compelled by law, and even then most won't do it. We had to sign NDAs to find out about severe Intel processor silicon bugs. Obviously you're not gonna read that info on Hackernews afterwards.
kibwen 2y ago
This is what forums like this one are for. Ordinary news isn't going to have more than a passing mention of the xz hack, or log4j, or meltdown, or heartbleed. Find (or start) a private group chat for technologists you know to share news like this.
ISL 2y ago
Yep. Chatting with other practitioners is a powerful way to learn how things actually work. There are tons of things that "everyone" knows that are not well documented, and therefore unavailable to people outside the network. This is a more-consequential example of the things you can learn by chatting with others; it is an extreme example of, "Hey, are you guys using components from Widget Inc.? Their datasheets are good, but sometimes we get a bad batch." Those little things can save you a ton of time. In this case, it may have prevented mission-failure. Part of the blame falls to NASA, too. If the outcome is your responsibility, then open-loop trust of a vendor for a known failure-mode may not be acceptable. Integration rad-hard testing may be requisite. In the spacecraft environment, qualifying components is very difficult -- there's a good chance that NASA has these MOSFETs on an approved list because they've worked well before and have had few (or known) faults. They're probably not on that list anymore.
mihaaly 2y ago
Or maybe those making a living on selling a product claiming certain parameters should raise their voice when those parameters are not met, regardless if that product is used for space travel or turn on and off a ketle for f's sake.

ChrisMarshallNY 2y ago

Well, I'm glad they didn't find out, by the board crapping out, around Jupiter.

sva_ 2y ago
> The transistors cannot simply be replaced. Clipper’s aluminum-zinc electronics vault, meant to provide a measure of radiation resistance, was sealed in October 2023. Barring an indication that the faulty MOSFETs will cause catastrophic failure, the agency will likely seek to continue with the launch—although backup windows are available the next 2 years.
- ChrisMarshallNY 2y ago
  Yeah, but if they figure out that the transistors are problematic, at least, they can do something about it, even if it does delay the launch. They may also use this as a spur to wargame failure mitigation strategies, so they'll be ready, if they do go belly-up.
DannyBee 2y ago
Right. Like for all the issues here, it's still better to find this out now.
m4rtink 2y ago
This happened already at least once with a couple Soviet Mars probes - they found a batch of transistors they used were faulty and deteriorate much too quickly in space - after the probes vere already launched. IIRC some did not even reach Mars while others failed soon after orbit insertion due to various sub-systems failing.

albumen 2y ago

>[Infineon] has already corrected the mistake, but Infineon did not report the flaw to NASA because the company did not know what the transistors would be used for, Fitzpatrick said. “They did not realize it was going to affect us.” Infineon did not respond to a request for comment. Not exactly responsible disclosure! NASA buys rad-hard transistors, and Infineon "didn't know what they'd be used for"?

nine_k 2y ago
I bet NASA buys rad-hardened electronics by a truckload, and buys from distributors, not Infineon directly. But it's a reasonable idea to notify all potential large consumers that are likely to have bought your specialty product; these are not numerous, and the impact may be large (as in this case).
- bangaladore 2y ago
  That's certainly possible, but the distributer should have notified NASA. Maybe that email is sitting in someone's inbox.
  - DannyBee 2y ago
    Distributors will notify you of recalls, but no distributor of electronics i've worked with notifies you of erratum (and it would be really annoying if they did, honestly :P)
    - bangaladore 2y ago
      This reads like it should have been a recall. But that's hard to tell unless I knew exactly the specifics of the issue. Did NASA assume these were rated higher then they were? Did Infineon make a mistake in documentation, or did they straight up not test them or test them incorrectly.
      - chmod775 2y ago

hoseja 2y ago

I am eagerly awaiting probe manufacturers learning anything at all from Ingenuity.

bloopernova 2y ago
Are you referring to off the shelf components used for the helicopter? https://en.wikipedia.org/wiki/Ingenuity_%28helicopter%29?wprov=sfla1 https://en.wikipedia.org/wiki/Ingenuity_%28helicopter%29?wpr... (in case anyone wanted to refresh their memory)
sdmike1 2y ago
The Ingenuity is a really interesting project with some important lessons for spaceflight with unhardened CotS parts. However, I would argue that it and the Europa Clipper are two very different designs for two very different environments. While mars is an elevated radiation environment when compared with earth, the Jovian radiation belts are on a whole other level, particles up to 1-2000 MeV are fairly common. To put that into context, a medical radiation beam therapy deals with 2-300 MeV on the absolute highest end. To get into the 1-2000 MeV range you generally are talking about energies found in the low end of particle accelerators. Ingenuity mostly had to worry about Total Lifetime Dose (TLD), one example of a TLD issue is dopant migration induced by high-energy heavy ion collisions which can change the on voltage of a transistor. At high energies you can have single events with enough energy to cause fatal latch-ups. For instance modern rad-hard FPGAs start encountering major issues around 60-70 MeV. Furthermore, these parts are power MOSFETs which control power for whole subsystems so their reliability is critical to the operation of the spacecraft. In addition, the biggest issue here is not just that there were issues that were addressed and fixed, it's that Infineon didn't issue an errata to the datasheet or inform NASA of the issue. As a result there are now transistors littered throughout the spacecraft which don't meet the radiation needs. This is going to require reworking the boards, re-validation of the subsystem, and re-integration of the subsystem into the spacecraft. This all comes at a non-trivial impact to budget and timelines which is to say nothing about what this does to the launch window the project was trying to hit for gravity assist / proximity. I hope you find this informative! :) EDITS: Spelling and an "is"

twh270 2y ago

From the article: "Infineon did not report the flaw to NASA because the company did not know what the transistors would be used for, Fitzpatrick said." They might not have "known", but come on, you're selling radiation-hardened chips to NASA. You can sure make an educated guess that they might be used for a probe. I'm guessing there's a clause missing in the contract that says Infineon must disclose all known problems to NASA regardless of how the chips will be used. Regardless, there are some people at NASA to whom 'Infineon' is now a curse word.

laurencei 2y ago
"They might not have "known", but come on, you're selling radiation-hardened chips to NASA. " But do people ever actually "invoice NASA" for components. It was probably one of 100 different sub contractors building the actual circuits to NASA specifications, i.e. it was lower in the chain rather than NASA itself. (Doesnt excuse the non-disclosure to those subcontractors)
- 0xffff2 2y ago
  >But do people ever actually "invoice NASA" for components Yes, absolutely they do. I'm not a part of this mission, but I'm currently working on another NASA spacecraft mission. I don't know the percentages off hand, but a substantial portion of our spacecraft is built in house with parts purchased directly by NASA from the manufacturer. Regardless, there are lines of communication to subcontractors. The mere fact that they found out about this at a conference is significant evidence that Infineon didn't notify who they should have.
  - deleted 2y ago
    [deleted]
  - hvs 2y ago
    Off-topic, but when components are sourced directly from the manufacturer do you have to buy in bulk? I figured you didn't just go on Mouser or DigiKey, but I would think manufacturers don't like dealing in small amounts.

indoordin0saur 2y ago

Is more shielding not the obvious answer? A thin sheet of lead around the sensitive parts should do the trick. (Note: I'm not a physicist and have no idea what I'm talking about in this domain)

klodolph 2y ago
1. You’d need more than a thin sheet of lead. The radiation in space can be very energetic. It can easily penetrate several cm of shielding and if it is absorbed, you get secondary radiation. 2. Even a thin sheet of lead may be too heavy.
JumpCrisscross 2y ago
> Is more shielding not the obvious answer? A thin sheet of lead around the sensitive parts should do the trick Lead "is effective at stopping gamma rays and x-rays" [1]. Jupiter's radiation comes from "trapped particles [that] are about ten times more energetic than the ones from the equivalent radiation belts of Earth" and "several orders of magnitude more abundant" [2]. When those encounter lead they cause bremsstrahlung radiation [3], a sort of subatomic shrapnel that can be more dangerous than the original radiation. Lead is also heavy, which means not only increasing the mass of the spacecraft, but its balance and thus propulsion profile. That might mean upgrading and moving thrusters and propellant tanks--in effect, a complete redesign. (It's a good question that doesn't deserve to be downvoted.) [1] https://en.wikipedia.org/wiki/Lead_shielding https://en.wikipedia.org/wiki/Lead_shielding [2] https://www.spenvis.oma.be/help/background/planetary/traprad_jup.html#TRPP https://www.spenvis.oma.be/help/background/planetary/traprad... [3] https://en.wikipedia.org/wiki/Bremsstrahlung https://en.wikipedia.org/wiki/Bremsstrahlung
- basementcat 2y ago
  Europa Clipper electronics are contained in a 9.2 mm thick aluminum-zinc vault. https://europa.nasa.gov/resources/342/electronics-vault/ https://europa.nasa.gov/resources/342/electronics-vault/
  - 2y ago

gene-h 2y ago

Europa Clipper also used a new approach for designing spacecraft. It's NASA's first major spacecraft designed with Model Based Systems Engineering(MBSE)[0]. Using diagrams in SysML to keep track of power use and interfaces is supposedly better than using spreadsheets [0]https://ses.gsfc.nasa.gov/ses_data_2021/210728_Bayer.pdf https://ses.gsfc.nasa.gov/ses_data_2021/210728_Bayer.pdf

PaulGaspardo 2y ago
Oh, I used to work on this :) For keeping track of power use and interfaces specifically it turns out doing it all with SysML diagrams wasn't so great. Aside from all the pointless futzing around with boxes and arrows the model eventually became so huge the authoring software could barely handle just opening it up. So it must have been shortly after these slides when all the power use tracking was shifted to a custom tool with a more tabular user interface that we were already using for tracking electrical interfaces (slide 15) with version control in git.
- crocal 2y ago
  Unrelated question: how did you manage tabular data in git? It’s always a struggle to diff and merge changes.
  - baq 2y ago
    not OP but the usual applies - data is not actually stored as a table in git, tables are an UI thing. git would store standard issue json, xml or whatever custom git-friendly format is used by the tool.
  - PaulGaspardo 2y ago
    Yeah, like 'baq said the data wasn't stored in a tabular form, it was actually XML. So sometimes you could just look at the textual diff and it would make perfect sense, although it wasn't expected users would work with XML at the source level. There was also a semantic object-level diff we got for "free" by virtue of building on top of the Eclipse Modeling Framework. It was integrated into the Eclipse git UI and could help resolve merge conflicts without having to touch the XML directly, but merge conflicts were still annoying to deal with so generally engineers coordinated with each other to not touch the same part of the model at the same time. Normally for review though I think users tended to compare reports generated from the model rather than trying to diff the source model files directly. There was a sort of automated build process that took care of that once you pushed your branch to Github.

metiscus 2y ago

Sounds like Infineon may owe someone a new satellite soon. At least if it can be shown that they sent NASA bad parts and didn't notify them in time to prevent this failure.

rkagerer 2y ago

Id like to know how this turns out - ie. hiw they decide to mitigate. Where's the easiest / best place to check in a couple months for a followup?

chrisjj 2y ago

> Infineon did not report the flaw to NASA because the company did not know what the transistors would be used for, Either the parts were in spec or they weren't. Which is it?

jjk166 2y ago
That's not how specs work. When the requirements for a part are specified, it is based on assumptions that may or may not hold true. For example, if an issue tends to be all or nothing, then testing a small percentage of a lot should reasonably be expected to catch an issue. So you might specify that 1% of these transistors be tested and so long as that 1% passes the rest are considered good. If let's say there's a process change and lots become more variable, the confidence with which you can say the others are good based on that 1% testing goes down, but you are still testing to the same standard that you were before, which is what the specification calls for. The issue gets even more thorny when issues are conditional. For example a part might meet the voltage specification, the temperature specification, and the radiation specification individually, but when you put that same part simultaneously in a high voltage, low temperature, and high radiation environment it doesn't perform as well. Or perhaps one component used downstream of a particular other component has an effect. Perhaps the most basic example is oversized but in tolerance shaft meets undersized but in tolerance hole.
- DoctorOetker 2y ago
  >For example a part might meet the voltage specification, the temperature specification, and the radiation specification individually, but when you put that same part simultaneously in a high voltage, low temperature, and high radiation environment it doesn't perform as well. I am not disagreeing, but at some point humanity should switch to actual probabilistic device models instead of vague datasheets. Imagine every datasheet has a .sample() function and you get a randomized SPICE model as if it came from the manufacturing line, you can draw a 100 and plot the properties. Want to measure the dynamic range of some ADC? A-weighted or not? Instead of specifying values highlighting figures of merit, each such figure of merit corresponds with an explicit SPICE circuit that measures that figure of merit on one and the same generator for random SPICE models with that specific part designation. If a brand tries to fool its customers by insinuating a desirably high or low value by changing the test method, its immediately clear. A user may specify his own test circuit, or reuse the test circuit from the interactive datasheet from a competitor etc in order to make apples to apples comparisons for different DUT's.

elzbardico 2y ago

1. Sell bunch of radiation-hardened parts to NASA. 2. Find out the parts you sold to NASA don't meet the specs. 3. Don't tell NASA, because NASA didn't tell you what those parts would be used for. This is criminally incompetent on the part of Infineon. WTF, NASA could use those transistors for a fancy inteliggent toilet FWIW, it doesn't matter, NASA doesn't have to tell you how they are going to use those parts. They bought parts based on a fucking SPECIFICATION, and if the parts you sold them don't meet the specs, you communicate immediatelly with the customer offering a replacement for free. Really, someone should be jailed for that.

hvs 2y ago
Jailing people for mistakes is a good way to ensure that mistakes are kept hidden.
- elzbardico 2y ago
  There are mistakes, and there is negligence.

jjk166 2y ago

> Infineon did not report the flaw to NASA because the company did not know what the transistors would be used for, People are reading this as Infineon didn't know that the parts were going into a probe when it's far more likely they meant they didn't know how the transistors are being used in that probe, which might have a large effect on whether or not the problem will affect them.

fergbrain 2y ago

Did Infineon not issue a GIDEP Notice for these parts? If not, does that mean that maybe NASA is using them outside of their designed spec? (See also: https://nodis3.gsfc.nasa.gov/displayDir.cfm?Internal_ID=N_PR_8735_001D_&page_name=Chapter3 https://nodis3.gsfc.nasa.gov/displayDir.cfm?Internal_ID=N_PR...)

brcmthrowaway 2y ago

Germans and space industry, what could go wrong

Space_rat66 2y ago
the Mosfets are actually from IR HiRel which is an American legacy space company

fsckboy 2y ago

DAE when i read a sentence like this > Engineers at NASA’s Jet Propulsion Laboratory (JPL), which leads the development of Clipper, discovered the problem in May after talking with colleagues about a classified satellite at a conference. ...I immediately think "that's what they want us to think" if classification of projects means anything, the true intent of the passing of dis information is something dat we can only guess at. fnord.

omneity 2y ago

So can we say the transistors of Clipper are ... clipping? More seriously, I'd be interested to learn what failed in the quality assurance process, as NASA & its suppliers have a legendary reputation in these topics. RCA will be enlightening.

rsynnott 2y ago

My favourite bit about this project: > In June 2022, project scientist Robert Pappalardo revealed that mission planners for Europa Clipper were considering disposing of the probe by crashing it into the surface of Ganymede for Europan protection purposes, in case an extended mission was not approved early. He noted that an impact would help the ESA's JUICE mission collect more information about Ganymede's surface chemistry What about Ganymedian protection, eh? The Ganymedians, should they exist, are going to be furious. Also it is weird that the outcome of the EJSM divorce (originally there was going to be a joint NASA-ESA mission to the Jupiter icy moons; Europa Clipper and JUICE are a result of the breakup), is that America explores Europa and Europe explores Ganymede, as the other way around would be less confusing.

amelius 2y ago
If the Ganymedians got angry and attacked us, that would be a great scientific advancement.

LeifCarrotson 2y ago

> The transistors cannot simply be replaced. Clipper’s aluminum-zinc electronics vault, meant to provide a measure of radiation resistance, was sealed in October 2023. Surely they can be replaced! Humans (in cleanrooms) can put their hands inches away from them. Here's a picture, admittedly from 2022, before it was sealed: https://europa.nasa.gov/resources/342/electronics-vault/ https://europa.nasa.gov/resources/342/electronics-vault/ It's not the same as saying they've failed and are in orbit around Jupiter and so can't be replaced. They simply need to open and re-seal the vault or build and seal a new electronics vault before October.

nebalee 2y ago
Here's a photo of the vault as it was about to be sealed in October last year: https://www.nasa.gov/missions/europa-clipper/how-nasa-is-protecting-europa-clipper-from-space-radiation/ https://www.nasa.gov/missions/europa-clipper/how-nasa-is-pro... In the meantime NASA built a whole spacecraft around it and the craft went through numerous tests. If they'd have to undo and redo all of that, they may even miss the 2025 backup launch date. I'm sure NASA is very eager to seek other ways to deal with this problem.
- rob74 2y ago
  Ok, that sounds a bit more convincing... I'm not the OP, but I was just trying to imagine why a "sealed vault" would work better against cosmic radiation than a more prosaic screwed shut container.
- deleted 2y ago
  [deleted]
- fbdab103 2y ago
  On one hand, it must be baller to work on assembling spacecraft. On the other, wearing those gowns all day must be terrible.
2o3jr2o34j 2y ago
Do you understand that testing and certifying everything takes months and is a significant portion of the cost of the mission? And plenty of these components and closures were designed for single use, so they would have to be replaced. This is not a desktop PC.