9 ms·

How NASA built Artemis II’s fault-tolerant computer

645 points by speckx 7mo ago

temptemptemp111 7mo ago
[dead]
starkparker 7mo ago
Headline needs its how-dectomy reverted to make sense
- arduanika 7mo ago
  (Off-topic:) Great word. Is that the usual word for it? Totally apt, and it should be the standard.
dmk 7mo ago
The quote from the CMU guy about modern Agile and DevOps approaches challenging architectural discipline is a nice way of saying most of us have completely forgotten how to build deterministic systems. Time-triggered Ethernet with strict frame scheduling feels like it's from a parallel universe compared to how we ship software now.
- tayk47999 7mo ago
  > “Modern Agile and DevOps approaches prioritize iteration, which can challenge architectural discipline,” Riley explained. “As a result, technical debt accumulates, and maintainability and system resiliency suffer.” Not sure i agree with the premise that "doing agile" implies decision making at odds with architecture: you can still iterate on architecture. Terraform etc make that very easy. Sure, tech debt accumulates naturally as a byproduct, but every team i've been on regularly does dedicated tech debt sprints. I don't think the average CRUD API or app needs "perfect determinism", as long as modifications are idempotent.
  - whiskey-one 7mo ago
    In theory, yes you could iterate on architecture and potentially even come up with better one with agile approach. In practice, so many aspects follow from it that it’s not practical to iterate with today’s tools.
  - crabbone 7mo ago
    Agile is like communism. Whenever something bad happens to people who practice agile, the explanation is that they did agile wrong, had they being doing the true agile, the problem would've been totally avoided. In reality, agile doesn't mean anything. Anyone can claim to do agile. Anyone can be blamed for only pretending to do agile. There's no yardstick. But it's also easy to understand what the author was trying to say, if we don't try to defend or blame a particular fashionable ideology. I've worked on projects that required high quality of code and product reliability and those that had no such requirement. There is, indeed, a very big difference in approach to the development process. Things that are often associated with agile and DevOps are bad for developing high-quality reliable programs. Here's why: The development process before DevOps looked like this: 1. Planning 2. Programming 3. QA 4. If QA found problems, goto 2 5. Release The "smart" idea behind DevOps, or, as it used to be called at the time "shift left" was to start QA before the whole of programming was done, in parallel with the development process, so that the testers wouldn't be idling for a year waiting for the developers to deliver the product to testers and the developers would have faster feedback to the changes they make. Iterating on this idea was the concept of "continuous delivery" (and that's where DevOps came into play: they are the ones, fundamentally, responsible to make this happen). Continuous delivery observed that since developers are getting feedback sooner in the development process, the release, too, may be "shifted left", thus starting the marketing and sales earlier. Back in those days, however, it was common to expect that testers will be conducting a kind of a double-blindfolded experiment. I.e. testers weren't supposed to know the ins and outs of the code intentionally, s.t. they don't, inadvertently, side with the developers on whatever issues they discover. Something that today, perhaps, would've been called "black-box testing". This became impossible with CD because testers would be incrementally exposed to the decisions governing the internal workings of the product. Another aspect of the more rigorous testing is the "mileage". Critical systems, normally, aren't released w/o being run intensively for a very long time, typically orders of magnitude longer than the single QA cycle (let's say, the QA gets a day of computer time to run their tests, then the mileage needs to be a month or so). This is a very inconvenient time for development, as feature freeze and code freeze are still in effect, so the coding can only happen in the next version of the product (provided it's even planned). But, the incremental approach used by CD managed to sell a lie that says that "we've ran the program for a substantial amount of time during all the increments we've made so far, therefore we don't need to collect more mileage". This, of course, overlooks the fact that changes in the program don't contribute proportionally to the program's quality or performance. In other words, what I'm trying to say is that agile or DevOps practices allowed to make the development process cheaper by making it faster while still maintaining some degree of quality control, however they are inadequate for products with high quality requirements because they don't address the worst case scenarios.
- ramraj07 7mo ago
  I take the opposite message from that line - out of touch teams working on something so over budget and so overdue, and so bureaucratic, and with such an insanely poor history of success, and they talk as if they have cured cancer. This is the equivalent of Altavista touting how amazing their custom server racks are when Google just starts up on a rack of naked motherboards and eats their lunch and then the world. Lets at least wait till the capsule comes back safely before touting how much better they are than "DevOps" teams running websites, apparently a comparison that's somehow relevant here to stoke egos.
  - danhon 7mo ago
    You mean like this? "With limited funds, Google founders Larry Page and Sergey Brin initially deployed this system of inexpensive, interconnected PCs to process many thousands of search requests per second from Google users. This hardware system reflected the Google search algorithm itself, which is based on tolerating multiple computer failures and optimizing around them. This production server was one of about thirty such racks in the first Google data center. Even though many of the installed PCs never worked and were difficult to repair, these racks provided Google with its first large-scale computing system and allowed the company to grow quickly and at minimal cost." https://blog.codinghorror.com/building-a-computer-the-google-way/ https://blog.codinghorror.com/building-a-computer-the-google...
    - 1970-01-01 7mo ago
      Google then had complete regret not doing this with ECC RAM: https://news.ycombinator.com/item?id=14206811 https://news.ycombinator.com/item?id=14206811
      - ramraj07 7mo ago
        It got them to where they need to be to then worry about ECC. This is like the dudes who deploy their blog on kubernetes just in case it hits front page of new york times or something.
        7mo ago
- arduanika 7mo ago
  You could even say that part of the value of Artemis is that we're remembering how to do some very hard things, including the software side. This is something that you can't fake. In a world where one of the more plausible threats of AI is the atrophy of real human skills -- the goose that lays the golden eggs that trains the models -- this is a software feat where I'd claim you couldn't rely on vibe code, at least not fully. That alone is worth my tax dollars.
  - randomNumber7 7mo ago
    Don’t count your chickens before they hatch.
    - arduanika 7mo ago
      I'm not sure you really understood my comment. A large portion of the kind of value I'm talking about comes from attempting the hard thing. If these chickens do not hatch that will be tragic, but we will still have learned something from it. In some ways, we will have learned even more, by getting taught about what we don't know. Anyway, let's all hope for a safe landing tonight.
- mvkel 7mo ago
  If you look at code as art, where its value is a measure of the effort it takes to make, sure.
  - couchand 7mo ago
    If your implication is that stencil art does not take effort then perhaps you may not fully appreciate Banksy. Works like Gaza Kitty or Flower Thrower don’t just appear haphazardly without effort.
  - BobbyTables2 7mo ago
    In that case, our test infrastructure belongs in the Louvre…
  - stodor89 7mo ago
    Or if you're building something important, like a spaceship.
- iknowstuff 7mo ago
  Tesla’s Cybertruck uses that in its ethernet as well!
  - carefree-bob 7mo ago
    All the ADAS automotive systems use this, there are several startups in this space as well, such as Ethernovia.
    - 21asdffdsa12 7mo ago
      All thanks to twisted pair https://de.wikipedia.org/wiki/BroadR-Reach https://de.wikipedia.org/wiki/BroadR-Reach
- dyauspitr 7mo ago
  Agile is not meant to make solid, robust products. It’s so you can make product fragments/iterations quickly, with okay quality and out to the customer asap to maximize profits.
  - nickff 7mo ago
    “Agile” doesn’t mean that you release the first iteration, it’s just a methodology that emphasizes short iteration loops. You can definitely develop reliable real-time systems with Agile.
    - kermatt 7mo ago
      > “Agile” doesn’t mean that you release the first iteration Someone needs to inform the management of the last three companies I worked for about this.
      - t43562 7mo ago
        Management understand it less than anyone else does.
    - tomasGiden 7mo ago
      I would differentiate between iterative development and incremental development. Incremental development is like panting a picture line by line like a printer where you add new pieces to the final result without affecting old pieces. Iterative is where you do the big brush strokes first and then add more and more detail dependent on what to learn from each previous brush strokes. You can also stop at any time when you think that the final result is good enough. If you are making a new type of system and don’t know what issues will come up and what customers will value (highly complex environment) iterative is the thing to do. But if you have a very predictable environment and you are implementing a standard or a very well specified system (van be highly complicated yet not very complex), you might as will do incremental development. Roughly speaking though as there is of course no perfect specification which is not the final implementation so there are always learnings so there is always some iterative parts of it.
    - 7mo ago
- vasco 7mo ago
  It's not like the approach they took is any different. Just slapped 8x the number of computers on it for calculating the same thing and wait to see if they disagree. Not the pinnacle of engineering. The equivalent of throwing money at the problem.
  - MikeTheGreat 7mo ago
    What happens when they don't?
    - vasco 7mo ago
      If you have a point to make, make it.
      - MikeTheGreat 7mo ago
        What my question is hinting at is that there's actually some really interesting engineering around resolving what happens when the systems disagree. Things like Paxos and Raft help make this much more tractable for mere mortals (like myself); the logic and reasoning behind them are cool and interesting.
        vasco 7mo ago
        [flagged]
        FabHK 7mo ago
        Though here the consensus algorithm seems totally different from Paxos/Raft. Rather it's a binary tree, where every non-leaf node compares the (non-silent) inputs from the leaf, and if they're different, it falls silent, else propagates the (identical) results up. Or something something.
        HeyLaughingBoy 7mo ago
        There really is. We designed a redundant system (software, hardware and mechanisms) a couple years ago. And the problems around figuring out who's in control and how to keep things synchronized across a number of potential failure modes gets really hairy. Sadly, the project was cancelled before we could complete the implementation.
- carefree-bob 7mo ago
  During the time of the first Apollo missions, a dominant portion of computing research was funded by the defense department and related arms of government, making this type of deterministic and WCET (worst case execution time) a dominant computing paradigm. Now that we have a huge free market for things like online shopping and social media, this is a bit of a neglected field and suffers from poor investment and mindshare, but I think it's still a fascinating field with some really interesting algorithms -- check out the work of Frank Mueller or Johann Blieberger.
  - therobots927 7mo ago
    Contrary to propaganda from the likes of Ludwig von Mises, the free market is not some kind of optimal solution to all of our problems. And it certainly does not produce excellent software.
    - psd1 7mo ago
      I can't think of a time when I've found an absolutist position useful or intelligent, in any field. Free-market absolutism is as stupid as totalitarianism. The content of economics papers does not need to be evaluated to discard an extreme position, one need merely say "there are more things in earth and heaven than are dreamed of in your philosophies"
      - staplers 7mo ago
        Great point, if the only constant is change, then philosophy should follow (or lead).
    - nairboon 7mo ago
      Propaganda is quite a strong term to describe the works of an economist. If one wants to debate the ideas of von Mises, it'd be useful to consider the Zeitgeist at that time. Von Mises preferred free markets in contrast to the planned economy of the communists. Partly because the latter has difficulties in proper resource allocation and pricing. Note that this was decades before we had working digital computers and digital communication systems, which, at least in theory, change the feasibility of a planned economy. Also, the last time I checked, the US government produced its goods and services using the free market. The government contractors (private enterprises) are usually tasked with building stuff, compared with the government itself in a non-free, purely planned economy (if you refer to von Mises). I assume that you originally meant to refer to the idea that without government intervention (funding for deep R&D), the free market itself would probably not have produced things like the internet or the moon landing (or at least not within the observed time span). That is, however,a rather interesting idea.
- pjmlp 7mo ago
  As 70's child that was there when the whole agile took over, and systems engineer got rebranded as devops, I fully agree with them. Add TDD, XP and mob programming as well. While in some ways better than pure waterfall, most companies never adopted them fully, while in some scenarios they are more fit to a Silicon Valley TV show than anything else.
- ggm 7mo ago
  Time triggered Ethernet is part of aircraft certified data bus and has a deep, decades long history. I believe INRIA did work on this, feeding Airbus maybe. It makes perfect sense when you can design for it. An aircraft is a bounded problem space of inputs and outputs which can have deterministic required minima and then you can build for it, and hopefully even have headroom for extras. Ethernet is such a misnomer for something which now is innately about a switching core ASIC or special purpose hardware, and direct (optical even) connects to a device. I'm sure there are also buses, dual redundant, master/slave failover, you name it. And given it's air or space probably a clockwork backup with a squirrel.
  - Arch-TK 7mo ago
    A real squirrel would need acorns, I would assume it's a clockwork squirrel too.
    - sqrrl-io 7mo ago
      A software squirrel, maybe? https://sqrrl.io https://sqrrl.io :-)
  - 21asdffdsa12 7mo ago
    Aircraft also have software and components, that form a "working" proclaimed eco-system in lockstep- a baseline. This is why there are paper "additions" on bug discovery until the bug is patched and the whole ecosystem of devices is lifted to the next "baseline".
- deleted 7mo ago
  [deleted]
- guenthert 7mo ago
  I think he refers to SpaceWire https://en.wikipedia.org/wiki/SpaceWire https://en.wikipedia.org/wiki/SpaceWire.
- anymouse123456 7mo ago
  Some of us still work on embedded systems with real-time guarantees. Believe it or not, at least some of those modern practices (unit testing, CI, etc) do make a big (positive) difference there.
  - cpgxiii 7mo ago
    The depressing part is that these "modern practices" were essentially invented in the 1960s by defense and aerospace projects like the NTDS, LLRV/LLTV, and Digital Fly-by-Wire to produce safety-critical software, and the rest of the software industry simply ignored them until the last couple of decades.
- globnomulous 7mo ago
  Microsoft fired all QA people ten or fifteen years ago. I'd imagine it's a similar a story: boxed software needed much higher guarantees of correctness. Digital deliver leaves much more room for error, because it leaves room for easier, cheaper fixes.
ConanRus 7mo ago
[dead]
object-a 7mo ago
How big of a challenge are hardware faults and radiation for orbital data centers? It seems like you’d eat a lot of capacity if you need 4x redundancy for everything
- totetsu 7mo ago
  They dont go into here.. but I thought that NASA also used like 250nm chips in space for radiation resistance. Are there even any radiation resistance GPUs out there?
  - pclmulqdq 7mo ago
    Absolutely not, although the latest fabs with rad-tolerant processors are at ~20 nm. There are FDSOI processes in that generation that I assume can be made radiation-tolerant.
  - linzhangrun 7mo ago
    [flagged]
    - tosapple 7mo ago
      [dead]
  - kersplody 7mo ago
    NOPE, RAD hardened space parts basically froze on mid 2000s tech: https://www.baesystems.com/en-us/product/radiation-hardened-electronics https://www.baesystems.com/en-us/product/radiation-hardened-...
- aidenn0 7mo ago
  You don't need 4x redundancy for everything. If no humans are aboard, you have 2x redundancy and immediately reboot if there is a disagreement.
- pjerem 7mo ago
  Orbital data centers are still nothing more than the current hyperloop.
jbritton 7mo ago
I wonder how often problems happen that the redundancy solves. Is radiation actually flipping bits and at what frequency. Can a sun flare cause all the computers to go haywire.
- BitsAndObjects 7mo ago
  Not a direct answer but probably as good information as you can get: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35162.pdf https://static.googleusercontent.com/media/research.google.c... Basically, yes, radiation does cause bit flips, more often than you might expect (but still a rare event in the grand scheme of things, but enough to matter). And radiation in space is much “worse” (in quotes because that word is glossing over a huge number of different problems, both just intensity).
  - BitsAndObjects 7mo ago
    Typo: “both” ~ “not”
- tosapple 7mo ago
  [dead]
- Tomte 7mo ago
  IEC 61508 estimates a soft error rate of about 700 to 1200 FIT (Failure in Time, i.e. 1E-9 failures/hour). That was in the 2000s though, and for embedded memory above 65nm. And obviously on earth.
y1n0 7mo ago
NASA didn't build this, Lockheed Martin and their subcontractors did. Articles and headlines like this make people think that NASA does a lot more than they actually do. This is like a CEO claiming credit for everything a company does.
- voodoo_child 7mo ago
  Nice “well, actually”. I’m sure Lockheed were building this quad-redundant, radiation-hardened PowerPC that costs millions of dollars and communicates via Time-Triggered Ethernet anyway, whether NASA needed one or not.
  - y1n0 7mo ago
    This is the equivalent of prompt engineering.
  - kube-system 7mo ago
    Probably, if it already wasn’t developed for DoD. For example, the OS it seems to be running is integrity 178. https://www.ghs.com/products/safety_critical/integrity_178_success-nasa_orion.html https://www.ghs.com/products/safety_critical/integrity_178_s... Aerospace tech is not entirely bespoke anymore, plenty of the foundational tech is off the shelf. Historically, the main difference between ICBM tech and human spaceflight tech is the payload and reentry system.
- Sebguer 7mo ago
  will nobody think of the megacorps!!!
- adrian_b 7mo ago
  Lockheed Martin and their subcontractors did the implementation. We do not know how much of the high-level architecture of the system has been specified by NASA and how much by Lockheed Martin.
  - y1n0 7mo ago
    I do.
seemaze 7mo ago
and yet.. https://news.ycombinator.com/item?id=47615490 https://news.ycombinator.com/item?id=47615490
- adrian_b 7mo ago
  That was a laptop, not one of the Artemis computers.
__d 7mo ago
Does anyone have pointers to some real information about this system? CPUs, RAM, storage, the networking, what OS, what language used for the software, etc etc? I’d love to know how often one of the FCMs has “failed silent”, and where they were in the route and so on too, but it’s probably a little soon for that.
- anthonj 7mo ago
  Nasa CFS, is written is plain C (trying to follow MISRA C, etc). It's open on girhub abd used by many companies. It's typically run over freertos or RTEMS, not sure here. Personally I find the project extremely messy, and kinda hate working with it.
  - sheepybloke 7mo ago
    It's most likely using vxworks for it's OS, since I believe it's one of the only fully certified ARINC653 OS's for human flight. It's used in most Aircraft and space missions.
    - __d 7mo ago
      Yeah, that was my guess too but the comment about separate implementation for the backup system made me wonder if there was a different OS, and the which was running where.
- j4k0bfr 7mo ago
  Not sure about the primary FSW but the BFS uses cFS[0]. As the sibling comment mentions, you can check it out on GitHub. Sadly I believe NASA keeps most of their best code private, probably siloed into mission-specific codebases. Still, the cFS repo is an awesome crash course on old-school Flight Software techniques. [0] https://youtu.be/4doI2iQe4Jk?si=ucMoIdw7x_QgZR32 https://youtu.be/4doI2iQe4Jk?si=ucMoIdw7x_QgZR32
  - __d 7mo ago
    Helpful video, thanks! At about 1:20, the presenter says the BFS uses a different OS and hardware (not sure if that means a different instance, or a different class, so to speak).
nickpsecurity 7mo ago
The ARINC scheduler, RTOS, and redundancy have been used in safety-critical for decades. ARINC to the 90's. Most safety-critical microkernels, like INTEGRITY-178B and LynxOS-178B, came with a layer for that. Their redundancy architecture is interesting. I'd be curious of what innovations went into rad-hard fabrication, too. Sandia Secure Processor (aka Score) was a neat example of rad-hard, secure processors. Their simulation systems might be helpful for others, too. We've seen more interest in that from FoundationDB to TigerBeetle.
geomark 7mo ago
I sure wish they would talk about the hardware. I spent a few years developing a radiation hardened fault tolerant computer back in the day. Adding redundancy at multiple levels was the usual solution. But there is another clever check on transient errors during process execution that we implemented that didn't involve any redundancy. Doesn't seem like they did anything like that. But can't tell since they don't mention the processor(s) they used.
- themafia 7mo ago
  One of the things I loved about the Shuttle is that all five computers were mounted not only in different locations but in different orientations in the shuttle. Providing some additional hardening against radiation by providing different cross sections to any incident event.
- MathMonkeyMan 7mo ago
  I read, for probe missions, that one technique is to get a bunch of consumer chips and irradiate the hell out of them. Now take the winner model and get a bunch of those. Irradiate them. The winner goes to Mars. The claim was that some plain old chips are exquisitely radiation resisitant, and it's not clear why.
- MutexMaven 7mo ago
  NASA actually publishes these things on their NTRS page. The Primary flight controller is rocking Green Hills INTEGRITY RTOS on BAE RAD750s in a quad redundant config, with a VxWorks backup on a Frontgrade Gaisler LEON4 (SPARC V8). This allowed for parts of the ARINC653 spec regarding time and space partitioning of the RTOS scheduler to be used. You can read more about it below (when the server throwing errors). https://ntrs.nasa.gov/api/citations/20190000011/downloads/20190000011.pdf https://ntrs.nasa.gov/api/citations/20190000011/downloads/20... https://ntrs.nasa.gov/api/citations/20230002185/downloads/FSW2023_Orion_BFS_Hirsh_final.pptx.pdf https://ntrs.nasa.gov/api/citations/20230002185/downloads/FS...
georgehm 7mo ago
>Effectively, eight CPUs run the flight software in parallel. The engineering philosophy hinges on a >“fail-silent” design. The self-checking pairs ensure that if a CPU performs an erroneous calculation >due to a radiation event, the error is detected immediately and the system responds. >“A faulty computer will fail silent, rather than transmit the ‘wrong answer,’” Uitenbroek explained. >This approach simplifies the complex task of the triplex “voting” mechanism that compares results. > >Instead of comparing three answers to find a majority, the system uses a priority-ordered source >selection algorithm among healthy channels that haven’t failed-silent. It picks the output from the >first available FCM in the priority list; if that module has gone silent due to a fault, it moves to >the second, third, or fourth. One part that seems omitted in the explanation is what happens if both CPUs in a pair for whatever reason performs an erroneous calculation and they both match, how will that source be silenced without comparing its results with other sources.
- themafia 7mo ago
  In the Shuttle they would use command averaging. All four computers would get access to an actuator which would tie into a manifold which delivered power to the flight control surface. If one disagreed then you'd get 25% less command authority to that element.
  - JumpCrisscross 7mo ago
    > In the Shuttle they would use command averaging I think the Shuttle, operating only in LEO, had more margin for error. Averaging a deep-space burn calculation is basically the same as killing the crew.
    - themafia 7mo ago
      The GNC loop runs several times per second. The desired output will consequently be increased by the working computers to achieve the target. The computer does not "dead reckon" anything. Travelling through Max-Q in Earth atmosphere on ascent is far more dangerous.
spaceman123 7mo ago
Probably same way they’ve built fault-tolerant toilet.
- jeron 7mo ago
  ctrl+f toilet, thank you for already commenting this
SeanAnderson 7mo ago
Typo in the first sentence of the first paragraph is oddly comforting since AI wouldn't make such a typo, heh. Typo in the first sentence of the second paragraph is sad though. C'mon, proofread a little.
- tux 7mo ago
  I think everyone should now make mistakes so we ca distinguish human vs ai.
  - zeristor 7mo ago
    This can be optimised for no doubt, adversarial training is like that
- deleted 7mo ago
  [deleted]
vhiremath4 7mo ago
> “Along with physically redundant wires, we have logically redundant network planes. We have redundant flight computers. All this is in place to cover for a hardware failure.” It would be really cool to see a visualization of redundancy measures/utilization over the course of the trip to get a more tangible feel for its importance. I'm hoping a bunch of interesting data is made public after this mission!
hulitu 7mo ago
They run 2 Outlook instances. For redundancy. /s
ajaystream 7mo ago
[flagged]
- adrian_b 7mo ago
  This is similar to the difference between using error-correcting codes and using erasure codes combined with error-detecting codes. The latter choice is frequently simpler and more reliable for preventing data corruption. (An erasure code can be as simple as having multiple copies and using the first good copy.)
- sammy2255 7mo ago
  Spoken like an LLM.
- high_na_euv 7mo ago
  How you can remove component from decision set if it is the only component in the whole decision set?
- randomNumber7 7mo ago
  > make each unit responsible for detecting its own faults and shutting up if it can't guarantee correctness Does this mean you have to trust the already compromised system?
perarneng 7mo ago
[dead]
JumpCrisscross 7mo ago
Does anyone know how this compares to Crew Dragon or HLS?
gambiting 7mo ago
So honest and perhaps a bit stupid question. Astronauts have actual phones with them - iPhones 17 I think? And a regular Thinkpad that they use to upload photos from the cameras. How does all of that equipment work fine with all the cosmic radiation floating about? With the iPhone's CPU in particular, shouldn't random bit flips be causing constant crashes due to errors? Or is it simply that these errors happen but nothing really detects them so the execution continues unhindered?
- BitsAndObjects 7mo ago
  They’re not mission-critical equipment. If they fail, nobody dies. They’re not radiation hardened, so given enough time, they’d be expected to fail. Rebooting them might clear the issue or it might not (soft vs hard faults). Also impossible to predict when a failure would happen, but NASA, ESA and others have data somewhere that makes them believe the risk is high enough that mission critical systems need this level of redundancy.
  - gambiting 7mo ago
    >>They’re not mission-critical equipment. If they fail, nobody dies. Yes, for sure, but that's not my question - it's not a "why is this allowed" but "why isn't this causing more visible problems with the iphones themselves". Like, do they need constant rebooting? Does this cause any noticable problems with their operation? Realistically, when would you expect a consumer grade phone to fail in these conditions?
    - BitsAndObjects 7mo ago
      Random bit flips due to radiation are infrequent - the stat is something like one but flip per megabyte per 40,000 data centre RAM modules per year - ie extremely uncommon, but common enough to matter at scale. Space is a harsher environment but they’re only up there for like a week. So, if there were an incident, it would be more likely to kill the devices, but it’s not very likely to happen during the short period of time (while still being more likely than on earth’s surface). That said, part of the point of them taking these devices up is to find out how well they perform in practice. We just don’t really know how these consumer devices perform in space. It will be interesting to see the results when they’re published!
GautamB13 7mo ago
It kinda crazy how this mission didn't become mainstream media until as of late.
kev009 7mo ago
Some people are claiming it's the good old RAD750 variant. Is there anything that talks about the actual computer architecture? The linked article is desperately void of technical details.
- u1hcw9nx 7mo ago
  It's a new (2002) variant of the same RAD750 architecture. CPUs: IBM PowerPC 750FX (Single-core, 900 MHz, 32-bit, radiation hardened) RAM: 256 MB (per processor) OS: VxWorks (Real-time OS) Network: TTEthernet (Time-Triggered Ethernet) at 1 Gbps programming: MISRA C++, flight control laws from Simulink adn MATLAB.
lrvick 7mo ago
NASA describes some impressive work for runtime integrity, but the lack of mention of build-time security is surprising. I would expect to see multi-party-signed deterministic builds etc. Anyone have any insight here?
- ranger207 7mo ago
  What would the threat profile be here to require that? Regardless, I'd be a little surprised if they didn't have anything like that; provenance is very important in aerospace, with hardware tracked to the point that NTSB investigators looking at a crash can tell what ingot a bolt was made from
  - lrvick 7mo ago
    In my experience government just uses RedHat which is -not- reproducible and -not- full source bootstrapped so a single person in the supply chain could maliciously or accidentally backdoor everything. Maybe the goal of the supply chain attacker is just embarrassing the Americans at best or cause a material loss of life at worst. I would -hope- NASA does not trust their OS supply chains to a single person for high risk applications, but given even major companies I audit do this with billions of dollars on the line, it would not shock me if NASA has the same stance which worries me a bit. They would need to be using something like heavily customized buildroot or stagex to produce deterministic OS images.
guenthert 7mo ago
Multiple and dissimilar redundancy is nice and all that, but is there a manual override? Apollo could be (and at least in Apollo 11 and 13 it had to), but is this still possible and feasible? I'd guess so, as it's still manned by (former) test pilots, much like Apollo.
TonyAlicea10 7mo ago
When I was first starting out as a professional developer 25 years ago doing web development, I had a friend who had retired from NASA and had worked on Apollo. I asked him “how did you deal with bugs”? He chuckled and said “we didn’t have them”. The average modern AI-prompting, React-using web developer could not fathom making software that killed people if it failed. We’ve normalized things not working well.
- xavortm 7mo ago
  there's a different level of 'good-enough' in each industry and that's normal. When your highest damage of a bad site is reduced revenue (or even just missed free user), you have lower motivation to do it right compared to a living human coming back in one piece.
  - TonyAlicea10 7mo ago
    Yes, of course, but a culture of “good enough” can go too far. One may work in a lower-risk context, but we can still learn a lot from robust architectural thinking. Edge cases, security, and more. Low quality for a shopping cart feels fine until someone steals all the credit card numbers.
    - jimbob21 7mo ago
      Likewise, perfectionism when it is unneeded can slow teams down to a halt for no reason. The balance in most cases is in the middle, and should shift towards 100% correctness as consequences get more dire. This is not to say your code should be a buggy mess, but 98% bug free when you're a SaaS product and pushing features is certainly better than 100% bug free and losing ground to competitors.
      - TonyAlicea10 7mo ago
        True, though I'd say more bug impact than bug free-ness. If the 2% of bugs is in the most critical area of your app and causes users to abandon your product then you're losing ground. That's one thing I think is good to learn from mission critical architecture: an awareness of the impact and risk tolerance of code and bugs, which means an awareness of how the software will be used and in what context by users.
huxleyFiddler 7mo ago
[flagged]
- Davidbrcz 7mo ago
  > Dissimilar redundancy eliminates that risk. A completely different OS, different codebase, different development team. Not entirely true. I've heard during my uni years of a case were two independent teams used the same textbook for implementing a feature, which had an error, and thus resulting in the same failure mode.
  - ragebol 7mo ago
    Ha, very curious what the issue was and what textbook
- treesknees 7mo ago
  Don't post generated comments or AI-edited comments. HN is for conversation between humans. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
dom111 7mo ago
I always wondered if the "radiation hardening" approaches of the challenges like this https://codegolf.stackexchange.com/questions/57257/radiation-hardened-quine https://codegolf.stackexchange.com/questions/57257/radiation... (see the tag for more https://codegolf.stackexchange.com/questions/tagged/radiation-hardening https://codegolf.stackexchange.com/questions/tagged/radiatio...) would be of any practical use... I assume not, as the problem is on too many levels, but still, seems at least tangentially relevant!
dojopico 7mo ago
I did VOS and database performance stuff at Stratus from 1989-95. Stratus was the hardware fault tolerant company. Tandem, our arch rivals, did software fault tolerance. Our architecture was “pair and spare”. Each board had redundant everything and was paired with a second board. Every pin out was compared on every tick. Boards that could not reset called home. The switch from Motorola 68K to Intel was a nightmare for the hardware group because some instructions had unused pins that could float.
- sillywalk 7mo ago
  > Stratus was the hardware fault tolerant company. Tandem, our arch rivals, did software fault tolerance. Our architecture was “pair and spare”. To expand on this, when Tandem switched MIPS from their proprietary processors, the CPUs were duplicated on a board and compared, and if they disagreed, the logical CPU would halt, similar to Stratus. The software-pair backup processes in a different logical CPU would then take over.
stevepotter 7mo ago
It would be nice to see some of the software source. I’m super interested and i think I helped pay for it
Schlagbohrer 7mo ago
"High-performance supercomputers are used for large-scale fault injection, emulating entire flight timelines where catastrophic hardware failures are introduced to see if the software can successfully ‘fail silent’ and recover." I assume this means they are using a digital twin simulation inside the HPC?
- MutexMaven 7mo ago
  Yes, they leveraged Intel Simics and many other tools like Matlab etc. to have "Digital Twin" simulations. The extensive use of simulators and emulators has been particularly critical, enabling parallel design and development workflows to compensate for the incredibly expensive and long-lead times of hardware. So this helped with bottlenecks in development too. https://ntrs.nasa.gov/api/citations/20190000011/downloads/20190000011.pdf https://ntrs.nasa.gov/api/citations/20190000011/downloads/20...
  - MutexMaven 7mo ago
    Wrong link, but still relevant. https://ntrs.nasa.gov/api/citations/20140009920/downloads/20140009920.pdf https://ntrs.nasa.gov/api/citations/20140009920/downloads/20...
albertzeyer 7mo ago
I'm curious: In the current moon flyby, how often did some of these fallback methods get active? Was the BFS ever in control at any point? How many bitflips were there during the flight so far?
- programmertote 7mo ago
  The same question I wanted to ask. I'd be very curious to learn about their post-mission analysis to find out how many bit flips occurred and how many times this redundant system prevented the mistakes from causing issues.
hpcgroup 7mo ago
[flagged]
bharat1010 7mo ago
The part about triple-redundant voting systems genuinely blew my mind — it's such a different world from how most of us write software day to day, and honestly kind of humbling.
- sebazzz 7mo ago
  I wonder how the voting components are protected from integrity failures?
- doublerabbit 7mo ago
  The Hyperia roller coaster ride at Thorpe Park uses triple-redundant voting. Which I thought was cool. > It’s a complex machine. There’s three computers all talking to each other for a start, and they have to agree on everything. Primary, Real-Time Secondary and Third for regulating votes. https://www.bbc.co.uk/news/articles/ckkknz9zpzgo https://www.bbc.co.uk/news/articles/ckkknz9zpzgo
ck2 7mo ago
if I remember correctly the space shuttle had four computers that all did the same processing and a fifth that decided what was the correct answer if they all didn't match or some went down can't find a wikipedia article on it but the times had an article in 1981 https://www.nytimes.com/1981/04/10/us/computers-to-have-the-last-word-on-shuttle-flight.html https://www.nytimes.com/1981/04/10/us/computers-to-have-the-... apparently the 5th was standby, not the decider
0xblinq 7mo ago
They should have also built a fault tolerant toilette.
PunchyHamster 7mo ago
I wonder how they made the voted-answer-picker fail-resistant
RobRivera 7mo ago
2 outlooks. 2. Two.
pbronez 7mo ago
The Artemis computer handles way more flight functions than Apollo did. What are the practical benefits of that? This electrify & integrate playbook has brought benefits to many industries, usually where better coordination unlocks efficiencies. Sometimes the smarts just add new failure modes and predatory vendor relationships. It’s showing up in space as more modular spacecraft, lower costs and more mission flexibility. But how is this playing out in manned space craft?
eggy 7mo ago
Some related good books I have been studying the past few years or so. The Spark book is written by people who've worked on Cube sats: * Logical Foundations of Cyber-Physical Systems * Building High Integrity Applications with SPARK * Analysable Real-Time Systems: Programmed in Ada * Control Systems Safety Evaluation and Reliability (William M. Goble) I am developing a high-integrity controls system for a prototype hoist to be certified for overhead hoisting with the highest safety standards and targeting aerospace, construction, entertainment, and defense.
veunes 7mo ago
[dead]
rurban 7mo ago
Raft consensus with pairs? I smell bulls*t there. Even when they say it's 8, it boils down to pair-wise checks, without any consensus. Just the consensus of wrong. Also https://en.wikipedia.org/wiki/TTEthernet https://en.wikipedia.org/wiki/TTEthernet looks like bolting time-guaranteed switching networks onto randomizing ethernet hardware. Sounds incredibly cheap and stupid. Either stay with guaranteed real-time switching, or give up on hard real-time guarantees and favor performance, simplicity and cheap stock hardware. Monkeys in space.
- estimator7292 7mo ago
  Extremely bold take to insist you're smarter than people who literally just flew to the actual moon and back.

645 points by speckx 7mo ago

temptemptemp111 7mo ago

[dead]

starkparker 7mo ago

Headline needs its how-dectomy reverted to make sense

arduanika 7mo ago
(Off-topic:) Great word. Is that the usual word for it? Totally apt, and it should be the standard.

dmk 7mo ago

The quote from the CMU guy about modern Agile and DevOps approaches challenging architectural discipline is a nice way of saying most of us have completely forgotten how to build deterministic systems. Time-triggered Ethernet with strict frame scheduling feels like it's from a parallel universe compared to how we ship software now.

tayk47999 7mo ago
> “Modern Agile and DevOps approaches prioritize iteration, which can challenge architectural discipline,” Riley explained. “As a result, technical debt accumulates, and maintainability and system resiliency suffer.” Not sure i agree with the premise that "doing agile" implies decision making at odds with architecture: you can still iterate on architecture. Terraform etc make that very easy. Sure, tech debt accumulates naturally as a byproduct, but every team i've been on regularly does dedicated tech debt sprints. I don't think the average CRUD API or app needs "perfect determinism", as long as modifications are idempotent.
- whiskey-one 7mo ago
  In theory, yes you could iterate on architecture and potentially even come up with better one with agile approach. In practice, so many aspects follow from it that it’s not practical to iterate with today’s tools.
- crabbone 7mo ago
  Agile is like communism. Whenever something bad happens to people who practice agile, the explanation is that they did agile wrong, had they being doing the true agile, the problem would've been totally avoided. In reality, agile doesn't mean anything. Anyone can claim to do agile. Anyone can be blamed for only pretending to do agile. There's no yardstick. But it's also easy to understand what the author was trying to say, if we don't try to defend or blame a particular fashionable ideology. I've worked on projects that required high quality of code and product reliability and those that had no such requirement. There is, indeed, a very big difference in approach to the development process. Things that are often associated with agile and DevOps are bad for developing high-quality reliable programs. Here's why: The development process before DevOps looked like this: 1. Planning 2. Programming 3. QA 4. If QA found problems, goto 2 5. Release The "smart" idea behind DevOps, or, as it used to be called at the time "shift left" was to start QA before the whole of programming was done, in parallel with the development process, so that the testers wouldn't be idling for a year waiting for the developers to deliver the product to testers and the developers would have faster feedback to the changes they make. Iterating on this idea was the concept of "continuous delivery" (and that's where DevOps came into play: they are the ones, fundamentally, responsible to make this happen). Continuous delivery observed that since developers are getting feedback sooner in the development process, the release, too, may be "shifted left", thus starting the marketing and sales earlier. Back in those days, however, it was common to expect that testers will be conducting a kind of a double-blindfolded experiment. I.e. testers weren't supposed to know the ins and outs of the code intentionally, s.t. they don't, inadvertently, side with the developers on whatever issues they discover. Something that today, perhaps, would've been called "black-box testing". This became impossible with CD because testers would be incrementally exposed to the decisions governing the internal workings of the product. Another aspect of the more rigorous testing is the "mileage". Critical systems, normally, aren't released w/o being run intensively for a very long time, typically orders of magnitude longer than the single QA cycle (let's say, the QA gets a day of computer time to run their tests, then the mileage needs to be a month or so). This is a very inconvenient time for development, as feature freeze and code freeze are still in effect, so the coding can only happen in the next version of the product (provided it's even planned). But, the incremental approach used by CD managed to sell a lie that says that "we've ran the program for a substantial amount of time during all the increments we've made so far, therefore we don't need to collect more mileage". This, of course, overlooks the fact that changes in the program don't contribute proportionally to the program's quality or performance. In other words, what I'm trying to say is that agile or DevOps practices allowed to make the development process cheaper by making it faster while still maintaining some degree of quality control, however they are inadequate for products with high quality requirements because they don't address the worst case scenarios.
ramraj07 7mo ago
I take the opposite message from that line - out of touch teams working on something so over budget and so overdue, and so bureaucratic, and with such an insanely poor history of success, and they talk as if they have cured cancer. This is the equivalent of Altavista touting how amazing their custom server racks are when Google just starts up on a rack of naked motherboards and eats their lunch and then the world. Lets at least wait till the capsule comes back safely before touting how much better they are than "DevOps" teams running websites, apparently a comparison that's somehow relevant here to stoke egos.
- danhon 7mo ago
  You mean like this? "With limited funds, Google founders Larry Page and Sergey Brin initially deployed this system of inexpensive, interconnected PCs to process many thousands of search requests per second from Google users. This hardware system reflected the Google search algorithm itself, which is based on tolerating multiple computer failures and optimizing around them. This production server was one of about thirty such racks in the first Google data center. Even though many of the installed PCs never worked and were difficult to repair, these racks provided Google with its first large-scale computing system and allowed the company to grow quickly and at minimal cost." https://blog.codinghorror.com/building-a-computer-the-google-way/ https://blog.codinghorror.com/building-a-computer-the-google...
  - 1970-01-01 7mo ago
    Google then had complete regret not doing this with ECC RAM: https://news.ycombinator.com/item?id=14206811 https://news.ycombinator.com/item?id=14206811
    - ramraj07 7mo ago
      It got them to where they need to be to then worry about ECC. This is like the dudes who deploy their blog on kubernetes just in case it hits front page of new york times or something.
      - 7mo ago
arduanika 7mo ago
You could even say that part of the value of Artemis is that we're remembering how to do some very hard things, including the software side. This is something that you can't fake. In a world where one of the more plausible threats of AI is the atrophy of real human skills -- the goose that lays the golden eggs that trains the models -- this is a software feat where I'd claim you couldn't rely on vibe code, at least not fully. That alone is worth my tax dollars.
- randomNumber7 7mo ago
  Don’t count your chickens before they hatch.
  - arduanika 7mo ago
    I'm not sure you really understood my comment. A large portion of the kind of value I'm talking about comes from attempting the hard thing. If these chickens do not hatch that will be tragic, but we will still have learned something from it. In some ways, we will have learned even more, by getting taught about what we don't know. Anyway, let's all hope for a safe landing tonight.
mvkel 7mo ago
If you look at code as art, where its value is a measure of the effort it takes to make, sure.
- couchand 7mo ago
  If your implication is that stencil art does not take effort then perhaps you may not fully appreciate Banksy. Works like Gaza Kitty or Flower Thrower don’t just appear haphazardly without effort.
- BobbyTables2 7mo ago
  In that case, our test infrastructure belongs in the Louvre…
- stodor89 7mo ago
  Or if you're building something important, like a spaceship.
iknowstuff 7mo ago
Tesla’s Cybertruck uses that in its ethernet as well!
- carefree-bob 7mo ago
  All the ADAS automotive systems use this, there are several startups in this space as well, such as Ethernovia.
  - 21asdffdsa12 7mo ago
    All thanks to twisted pair https://de.wikipedia.org/wiki/BroadR-Reach https://de.wikipedia.org/wiki/BroadR-Reach
dyauspitr 7mo ago
Agile is not meant to make solid, robust products. It’s so you can make product fragments/iterations quickly, with okay quality and out to the customer asap to maximize profits.
- nickff 7mo ago
  “Agile” doesn’t mean that you release the first iteration, it’s just a methodology that emphasizes short iteration loops. You can definitely develop reliable real-time systems with Agile.
  - kermatt 7mo ago
    > “Agile” doesn’t mean that you release the first iteration Someone needs to inform the management of the last three companies I worked for about this.
    - t43562 7mo ago
      Management understand it less than anyone else does.
  - tomasGiden 7mo ago
    I would differentiate between iterative development and incremental development. Incremental development is like panting a picture line by line like a printer where you add new pieces to the final result without affecting old pieces. Iterative is where you do the big brush strokes first and then add more and more detail dependent on what to learn from each previous brush strokes. You can also stop at any time when you think that the final result is good enough. If you are making a new type of system and don’t know what issues will come up and what customers will value (highly complex environment) iterative is the thing to do. But if you have a very predictable environment and you are implementing a standard or a very well specified system (van be highly complicated yet not very complex), you might as will do incremental development. Roughly speaking though as there is of course no perfect specification which is not the final implementation so there are always learnings so there is always some iterative parts of it.
  - 7mo ago
vasco 7mo ago
It's not like the approach they took is any different. Just slapped 8x the number of computers on it for calculating the same thing and wait to see if they disagree. Not the pinnacle of engineering. The equivalent of throwing money at the problem.
- MikeTheGreat 7mo ago
  What happens when they don't?
  - vasco 7mo ago
    If you have a point to make, make it.
    - MikeTheGreat 7mo ago
      What my question is hinting at is that there's actually some really interesting engineering around resolving what happens when the systems disagree. Things like Paxos and Raft help make this much more tractable for mere mortals (like myself); the logic and reasoning behind them are cool and interesting.
      - vasco 7mo ago
        [flagged]
      - FabHK 7mo ago
        Though here the consensus algorithm seems totally different from Paxos/Raft. Rather it's a binary tree, where every non-leaf node compares the (non-silent) inputs from the leaf, and if they're different, it falls silent, else propagates the (identical) results up. Or something something.
      - HeyLaughingBoy 7mo ago
        There really is. We designed a redundant system (software, hardware and mechanisms) a couple years ago. And the problems around figuring out who's in control and how to keep things synchronized across a number of potential failure modes gets really hairy. Sadly, the project was cancelled before we could complete the implementation.
carefree-bob 7mo ago
During the time of the first Apollo missions, a dominant portion of computing research was funded by the defense department and related arms of government, making this type of deterministic and WCET (worst case execution time) a dominant computing paradigm. Now that we have a huge free market for things like online shopping and social media, this is a bit of a neglected field and suffers from poor investment and mindshare, but I think it's still a fascinating field with some really interesting algorithms -- check out the work of Frank Mueller or Johann Blieberger.
- therobots927 7mo ago
  Contrary to propaganda from the likes of Ludwig von Mises, the free market is not some kind of optimal solution to all of our problems. And it certainly does not produce excellent software.
  - psd1 7mo ago
    I can't think of a time when I've found an absolutist position useful or intelligent, in any field. Free-market absolutism is as stupid as totalitarianism. The content of economics papers does not need to be evaluated to discard an extreme position, one need merely say "there are more things in earth and heaven than are dreamed of in your philosophies"
    - staplers 7mo ago
      Great point, if the only constant is change, then philosophy should follow (or lead).
  - nairboon 7mo ago
    Propaganda is quite a strong term to describe the works of an economist. If one wants to debate the ideas of von Mises, it'd be useful to consider the Zeitgeist at that time. Von Mises preferred free markets in contrast to the planned economy of the communists. Partly because the latter has difficulties in proper resource allocation and pricing. Note that this was decades before we had working digital computers and digital communication systems, which, at least in theory, change the feasibility of a planned economy. Also, the last time I checked, the US government produced its goods and services using the free market. The government contractors (private enterprises) are usually tasked with building stuff, compared with the government itself in a non-free, purely planned economy (if you refer to von Mises). I assume that you originally meant to refer to the idea that without government intervention (funding for deep R&D), the free market itself would probably not have produced things like the internet or the moon landing (or at least not within the observed time span). That is, however,a rather interesting idea.
pjmlp 7mo ago
As 70's child that was there when the whole agile took over, and systems engineer got rebranded as devops, I fully agree with them. Add TDD, XP and mob programming as well. While in some ways better than pure waterfall, most companies never adopted them fully, while in some scenarios they are more fit to a Silicon Valley TV show than anything else.
ggm 7mo ago
Time triggered Ethernet is part of aircraft certified data bus and has a deep, decades long history. I believe INRIA did work on this, feeding Airbus maybe. It makes perfect sense when you can design for it. An aircraft is a bounded problem space of inputs and outputs which can have deterministic required minima and then you can build for it, and hopefully even have headroom for extras. Ethernet is such a misnomer for something which now is innately about a switching core ASIC or special purpose hardware, and direct (optical even) connects to a device. I'm sure there are also buses, dual redundant, master/slave failover, you name it. And given it's air or space probably a clockwork backup with a squirrel.
- Arch-TK 7mo ago
  A real squirrel would need acorns, I would assume it's a clockwork squirrel too.
  - sqrrl-io 7mo ago
    A software squirrel, maybe? https://sqrrl.io https://sqrrl.io :-)
- 21asdffdsa12 7mo ago
  Aircraft also have software and components, that form a "working" proclaimed eco-system in lockstep- a baseline. This is why there are paper "additions" on bug discovery until the bug is patched and the whole ecosystem of devices is lifted to the next "baseline".
deleted 7mo ago
[deleted]
guenthert 7mo ago
I think he refers to SpaceWire https://en.wikipedia.org/wiki/SpaceWire https://en.wikipedia.org/wiki/SpaceWire.
anymouse123456 7mo ago
Some of us still work on embedded systems with real-time guarantees. Believe it or not, at least some of those modern practices (unit testing, CI, etc) do make a big (positive) difference there.
- cpgxiii 7mo ago
  The depressing part is that these "modern practices" were essentially invented in the 1960s by defense and aerospace projects like the NTDS, LLRV/LLTV, and Digital Fly-by-Wire to produce safety-critical software, and the rest of the software industry simply ignored them until the last couple of decades.
globnomulous 7mo ago
Microsoft fired all QA people ten or fifteen years ago. I'd imagine it's a similar a story: boxed software needed much higher guarantees of correctness. Digital deliver leaves much more room for error, because it leaves room for easier, cheaper fixes.

ConanRus 7mo ago

[dead]

object-a 7mo ago

How big of a challenge are hardware faults and radiation for orbital data centers? It seems like you’d eat a lot of capacity if you need 4x redundancy for everything

totetsu 7mo ago
They dont go into here.. but I thought that NASA also used like 250nm chips in space for radiation resistance. Are there even any radiation resistance GPUs out there?
- pclmulqdq 7mo ago
  Absolutely not, although the latest fabs with rad-tolerant processors are at ~20 nm. There are FDSOI processes in that generation that I assume can be made radiation-tolerant.
- linzhangrun 7mo ago
  [flagged]
  - tosapple 7mo ago
    [dead]
- kersplody 7mo ago
  NOPE, RAD hardened space parts basically froze on mid 2000s tech: https://www.baesystems.com/en-us/product/radiation-hardened-electronics https://www.baesystems.com/en-us/product/radiation-hardened-...
aidenn0 7mo ago
You don't need 4x redundancy for everything. If no humans are aboard, you have 2x redundancy and immediately reboot if there is a disagreement.
pjerem 7mo ago
Orbital data centers are still nothing more than the current hyperloop.

jbritton 7mo ago

I wonder how often problems happen that the redundancy solves. Is radiation actually flipping bits and at what frequency. Can a sun flare cause all the computers to go haywire.

BitsAndObjects 7mo ago
Not a direct answer but probably as good information as you can get: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35162.pdf https://static.googleusercontent.com/media/research.google.c... Basically, yes, radiation does cause bit flips, more often than you might expect (but still a rare event in the grand scheme of things, but enough to matter). And radiation in space is much “worse” (in quotes because that word is glossing over a huge number of different problems, both just intensity).
- BitsAndObjects 7mo ago
  Typo: “both” ~ “not”
tosapple 7mo ago
[dead]
Tomte 7mo ago
IEC 61508 estimates a soft error rate of about 700 to 1200 FIT (Failure in Time, i.e. 1E-9 failures/hour). That was in the 2000s though, and for embedded memory above 65nm. And obviously on earth.

y1n0 7mo ago

NASA didn't build this, Lockheed Martin and their subcontractors did. Articles and headlines like this make people think that NASA does a lot more than they actually do. This is like a CEO claiming credit for everything a company does.

voodoo_child 7mo ago
Nice “well, actually”. I’m sure Lockheed were building this quad-redundant, radiation-hardened PowerPC that costs millions of dollars and communicates via Time-Triggered Ethernet anyway, whether NASA needed one or not.
- y1n0 7mo ago
  This is the equivalent of prompt engineering.
- kube-system 7mo ago
  Probably, if it already wasn’t developed for DoD. For example, the OS it seems to be running is integrity 178. https://www.ghs.com/products/safety_critical/integrity_178_success-nasa_orion.html https://www.ghs.com/products/safety_critical/integrity_178_s... Aerospace tech is not entirely bespoke anymore, plenty of the foundational tech is off the shelf. Historically, the main difference between ICBM tech and human spaceflight tech is the payload and reentry system.
Sebguer 7mo ago
will nobody think of the megacorps!!!
adrian_b 7mo ago
Lockheed Martin and their subcontractors did the implementation. We do not know how much of the high-level architecture of the system has been specified by NASA and how much by Lockheed Martin.
- y1n0 7mo ago
  I do.

seemaze 7mo ago

and yet.. https://news.ycombinator.com/item?id=47615490 https://news.ycombinator.com/item?id=47615490

adrian_b 7mo ago
That was a laptop, not one of the Artemis computers.

__d 7mo ago

Does anyone have pointers to some real information about this system? CPUs, RAM, storage, the networking, what OS, what language used for the software, etc etc? I’d love to know how often one of the FCMs has “failed silent”, and where they were in the route and so on too, but it’s probably a little soon for that.

anthonj 7mo ago
Nasa CFS, is written is plain C (trying to follow MISRA C, etc). It's open on girhub abd used by many companies. It's typically run over freertos or RTEMS, not sure here. Personally I find the project extremely messy, and kinda hate working with it.
- sheepybloke 7mo ago
  It's most likely using vxworks for it's OS, since I believe it's one of the only fully certified ARINC653 OS's for human flight. It's used in most Aircraft and space missions.
  - __d 7mo ago
    Yeah, that was my guess too but the comment about separate implementation for the backup system made me wonder if there was a different OS, and the which was running where.
j4k0bfr 7mo ago
Not sure about the primary FSW but the BFS uses cFS[0]. As the sibling comment mentions, you can check it out on GitHub. Sadly I believe NASA keeps most of their best code private, probably siloed into mission-specific codebases. Still, the cFS repo is an awesome crash course on old-school Flight Software techniques. [0] https://youtu.be/4doI2iQe4Jk?si=ucMoIdw7x_QgZR32 https://youtu.be/4doI2iQe4Jk?si=ucMoIdw7x_QgZR32
- __d 7mo ago
  Helpful video, thanks! At about 1:20, the presenter says the BFS uses a different OS and hardware (not sure if that means a different instance, or a different class, so to speak).

nickpsecurity 7mo ago

The ARINC scheduler, RTOS, and redundancy have been used in safety-critical for decades. ARINC to the 90's. Most safety-critical microkernels, like INTEGRITY-178B and LynxOS-178B, came with a layer for that. Their redundancy architecture is interesting. I'd be curious of what innovations went into rad-hard fabrication, too. Sandia Secure Processor (aka Score) was a neat example of rad-hard, secure processors. Their simulation systems might be helpful for others, too. We've seen more interest in that from FoundationDB to TigerBeetle.

geomark 7mo ago

I sure wish they would talk about the hardware. I spent a few years developing a radiation hardened fault tolerant computer back in the day. Adding redundancy at multiple levels was the usual solution. But there is another clever check on transient errors during process execution that we implemented that didn't involve any redundancy. Doesn't seem like they did anything like that. But can't tell since they don't mention the processor(s) they used.

themafia 7mo ago
One of the things I loved about the Shuttle is that all five computers were mounted not only in different locations but in different orientations in the shuttle. Providing some additional hardening against radiation by providing different cross sections to any incident event.
MathMonkeyMan 7mo ago
I read, for probe missions, that one technique is to get a bunch of consumer chips and irradiate the hell out of them. Now take the winner model and get a bunch of those. Irradiate them. The winner goes to Mars. The claim was that some plain old chips are exquisitely radiation resisitant, and it's not clear why.
MutexMaven 7mo ago
NASA actually publishes these things on their NTRS page. The Primary flight controller is rocking Green Hills INTEGRITY RTOS on BAE RAD750s in a quad redundant config, with a VxWorks backup on a Frontgrade Gaisler LEON4 (SPARC V8). This allowed for parts of the ARINC653 spec regarding time and space partitioning of the RTOS scheduler to be used. You can read more about it below (when the server throwing errors). https://ntrs.nasa.gov/api/citations/20190000011/downloads/20190000011.pdf https://ntrs.nasa.gov/api/citations/20190000011/downloads/20... https://ntrs.nasa.gov/api/citations/20230002185/downloads/FSW2023_Orion_BFS_Hirsh_final.pptx.pdf https://ntrs.nasa.gov/api/citations/20230002185/downloads/FS...

georgehm 7mo ago

>Effectively, eight CPUs run the flight software in parallel. The engineering philosophy hinges on a >“fail-silent” design. The self-checking pairs ensure that if a CPU performs an erroneous calculation >due to a radiation event, the error is detected immediately and the system responds. >“A faulty computer will fail silent, rather than transmit the ‘wrong answer,’” Uitenbroek explained. >This approach simplifies the complex task of the triplex “voting” mechanism that compares results. > >Instead of comparing three answers to find a majority, the system uses a priority-ordered source >selection algorithm among healthy channels that haven’t failed-silent. It picks the output from the >first available FCM in the priority list; if that module has gone silent due to a fault, it moves to >the second, third, or fourth. One part that seems omitted in the explanation is what happens if both CPUs in a pair for whatever reason performs an erroneous calculation and they both match, how will that source be silenced without comparing its results with other sources.

themafia 7mo ago
In the Shuttle they would use command averaging. All four computers would get access to an actuator which would tie into a manifold which delivered power to the flight control surface. If one disagreed then you'd get 25% less command authority to that element.
- JumpCrisscross 7mo ago
  > In the Shuttle they would use command averaging I think the Shuttle, operating only in LEO, had more margin for error. Averaging a deep-space burn calculation is basically the same as killing the crew.
  - themafia 7mo ago
    The GNC loop runs several times per second. The desired output will consequently be increased by the working computers to achieve the target. The computer does not "dead reckon" anything. Travelling through Max-Q in Earth atmosphere on ascent is far more dangerous.

spaceman123 7mo ago

Probably same way they’ve built fault-tolerant toilet.

jeron 7mo ago
ctrl+f toilet, thank you for already commenting this

SeanAnderson 7mo ago

Typo in the first sentence of the first paragraph is oddly comforting since AI wouldn't make such a typo, heh. Typo in the first sentence of the second paragraph is sad though. C'mon, proofread a little.

tux 7mo ago
I think everyone should now make mistakes so we ca distinguish human vs ai.
- zeristor 7mo ago
  This can be optimised for no doubt, adversarial training is like that
deleted 7mo ago
[deleted]

vhiremath4 7mo ago

> “Along with physically redundant wires, we have logically redundant network planes. We have redundant flight computers. All this is in place to cover for a hardware failure.” It would be really cool to see a visualization of redundancy measures/utilization over the course of the trip to get a more tangible feel for its importance. I'm hoping a bunch of interesting data is made public after this mission!

hulitu 7mo ago

They run 2 Outlook instances. For redundancy. /s

ajaystream 7mo ago

[flagged]

adrian_b 7mo ago
This is similar to the difference between using error-correcting codes and using erasure codes combined with error-detecting codes. The latter choice is frequently simpler and more reliable for preventing data corruption. (An erasure code can be as simple as having multiple copies and using the first good copy.)
sammy2255 7mo ago
Spoken like an LLM.
high_na_euv 7mo ago
How you can remove component from decision set if it is the only component in the whole decision set?
randomNumber7 7mo ago
> make each unit responsible for detecting its own faults and shutting up if it can't guarantee correctness Does this mean you have to trust the already compromised system?

perarneng 7mo ago

[dead]

JumpCrisscross 7mo ago

Does anyone know how this compares to Crew Dragon or HLS?

gambiting 7mo ago

So honest and perhaps a bit stupid question. Astronauts have actual phones with them - iPhones 17 I think? And a regular Thinkpad that they use to upload photos from the cameras. How does all of that equipment work fine with all the cosmic radiation floating about? With the iPhone's CPU in particular, shouldn't random bit flips be causing constant crashes due to errors? Or is it simply that these errors happen but nothing really detects them so the execution continues unhindered?

BitsAndObjects 7mo ago
They’re not mission-critical equipment. If they fail, nobody dies. They’re not radiation hardened, so given enough time, they’d be expected to fail. Rebooting them might clear the issue or it might not (soft vs hard faults). Also impossible to predict when a failure would happen, but NASA, ESA and others have data somewhere that makes them believe the risk is high enough that mission critical systems need this level of redundancy.
- gambiting 7mo ago
  >>They’re not mission-critical equipment. If they fail, nobody dies. Yes, for sure, but that's not my question - it's not a "why is this allowed" but "why isn't this causing more visible problems with the iphones themselves". Like, do they need constant rebooting? Does this cause any noticable problems with their operation? Realistically, when would you expect a consumer grade phone to fail in these conditions?
  - BitsAndObjects 7mo ago
    Random bit flips due to radiation are infrequent - the stat is something like one but flip per megabyte per 40,000 data centre RAM modules per year - ie extremely uncommon, but common enough to matter at scale. Space is a harsher environment but they’re only up there for like a week. So, if there were an incident, it would be more likely to kill the devices, but it’s not very likely to happen during the short period of time (while still being more likely than on earth’s surface). That said, part of the point of them taking these devices up is to find out how well they perform in practice. We just don’t really know how these consumer devices perform in space. It will be interesting to see the results when they’re published!

GautamB13 7mo ago

It kinda crazy how this mission didn't become mainstream media until as of late.

kev009 7mo ago

Some people are claiming it's the good old RAD750 variant. Is there anything that talks about the actual computer architecture? The linked article is desperately void of technical details.

u1hcw9nx 7mo ago
It's a new (2002) variant of the same RAD750 architecture. CPUs: IBM PowerPC 750FX (Single-core, 900 MHz, 32-bit, radiation hardened) RAM: 256 MB (per processor) OS: VxWorks (Real-time OS) Network: TTEthernet (Time-Triggered Ethernet) at 1 Gbps programming: MISRA C++, flight control laws from Simulink adn MATLAB.

lrvick 7mo ago

NASA describes some impressive work for runtime integrity, but the lack of mention of build-time security is surprising. I would expect to see multi-party-signed deterministic builds etc. Anyone have any insight here?

ranger207 7mo ago
What would the threat profile be here to require that? Regardless, I'd be a little surprised if they didn't have anything like that; provenance is very important in aerospace, with hardware tracked to the point that NTSB investigators looking at a crash can tell what ingot a bolt was made from
- lrvick 7mo ago
  In my experience government just uses RedHat which is -not- reproducible and -not- full source bootstrapped so a single person in the supply chain could maliciously or accidentally backdoor everything. Maybe the goal of the supply chain attacker is just embarrassing the Americans at best or cause a material loss of life at worst. I would -hope- NASA does not trust their OS supply chains to a single person for high risk applications, but given even major companies I audit do this with billions of dollars on the line, it would not shock me if NASA has the same stance which worries me a bit. They would need to be using something like heavily customized buildroot or stagex to produce deterministic OS images.

guenthert 7mo ago

Multiple and dissimilar redundancy is nice and all that, but is there a manual override? Apollo could be (and at least in Apollo 11 and 13 it had to), but is this still possible and feasible? I'd guess so, as it's still manned by (former) test pilots, much like Apollo.

TonyAlicea10 7mo ago

When I was first starting out as a professional developer 25 years ago doing web development, I had a friend who had retired from NASA and had worked on Apollo. I asked him “how did you deal with bugs”? He chuckled and said “we didn’t have them”. The average modern AI-prompting, React-using web developer could not fathom making software that killed people if it failed. We’ve normalized things not working well.

xavortm 7mo ago
there's a different level of 'good-enough' in each industry and that's normal. When your highest damage of a bad site is reduced revenue (or even just missed free user), you have lower motivation to do it right compared to a living human coming back in one piece.
- TonyAlicea10 7mo ago
  Yes, of course, but a culture of “good enough” can go too far. One may work in a lower-risk context, but we can still learn a lot from robust architectural thinking. Edge cases, security, and more. Low quality for a shopping cart feels fine until someone steals all the credit card numbers.
  - jimbob21 7mo ago
    Likewise, perfectionism when it is unneeded can slow teams down to a halt for no reason. The balance in most cases is in the middle, and should shift towards 100% correctness as consequences get more dire. This is not to say your code should be a buggy mess, but 98% bug free when you're a SaaS product and pushing features is certainly better than 100% bug free and losing ground to competitors.
    - TonyAlicea10 7mo ago
      True, though I'd say more bug impact than bug free-ness. If the 2% of bugs is in the most critical area of your app and causes users to abandon your product then you're losing ground. That's one thing I think is good to learn from mission critical architecture: an awareness of the impact and risk tolerance of code and bugs, which means an awareness of how the software will be used and in what context by users.

huxleyFiddler 7mo ago

[flagged]

Davidbrcz 7mo ago
> Dissimilar redundancy eliminates that risk. A completely different OS, different codebase, different development team. Not entirely true. I've heard during my uni years of a case were two independent teams used the same textbook for implementing a feature, which had an error, and thus resulting in the same failure mode.
- ragebol 7mo ago
  Ha, very curious what the issue was and what textbook
treesknees 7mo ago
Don't post generated comments or AI-edited comments. HN is for conversation between humans. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

dom111 7mo ago

I always wondered if the "radiation hardening" approaches of the challenges like this https://codegolf.stackexchange.com/questions/57257/radiation-hardened-quine https://codegolf.stackexchange.com/questions/57257/radiation... (see the tag for more https://codegolf.stackexchange.com/questions/tagged/radiation-hardening https://codegolf.stackexchange.com/questions/tagged/radiatio...) would be of any practical use... I assume not, as the problem is on too many levels, but still, seems at least tangentially relevant!

dojopico 7mo ago

I did VOS and database performance stuff at Stratus from 1989-95. Stratus was the hardware fault tolerant company. Tandem, our arch rivals, did software fault tolerance. Our architecture was “pair and spare”. Each board had redundant everything and was paired with a second board. Every pin out was compared on every tick. Boards that could not reset called home. The switch from Motorola 68K to Intel was a nightmare for the hardware group because some instructions had unused pins that could float.

sillywalk 7mo ago
> Stratus was the hardware fault tolerant company. Tandem, our arch rivals, did software fault tolerance. Our architecture was “pair and spare”. To expand on this, when Tandem switched MIPS from their proprietary processors, the CPUs were duplicated on a board and compared, and if they disagreed, the logical CPU would halt, similar to Stratus. The software-pair backup processes in a different logical CPU would then take over.

stevepotter 7mo ago

It would be nice to see some of the software source. I’m super interested and i think I helped pay for it

Schlagbohrer 7mo ago

"High-performance supercomputers are used for large-scale fault injection, emulating entire flight timelines where catastrophic hardware failures are introduced to see if the software can successfully ‘fail silent’ and recover." I assume this means they are using a digital twin simulation inside the HPC?

MutexMaven 7mo ago
Yes, they leveraged Intel Simics and many other tools like Matlab etc. to have "Digital Twin" simulations. The extensive use of simulators and emulators has been particularly critical, enabling parallel design and development workflows to compensate for the incredibly expensive and long-lead times of hardware. So this helped with bottlenecks in development too. https://ntrs.nasa.gov/api/citations/20190000011/downloads/20190000011.pdf https://ntrs.nasa.gov/api/citations/20190000011/downloads/20...
- MutexMaven 7mo ago
  Wrong link, but still relevant. https://ntrs.nasa.gov/api/citations/20140009920/downloads/20140009920.pdf https://ntrs.nasa.gov/api/citations/20140009920/downloads/20...

albertzeyer 7mo ago

I'm curious: In the current moon flyby, how often did some of these fallback methods get active? Was the BFS ever in control at any point? How many bitflips were there during the flight so far?

programmertote 7mo ago
The same question I wanted to ask. I'd be very curious to learn about their post-mission analysis to find out how many bit flips occurred and how many times this redundant system prevented the mistakes from causing issues.

hpcgroup 7mo ago

[flagged]

bharat1010 7mo ago

The part about triple-redundant voting systems genuinely blew my mind — it's such a different world from how most of us write software day to day, and honestly kind of humbling.

sebazzz 7mo ago
I wonder how the voting components are protected from integrity failures?
doublerabbit 7mo ago
The Hyperia roller coaster ride at Thorpe Park uses triple-redundant voting. Which I thought was cool. > It’s a complex machine. There’s three computers all talking to each other for a start, and they have to agree on everything. Primary, Real-Time Secondary and Third for regulating votes. https://www.bbc.co.uk/news/articles/ckkknz9zpzgo https://www.bbc.co.uk/news/articles/ckkknz9zpzgo

ck2 7mo ago

if I remember correctly the space shuttle had four computers that all did the same processing and a fifth that decided what was the correct answer if they all didn't match or some went down can't find a wikipedia article on it but the times had an article in 1981 https://www.nytimes.com/1981/04/10/us/computers-to-have-the-last-word-on-shuttle-flight.html https://www.nytimes.com/1981/04/10/us/computers-to-have-the-... apparently the 5th was standby, not the decider

0xblinq 7mo ago

They should have also built a fault tolerant toilette.

PunchyHamster 7mo ago

I wonder how they made the voted-answer-picker fail-resistant

RobRivera 7mo ago

2 outlooks. 2. Two.

pbronez 7mo ago

The Artemis computer handles way more flight functions than Apollo did. What are the practical benefits of that? This electrify & integrate playbook has brought benefits to many industries, usually where better coordination unlocks efficiencies. Sometimes the smarts just add new failure modes and predatory vendor relationships. It’s showing up in space as more modular spacecraft, lower costs and more mission flexibility. But how is this playing out in manned space craft?

eggy 7mo ago

Some related good books I have been studying the past few years or so. The Spark book is written by people who've worked on Cube sats: * Logical Foundations of Cyber-Physical Systems * Building High Integrity Applications with SPARK * Analysable Real-Time Systems: Programmed in Ada * Control Systems Safety Evaluation and Reliability (William M. Goble) I am developing a high-integrity controls system for a prototype hoist to be certified for overhead hoisting with the highest safety standards and targeting aerospace, construction, entertainment, and defense.

veunes 7mo ago

[dead]

rurban 7mo ago

Raft consensus with pairs? I smell bulls*t there. Even when they say it's 8, it boils down to pair-wise checks, without any consensus. Just the consensus of wrong. Also https://en.wikipedia.org/wiki/TTEthernet https://en.wikipedia.org/wiki/TTEthernet looks like bolting time-guaranteed switching networks onto randomizing ethernet hardware. Sounds incredibly cheap and stupid. Either stay with guaranteed real-time switching, or give up on hard real-time guarantees and favor performance, simplicity and cheap stock hardware. Monkeys in space.

estimator7292 7mo ago
Extremely bold take to insist you're smarter than people who literally just flew to the actual moon and back.