13 ms·
How to avoid a BSOD on your 2B dollar spacecraft
- sharpshadow 2y agoOne must have balls of steel to run windows on a spaceship.
- perching_aix 2y agoPersonally, I wouldn't be stoked to run Linux on them either to be honest. But both are being done. Practicality rules I suppose.
- anonzzzies 2y agoHaving sourcecode to everything would make me trust things more as at least we could fix things without calling MS. But what would you run? QNX? BSD?
- withinboredom 2y agoJust because you have the source code, doesn't mean you have the knowledge to fix it before you die.
- anonzzzies 2y agoSure, but without it, you stand no chance at all.
- exe34 2y agowindows is source-available if you have deep enough pockets: https://www.microsoft.com/en-us/sharedsource/enterprise-source-licensing-program.aspx https://www.microsoft.com/en-us/sharedsource/enterprise-sour...
- anonzzzies 2y agoAnyway, I was responding to someone who wouldn't run either on a spaceship so I still would like to know what they would want to run. I am from a formal verification school of thought, so I would want something sel4.
- exe34 2y agoI'd want manual backup pushbuttons.
- foobar1962 2y agoAnd another box close enough so the CD tray can press them when it opens.
- 0cf8612b2e1e 2y agoDoes Microsoft also provide you the tools to build it? I assume there are many Microsoft internal tools, libraries, etc required to compile anything of note. Presumably it has been dog fooded for so long it would be impossible to bootstrap without some number of binary artifacts in hand.
- withinboredom 2y agoI feel like this misses the point so much that it might as well be nonsense. A better way to put my argument is: could an average mom build Linux on specialized hardware in space? If the answer is "yes", then you may have a point. I don't think the answer is yes.
- johnisgood 2y agoNo, but you can review the source code before (or at any point), whereas with Windows, you cannot even do that.
- withinboredom 2y agoI've occasionally worked on drivers for windows and linux. In either case, I didn't really need to read the source code; neither was it a valuable proposition. If the advertised API didn't do what it was supposed to, I likely wouldn't have understood enough to fix it: and this is my point. Just because you can read it, doesn't mean you can or will be able to actually fix it; not because of technicality, but because of personal knowledge. In this case, they are both black boxes.
- johnisgood 2y agoI mean, I agree, I am just saying that it is better (in general) to have the source than not having it.
- withinboredom 2y agoHow so? I once spent three days trying to figure out an issue, stepping line by line through hadoop (after figuring out the issue was in hadoop and not my own code). Yay, I proved the issue was actually in Java itself. Guess what happened next? We avoided the bug. Why? - We couldn't update Java. - We couldn't change hadoop because we were using a packaged solution. So, we just filed a bug with them. Had the source not been available, we would have just skipped all of that, and it would have been our vendor's problem 3 days earlier.
- icehawk 2y agoFor something like the spacecraft in the article, you absolutely would have the ability to get access to the Windows source code.
- exe34 2y agowindows is source-available if you have deep enough pockets: https://www.microsoft.com/en-us/sharedsource/enterprise-source-licensing-program.aspx https://www.microsoft.com/en-us/sharedsource/enterprise-sour...
- GlenTheMachine 2y agoVxWorks, usually
- redleader55 2y agoIf you ever count how many patches are in between stable point releases - eg. 6.10.5 to 6.10.6, you'll see having the source code is not enough. All of these patches are fixes for something or another, not features, but fixes. If you look at an LTS branch, you'll see there are hundreds of point releases. Usually a point release is created once every 5-10 days. I interpret that to mean bugs were not found until many weeks after the LTS branch was cut. Obviously, not all of them affect you, but many patches apply important subsystems which affect you.
- ssrc 2y agoVxWorks, LynxOS or RTEMS. RTEMS is open source.
- inamberclad 2y ago+1 for RTEMS! Great fully featured RTOS
- perching_aix 2y agoMaybe sel4? I'm not in embedded so maybe I'm being pretty silly here, but I think being formally proven and realtime are pretty good things to have in the case of an expensive, long lived space project. That said, this is on the scale that I'm surprised off-the-shelf things are even being considered. I'd have thought they'd just roll something bespoke and that's the end of that.
- rurban 2y agoBaremetal of course. If it gets complicated (ie threads): QNX, vxworks, L4.
- wubrr 2y agoWhat's the practical benefit of using Windows here?
- wil421 2y agoThe new galactic empire has an enterprise licensing agreement with Microsoft.
- jandrese 2y agoEasier to find developers who are comfortable with Windows.
- wubrr 2y agoProbably not the case in spacecraft engineering. https://news.ycombinator.com/item?id=41650534#41651310 https://news.ycombinator.com/item?id=41650534#41651310
- gosub100 2y agoMaybe technically you could save $89 and run pirated windows because of (possibly?) no jurisdiction in space?
- jmclnx 2y agoReally, with Linux (or a BSD), you can make tweaks for free to save memory and to only allow specific tasks and hardware. Plus you could publish these changes and maybe they will be accepted by upstream. With Windows, you need to beg and pay Microsoft for customizations and hope these changes will not cause other issues. Plus, most space projects are on a tight and limited budgets where management would rather spend on hardware than software.
- dylan604 2y agoWell, if your account is only a Home Edition, you will not get the same support as if you upgrade to Universal Galactic Edition which has a LTS measured in generations.
- freedomben 2y agoAlso, if you want RDP then I think you need to upgrade to Universal Galactic Edition
- pjmlp 2y agoOnly if the ground control station has more than two persons.
- hypeatei 2y agoWonder if it's LTSC or the standard image with candy crush, windows store, and Xbox game app bloat?
- lpribis 2y agoDon't know if it's publically available, but MS makes Windows IoT which is a stripped down distro for embedded systems.
- geepytee 2y agoGenuine question, what would you use instead and why?
- MPSimmons 2y agoDepending on the complexity of the satellite, a linux node, several linux nodes, or a LOT of linux nodes. Or a very small embedded SoC if your satellite is very simple or has a segregated payload.
- jasonwatkinspdx 2y agoReal Time Operating System(s) like VxWorks. Commodity equipment running specialized distros of Linux is a growing thing however.
- MPSimmons 2y agoLinux just mainlined the RTOS patch that a lot of people have used. Including SpaceX - https://ntrs.nasa.gov/api/citations/20200002390/downloads/20200002390.pdf https://ntrs.nasa.gov/api/citations/20200002390/downloads/20... Linux RT merge: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=baeb9a7d8b60b021d907127509c44507539c15e5 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
- gosub100 2y agoWhat parts would actually need realtime response? Only thing I can think of would be thrusters, but wouldn't that be solved by an asic? Essentially a network command where you say "I don't care specifically when you do this, but I'm requesting a correction of M-magnitude in this 3-vector. If I don't get an ack within 500ms cancel. "
- jasonwatkinspdx 2y agoNavigation / attitude control needs to be real time. Even if there's a microcontroller at the device, the guidance system receives telemetry and then sends commands. If the send is arbitrarily delayed then the resulting motion won't be what the system desired. The microcontroller can't do the complex calculations on its own because it doesn't have all the telemetry from multiple sensors. Also RTOS tend to have a lot of verification work applied so that you can be confident some facilities just won't fail.
- wongarsu 2y agoThe article doesn't actually involve Windows. They wanted to avoid a satellite going into safemode, which they describe as "the satellite equivalent of a blue screen of death". That's honestly not even a good analogy. The headline is just bad
- hunter2_ 2y agoFor anyone reading only headlines and comments, from TFA: > If the watchdog timer has not been restarted and instead times out after ~30 seconds, the satellite enters something called safemode. Safemode is when all non critical functions are automatically shut down and the satellite becomes entirely focused on generating power by pointing its solar panels towards the Sun and trying to reestablish any communication that was lost. It’s a state the vehicle goes into when something bad happens [...] the satellite equivalent of a blue screen of death. If only Windows would be so kind!
- neuralRiot 2y agoInstalling update 1 of 456. Please don’t turn off your spaceship.
- lisper 2y agoThey're not running Windows. https://news.ycombinator.com/item?id=41651715 https://news.ycombinator.com/item?id=41651715
- dekhn 2y ago1998: Windows 95 on Thinkpads (not running system-critical) https://www.nytimes.com/1998/11/05/technology/laptops-on-the-shuttle-age-does-not-matter.html https://www.nytimes.com/1998/11/05/technology/laptops-on-the... Detailed writeup by project manager: https://forum.nasaspaceflight.com/index.php?topic=27043.0 https://forum.nasaspaceflight.com/index.php?topic=27043.0 (doesn't mention the OS) Some indication the laptops on ISS run linux on HP ZBooks: https://training.linuxfoundation.org/solutions/corporate-solutions/success-stories/linux-foundation-training-prepares-the-international-space-station-for-linux-migration/ https://training.linuxfoundation.org/solutions/corporate-sol...
- GlenTheMachine 2y agoThee are a bunch of comments here asking why one would run Windows on a spacecraft. I am a spacecraft engineer. I don’t see anything in the linked article indicating that they are actually running Windows - the BSOD claim is tongue-in-cheek, or at least that’s how I read it. I also don’t know of anyone anywhere that runs Windows on a spacecraft, with the exception of laptops used by astronauts. Typically one runs vxWorks, or maybe QNX. Some experimental (high risk, low cost) systems run Linux. Older spacecraft don't run any OS at all, everything is running on bare metal, and that may be true for a handful of current spacecraft as well. Windows is used in some places by ground controllers, but these days they tend to be running Linux a lot more often.
- TrueDuality 2y agoSeconding the vxWorks and bare metal. Never seen Windows or Linux on a satellite bus. Haven't really touched payloads but I've seen some wonky things shipped to orbit by universities and not all them have been cubesat student projects.
- nicce 2y agoEvery Starlink runs with Linux. The license list is a bit long: https://www.starlink.com/assets/pdfs/Starlink-Open-Source-Components-License-Notices.pdf https://www.starlink.com/assets/pdfs/Starlink-Open-Source-Co...
- MichaelZuo 2y agoIt's surprising they didn't pick a RTOS.
- gosub100 2y agoWhy? What benefit would that give them?
- deleted 2y ago[deleted]
- aghilmort 2y ago[flagged]
- farceSpherule 2y agoOr you can avoid contracting with Boeing.
- rdist 2y agoAnd here I thought we were going to rehash Crowdstrike ;-)
- TrueDuality 2y agoJust a tactful reference hahah > the US government isn’t burning taxpayer dollars on a ten figure spaceship just to have us push a Crowdstrike update on it.
- gosub100 2y agoI know you kid, but theoretically running crowdstrike-susceptible windows on a spacecraft would work fine (at least I claim so), because you'd need a robust backdoor / OOB into it anywa (And I'm no windows fanboy, I hate them just as much as the next guy). Crowdstrike bug would cause an N-day loss of comms just like a thousand other things they plan for in a spacecraft.
- linebeck 2y agoAuthor here: I should clarify the satellite is not running Windows. Instead, it’s running its own custom OS written in C called Flight Software (FSW) specifically designed for the satellite onboard computer. Re-reading the post, I see how the title, my analogies, and poor attempts at humor would give the incorrect description of what’s happening with the satellite when it enters safemode. I’ll amend the post soon. Thanks for the feedback, I’ll be better next time.
- barbegal 2y agoCould I ask you to clarify why avoiding safemode is so important? In a non satellite system safemode means everything is driven to a safe state which is fine during testing in the lab. Also do you not run these tests in an even more simulated environment where there is only the flight computer and no real hardware at all?
- linebeck 2y agoHaving discussed this same question with the more experienced members of my team, the only conclusion I can draw is that the customer (US Government) is incredibly risk averse. Any unexpected entry into safemode would require a report, multiple meetings with the customer, and them being pretty angry. Their line of reasoning seems to be "Safemode->Something is wrong->Why is something wrong? We're not paying you to be wrong". I'm personally of the opinion that safemode isn't that bad. It's fully recoverable and shows the system is working properly. We normally have a Functional Test Assembly (real computer and some other hardware for testing) to run our tests against, but we only have one setup and it is consistently unreliable. This particular CLT was unable to get a clean run in the lab but it was decided that the issues were related to the lab setup rather than the actual test, so we moved forward to run on the satellite (against our team's protests). This to me is the real crux of the issue: if we can't even trust our own testing environment, what's the point of having it at all? If the customer is so risk averse, why would we take this chance? Needless to say, I don't think we'll be running anything on the satellite without full FTA vetting anytime in the near future.
- 2y ago
- dangoodmanUT 2y agoStep 1: Use linux
- ksajh 2y agoStep 1: Read and understand the article
- imoverclocked 2y agoStep 2: install vxworks
- rurban 2y agoSuch systems need to use a secure and trusted OS, if an OS is actually needed. Many get away without one. The problem was that their parameter file had no header for the machine type. Unchecked memcpy in space is seldom good, and you are lucky if it trips a shutdown. Rotating satellites don't tend to unrotate in time, and very soon they are gone.
- jesprenj 2y agoWas the spacecraft from the event described in the article an actual spacecraft in space or a simulation of a space mission on the ground?
- MadnessASAP 2y agoA simulation of a space mission on the ground with a satellite that will eventually be in space. Take your satellite, replace it's navigation/communication inputs with ones generated by a reasonably high fidelity real time physical simulation. Feed it's outputs back into said simulation. Ensure the satellite does the right things at the right time.
- LorenPechtel 2y agoWhy is it using memory-mapped stuff in the first place rather than some sort of messaging system that would allow more defensive programming?
- gavinsyancey 2y agoIf I were to guess -- At the lowest level, that's what the hardware does. I wouldn't be surprised if they have a library to wrap that with something more friendly, but something has to translate that into writes to memory-mapped addresses, and whatever that is was configured with the wrong addresses...
- jdiez17 2y agoTo the shock and horror of many programming-inclined people, it turns out that having "arbitrary memory read/write" commands on a remote computer that *must* keep the mission going is quite useful. I can tell you a little first-hand account of where this helped a satellite formation flight mission I worked on. The communication system was working fine in terms of signal strength, but many command packets were ignored (no response). We were able to figure out that a message queue in the processing pipeline was considered full, by strategically reading certain memory locations. We then sent a memory patch to the satellite to skip some of the processing steps and this improved the communication system dramatically. We also brought back to life the first CubeSat launched by my university, BEESAT-1, by using arbitrary memory write to patch some telemetry collection software to avoid a damaged section of the onboard flash. Pretty cool story actually.
- 0xDEAFBEAD 2y agoIt occurs to me that writing software for spacecraft could demand an entirely different paradigm than writing software for traditional applications. For example, you could use an OS that is deterministic down to the last detail, and have a "digital twin" / virtual machine of the spacecraft computer here on Earth, kept in sync with all sensor and actuator activity out in space. Before issuing any command to the spacecraft, you branch the digital twin, issue the command on the branch, and make sure everything looks good. With this method, you wouldn't even need to read memory locations on the spacecraft, you could just read them on the digital twin. Then test the memory patch on the digital twin and make very sure that it won't brick your spacecraft before you transmit it.
- joelkevinjones 2y agoAs much as I hate writing "getter" functions for referencing global variables, I would when I knew I didn't have the right address yet. Write them first to error out loudly, then when you have the actual addresses replace the error out code.
- jwrallie 2y agoI would bet the schedule didn't allow much time to doing subsystem level test with on-board computer, so everyone went to the big test praying for the best. That or inexperienced programmers were involved, assuming they were not scared of modifying memory addresses directly. As for the safe-mode, if it happened maybe you could say you were randomly injecting errors in the memory during runtime and spacecraft entered safe mode as expected, would not be far off from the truth, just do not mention it was unintended :)
- taspeotis 2y agohttps://www.fastcompany.com/28121/they-write-right-stuff https://www.fastcompany.com/28121/they-write-right-stuff
- PoignardAzur 2y ago> I think what surprised me the most was how nonchalant the response was. We had documented all of our actions, so other people had read what happened and knew something had gone on. I wasn’t expecting any fanfare but we weren’t even debriefed on what happened. That's... Concerning. No root cause analysis? Not even an internal one?
- bronlund 2y agoClickbait. Unlike british missile submarines, they are not using Windows.
- pif 2y agoVery simple: just Write the Right Stuff! https://www.eng.auburn.edu/~kchang/comp6710/readings/They%20Write%20the%20Right%20Stuff.pdf https://www.eng.auburn.edu/~kchang/comp6710/readings/They%20...
- egberts1 2y agoYou can always run Minix3 which basically keeps on running after a kernel OOPS.