6 ms·

I wonder why they used Linux? I'm not that familiar with engineering at NASA or JPL, but I thought computers used in exploration were running real-time operati

I wonder why they used Linux? I'm not that familiar with engineering at NASA or JPL, but I thought computers used in exploration were running real-time operating systems. Is Linux capable of this?

janderland 6y ago
When I worked on military autonomous vehicles (which I’d expect to be similar to these) we always had at least two system on board: 1) A real-time flight controller, 2) One or more Linux computers, networked via an on-board LAN which performed all the other tasks.
- senden9 6y ago
  I worked with some people of that particular JPL software team to setup a similar setup like the Mars helicopter for a earth UAV. Can confirm that multiple real time systems are used. They are controlled by a non real time Linux system.
kashyapc 6y ago
Real-time Linux is a thing, although it was maintained separately, outside mainline Linux, for a long time. About 4-ish years ago, the project got some decent funding and is now part of Linux Foundation: https://www.linuxfoundation.org/en/blog/real-time-linux-continues-its-way-to-mainline-development-and-beyond/ https://www.linuxfoundation.org/en/blog/real-time-linux-cont...
redis_mlc 6y ago
The description "real-time" varies in meaning depending on the application and acceptable average and maximum latency. The un-informed usually thinks "real-time" means "hard real-time", but that's seldom necessary, so one can save on additional expense and effort by using a regular linux distro and removing most of the daemons and file location indexing. I've done real-time development (for Space Shuttle, rocket and balloon projects), and largely all I care about is if a circular buffer can be emptied in time before it fills. That's one technique for avoiding latency variation issues. The versions of linux that you would normally encounter aim for music real-time, which is about 10 ms latency. 30 ms is considered to be bad. Most of the pro Yamaha synths use linux as the embedded OS, and some of the code is downloadable (they attempt to comply with the letter of the GPL.) So you can go down to Guitar Center and do a real-time test anytime yourself. :) The iPhone is pretty good for music, because it was designed to have low latency when playing. https://www.synthtopia.com/content/2018/02/17/10-years-later-android-audio-quality-still-lags/ https://www.synthtopia.com/content/2018/02/17/10-years-later... https://superpowered.com/androidaudiopathlatency https://superpowered.com/androidaudiopathlatency Looks like Wind River discontinued RTLinux, which was hard real-time (a real shame actually, as it removes one of the few hard real-time options): https://en.wikipedia.org/wiki/RTLinux https://en.wikipedia.org/wiki/RTLinux
- codetrotter 6y ago
  The Akai MPC X and its siblings run Linux too. Based on buildroot according to someone that looked into it. https://niklasnisbeth.gitlab.io/mpc-internals/ https://niklasnisbeth.gitlab.io/mpc-internals/ However I don’t remember the GPL being mentioned anywhere when I had my MPC X. I sold it recently because of not having any money. But I hope to own one again in the future. If I ever do have one again I will probably have a closer look at what it says about the GPL, and then try and get a copy of all of the open source portions of the firmware directly from Akai.
  - aasasd 6y ago
    Just as an aside, you probably (and hopefully) don't need to have the unit to download the manual—which is where the licenses are likely mentioned. Though this one seems to direct the user to Akai's site. E.g. https://www.mpc-tutor.com/akai-mpc-manuals/ https://www.mpc-tutor.com/akai-mpc-manuals/
- Darkphibre 6y ago
  As someone that worked at Microsoft on the XAudio 2 for the Xbox 360 & Windows Phone stacks... holy heck that android audio path is painful to read. >.<;; We went with a 5.33ms quantum on the Xbox 360 (vs. Window's 10ms), and tried to stay out of the way as much as possible to ensure minimal latencies: https://docs.microsoft.com/en-us/windows/win32/xaudio2/xaudio2-audio-graph https://docs.microsoft.com/en-us/windows/win32/xaudio2/xaudi...
- kijiki 6y ago
  > Looks like Wind River discontinued RTLinux Xenomai is still under somewhat active development.
  - redis_mlc 6y ago
    https://gitlab.denx.de/Xenomai/xenomai/-/wikis/home https://gitlab.denx.de/Xenomai/xenomai/-/wikis/home
mvh 6y ago
I worked at NASA Armstrong for a summer. The have an entire team there that at least a few years ago worked full time on real time Linux. IDK if that was used in this case or not.
jagger27 6y ago
The helicopter uses a Qualcomm Snapdragon 801, which doesn't appear to be radiation hardened. I'm guessing their usual BAE RAD750 PowerPC rad hardened CPU was too heavy to put on the helicopter.
- zaphod12 6y ago
  another thread noted that the hardened CPU was not fast enough to do the required sensor fusion and responses. In the case of a severe fault, the 801 can be restarted fast enough (while in mid flight!) to not crash the vehicle.
- abfan1127 6y ago
  apparently list price for the RAD750 is $200,000! could you not externally harden the snapdragon for less?
  - ChuckNorris89 6y ago
    That's not how hardening chips works. You need a specialized fab manufacturing, etching and packaging process for this and it's not like regular fabs are cheap to begin with. Plus you're working from the start with much larger nodes with dedicated cell libraries so everything has to be designed from scratch to fit that node which means you can't reuse consumer off the shelf designs very easily. Then there's the lack of economies of scale in building such custom parts in small numbers. I imagine if Apple would only order 100 5nm chips per year from TSMC, the unit price would be equally eye watering.
    - abfan1127 6y ago
      I've only read this doc quickly [1]. I would have guessed a lead case of a particular thickness would do the trick. But I don't know the thickness needed, maybe too much? [1] https://nepp.nasa.gov/files/25295/MRS04_LaBel.pdf https://nepp.nasa.gov/files/25295/MRS04_LaBel.pdf
      - ChuckNorris89 6y ago
peter303 6y ago
VxWorks has been battletested in NASA space probes for over 30 years. https://en.wikipedia.org/wiki/Comparison_of_embedded_computer_systems_on_board_the_Mars_rovers https://en.wikipedia.org/wiki/Comparison_of_embedded_compute... I recall an early bug in the 2004 MERS duo. They were the first to use flash memory and its new drivers. The file freelist busted and the OS kept on rebooting. Fortunately a fix was uploaded from Earth. https://en.wikipedia.org/wiki/Spirit_(rover)#Sol_17_flash_memory_management_anomaly https://en.wikipedia.org/wiki/Spirit_(rover)#Sol_17_flash_me...
- a-dub 6y ago
  the pathfinder also had a priority inversion bug resulting in deadline trips. same story, they patched it...
- Darkphibre 6y ago
  Fun read, tyvm! I loved this anecdote from the second link: > On sol 20, the command team sent it the command SHUTDWN_DMT_TIL ("Shutdown Dammit Until") to try to cause it to suspend itself until a given time. > > It seemingly ignored the command.
- kumarvvr 6y ago
  The fix was basically, there are too many files in the flash drive, causing it to reboot endlessly because the booting process could not handle so many files in the flash. Solution was to delete some files. Mo' files, Mo' problems.
ohmywhat 6y ago
Is it known what flavor of Linux they're running? E.g. are they using real time modifications to the kernel or something?

I wonder why they used Linux? I'm not that familiar with engineering at NASA or JPL, but I thought computers used in exploration were running real-time operating systems. Is Linux capable of this?

janderland 6y ago

When I worked on military autonomous vehicles (which I’d expect to be similar to these) we always had at least two system on board: 1) A real-time flight controller, 2) One or more Linux computers, networked via an on-board LAN which performed all the other tasks.

senden9 6y ago
I worked with some people of that particular JPL software team to setup a similar setup like the Mars helicopter for a earth UAV. Can confirm that multiple real time systems are used. They are controlled by a non real time Linux system.

kashyapc 6y ago

Real-time Linux is a thing, although it was maintained separately, outside mainline Linux, for a long time. About 4-ish years ago, the project got some decent funding and is now part of Linux Foundation: https://www.linuxfoundation.org/en/blog/real-time-linux-continues-its-way-to-mainline-development-and-beyond/ https://www.linuxfoundation.org/en/blog/real-time-linux-cont...

redis_mlc 6y ago

The description "real-time" varies in meaning depending on the application and acceptable average and maximum latency. The un-informed usually thinks "real-time" means "hard real-time", but that's seldom necessary, so one can save on additional expense and effort by using a regular linux distro and removing most of the daemons and file location indexing. I've done real-time development (for Space Shuttle, rocket and balloon projects), and largely all I care about is if a circular buffer can be emptied in time before it fills. That's one technique for avoiding latency variation issues. The versions of linux that you would normally encounter aim for music real-time, which is about 10 ms latency. 30 ms is considered to be bad. Most of the pro Yamaha synths use linux as the embedded OS, and some of the code is downloadable (they attempt to comply with the letter of the GPL.) So you can go down to Guitar Center and do a real-time test anytime yourself. :) The iPhone is pretty good for music, because it was designed to have low latency when playing. https://www.synthtopia.com/content/2018/02/17/10-years-later-android-audio-quality-still-lags/ https://www.synthtopia.com/content/2018/02/17/10-years-later... https://superpowered.com/androidaudiopathlatency https://superpowered.com/androidaudiopathlatency Looks like Wind River discontinued RTLinux, which was hard real-time (a real shame actually, as it removes one of the few hard real-time options): https://en.wikipedia.org/wiki/RTLinux https://en.wikipedia.org/wiki/RTLinux

codetrotter 6y ago
The Akai MPC X and its siblings run Linux too. Based on buildroot according to someone that looked into it. https://niklasnisbeth.gitlab.io/mpc-internals/ https://niklasnisbeth.gitlab.io/mpc-internals/ However I don’t remember the GPL being mentioned anywhere when I had my MPC X. I sold it recently because of not having any money. But I hope to own one again in the future. If I ever do have one again I will probably have a closer look at what it says about the GPL, and then try and get a copy of all of the open source portions of the firmware directly from Akai.
- aasasd 6y ago
  Just as an aside, you probably (and hopefully) don't need to have the unit to download the manual—which is where the licenses are likely mentioned. Though this one seems to direct the user to Akai's site. E.g. https://www.mpc-tutor.com/akai-mpc-manuals/ https://www.mpc-tutor.com/akai-mpc-manuals/
Darkphibre 6y ago
As someone that worked at Microsoft on the XAudio 2 for the Xbox 360 & Windows Phone stacks... holy heck that android audio path is painful to read. >.<;; We went with a 5.33ms quantum on the Xbox 360 (vs. Window's 10ms), and tried to stay out of the way as much as possible to ensure minimal latencies: https://docs.microsoft.com/en-us/windows/win32/xaudio2/xaudio2-audio-graph https://docs.microsoft.com/en-us/windows/win32/xaudio2/xaudi...
kijiki 6y ago
> Looks like Wind River discontinued RTLinux Xenomai is still under somewhat active development.
- redis_mlc 6y ago
  https://gitlab.denx.de/Xenomai/xenomai/-/wikis/home https://gitlab.denx.de/Xenomai/xenomai/-/wikis/home

mvh 6y ago

I worked at NASA Armstrong for a summer. The have an entire team there that at least a few years ago worked full time on real time Linux. IDK if that was used in this case or not.

jagger27 6y ago

The helicopter uses a Qualcomm Snapdragon 801, which doesn't appear to be radiation hardened. I'm guessing their usual BAE RAD750 PowerPC rad hardened CPU was too heavy to put on the helicopter.

zaphod12 6y ago
another thread noted that the hardened CPU was not fast enough to do the required sensor fusion and responses. In the case of a severe fault, the 801 can be restarted fast enough (while in mid flight!) to not crash the vehicle.
abfan1127 6y ago
apparently list price for the RAD750 is $200,000! could you not externally harden the snapdragon for less?
- ChuckNorris89 6y ago
  That's not how hardening chips works. You need a specialized fab manufacturing, etching and packaging process for this and it's not like regular fabs are cheap to begin with. Plus you're working from the start with much larger nodes with dedicated cell libraries so everything has to be designed from scratch to fit that node which means you can't reuse consumer off the shelf designs very easily. Then there's the lack of economies of scale in building such custom parts in small numbers. I imagine if Apple would only order 100 5nm chips per year from TSMC, the unit price would be equally eye watering.
  - abfan1127 6y ago
    I've only read this doc quickly [1]. I would have guessed a lead case of a particular thickness would do the trick. But I don't know the thickness needed, maybe too much? [1] https://nepp.nasa.gov/files/25295/MRS04_LaBel.pdf https://nepp.nasa.gov/files/25295/MRS04_LaBel.pdf
    - ChuckNorris89 6y ago

peter303 6y ago

VxWorks has been battletested in NASA space probes for over 30 years. https://en.wikipedia.org/wiki/Comparison_of_embedded_computer_systems_on_board_the_Mars_rovers https://en.wikipedia.org/wiki/Comparison_of_embedded_compute... I recall an early bug in the 2004 MERS duo. They were the first to use flash memory and its new drivers. The file freelist busted and the OS kept on rebooting. Fortunately a fix was uploaded from Earth. https://en.wikipedia.org/wiki/Spirit_(rover)#Sol_17_flash_memory_management_anomaly https://en.wikipedia.org/wiki/Spirit_(rover)#Sol_17_flash_me...

a-dub 6y ago
the pathfinder also had a priority inversion bug resulting in deadline trips. same story, they patched it...
Darkphibre 6y ago
Fun read, tyvm! I loved this anecdote from the second link: > On sol 20, the command team sent it the command SHUTDWN_DMT_TIL ("Shutdown Dammit Until") to try to cause it to suspend itself until a given time. > > It seemingly ignored the command.
kumarvvr 6y ago
The fix was basically, there are too many files in the flash drive, causing it to reboot endlessly because the booting process could not handle so many files in the flash. Solution was to delete some files. Mo' files, Mo' problems.

ohmywhat 6y ago

Is it known what flavor of Linux they're running? E.g. are they using real time modifications to the kernel or something?