10 ms·

As a person who used to write navigation and control software for autonomous vehicles who occasionally gets downvoted for telling people that you really don't n

As a person who used to write navigation and control software for autonomous vehicles who occasionally gets downvoted for telling people that you really don't need a fancy RTOS for this stuff, you really don't need a fancy RTOS for this stuff. Linux is a very common platform for highly responsive robotic systems. I promise that their pid control isn't an Electron app.

mlaretallack 8y ago
Agree completely. Hard real time is possible with Linux, we use it for sub millisecond control of Traffic Lights. The only issue we ever hit is proving that the code running is the stuff we expected to run.
- mlaretallack 8y ago
  But I have to add we have a real embedded cortex checking everything, just in case ......
- arcticfox 8y ago
  > Agree completely. Hard real time is possible with Linux, we use it for sub millisecond control of Traffic Lights. Out of curiosity, why do traffic light controls need to be that precise?
  - huehehue 8y ago
    Venturing a guess -- there's either some X% accuracy standard required by the government for who-knows-what reason, or it's for red light cameras and ticketing systems. "Ironically, the biggest concern with red-light camera systems is that they are so precise. They measure a driver’s speed and exact location within a fraction of a second — but do not leave any wiggle room for the errors of traffic signals such as inconsistent yellow light times"[0] If there's not an accuracy threshold for safety reasons, there's gotta be one when traffic ticketing revenue is on the line (also I guess determining fault at accidents, vehicular manslaughter cases, etc.) [0] https://www.mercurynews.com/2014/06/06/red-light-cameras-how-accurate-are-they/ https://www.mercurynews.com/2014/06/06/red-light-cameras-how...
  - loourr 8y ago
    I don't actually know, but my guess is time drift with respect to other lights.
  - jackhack 8y ago
    resume-driven development?
    - lifeisstillgood 8y ago
      everything is revenue driven development
      - jethro_tell 8y ago
        he said resume, i.e. working with tech you want to use at your next gig not the tech you need at this gig.
        lifeisstillgood 8y ago
        could have sworn i read that right ... ah well still, revenue driven development actually makes sense as a thing too ...
- coldtea 8y ago
  >we use it for sub millisecond control of Traffic Lights Why would anyone need "sub millisecond control of Traffic Lights"? Traffic lights are mission critical systems, of course, but even millisecond precision should be more than enough, and possibly even 0.5x-1 second precision...
  - brbrodude 8y ago
    if it controls a ton of them, then small stuff could get them to be intolerably out of sync maybe? I'm just guessing :p
  - encoderer 8y ago
    The sub-millisecond precision ensures they can totally destroy your day with short left turn arrows and yellow lights.
    - xg15 8y ago
      Yes, some green phases definitely feel sub-millisecond.
  - dnautics 8y ago
    For legal reasons. Regs say that stop cycles have to meet minimum times in each state, if you're even off by a Tony bit you could potentially challenge it legally in court.
  - Cyph0n 8y ago
    The real question is: why do you need an entire OS kernel to control traffic lights?
    - noselasd 8y ago
      To communicate with all the other traffic lights and sensors at the other intersections on the road to optimize traffic flow.
      - 8y ago
- samstave 8y ago
  On a tangent, but still germane to the thread: Why arent their efforts to create "autonomous only" traffic managment scenarios, where people drive into a given, known area, and the area then takes control of managing the traffic and vehicles. Such that you relinquish control of the vehicle to that area's control system, with your destination stated and then your vehicle is managed accordingly. For example, a parking lot for a really large venue with an autonomous valet system. YOu drive up and get out and then the system takes over your car and drives off iwth it and parks it and you recall it when needed... Or managing traffic in a very heavily trafficed bottle-neck of a grid; such as the baybridge merging egress from SF financial district. If you put in your destination, and join the group, all the cars could then be managed for getting onto the bridge more rapidiously ... Autonomous doesnt need to drive me from SF to LA, but it would be great if an autonomous hive mind could get all the cars to up throughput in given situations, no?
  - forapurpose 8y ago
    That seems to match the right tool (AI driving) to the right job (well-defined, well-controlled situations). I seem to recall that similar ideas go back to the early 1990s, at least, for highways: Drive your car to the entrance ramp, plug in your destination, and the autonomous system takes it from there. But for many of these things, such as the Bay Bridge or a highway, it seems like there is a simpler solution: Put the cars on a train and take them across by rail. I suspect I'm not the first person to think of it so I wonder why it's never been done (i.e., what problem I'm overlooking).
    - samstave 8y ago
      Ive been picturing the rail problem for some time as well. Not just for cars, but also for cargo... just have a constant gondola-like conveyer that detaches a platform from the line to slow it enough to allow for cargo to get on, then re-zip-it backinto the line and speed it along, de-rail it once it hits its exit/location... ideally though, in cities, there would be no surface streets and all cars would have their own level below that of bikes pedestrians. What would SF look like if a superstructure was built above all streets and all pedestrian and bike traffic was moved up there? (sure, SF may be a poor example, so just select [city]) Look at Singapore's vast underground connecting malls between facilities. Those are pretty amazing. I grade US urban/city planners rather poorly.
- atomicthumbs 8y ago
  So what sort of interlocks do you have on top of that to prevent simultaneous greens?
  - amanaplanacanal 8y ago
    Traffic signals have hardware interlocks so that can never happen. I don't know if they are using mechanical relays or what, but you can't turn both sides green in software.
js2 8y ago
Also, an RTOS is susceptible to priority inversion[0] so it’s not necessarily a panacea. 0. which infamously occurred on the Mars Pathfinder. https://www.rapitasystems.com/blog/what-really-happened-to-the-software-on-the-mars-pathfinder-spacecraft https://www.rapitasystems.com/blog/what-really-happened-to-t...
- gapo 8y ago
  There are solutions to priority inversion. This is an old limitation that is now days even taught in school. Any decent RTOS should have Priority inheritance that should avoid this. Pointing to this one things as RTOS issue isn't really an accurate portrayal of current RTOS capabilities.
  - js2 8y ago
    That’s good to know. Indeed, the Pathfinder’s OS (VxWorks) had priority inheritance but it wasn’t enabled on a particular mutex and enabling it was the fix. Priority inversion had been known about since the 70s. Priority inheritance seems to have first been proposed in 1990: https://www3.nd.edu/~dwang5/courses/spring18/papers/real-time/pip.pdf https://www3.nd.edu/~dwang5/courses/spring18/papers/real-tim... (Priority Inheritance Protocols: An Approach to Real-Time Synchronization) The Pathfinder engineers were apparently unaware of the priority inheritance option available in VxWorks until they had to debug the issue live from a few hundred million km away.
flamedoge 8y ago
muh arteries...
coldtea 8y ago
>I promise that their pid control isn't an Electron app. Given all the wtf stuff in TFA, it might very well be...
8_hours_ago 8y ago
Automotive ADAS systems generally require ASIL-D certification, which is much easier with an RTOS than Linux. I don't have much experience with real-time embedded Linux, but my understanding is that it is very difficult or impossible to certify to ASIL-D. Can someone correct me?
- yitchelle 8y ago
  A RTOS helps because the vendor will usually provide the RTOS already certified for ASIL-D application. The rest of the software components will also need to reach ASIL-D, but getting the RTOS to ASIL-D makes things a little bit easier.
- balp 8y ago
  Usually this boils down to the ASIL-D RTOS systems are much smaller, e.g. much simpler to verify, leaving the developers of the systems above with much more job to verify there parts. Also in my experience it might have been easier to reach the ASIL-D requirements, using a smarter combination of a Limiter on RTOS and using more generic code on something like linux for more of the code. This probably also would end up in more used and tested applications reaching more stability. (That's is partly outside ASIL-D). Functional safety and ISO-26262 is much misunderstood in automotive development and architecture. Also imho the certifications, well with out the safety case are kind of useless. You still have to make the assessment how you will find the problems with it in your use case. That might ever so slightly differ from what they certified. The automotive industry thou loves to have someone else to blame, e.g. the supplier of the RTOS, Compiler etc. Using Linux makes the blame game hard.
Keyframe 8y ago
you really don't need a fancy RTOS for this stuff maybe you don't, but genuinely curious how do you validate/guarantee scenarios then?
- BugsJustFindMe 8y ago
  I suppose my best questions back to you might be: 1) What concrete guarantees do you think you get from a special RTOS? 2) Which of those guarantees are meaningful to the scenario? 3) How (by what mechanisms) do you think the special RTOS guarantees the things that it guarantees? 4) Which of those behaviors are something that only an RTOS can provide?
  - Keyframe 8y ago
    Among other things, determinism, since timing can be guaranteed (within a margin). RTOS will run with consistent timing, which is guaranteed and facilitated by control over tasks priority and checks whether timings are met. You could probably do that without a hard RTOS, but without any sort of formal guarantee. So, it might work all fine and dandy, until it doesn't. Doesn't should not exist in a hard RTOS by definition and proofing of it.
    - BugsJustFindMe 8y ago
      You don't need an RTOS to know that a system without extraneous background processes isn't doing extraneous background processing. Where precisely do you think think your timings are going? You're not also running a Minecraft server on the nav computer. Using Linux doesn't mean you also need to enable stupid PC things like search indexing or seti@home. Navigation isn't resource constrained. The only resource constrained component in the whole system is the computer vision module and it's going to be on its own processors, and the rate at which it can hand off new output is on far faaaarrrr longer timescales than your interprocess communication latency.
    - balp 8y ago
      Yes, thata one of the RTOS thing, but almost always, when trying to build this down into what these timing limits are, they are kinda arbitrary, gut feeling put into a number. There are exceptions, yes, usally these exceptions as well are only for a limity subpart of the system. So for this using RTOS for everything is as stupid as not using RTOS, (or possibly even puting these parts into hardware, ASIC of FPGA) for these small subsystems. These timing used, also is mostly up to scheduler and Linux is possible to run with a scheduler that gives it similar capabilities ans most RTOS systems. It's a blury grey area at best.
gascan 8y ago
Is that like, you really don't need a fancy RTOS for this stuff 99.999% of the time, although sometimes you do? Or truly, despite the life safety element, there is never any need for RTOS.
- BugsJustFindMe 8y ago
  In every application I can think of off the top of my head, but mostly in the ones that apply to Tesla, you truly don't, except when the law or a contract says otherwise. I'm sure there are exceptions for things that don't apply to Tesla (or for that matter SpaceX). Here's what's inside of every autonomous vehicle ever made: a message-passing subsystem, sensors, fusers, navigation, dynamic control, actuator device drivers, and thruster device drivers. Sensors measure things and emit readings. Your most expensive, highest frequency general purpose sensors emit new readings at something too fast for a human but hella slow for a computer, like 100Hz-5KHz. Your common sensors, a video camera for instance, don't get even close to that. These sensors are often connected, even today because milspec companies hate modernity, via RS-232 serial cables. For those younger than 30, RS-232 is what non-Apple computers used for non-keyboard/mouse peripherals prior to the introduction of the first iMac in 1998 because USB didn't really take off until then. Sensors send their readings via the message-passing subsystem to fusers. Fusers take the readings from the sensors and, hur hur, "fuse" them together into a description of where the vehicle is and what the environment is like. This usually involves something like a kalman filter. Fusing even your very fastest sensors, the 5KHz IMUs of the world, is just a small bit of math and basically takes no time at all. Fusers send their fused states via the message-passing subsystem to navigation. Navigation takes the fused sense of self and the world and decides which direction to head and how fast to go. The objective could be something like hitting route waypoints or it could be something like staying in a lane and not being rear-ended and avoiding obstacles. Car navigation probably doesn't act on new input more frequently than 100Hz, you certainly can't act on new input more frequently than 100Hz, and it takes basically no time at all. Navigation sends its directives via the message-passing subsystem to dynamic control. Dynamic control takes navigation's "which way" and "how fast" directives and turns them into more realistic short-term goals accounting for hysteresis and other physical limitations of the system like minimum turn radius. This is just a small bit of math and basically takes no time at all. Dynamic control sends its directives via the message-passing subsystem to the actuator and thruster drivers. Actuator drivers convert dynamic control's "go more left" message into trying to go more left. Thruster drivers convert dynamic control's "go more fast" message into trying to go more fast. Actuator and thruster drivers send readings (hopefully) from the actuators and thrusters, because those are also sensors, back to dynamic control and fusion. Sensors feed into fusers, fusers feed into nav, nav feeds into dynamic control, dynamic control feeds into actuation and thrust. When you have new data, you do something new with it which is technically doing the same old thing with it and just producing new output. Now there aren't that many sensors. There are way fewer fusers. There's only one navigation. There's probably only one dynamic control, though there could be a couple. Anything else that I haven't already described, like Waymo's machine learning object classifying 4D mustache adding hotdog detectors, are just sensors and fusers sitting on their own computers feeding new lat/lng/heading/speed to navigation at a rate that is hella slow for a computer. And for sure Waymo's convolutional neural network middle-out jaywalking yoga mom detector takes a lot of processing, but it's running on its own computer, not competing for resources, and emitting its fused readings at some hella slow for a computer rate.
dnbgfher 8y ago
Having done just enough robotics stuff to have used linux before (though not controlling anything even kind of large), I find this an odd notion. It's not that I think linux would go wrong for this sort of thing often, but when dealing with things like rockets or self-driving cars I'd think you want more assurance than "well we haven't had it be slow yet." I haven't messed with an RTOS before but have done some fooling around with scheduling on microcontrollers and I can see why linux is tempting for ease and speed. But we're talking rockets and self-driving cars. These things are expensive as hell, can easily kill people, or both. It seems like the exact sort of place you'd want to take the time and effort to be sure.
zachd1_618 8y ago
+1. I wrote some near flight software for an instrument using Ubuntu with a RT patch on modern hardware, in parallel with another team that took the traditional approach. Not to brag, but our “undeterministic” system ran far more reliably than the real time one that didn’t have the advantage of modern application libraries. Plus, even though we had that pesky operating system in the way, we ran on blazing fast modern architectures and were actually more deterministic than the slow as hell hardware we were benchmarking against. I’ve seen the same pattern a few other times. Slow, hand built, rad hard systems CAN be more stable and demonstratively safer... but that is rarely the case and the effort required to get such a system right is orders of magnitude greater than using standard “undeterministic” systems. That engineering effort can be better spent innovating and building fundamentally more advanced solutions. Just my experience. Just my opinion
agitator 8y ago
As someone who worked on RTOS systems and now works on autonomy. These things frighten me everyday. What frightens me even more, is the people who work on autonomy without a real grasp on determinism. It's unfortunate that the people who have the most high tech backgrounds (phds in computer vision AI, etc) applicable to autonomy, have never implemented safety critical autonomous systems outside of a research project that tested some aspect of detection or control in a test environment and had to only work once to get a paper published. For general robotics linux is great. But there is an enormous difference between a robot roaming around your house bumping off walls, and a vehicle carrying a whole family at 70mph. Most of the linux based systems I have worked with have some form of redundancy, whether it be other chips running linux, or ideally ECU's running an RTOS that perform monitoring, gating, and/or some level of safety fallback control. The RTOS based redundancies often are what provide ASILD. Trusting a single linux processor is what everyone does to get funding, but when you go out and test on public roads with human lives at stake, or start selling a product, you better have some quantitative guarantees other than "It's been fine so far..." That kind of stuff makes me angry.
- crubier 8y ago
  Having worked several years on critical embedded systems in aerospace, I would tend to agree with you. But on the other hand, has any Tesla car ever had an accident because of this? At some point, "heavily tested and validated end to end in real-life conditions for years" and "formally proven on a simplified model using reasonable assumptions made by human engineers" become relatively close in terms of how much trust you can put in a system. But somehow we tend to prefer to later. I am not sure if this paradigm is still relevant these days.
  - xg15 8y ago
    > "heavily tested and validated end to end in real-life conditions for years" That seems like a more wordy way to state "It's been fine so far". > "formally proven on a simplified model using reasonable assumptions made by human engineers" I didn't know about that. What kind of formal proofs did they do? Did they involve the linux scheduler?
- 8y ago
hatsubai 8y ago
Agreed. I work on both vxWorks and Linux in the defense industry for a very popular armored fighting vehicle, and despite popular belief, the Linux kernel with the RT patch works well enough that both the cost of vxWorks and the issue of finding developers to maintain it isn't exactly justifiable anymore. Without going into too much detail, there have been a good few studies internally to show that our current fire control unit doesn't need the hard time precision it once required on legacy hardware, and all of the Linux ports with the RT patch perform just as well. The biggest hurdle, of course, is not exactly the performance. It's the certification process.

mlaretallack 8y ago

Agree completely. Hard real time is possible with Linux, we use it for sub millisecond control of Traffic Lights. The only issue we ever hit is proving that the code running is the stuff we expected to run.

mlaretallack 8y ago
But I have to add we have a real embedded cortex checking everything, just in case ......
arcticfox 8y ago
> Agree completely. Hard real time is possible with Linux, we use it for sub millisecond control of Traffic Lights. Out of curiosity, why do traffic light controls need to be that precise?
- huehehue 8y ago
  Venturing a guess -- there's either some X% accuracy standard required by the government for who-knows-what reason, or it's for red light cameras and ticketing systems. "Ironically, the biggest concern with red-light camera systems is that they are so precise. They measure a driver’s speed and exact location within a fraction of a second — but do not leave any wiggle room for the errors of traffic signals such as inconsistent yellow light times"[0] If there's not an accuracy threshold for safety reasons, there's gotta be one when traffic ticketing revenue is on the line (also I guess determining fault at accidents, vehicular manslaughter cases, etc.) [0] https://www.mercurynews.com/2014/06/06/red-light-cameras-how-accurate-are-they/ https://www.mercurynews.com/2014/06/06/red-light-cameras-how...
- loourr 8y ago
  I don't actually know, but my guess is time drift with respect to other lights.
- jackhack 8y ago
  resume-driven development?
  - lifeisstillgood 8y ago
    everything is revenue driven development
    - jethro_tell 8y ago
      he said resume, i.e. working with tech you want to use at your next gig not the tech you need at this gig.
      - lifeisstillgood 8y ago
        could have sworn i read that right ... ah well still, revenue driven development actually makes sense as a thing too ...
coldtea 8y ago
>we use it for sub millisecond control of Traffic Lights Why would anyone need "sub millisecond control of Traffic Lights"? Traffic lights are mission critical systems, of course, but even millisecond precision should be more than enough, and possibly even 0.5x-1 second precision...
- brbrodude 8y ago
  if it controls a ton of them, then small stuff could get them to be intolerably out of sync maybe? I'm just guessing :p
- encoderer 8y ago
  The sub-millisecond precision ensures they can totally destroy your day with short left turn arrows and yellow lights.
  - xg15 8y ago
    Yes, some green phases definitely feel sub-millisecond.
- dnautics 8y ago
  For legal reasons. Regs say that stop cycles have to meet minimum times in each state, if you're even off by a Tony bit you could potentially challenge it legally in court.
- Cyph0n 8y ago
  The real question is: why do you need an entire OS kernel to control traffic lights?
  - noselasd 8y ago
    To communicate with all the other traffic lights and sensors at the other intersections on the road to optimize traffic flow.
    - 8y ago
samstave 8y ago
On a tangent, but still germane to the thread: Why arent their efforts to create "autonomous only" traffic managment scenarios, where people drive into a given, known area, and the area then takes control of managing the traffic and vehicles. Such that you relinquish control of the vehicle to that area's control system, with your destination stated and then your vehicle is managed accordingly. For example, a parking lot for a really large venue with an autonomous valet system. YOu drive up and get out and then the system takes over your car and drives off iwth it and parks it and you recall it when needed... Or managing traffic in a very heavily trafficed bottle-neck of a grid; such as the baybridge merging egress from SF financial district. If you put in your destination, and join the group, all the cars could then be managed for getting onto the bridge more rapidiously ... Autonomous doesnt need to drive me from SF to LA, but it would be great if an autonomous hive mind could get all the cars to up throughput in given situations, no?
- forapurpose 8y ago
  That seems to match the right tool (AI driving) to the right job (well-defined, well-controlled situations). I seem to recall that similar ideas go back to the early 1990s, at least, for highways: Drive your car to the entrance ramp, plug in your destination, and the autonomous system takes it from there. But for many of these things, such as the Bay Bridge or a highway, it seems like there is a simpler solution: Put the cars on a train and take them across by rail. I suspect I'm not the first person to think of it so I wonder why it's never been done (i.e., what problem I'm overlooking).
  - samstave 8y ago
    Ive been picturing the rail problem for some time as well. Not just for cars, but also for cargo... just have a constant gondola-like conveyer that detaches a platform from the line to slow it enough to allow for cargo to get on, then re-zip-it backinto the line and speed it along, de-rail it once it hits its exit/location... ideally though, in cities, there would be no surface streets and all cars would have their own level below that of bikes pedestrians. What would SF look like if a superstructure was built above all streets and all pedestrian and bike traffic was moved up there? (sure, SF may be a poor example, so just select [city]) Look at Singapore's vast underground connecting malls between facilities. Those are pretty amazing. I grade US urban/city planners rather poorly.
atomicthumbs 8y ago
So what sort of interlocks do you have on top of that to prevent simultaneous greens?
- amanaplanacanal 8y ago
  Traffic signals have hardware interlocks so that can never happen. I don't know if they are using mechanical relays or what, but you can't turn both sides green in software.

js2 8y ago

Also, an RTOS is susceptible to priority inversion[0] so it’s not necessarily a panacea. 0. which infamously occurred on the Mars Pathfinder. https://www.rapitasystems.com/blog/what-really-happened-to-the-software-on-the-mars-pathfinder-spacecraft https://www.rapitasystems.com/blog/what-really-happened-to-t...

gapo 8y ago
There are solutions to priority inversion. This is an old limitation that is now days even taught in school. Any decent RTOS should have Priority inheritance that should avoid this. Pointing to this one things as RTOS issue isn't really an accurate portrayal of current RTOS capabilities.
- js2 8y ago
  That’s good to know. Indeed, the Pathfinder’s OS (VxWorks) had priority inheritance but it wasn’t enabled on a particular mutex and enabling it was the fix. Priority inversion had been known about since the 70s. Priority inheritance seems to have first been proposed in 1990: https://www3.nd.edu/~dwang5/courses/spring18/papers/real-time/pip.pdf https://www3.nd.edu/~dwang5/courses/spring18/papers/real-tim... (Priority Inheritance Protocols: An Approach to Real-Time Synchronization) The Pathfinder engineers were apparently unaware of the priority inheritance option available in VxWorks until they had to debug the issue live from a few hundred million km away.

flamedoge 8y ago

muh arteries...

coldtea 8y ago

>I promise that their pid control isn't an Electron app. Given all the wtf stuff in TFA, it might very well be...

8_hours_ago 8y ago

Automotive ADAS systems generally require ASIL-D certification, which is much easier with an RTOS than Linux. I don't have much experience with real-time embedded Linux, but my understanding is that it is very difficult or impossible to certify to ASIL-D. Can someone correct me?

yitchelle 8y ago
A RTOS helps because the vendor will usually provide the RTOS already certified for ASIL-D application. The rest of the software components will also need to reach ASIL-D, but getting the RTOS to ASIL-D makes things a little bit easier.
balp 8y ago
Usually this boils down to the ASIL-D RTOS systems are much smaller, e.g. much simpler to verify, leaving the developers of the systems above with much more job to verify there parts. Also in my experience it might have been easier to reach the ASIL-D requirements, using a smarter combination of a Limiter on RTOS and using more generic code on something like linux for more of the code. This probably also would end up in more used and tested applications reaching more stability. (That's is partly outside ASIL-D). Functional safety and ISO-26262 is much misunderstood in automotive development and architecture. Also imho the certifications, well with out the safety case are kind of useless. You still have to make the assessment how you will find the problems with it in your use case. That might ever so slightly differ from what they certified. The automotive industry thou loves to have someone else to blame, e.g. the supplier of the RTOS, Compiler etc. Using Linux makes the blame game hard.

Keyframe 8y ago

you really don't need a fancy RTOS for this stuff maybe you don't, but genuinely curious how do you validate/guarantee scenarios then?

BugsJustFindMe 8y ago
I suppose my best questions back to you might be: 1) What concrete guarantees do you think you get from a special RTOS? 2) Which of those guarantees are meaningful to the scenario? 3) How (by what mechanisms) do you think the special RTOS guarantees the things that it guarantees? 4) Which of those behaviors are something that only an RTOS can provide?
- Keyframe 8y ago
  Among other things, determinism, since timing can be guaranteed (within a margin). RTOS will run with consistent timing, which is guaranteed and facilitated by control over tasks priority and checks whether timings are met. You could probably do that without a hard RTOS, but without any sort of formal guarantee. So, it might work all fine and dandy, until it doesn't. Doesn't should not exist in a hard RTOS by definition and proofing of it.
  - BugsJustFindMe 8y ago
    You don't need an RTOS to know that a system without extraneous background processes isn't doing extraneous background processing. Where precisely do you think think your timings are going? You're not also running a Minecraft server on the nav computer. Using Linux doesn't mean you also need to enable stupid PC things like search indexing or seti@home. Navigation isn't resource constrained. The only resource constrained component in the whole system is the computer vision module and it's going to be on its own processors, and the rate at which it can hand off new output is on far faaaarrrr longer timescales than your interprocess communication latency.
  - balp 8y ago
    Yes, thata one of the RTOS thing, but almost always, when trying to build this down into what these timing limits are, they are kinda arbitrary, gut feeling put into a number. There are exceptions, yes, usally these exceptions as well are only for a limity subpart of the system. So for this using RTOS for everything is as stupid as not using RTOS, (or possibly even puting these parts into hardware, ASIC of FPGA) for these small subsystems. These timing used, also is mostly up to scheduler and Linux is possible to run with a scheduler that gives it similar capabilities ans most RTOS systems. It's a blury grey area at best.

gascan 8y ago

Is that like, you really don't need a fancy RTOS for this stuff 99.999% of the time, although sometimes you do? Or truly, despite the life safety element, there is never any need for RTOS.

BugsJustFindMe 8y ago
In every application I can think of off the top of my head, but mostly in the ones that apply to Tesla, you truly don't, except when the law or a contract says otherwise. I'm sure there are exceptions for things that don't apply to Tesla (or for that matter SpaceX). Here's what's inside of every autonomous vehicle ever made: a message-passing subsystem, sensors, fusers, navigation, dynamic control, actuator device drivers, and thruster device drivers. Sensors measure things and emit readings. Your most expensive, highest frequency general purpose sensors emit new readings at something too fast for a human but hella slow for a computer, like 100Hz-5KHz. Your common sensors, a video camera for instance, don't get even close to that. These sensors are often connected, even today because milspec companies hate modernity, via RS-232 serial cables. For those younger than 30, RS-232 is what non-Apple computers used for non-keyboard/mouse peripherals prior to the introduction of the first iMac in 1998 because USB didn't really take off until then. Sensors send their readings via the message-passing subsystem to fusers. Fusers take the readings from the sensors and, hur hur, "fuse" them together into a description of where the vehicle is and what the environment is like. This usually involves something like a kalman filter. Fusing even your very fastest sensors, the 5KHz IMUs of the world, is just a small bit of math and basically takes no time at all. Fusers send their fused states via the message-passing subsystem to navigation. Navigation takes the fused sense of self and the world and decides which direction to head and how fast to go. The objective could be something like hitting route waypoints or it could be something like staying in a lane and not being rear-ended and avoiding obstacles. Car navigation probably doesn't act on new input more frequently than 100Hz, you certainly can't act on new input more frequently than 100Hz, and it takes basically no time at all. Navigation sends its directives via the message-passing subsystem to dynamic control. Dynamic control takes navigation's "which way" and "how fast" directives and turns them into more realistic short-term goals accounting for hysteresis and other physical limitations of the system like minimum turn radius. This is just a small bit of math and basically takes no time at all. Dynamic control sends its directives via the message-passing subsystem to the actuator and thruster drivers. Actuator drivers convert dynamic control's "go more left" message into trying to go more left. Thruster drivers convert dynamic control's "go more fast" message into trying to go more fast. Actuator and thruster drivers send readings (hopefully) from the actuators and thrusters, because those are also sensors, back to dynamic control and fusion. Sensors feed into fusers, fusers feed into nav, nav feeds into dynamic control, dynamic control feeds into actuation and thrust. When you have new data, you do something new with it which is technically doing the same old thing with it and just producing new output. Now there aren't that many sensors. There are way fewer fusers. There's only one navigation. There's probably only one dynamic control, though there could be a couple. Anything else that I haven't already described, like Waymo's machine learning object classifying 4D mustache adding hotdog detectors, are just sensors and fusers sitting on their own computers feeding new lat/lng/heading/speed to navigation at a rate that is hella slow for a computer. And for sure Waymo's convolutional neural network middle-out jaywalking yoga mom detector takes a lot of processing, but it's running on its own computer, not competing for resources, and emitting its fused readings at some hella slow for a computer rate.

dnbgfher 8y ago

Having done just enough robotics stuff to have used linux before (though not controlling anything even kind of large), I find this an odd notion. It's not that I think linux would go wrong for this sort of thing often, but when dealing with things like rockets or self-driving cars I'd think you want more assurance than "well we haven't had it be slow yet." I haven't messed with an RTOS before but have done some fooling around with scheduling on microcontrollers and I can see why linux is tempting for ease and speed. But we're talking rockets and self-driving cars. These things are expensive as hell, can easily kill people, or both. It seems like the exact sort of place you'd want to take the time and effort to be sure.

zachd1_618 8y ago

+1. I wrote some near flight software for an instrument using Ubuntu with a RT patch on modern hardware, in parallel with another team that took the traditional approach. Not to brag, but our “undeterministic” system ran far more reliably than the real time one that didn’t have the advantage of modern application libraries. Plus, even though we had that pesky operating system in the way, we ran on blazing fast modern architectures and were actually more deterministic than the slow as hell hardware we were benchmarking against. I’ve seen the same pattern a few other times. Slow, hand built, rad hard systems CAN be more stable and demonstratively safer... but that is rarely the case and the effort required to get such a system right is orders of magnitude greater than using standard “undeterministic” systems. That engineering effort can be better spent innovating and building fundamentally more advanced solutions. Just my experience. Just my opinion

agitator 8y ago

As someone who worked on RTOS systems and now works on autonomy. These things frighten me everyday. What frightens me even more, is the people who work on autonomy without a real grasp on determinism. It's unfortunate that the people who have the most high tech backgrounds (phds in computer vision AI, etc) applicable to autonomy, have never implemented safety critical autonomous systems outside of a research project that tested some aspect of detection or control in a test environment and had to only work once to get a paper published. For general robotics linux is great. But there is an enormous difference between a robot roaming around your house bumping off walls, and a vehicle carrying a whole family at 70mph. Most of the linux based systems I have worked with have some form of redundancy, whether it be other chips running linux, or ideally ECU's running an RTOS that perform monitoring, gating, and/or some level of safety fallback control. The RTOS based redundancies often are what provide ASILD. Trusting a single linux processor is what everyone does to get funding, but when you go out and test on public roads with human lives at stake, or start selling a product, you better have some quantitative guarantees other than "It's been fine so far..." That kind of stuff makes me angry.

crubier 8y ago
Having worked several years on critical embedded systems in aerospace, I would tend to agree with you. But on the other hand, has any Tesla car ever had an accident because of this? At some point, "heavily tested and validated end to end in real-life conditions for years" and "formally proven on a simplified model using reasonable assumptions made by human engineers" become relatively close in terms of how much trust you can put in a system. But somehow we tend to prefer to later. I am not sure if this paradigm is still relevant these days.
- xg15 8y ago
  > "heavily tested and validated end to end in real-life conditions for years" That seems like a more wordy way to state "It's been fine so far". > "formally proven on a simplified model using reasonable assumptions made by human engineers" I didn't know about that. What kind of formal proofs did they do? Did they involve the linux scheduler?
8y ago

hatsubai 8y ago

Agreed. I work on both vxWorks and Linux in the defense industry for a very popular armored fighting vehicle, and despite popular belief, the Linux kernel with the RT patch works well enough that both the cost of vxWorks and the issue of finding developers to maintain it isn't exactly justifiable anymore. Without going into too much detail, there have been a good few studies internally to show that our current fire control unit doesn't need the hard time precision it once required on legacy hardware, and all of the Linux ports with the RT patch perform just as well. The biggest hurdle, of course, is not exactly the performance. It's the certification process.