16 ms·

Why collision avoidance is harder for an AI-based system

118 points by IndrekR 9y ago

jdmichal 9y ago
> AI is not preprogrammed to monitor a known input from a sensor to take a predefined action. I guess one of my outstanding questions, which reading this only confirmed, is why this is the case? I mean, humans are pretty good examples of intelligence. And yet we still have and use these anti-collision systems. Because, in the end, when wrong decisions are made these systems save lives. Why would AI-driven vehicles not have dedicated, single-purpose subsystems such as anti-collision? I mean, are we going to also remove ABS, because the AI could learn to modulate the brakes itself? How much are we going to push into AI, when the purpose-built systems are both functional and effective?
- SlowRobotAhead 9y ago
  >I mean, are we going to also remove ABS, because the AI could learn to modulate the brakes itself? This is the topic that I feel SDC enthusiasts forget. Not everything HAS TO BE AI. And we could maybe make steps towards SDC, not fantastic leaps that get people killed and really just get government involved where it doesn't need to be yet. We could replace the ABS/ESP/traction control systems in current vehicles with a machine learning / deep learning system absolutely - that's not sexy though! No one wants increments, they want whole self-driving cars right now - and while I have opinions about that, there is no doubt it's driving (pun intended) the industry. Ideally, at first, we'd see components in consumer vehicles, and completely automated long haul trucks from A to B determined routes - but like I said, not sexy.
  - couchand 9y ago
    > Ideally, at first, we'd see components in consumer vehicles We do, though. Look at the cruise control or auto-park on a vehicle produced in the last few years. The totally autonomous car may make headlines, but these sorts of features will be what really make the technology ubiquitous.
    - SlowRobotAhead 9y ago
      I work on ECUs like this. High end stuff is doing computer vision, but 99% of what is on the road is pre-calibrated motor/solenoid control and that’s all. Even adaptive cruise, steel camera object “detection” are just pretty simple systems. Almost nothing is doing even pieces of what the whole-package SDCs are.
- mulmen 9y ago
  I don't know much about AI or ML and even less about how it relates to automotive engineering but I imagine this is already how it works. I don't think there is one model for everything as much as there are models for individual components of driving. Something for steering, something(s?) for identifying obstructions, etc. In this case maybe letting a model modulate brake pulses based on conditions like temperature and velocity makes more sense? Hopefully someone with more experience in this domain can enlighten us.
  - skgoa 9y ago
    It is exactly how it works. There ~40-115 different computers in a modern car that often have completely walled-off modules running on them. Most 'models' you will find in a car's software aren't even generated through machine learning, but tried and true engineering/statistics. No one in their right mind is proposing to replace those with one big AI monolith. That's marketing BS from the likes of NVidia, who would stand to gain from it. AI is being used in certain subfunctions like visual object detection. In the future, AI will be used to make higher level decisions like trajectory (lane choice, overtaking etc.) and route planning. But it will only hand those higher level plans down to the mostly 'dumb' computer systems of the car to be carried out just as with a manually driven car.
- phaedrus 9y ago
  This is a design trade-off that comes up all the time in automotive engineering. I own a couple of 90's-era Honda vehicles and what's interesting is although the cars are computerized, they also employ a number of semi-independent systems such as instrument gauges which directly connect to their own sensors even though a similar sensor connected to the engine control unit is right next to it, mechanical choke and coolant temperature modulators to intake air despite fuel injection, an "hydraulic computer" controlling part of the automatic transmission even though there is also a dedicated transmission control computer (which is itself redundant to making the transmission control an integrated feature of the engine control module). To me this seems to make no design sense; why not bring all of the inputs into one central box with one computer control, and drive all of the outputs from this central box? But then who am I to argue that the separate / bespoke approach is not better, given Honda's reputation for reliability?
  - majormajor 9y ago
    Seems like classic, prudent, avoiding of single points of failure. Sounds reasonable, especially since computerization in cars was still pretty new back then. For something like ABS, or emergency braking, we have established algorithms for this already, so why be in a hurry to offload that to a machine-learning black box? Is the math behind "we need to stop this car" so complex that AI is needed?
- martinrebane 9y ago
  Author here :) I cannot answer for Uber, but the grand reason why we cannot put deterministic parts into the AI is the same as for humans - we do not control the internals of it, the complexity is too high. Although we can build systems that are composed of both AI and deterministic collision avoidance (as seen on new cars today). It is what I also suggested in the article.
  - wnoise 9y ago
    Ah, then I should complain to you that "AI" is a broad suite of techniques and goals, rather than a specific black-box technique which has these flaws.
    - andreyk 9y ago
      Yes - you really seem to be talking about Machine Learning (a subset of AI). And as another comment says 'This makes me want to pull my hair out. No reasonable system relies on AI [or ML] from top to bottom. Take this article with a grain of salt, it is attacking a straw-man.'
- CardenB 9y ago
  The false positive rate of this system could be very noisy. At what point should you brake, even if the AI says not to? When you might collide with a lidar point? You want to collide with lidar points all the time (the ground beneath the vehicle, for example, or any reflective noise). The AI is there to tell you what lidar points are actual obstacles that you can't collide with, otherwise it's not really meaningful.
  - jdmichal 9y ago
    This line of argument doesn't mean anything when we already have cars with intelligent, independent drivers, and these systems. And yes, their purpose would be to override the AI control when they trigger. Exactly the same as they do right now.
- jmalicki 9y ago
  " I mean, are we going to also remove ABS, because the AI could learn to modulate the brakes itself?" I think that's a reasonable definition of ABS - an AI (in most cases for ABS, an expert system AI) that knows how to best mediate the brakes, with levels of performance that exceed the vast majority of humans.
  - ams6110 9y ago
    The conventional ABS system is not AI in any sense. It is a few sensors and maybe some look up tables of what to do under various speed / brake pressure / wheel rotation situations
    - randomdata 9y ago
      Does that not fit the definition of artificial intelligence? The system makes decisions – ones that would be otherwise done by a human – based on its perception of the world. That is what I have always understood AI to mean. To be sure, a conventional ABS system does not use machine learning, but machine learning is only a subset of AI.
      - skgoa 9y ago
        The core of this debate is the matter of what intelligence actually is. We don't really know, but the most common definition I have heard/read is that intelligence is the ability to generate new solutions when presented with previously unseen input. An ABS would not qualify under this definition, because its output is predefined rigorously for all possible combinations of input values.
        randomdata 9y ago
        On the other hand, artificial intelligence is not actual intelligence. The definition of artificial intelligence is the ability to perform a task that would normally require human intelligence. An ABS system would fit into this definition as it is a task that normally would require human sensing the conditions to know how to apply the breaks. Although I can see why some point out that AI is a moving target, representing only what still seems 'magical'. If you showed an ABS system to someone in the early 1900s, I truly believe they would see it as some kind of intelligence. Now that we have acclimated to the technology, we don't see it the same way.
- jvanderbot 9y ago
  > Why would AI-driven vehicles not have dedicated, single-purpose subsystems such as anti-collision? I mean, are we going to also remove ABS, because the AI could learn to modulate the brakes itself? How much are we going to push into AI, when the purpose-built systems are both functional and effective? This makes me want to pull my hair out. No reasonable system relies on AI from top to bottom. Take this article with a grain of salt, it is attacking a straw-man.
- andreyk 9y ago
  "Why would AI-driven vehicles not have dedicated, single-purpose subsystems such as anti-collision? " The answer is that they will. And probably already mostly do (at least for Waymo/Cruise/most of them). It's not clear why Uber's vehicle did not brake.
mtgx 9y ago
I don't know what would have happened if "this guy programmed it", but the answer should be yes, the car should have seen the cyclist and it should have pressed the brakes. This was an interesting post, too, by Brad Templeton who worked on Google's self-driving car project for a while: http://ideas.4brad.com/almost-every-thing-went-wrong-uber-fatality-both-terrible-and-expected http://ideas.4brad.com/almost-every-thing-went-wrong-uber-fa...
- decacorn 9y ago
  in response to that piece by Brad, I sincerely hope that the "safety" driver in the uber accident was fired.
  - mulmen 9y ago
    Why? If it is not proven the driver was negligent why fire this individual? Presumably this individual has a lot of experience and domain knowledge in testing self driving cars so replacing them with someone else may not be an improvement. If they did make a mistake that got someone killed that could change things but I would hope we wait to find out if the driver was actually at fault.
    - amaranth 9y ago
      I don't think Uber's safety drivers have any domain knowledge. The law says you need bodies in seats so Uber is paying as little as possible to put bodies in seats. They aren't engineers or professional (as in stunt/trick/etc) drivers.
      - deleted 9y ago
        [deleted]
      - deleted 9y ago
        [deleted]
      - mulmen 9y ago
        I’m not sure stunt or trick drivers are the best choice anyway but you make a good point. Do we know this individual’s qualifications or are we just guessing at Uber’s standards?
SlowRobotAhead 9y ago
Has anyone been able to even remotely explain why the LIDAR system wasn't going nuts? I saw the "it was dark" nonsense, but I assume this vehicle had laser and IR right? The camera footage was released, I'd like to see the lidar representation.
- icc97 9y ago
  I thought there were suggestions that the LIDAR was turned off for testing.
- blensor 9y ago
  The closest I can give you is a simulation of a scene modeled after the video using a very simple reconstruction of the site of the accident (based on google maps and street view) http://www.blensor.org/lidar_accident.html http://www.blensor.org/lidar_accident.html
- InclinedPlane 9y ago
  A couple possibilities here. The LIDAR was turned off. Or the system just blatantly failed to categorize the pedestrian properly. Given that she was walking a bike that had bags on the handlebars this is my guess, the system just choked on that and was too stupid to brake for any object in the road that it might hit.
- justin66 9y ago
  What makes you think it wasn't? You're making some assumptions about which of the many systems failed, I think.
  - SlowRobotAhead 9y ago
    You know what makes me thing the LIDAR wasn’t working, the part about the dead woman.
- martinrebane 9y ago
  Author here :) LIDAR itself is just a sensor, it does not process the data nor does it output a directly usable image like a camera more or less does. LIDAR in this case is a rotating laser and while it scans, the vehicle moves (imagine moving a paper when a copier scans it). All processing is done later, first to construct an image and then to understand and use it. Part of why I wrote the piece was to explain how things can go wrong even if your LIDAR works fine.
blensor 9y ago
If the technology were as undeterministic as this post makes it seem to be, there should not be any self driving car allowed even under the best conditions. I fully agree that an AI can be fooled, but that is high level logic (path planing), the system should be designed to have a fallback that does emergency braking if all else fails. There simply is a point where the high-level AI does not matter any more. And that is if I (the car) am moving at 45 mph towards an obstacle that is in the middle of the road less than 2 meters from my projected path. This does not mean that a full brake is required but the speed definitely needs to be reduced to account for the uncertainty, and once it is determined that it is physically impossible to miss the obstacle the system must do a full stop to reduce the impact velocity as much as possible. It's fine, if the LIDAR data is plugged into a machine learning algorithm, and you will probably get less than the 10-20 Hz the scanner can produce, but at the same time it is probably also used by a much simpler obstacle tracking algorithm that can run at near real-time speeds.
- dclowd9901 9y ago
  Right; if we want to compare it to ourselves, maybe the various minds that control our impulses. We can hold a knife over our finger with our high level cognitive mind, but (hopefully) our low level brain will prevent us from bringing it down and chopping our finger off, even if we _really really_ wanted to do it. I know that's sort of a weird example, but I think it's really illustrative of a dual/multiple mind scenario playing out in our own understanding.
  - blensor 9y ago
    Or reflex. We act on an input on a lower cognitive level first before a higher level function even has the chance to intervene. If someone is throwing a ball at my face, my body hopefully reacts before my higher level functions had the chance to evaluate if the ball will really hit the face and if I may look silly if I wave my hands in the air while no actual danger exists. Because it actually IS on a trajectory to my face the benefit far outweigh the risk of looking silly
wffurr 9y ago
They should release all of their sensor data and logs from the incident, and then some of us could actually answer the question in the headline. This should be standard procedure for any future incidents.
- bkanber 9y ago
  The NTSB has control of the investigation and the data for now. That should be standard procedure IMO.
icc97 9y ago
> Also, lets back up our AI’s with old school collision avoidance! Intelligence is not the same as perfection, at least for now. The car was travelling at 38mph and never braked. Even if the collision-avoidance only saw it at the last moment it would still have braked and potentially slowed the car enough so that the woman was injured instead of being killed. I'm all for self-driving and fully believe it can improve on humans, but I don't see how it's possible for self-driving cars to be on the road if they can't properly detect the most vulnerable users in all conditions.
- jvanderbot 9y ago
  This is absolutely a case where an autonomous system should have out-performed a manned system. Lasers see through darkness very well. A camera-only system is insufficient.
  - daveguy 9y ago
    Not to mention it wasn't even very dark on the road. If you look at other video and images, that stretch is very well lit. The dash cam video was poorly calibrated / selected (but probably not used in the decision process of the AI).
- ltrcola 9y ago
  Humans can't properly detect the most vulnerable users in all conditions. Not an excuse for this system failing to perform, but all self-driving cars have to do is be a little bit better for net safety to improve. People are notoriously unreliable. Computers don't get drunk or sleepy, for instance.
  - couchand 9y ago
    > all self-driving cars have to do is be a little bit better for net safety to improve. That's true abstractly but ignores several important real-world factors about the adoption of self-driving cars. On the one had, autonomous cars have to be a lot better than humans to prevent these sorts of PR trash can fires or they won't be given the opportunity to improve net safety. On the other hand, people are so bad that we're liable to soon live in a world of autonomous cars, regardless of the effect on net safety. I hope they can be made safe, because it's vital for the future of our car-obsessed culture. But I don't have as much faith as you.
lukemunn 9y ago
I've been really interested in exactly this question - the technical drive to revisit moments when contingency (and tragedy) emerges. I've been working on an artwork around this, "Iterated Accident", which I just put online this morning, http://darkmttr.xyz/16/ http://darkmttr.xyz/16/
natch 9y ago
I'd like to see a law that says: to obtain certification to operate a self-driving car on a roadway, an organization must agree to: 1) Conform to a standard set of protocols for how sensors provide data to a self-driving software system. 2) Log data in a form that could be submitted to any conforming self-driving software system, to obtain results from that system reporting what the system would do given these inputs. With this in place, it would be easy to do after-the-fact comparisons of data leading up to incidents, and learn from the differences in results between systems. It could be taken a step further if the car makers would also share data on near misses, which could uncover cases where other car makers' systems did not handle the situation as well. Even if the sensors are different, I suspect some good mileage could be gotten out of this. The fact that learning opportunities are not perfect, is not always a good reason to pass them up.
- lighthazard 9y ago
  Perhaps some sort of logging standard that is enforced across all systems? A black box of some sort?
  - natch 9y ago
    Yes, exactly, but with the additional factors I mentioned.
- couchand 9y ago
  Oh no, we could never have that. It would squash innovation! We can't possibly have any regulation of self-driving cars or we'll lose to China.
- skgoa 9y ago
  > 1) Conform to a standard set of protocols for how sensors provide data to a self-driving software system. Doesn't exist yet, though.
amanzi 9y ago
All I'm going to say is that I could tell this was a Medium post by the title...
- phyzome 9y ago
  I couldn't, but I could immediately tell once I clicked and the page was half-covered by persistent sharing dickbars. -.-
arithma 9y ago
Why isn't car driving AI put to the same rigour of testing and approval as that of airplanes? There are far more people on the road than in the sky, and yet, there's formal verification for airplane autopilot code but an accident like this is supposed to be "you couldn't have prevented it either?" If no one could prevent such a thing, this AI should never drive a car!
- ltrcola 9y ago
  More rigor is definitely required, but it may not be possible to formally verify AI driven code to the same extent. Aviation software is driven from formal requirements, has a strict set of coding rules, and the generated assembly code is compared to the original source code. Within the coding rules, the software isn't even allowed to dynamically allocate memory. I have no idea how you'd build an AI system with those constraints, given that the computer essentially programmed the model itself by learning.
hangonhn 9y ago
Hang on. That argument makes no sense. We've had subsumption architecture for a long time now( https://en.wikipedia.org/wiki/Subsumption_architecture https://en.wikipedia.org/wiki/Subsumption_architecture ). Subsumption architecture puts some of the intelligence in the lower level systems. While the higher level controls can tell the lower level systems what it wants, it can't do things that the lower system determines is dangerous. So if a normal Mercedez has a collision avoidance system that automatically brakes, there is no reason why an AI based system can't be built on top of that and the collision avoidance system automatically braking without intervention from the higher AI systems. A subsumption system prevents higher level controls from doing something catastrophic like hitting a pedestrian or, in biology, a person holding their breath until they die.
- btrettel 9y ago
  As I was skimming the article, I was thinking some sort of system like this could be made, and I was pleasantly surprised to see the approach has a long history. This is one reason I like Hacker News.
- saetaes 9y ago
  Doesn’t the author recommend exactly that in the last sentence of the article?
  - jfrankamp 9y ago
    Yes. For new readers, suggestion for this article: start with the last sentence. Then skim the article re AI. Then respond. So we don't have to keep saying/hearing "Why not both?". The author's take is: both!
    - TTPrograms 9y ago
      But let's make sure that the definition of AI doesn't encompass this because...?
      - glaberficken 9y ago
et2o 9y ago
This is an extremely unconvincing argument
jvanderbot 9y ago
Either the author works on toy systems, or is being disingenuous. These are not problems with AI, they are problems with using only AI, and no viable-for-development system that I'm aware of in industry does that.
jvanderbot 9y ago
I smell a straw man argument.
deleted 9y ago
[deleted]
dreamcompiler 9y ago
I am freaking sick of this notion that AI==ML. AI is a much bigger field than neural nets. AI can be programmed with rules, with logic, with symbols, with subsumption, and with a thousand other things that are both deterministic and don't require huge training sets. If you're making a living doing AI, you damn well should know this.
- akvadrako 9y ago
  If you define AI like that, then basically every decision making program, i.e. with branches, is AI.
  - 8note 9y ago
    how else are you going to include pacman's ghosts as using AI?
    - akvadrako 9y ago
      You shouldn't because those ghosts are stupid. If intelligence just means "can make decisions based on inputs" then everything is intelligent. That's a useless definition.
Animats 9y ago
This is looking at it all wrong. Watch Chris Urmson's video from SXSW on how Waymo does it. I've mentioned that before. You build a map of what's around the vehicle. Then you try to tag some obstacles to help predict their behavior. But the obstacle detection is geometric, and does not depend on the tagging, which uses machine learning. If there's an obstacle, the system doesn't hit it, even if it has no clue how to identify it. Tesla/Mobileye managed to get that backwards. Their original system was "recognize vehicle visually, compute distance and closing rate to vehicle". If it didn't recognize an obstacle as a vehicle, it ignored it. We know this for sure, because you can buy a Mobileye unit as a dashcam-like warning device and many people have seen how they work. That led to three collisions with vehicles partly blocking the left side of a lane. One death ramming a street sweeper, one collision with a stopped fire truck, one sideswipe. The NTSB is investigating the fire truck collision. The NTSB is now investigating the Uber collision.[1] As they usually do, the first thing they did was to get control of the wreckage.[2] Uber does not have control of the investigation. The NTSB investigators are working this like an air crash. They are "beginning collection of any and all electronic data stored on the test vehicle or transmitted to Uber". As usual, they haven't announced much, but they have mentioned that the video seen publicly is from a third-party dashcam, not the vehicle sensors. [1] https://www.ntsb.gov/news/press-releases/Pages/NR20180320.aspx https://www.ntsb.gov/news/press-releases/Pages/NR20180320.as... [2] http://wsau.com/news/articles/2018/mar/21/arizona-police-release-video-of-fatal-collision-with-uber-self-driving-suv/ http://wsau.com/news/articles/2018/mar/21/arizona-police-rel...
- philwelch 9y ago
  Devils advocate here, but what if a supposed “obstacle” is a plastic bag or a jet of steam coming out of a sewer grate?
  - dreamcompiler 9y ago
    Then you stop. Type 2 errors for cars are much less dangerous than Type 1 errors.
    - philwelch 9y ago
justicezyx 9y ago
This guy seems specificially talking about a certain style of ai approach.
magoon 9y ago
Unpopular question but here goes: What if some crashes are unavoidable? e.g. somebody darts out in front of a mobile vehicle. We accept that trains are not at fault for striking “trespassers” on their railways. Also, when we all drive cars with collision avoidance systems, who gets sued by whom? If my car e-brakes for no reason and I get rear-ended, is the guy who hits me still at fault like usual? I believe computer control will be super helpful and is here to stay, but it’s interesting to see it implemented in cars as emergency help versus in modern commercial airliners (where autopilot and landing control systems are ubiquitous) when it is only relied on for the most routine and straight-shot ability.
- jsjohnst 9y ago
  > If my car e-brakes for no reason and I get rear-ended, is the guy who hits me still at fault like usual? What’s different about your SDC slamming on its brakes vs you doing the same thing? That’s why the law says how much spacing is to be between cars! > What if some crashes are unavoidable? From all evidence which is publicly available at this time, it appears this accident was anything but unavoidable, so the question is a red herring.
- cesarb 9y ago
  > What if some crashes are unavoidable? e.g. somebody darts out in front of a mobile vehicle. It should brake as much as possible. Even if the collision is unavoidable, reducing the kinetic energy available for the collision is still a good idea. > We accept that trains are not at fault for striking “trespassers” on their railways. When a train "detects" an "obstacle" (the train driver sees the "trespasser"), it goes into emergency braking. > Also, when we all drive cars with collision avoidance systems, who gets sued by whom? If my car e-brakes for no reason and I get rear-ended, is the guy who hits me still at fault like usual? When we all drive cars with enough automation, the automation should keep enough distance that the following car can brake without colliding. > but it’s interesting to see it implemented in cars as emergency help versus in modern commercial airliners (where autopilot and landing control systems are ubiquitous) when it is only relied on for the most routine and straight-shot ability. Airplanes have the unfortunate property that they can't simply stop in an emergency; stopping would be an even bigger emergency. Cars (and trains) can simply stop, they won't fall from the sky in that case. But even then, airplanes do have emergency help from their automation: the TCAS has a similar purpose to a car's automated emergency braking, that is, to prevent a collision.
andreyk 9y ago
This is shoddily written bunk. I would downvote if I could. Object detection from LIDAR/Radar data plus path planning is 'AI-based', and I guarantee you Waymo's systems (which are 'AI-based', though more old school and less ML based) would have done collision avoidance better than Uber.
wirrbel 9y ago
I admit I stopped reading halfway through, but managed to scroll to the last paragraph. This article is a dance through many important topics in a AV. Yet, it fails to actually answer the "Why collision avoidance is harder fo an AI-based system" question, really. Some arguments argue that systems with a smaller scope are easier, systems with a larger scope are more difficult, it brings on arguments about determinism in decision making. It brings on sensor sets, neither is really about AI or hand-crafted rules, but about problems inherent to robotics as a whole. Again, it is a fine example why the therm AI is useless and harmful for discussions, as it blurs what is talked about considerably.
kristoffer 9y ago
The author writes a long text about how this is "not as easy for an autonomous system to solve" just to make the obvious point at the end of the article. It should be obvious to anyone that you need to compose systems of different criticality to build a safe autonomous vehicle. Of course the "AI" system needs to be complemented with a safety critical auto brake and other fail safes.
cameldrv 9y ago
My educated guess: The somewhat unusual shape of the combination of the woman and the bicycle, combined with the uneven lighting caused the vehicle to misclassify her as lightweight road debris. To a real degree, this is a downfall of machine learning. Every distribution has tails. If we learn purely from data, rather than from principle, we will necessarily make mistakes on the tails. For problems that can be effectively solved with 99% accuracy, this is fine, and we just deal with a few mistakes. With more data, our accuracy will improve anyhow. If a datapoint costs a human life though, we can't afford to collect enough data. We must have a more sophisticated model of the world in order to operate on the tails without killing people. I think that this might actually be a watershed moment for ML. Supervised learning is not adequate to this type of task. Either the computer does low level perception, and a human writes a high level algorithm to manage the risk, or datapoints have to contain a lot more information than just safe/unsafe. When you made a mistake as a child, your parents didn't just punish you, they explained what you did wrong and why, and a rule to follow to do better next time.
- invalidusernam3 9y ago
  > misclassify her as lightweight road debris. Surely the classification has a confidence level, and a low confidence score should cause the vehicle to slow down if it's not confident in knowing what it's looking at? Also, the size of the "lightweight road debris" should have made the vehicle slow down slightly at least, because hitting a 6ft pile of paper wouldn't be great at even a low speed

118 points by IndrekR 9y ago

> AI is not preprogrammed to monitor a known input from a sensor to take a predefined action. I guess one of my outstanding questions, which reading this only confirmed, is why this is the case? I mean, humans are pretty good examples of intelligence. And yet we still have and use these anti-collision systems. Because, in the end, when wrong decisions are made these systems save lives. Why would AI-driven vehicles not have dedicated, single-purpose subsystems such as anti-collision? I mean, are we going to also remove ABS, because the AI could learn to modulate the brakes itself? How much are we going to push into AI, when the purpose-built systems are both functional and effective?

SlowRobotAhead 9y ago
>I mean, are we going to also remove ABS, because the AI could learn to modulate the brakes itself? This is the topic that I feel SDC enthusiasts forget. Not everything HAS TO BE AI. And we could maybe make steps towards SDC, not fantastic leaps that get people killed and really just get government involved where it doesn't need to be yet. We could replace the ABS/ESP/traction control systems in current vehicles with a machine learning / deep learning system absolutely - that's not sexy though! No one wants increments, they want whole self-driving cars right now - and while I have opinions about that, there is no doubt it's driving (pun intended) the industry. Ideally, at first, we'd see components in consumer vehicles, and completely automated long haul trucks from A to B determined routes - but like I said, not sexy.
- couchand 9y ago
  > Ideally, at first, we'd see components in consumer vehicles We do, though. Look at the cruise control or auto-park on a vehicle produced in the last few years. The totally autonomous car may make headlines, but these sorts of features will be what really make the technology ubiquitous.
  - SlowRobotAhead 9y ago
    I work on ECUs like this. High end stuff is doing computer vision, but 99% of what is on the road is pre-calibrated motor/solenoid control and that’s all. Even adaptive cruise, steel camera object “detection” are just pretty simple systems. Almost nothing is doing even pieces of what the whole-package SDCs are.
mulmen 9y ago
I don't know much about AI or ML and even less about how it relates to automotive engineering but I imagine this is already how it works. I don't think there is one model for everything as much as there are models for individual components of driving. Something for steering, something(s?) for identifying obstructions, etc. In this case maybe letting a model modulate brake pulses based on conditions like temperature and velocity makes more sense? Hopefully someone with more experience in this domain can enlighten us.
- skgoa 9y ago
  It is exactly how it works. There ~40-115 different computers in a modern car that often have completely walled-off modules running on them. Most 'models' you will find in a car's software aren't even generated through machine learning, but tried and true engineering/statistics. No one in their right mind is proposing to replace those with one big AI monolith. That's marketing BS from the likes of NVidia, who would stand to gain from it. AI is being used in certain subfunctions like visual object detection. In the future, AI will be used to make higher level decisions like trajectory (lane choice, overtaking etc.) and route planning. But it will only hand those higher level plans down to the mostly 'dumb' computer systems of the car to be carried out just as with a manually driven car.
phaedrus 9y ago
This is a design trade-off that comes up all the time in automotive engineering. I own a couple of 90's-era Honda vehicles and what's interesting is although the cars are computerized, they also employ a number of semi-independent systems such as instrument gauges which directly connect to their own sensors even though a similar sensor connected to the engine control unit is right next to it, mechanical choke and coolant temperature modulators to intake air despite fuel injection, an "hydraulic computer" controlling part of the automatic transmission even though there is also a dedicated transmission control computer (which is itself redundant to making the transmission control an integrated feature of the engine control module). To me this seems to make no design sense; why not bring all of the inputs into one central box with one computer control, and drive all of the outputs from this central box? But then who am I to argue that the separate / bespoke approach is not better, given Honda's reputation for reliability?
- majormajor 9y ago
  Seems like classic, prudent, avoiding of single points of failure. Sounds reasonable, especially since computerization in cars was still pretty new back then. For something like ABS, or emergency braking, we have established algorithms for this already, so why be in a hurry to offload that to a machine-learning black box? Is the math behind "we need to stop this car" so complex that AI is needed?
martinrebane 9y ago
Author here :) I cannot answer for Uber, but the grand reason why we cannot put deterministic parts into the AI is the same as for humans - we do not control the internals of it, the complexity is too high. Although we can build systems that are composed of both AI and deterministic collision avoidance (as seen on new cars today). It is what I also suggested in the article.
- wnoise 9y ago
  Ah, then I should complain to you that "AI" is a broad suite of techniques and goals, rather than a specific black-box technique which has these flaws.
  - andreyk 9y ago
    Yes - you really seem to be talking about Machine Learning (a subset of AI). And as another comment says 'This makes me want to pull my hair out. No reasonable system relies on AI [or ML] from top to bottom. Take this article with a grain of salt, it is attacking a straw-man.'
CardenB 9y ago
The false positive rate of this system could be very noisy. At what point should you brake, even if the AI says not to? When you might collide with a lidar point? You want to collide with lidar points all the time (the ground beneath the vehicle, for example, or any reflective noise). The AI is there to tell you what lidar points are actual obstacles that you can't collide with, otherwise it's not really meaningful.
- jdmichal 9y ago
  This line of argument doesn't mean anything when we already have cars with intelligent, independent drivers, and these systems. And yes, their purpose would be to override the AI control when they trigger. Exactly the same as they do right now.
jmalicki 9y ago
" I mean, are we going to also remove ABS, because the AI could learn to modulate the brakes itself?" I think that's a reasonable definition of ABS - an AI (in most cases for ABS, an expert system AI) that knows how to best mediate the brakes, with levels of performance that exceed the vast majority of humans.
- ams6110 9y ago
  The conventional ABS system is not AI in any sense. It is a few sensors and maybe some look up tables of what to do under various speed / brake pressure / wheel rotation situations
  - randomdata 9y ago
    Does that not fit the definition of artificial intelligence? The system makes decisions – ones that would be otherwise done by a human – based on its perception of the world. That is what I have always understood AI to mean. To be sure, a conventional ABS system does not use machine learning, but machine learning is only a subset of AI.
    - skgoa 9y ago
      The core of this debate is the matter of what intelligence actually is. We don't really know, but the most common definition I have heard/read is that intelligence is the ability to generate new solutions when presented with previously unseen input. An ABS would not qualify under this definition, because its output is predefined rigorously for all possible combinations of input values.
      - randomdata 9y ago
        On the other hand, artificial intelligence is not actual intelligence. The definition of artificial intelligence is the ability to perform a task that would normally require human intelligence. An ABS system would fit into this definition as it is a task that normally would require human sensing the conditions to know how to apply the breaks. Although I can see why some point out that AI is a moving target, representing only what still seems 'magical'. If you showed an ABS system to someone in the early 1900s, I truly believe they would see it as some kind of intelligence. Now that we have acclimated to the technology, we don't see it the same way.
jvanderbot 9y ago
> Why would AI-driven vehicles not have dedicated, single-purpose subsystems such as anti-collision? I mean, are we going to also remove ABS, because the AI could learn to modulate the brakes itself? How much are we going to push into AI, when the purpose-built systems are both functional and effective? This makes me want to pull my hair out. No reasonable system relies on AI from top to bottom. Take this article with a grain of salt, it is attacking a straw-man.
andreyk 9y ago
"Why would AI-driven vehicles not have dedicated, single-purpose subsystems such as anti-collision? " The answer is that they will. And probably already mostly do (at least for Waymo/Cruise/most of them). It's not clear why Uber's vehicle did not brake.

mtgx 9y ago

I don't know what would have happened if "this guy programmed it", but the answer should be yes, the car should have seen the cyclist and it should have pressed the brakes. This was an interesting post, too, by Brad Templeton who worked on Google's self-driving car project for a while: http://ideas.4brad.com/almost-every-thing-went-wrong-uber-fatality-both-terrible-and-expected http://ideas.4brad.com/almost-every-thing-went-wrong-uber-fa...

decacorn 9y ago
in response to that piece by Brad, I sincerely hope that the "safety" driver in the uber accident was fired.
- mulmen 9y ago
  Why? If it is not proven the driver was negligent why fire this individual? Presumably this individual has a lot of experience and domain knowledge in testing self driving cars so replacing them with someone else may not be an improvement. If they did make a mistake that got someone killed that could change things but I would hope we wait to find out if the driver was actually at fault.
  - amaranth 9y ago
    I don't think Uber's safety drivers have any domain knowledge. The law says you need bodies in seats so Uber is paying as little as possible to put bodies in seats. They aren't engineers or professional (as in stunt/trick/etc) drivers.
    - deleted 9y ago
      [deleted]
    - deleted 9y ago
      [deleted]
    - mulmen 9y ago
      I’m not sure stunt or trick drivers are the best choice anyway but you make a good point. Do we know this individual’s qualifications or are we just guessing at Uber’s standards?

SlowRobotAhead 9y ago

Has anyone been able to even remotely explain why the LIDAR system wasn't going nuts? I saw the "it was dark" nonsense, but I assume this vehicle had laser and IR right? The camera footage was released, I'd like to see the lidar representation.

icc97 9y ago
I thought there were suggestions that the LIDAR was turned off for testing.
blensor 9y ago
The closest I can give you is a simulation of a scene modeled after the video using a very simple reconstruction of the site of the accident (based on google maps and street view) http://www.blensor.org/lidar_accident.html http://www.blensor.org/lidar_accident.html
InclinedPlane 9y ago
A couple possibilities here. The LIDAR was turned off. Or the system just blatantly failed to categorize the pedestrian properly. Given that she was walking a bike that had bags on the handlebars this is my guess, the system just choked on that and was too stupid to brake for any object in the road that it might hit.
justin66 9y ago
What makes you think it wasn't? You're making some assumptions about which of the many systems failed, I think.
- SlowRobotAhead 9y ago
  You know what makes me thing the LIDAR wasn’t working, the part about the dead woman.
martinrebane 9y ago
Author here :) LIDAR itself is just a sensor, it does not process the data nor does it output a directly usable image like a camera more or less does. LIDAR in this case is a rotating laser and while it scans, the vehicle moves (imagine moving a paper when a copier scans it). All processing is done later, first to construct an image and then to understand and use it. Part of why I wrote the piece was to explain how things can go wrong even if your LIDAR works fine.

blensor 9y ago

If the technology were as undeterministic as this post makes it seem to be, there should not be any self driving car allowed even under the best conditions. I fully agree that an AI can be fooled, but that is high level logic (path planing), the system should be designed to have a fallback that does emergency braking if all else fails. There simply is a point where the high-level AI does not matter any more. And that is if I (the car) am moving at 45 mph towards an obstacle that is in the middle of the road less than 2 meters from my projected path. This does not mean that a full brake is required but the speed definitely needs to be reduced to account for the uncertainty, and once it is determined that it is physically impossible to miss the obstacle the system must do a full stop to reduce the impact velocity as much as possible. It's fine, if the LIDAR data is plugged into a machine learning algorithm, and you will probably get less than the 10-20 Hz the scanner can produce, but at the same time it is probably also used by a much simpler obstacle tracking algorithm that can run at near real-time speeds.

dclowd9901 9y ago
Right; if we want to compare it to ourselves, maybe the various minds that control our impulses. We can hold a knife over our finger with our high level cognitive mind, but (hopefully) our low level brain will prevent us from bringing it down and chopping our finger off, even if we _really really_ wanted to do it. I know that's sort of a weird example, but I think it's really illustrative of a dual/multiple mind scenario playing out in our own understanding.
- blensor 9y ago
  Or reflex. We act on an input on a lower cognitive level first before a higher level function even has the chance to intervene. If someone is throwing a ball at my face, my body hopefully reacts before my higher level functions had the chance to evaluate if the ball will really hit the face and if I may look silly if I wave my hands in the air while no actual danger exists. Because it actually IS on a trajectory to my face the benefit far outweigh the risk of looking silly

wffurr 9y ago

They should release all of their sensor data and logs from the incident, and then some of us could actually answer the question in the headline. This should be standard procedure for any future incidents.

bkanber 9y ago
The NTSB has control of the investigation and the data for now. That should be standard procedure IMO.

icc97 9y ago

> Also, lets back up our AI’s with old school collision avoidance! Intelligence is not the same as perfection, at least for now. The car was travelling at 38mph and never braked. Even if the collision-avoidance only saw it at the last moment it would still have braked and potentially slowed the car enough so that the woman was injured instead of being killed. I'm all for self-driving and fully believe it can improve on humans, but I don't see how it's possible for self-driving cars to be on the road if they can't properly detect the most vulnerable users in all conditions.

jvanderbot 9y ago
This is absolutely a case where an autonomous system should have out-performed a manned system. Lasers see through darkness very well. A camera-only system is insufficient.
- daveguy 9y ago
  Not to mention it wasn't even very dark on the road. If you look at other video and images, that stretch is very well lit. The dash cam video was poorly calibrated / selected (but probably not used in the decision process of the AI).
ltrcola 9y ago
Humans can't properly detect the most vulnerable users in all conditions. Not an excuse for this system failing to perform, but all self-driving cars have to do is be a little bit better for net safety to improve. People are notoriously unreliable. Computers don't get drunk or sleepy, for instance.
- couchand 9y ago
  > all self-driving cars have to do is be a little bit better for net safety to improve. That's true abstractly but ignores several important real-world factors about the adoption of self-driving cars. On the one had, autonomous cars have to be a lot better than humans to prevent these sorts of PR trash can fires or they won't be given the opportunity to improve net safety. On the other hand, people are so bad that we're liable to soon live in a world of autonomous cars, regardless of the effect on net safety. I hope they can be made safe, because it's vital for the future of our car-obsessed culture. But I don't have as much faith as you.

lukemunn 9y ago

I've been really interested in exactly this question - the technical drive to revisit moments when contingency (and tragedy) emerges. I've been working on an artwork around this, "Iterated Accident", which I just put online this morning, http://darkmttr.xyz/16/ http://darkmttr.xyz/16/

natch 9y ago

I'd like to see a law that says: to obtain certification to operate a self-driving car on a roadway, an organization must agree to: 1) Conform to a standard set of protocols for how sensors provide data to a self-driving software system. 2) Log data in a form that could be submitted to any conforming self-driving software system, to obtain results from that system reporting what the system would do given these inputs. With this in place, it would be easy to do after-the-fact comparisons of data leading up to incidents, and learn from the differences in results between systems. It could be taken a step further if the car makers would also share data on near misses, which could uncover cases where other car makers' systems did not handle the situation as well. Even if the sensors are different, I suspect some good mileage could be gotten out of this. The fact that learning opportunities are not perfect, is not always a good reason to pass them up.

lighthazard 9y ago
Perhaps some sort of logging standard that is enforced across all systems? A black box of some sort?
- natch 9y ago
  Yes, exactly, but with the additional factors I mentioned.
couchand 9y ago
Oh no, we could never have that. It would squash innovation! We can't possibly have any regulation of self-driving cars or we'll lose to China.
skgoa 9y ago
> 1) Conform to a standard set of protocols for how sensors provide data to a self-driving software system. Doesn't exist yet, though.

amanzi 9y ago

All I'm going to say is that I could tell this was a Medium post by the title...

phyzome 9y ago
I couldn't, but I could immediately tell once I clicked and the page was half-covered by persistent sharing dickbars. -.-

arithma 9y ago

Why isn't car driving AI put to the same rigour of testing and approval as that of airplanes? There are far more people on the road than in the sky, and yet, there's formal verification for airplane autopilot code but an accident like this is supposed to be "you couldn't have prevented it either?" If no one could prevent such a thing, this AI should never drive a car!

ltrcola 9y ago
More rigor is definitely required, but it may not be possible to formally verify AI driven code to the same extent. Aviation software is driven from formal requirements, has a strict set of coding rules, and the generated assembly code is compared to the original source code. Within the coding rules, the software isn't even allowed to dynamically allocate memory. I have no idea how you'd build an AI system with those constraints, given that the computer essentially programmed the model itself by learning.

hangonhn 9y ago

Hang on. That argument makes no sense. We've had subsumption architecture for a long time now( https://en.wikipedia.org/wiki/Subsumption_architecture https://en.wikipedia.org/wiki/Subsumption_architecture ). Subsumption architecture puts some of the intelligence in the lower level systems. While the higher level controls can tell the lower level systems what it wants, it can't do things that the lower system determines is dangerous. So if a normal Mercedez has a collision avoidance system that automatically brakes, there is no reason why an AI based system can't be built on top of that and the collision avoidance system automatically braking without intervention from the higher AI systems. A subsumption system prevents higher level controls from doing something catastrophic like hitting a pedestrian or, in biology, a person holding their breath until they die.

btrettel 9y ago
As I was skimming the article, I was thinking some sort of system like this could be made, and I was pleasantly surprised to see the approach has a long history. This is one reason I like Hacker News.
saetaes 9y ago
Doesn’t the author recommend exactly that in the last sentence of the article?
- jfrankamp 9y ago
  Yes. For new readers, suggestion for this article: start with the last sentence. Then skim the article re AI. Then respond. So we don't have to keep saying/hearing "Why not both?". The author's take is: both!
  - TTPrograms 9y ago
    But let's make sure that the definition of AI doesn't encompass this because...?
    - glaberficken 9y ago

et2o 9y ago

This is an extremely unconvincing argument

jvanderbot 9y ago

Either the author works on toy systems, or is being disingenuous. These are not problems with AI, they are problems with using only AI, and no viable-for-development system that I'm aware of in industry does that.

jvanderbot 9y ago

I smell a straw man argument.

deleted 9y ago

[deleted]

dreamcompiler 9y ago

I am freaking sick of this notion that AI==ML. AI is a much bigger field than neural nets. AI can be programmed with rules, with logic, with symbols, with subsumption, and with a thousand other things that are both deterministic and don't require huge training sets. If you're making a living doing AI, you damn well should know this.

akvadrako 9y ago
If you define AI like that, then basically every decision making program, i.e. with branches, is AI.
- 8note 9y ago
  how else are you going to include pacman's ghosts as using AI?
  - akvadrako 9y ago
    You shouldn't because those ghosts are stupid. If intelligence just means "can make decisions based on inputs" then everything is intelligent. That's a useless definition.

Animats 9y ago

This is looking at it all wrong. Watch Chris Urmson's video from SXSW on how Waymo does it. I've mentioned that before. You build a map of what's around the vehicle. Then you try to tag some obstacles to help predict their behavior. But the obstacle detection is geometric, and does not depend on the tagging, which uses machine learning. If there's an obstacle, the system doesn't hit it, even if it has no clue how to identify it. Tesla/Mobileye managed to get that backwards. Their original system was "recognize vehicle visually, compute distance and closing rate to vehicle". If it didn't recognize an obstacle as a vehicle, it ignored it. We know this for sure, because you can buy a Mobileye unit as a dashcam-like warning device and many people have seen how they work. That led to three collisions with vehicles partly blocking the left side of a lane. One death ramming a street sweeper, one collision with a stopped fire truck, one sideswipe. The NTSB is investigating the fire truck collision. The NTSB is now investigating the Uber collision.[1] As they usually do, the first thing they did was to get control of the wreckage.[2] Uber does not have control of the investigation. The NTSB investigators are working this like an air crash. They are "beginning collection of any and all electronic data stored on the test vehicle or transmitted to Uber". As usual, they haven't announced much, but they have mentioned that the video seen publicly is from a third-party dashcam, not the vehicle sensors. [1] https://www.ntsb.gov/news/press-releases/Pages/NR20180320.aspx https://www.ntsb.gov/news/press-releases/Pages/NR20180320.as... [2] http://wsau.com/news/articles/2018/mar/21/arizona-police-release-video-of-fatal-collision-with-uber-self-driving-suv/ http://wsau.com/news/articles/2018/mar/21/arizona-police-rel...

philwelch 9y ago
Devils advocate here, but what if a supposed “obstacle” is a plastic bag or a jet of steam coming out of a sewer grate?
- dreamcompiler 9y ago
  Then you stop. Type 2 errors for cars are much less dangerous than Type 1 errors.
  - philwelch 9y ago

justicezyx 9y ago

This guy seems specificially talking about a certain style of ai approach.

magoon 9y ago

Unpopular question but here goes: What if some crashes are unavoidable? e.g. somebody darts out in front of a mobile vehicle. We accept that trains are not at fault for striking “trespassers” on their railways. Also, when we all drive cars with collision avoidance systems, who gets sued by whom? If my car e-brakes for no reason and I get rear-ended, is the guy who hits me still at fault like usual? I believe computer control will be super helpful and is here to stay, but it’s interesting to see it implemented in cars as emergency help versus in modern commercial airliners (where autopilot and landing control systems are ubiquitous) when it is only relied on for the most routine and straight-shot ability.

jsjohnst 9y ago
> If my car e-brakes for no reason and I get rear-ended, is the guy who hits me still at fault like usual? What’s different about your SDC slamming on its brakes vs you doing the same thing? That’s why the law says how much spacing is to be between cars! > What if some crashes are unavoidable? From all evidence which is publicly available at this time, it appears this accident was anything but unavoidable, so the question is a red herring.
cesarb 9y ago
> What if some crashes are unavoidable? e.g. somebody darts out in front of a mobile vehicle. It should brake as much as possible. Even if the collision is unavoidable, reducing the kinetic energy available for the collision is still a good idea. > We accept that trains are not at fault for striking “trespassers” on their railways. When a train "detects" an "obstacle" (the train driver sees the "trespasser"), it goes into emergency braking. > Also, when we all drive cars with collision avoidance systems, who gets sued by whom? If my car e-brakes for no reason and I get rear-ended, is the guy who hits me still at fault like usual? When we all drive cars with enough automation, the automation should keep enough distance that the following car can brake without colliding. > but it’s interesting to see it implemented in cars as emergency help versus in modern commercial airliners (where autopilot and landing control systems are ubiquitous) when it is only relied on for the most routine and straight-shot ability. Airplanes have the unfortunate property that they can't simply stop in an emergency; stopping would be an even bigger emergency. Cars (and trains) can simply stop, they won't fall from the sky in that case. But even then, airplanes do have emergency help from their automation: the TCAS has a similar purpose to a car's automated emergency braking, that is, to prevent a collision.

andreyk 9y ago

This is shoddily written bunk. I would downvote if I could. Object detection from LIDAR/Radar data plus path planning is 'AI-based', and I guarantee you Waymo's systems (which are 'AI-based', though more old school and less ML based) would have done collision avoidance better than Uber.

wirrbel 9y ago

I admit I stopped reading halfway through, but managed to scroll to the last paragraph. This article is a dance through many important topics in a AV. Yet, it fails to actually answer the "Why collision avoidance is harder fo an AI-based system" question, really. Some arguments argue that systems with a smaller scope are easier, systems with a larger scope are more difficult, it brings on arguments about determinism in decision making. It brings on sensor sets, neither is really about AI or hand-crafted rules, but about problems inherent to robotics as a whole. Again, it is a fine example why the therm AI is useless and harmful for discussions, as it blurs what is talked about considerably.

kristoffer 9y ago

The author writes a long text about how this is "not as easy for an autonomous system to solve" just to make the obvious point at the end of the article. It should be obvious to anyone that you need to compose systems of different criticality to build a safe autonomous vehicle. Of course the "AI" system needs to be complemented with a safety critical auto brake and other fail safes.

cameldrv 9y ago

My educated guess: The somewhat unusual shape of the combination of the woman and the bicycle, combined with the uneven lighting caused the vehicle to misclassify her as lightweight road debris. To a real degree, this is a downfall of machine learning. Every distribution has tails. If we learn purely from data, rather than from principle, we will necessarily make mistakes on the tails. For problems that can be effectively solved with 99% accuracy, this is fine, and we just deal with a few mistakes. With more data, our accuracy will improve anyhow. If a datapoint costs a human life though, we can't afford to collect enough data. We must have a more sophisticated model of the world in order to operate on the tails without killing people. I think that this might actually be a watershed moment for ML. Supervised learning is not adequate to this type of task. Either the computer does low level perception, and a human writes a high level algorithm to manage the risk, or datapoints have to contain a lot more information than just safe/unsafe. When you made a mistake as a child, your parents didn't just punish you, they explained what you did wrong and why, and a rule to follow to do better next time.

invalidusernam3 9y ago
> misclassify her as lightweight road debris. Surely the classification has a confidence level, and a low confidence score should cause the vehicle to slow down if it's not confident in knowing what it's looking at? Also, the size of the "lightweight road debris" should have made the vehicle slow down slightly at least, because hitting a 6ft pile of paper wouldn't be great at even a low speed