14 ms·

This is why I believe the approach of incremental improvement towards full self driving is fundamentally flawed. These advanced driver assist tools are good eno

by gundmc 5y ago

This is why I believe the approach of incremental improvement towards full self driving is fundamentally flawed. These advanced driver assist tools are good enough to lull users into a false sense of security. No amount of "but our terms and conditions say you need to always pay attention!" will overcome human nature building that trust and dependence.

judge2020 5y ago
Your reasoning doesn’t apply to the incremental improvements to self-driving, rather Tesla’s decision to allow all cars to use TACC/auto-steer. They haven’t even given people “the button” to enroll in FSD beta, likely because they know it would be extremely bad PR when a bunch of people use it without paying attention.
nathias 5y ago
that's not human nature, that's user stupidity
- weird-eye-issue 5y ago
  What's the... Difference?
  - nathias 5y ago
    One is a permanent property of our nature the other a choice.
    - weird-eye-issue 5y ago
      No, one is a fact and the other is a consequence
- formerly_proven 5y ago
  The design is fine, it's all the users who are idiots. P.S. /s. Obviously, Mr. Poe.
  - rvz 5y ago
    Well they did not 'pay' attention. They 'paid' for the "Fools Self Driving" package. This is why 'attention' and 'driver monitoring' was not included.
  - salawat 5y ago
    T. Every maligned designer when someone points out a flaw It's okay. I do it too. Really need to work on seeing yourself making that argument as a starting point and not an endpoint.
- deleted 5y ago
  [deleted]
mnmmn123456 5y ago
Today, there is at least one of the most advanced Neural Networks entering each car: A human being. If we could just implement the AI to add to this person and not replace it...
- salawat 5y ago
  This is the bit nobody likes to realize. FSD at it's best...is still about as fallible as a human driver. Minus free will (it is hoped). I will be amused, intrigued, and possibly a bit horrified if by the time FSD hits level 5, and they stick with the Neural Net of Neural Nets architecture if there isn't a rash of system induced variance in behavior as emergent phenomena take shape. Imagined news: All Tesla's on I-95 engaged in creating patterns whereby all non-Teala traffic was bordered by a Tesla on each side. Almost like a game of Go, says expert. Researchers stumped. Then again, that's imply you had an NN capable of retraining itself on the fly to some limited degree, which I assume no one sane would put into service... Hopefully this comment doesn't suffer a date of not aging well.
- ben_w 5y ago
  What would such an AI even look like? If it spots every real danger but also hallucinates even a few dangers that aren’t really there, it gets ignored or switched off for needlessly slowing the traveler down (false positives, apparently an issue with early Google examples [0]); if it only spots real dangers but misses most of them, it is not helping (false negatives, even worse if a human is blindly assuming the machine knows best and what happened with e.g. Uber [1]); if it’s about the same as humans overall but makes different types of mistake, people rely on it right up until it crashes then go apoplectic because it didn’t see something any human would consider obvious (e.g. Tesla, which gets slightly safer when the AI is active, but people keep showing the AI getting confused about things that they consider obvious [2]). [0] https://theoatmeal.com/blog/google_self_driving_car https://theoatmeal.com/blog/google_self_driving_car [1] https://en.wikipedia.org/wiki/Death_of_Elaine_Herzberg https://en.wikipedia.org/wiki/Death_of_Elaine_Herzberg [2] https://youtube.com/watch?v=Wz6Ins1D9ak https://youtube.com/watch?v=Wz6Ins1D9ak
cmpb 5y ago
I disagree. One feature my car has is to pull me back into the lane when I veer out of it (Subaru's lane keep assist). That is still incremental improvement towards "full self driving". I agree, however, that Tesla's Autopilot is not functional enough, and any tool designed to allow humans to remove their hands from the wheel should not require their immediate attention in any way.
- tapoxi 5y ago
  I think people just assume Tesla's Autopilot is more capable than it really is. My car has adaptive cruise control and lane keep assist, but I'm not relying on either for anything more complex than sipping a drink while on the highway.
  - rmckayfleming 5y ago
    Yep, if anything they’re just a way to make long drives or stop and go highway traffic more tolerable. When I got my first car with those features it seemed like a gimmick, but they really help to reduce fatigue.
- Robotbeat 5y ago
  Tesla’s autopilot does not allow you to remove your hands from the wheel. You must keep them on, and apply torque occasionally, to keep it engaged.
  - Karunamon 5y ago
    In reality, it'll let you get away with going handsfree for upwards of 30 seconds. That's more than long enough to lose your attention.
- darkerside 5y ago
  I think that's actually a step towards a local maximum that makes us less likely to achieve actual FSD. The safer we can make AI-guided driving where a person is still in control, the higher the bar becomes for a solo AI to be significantly safer than the alternatives.
mrfusion 5y ago
I’d be curious if there are studies out there on how to do automated assists in machines that require vigilance that don’t have this problem.
- jdavis703 5y ago
  Both airplanes and trains have automated “assist.” At least in the case of WMATA they give up on automatic train control after a fatal crash.
  - userbinator 5y ago
    In a plane you also have far more time to react once the autopilot disconnects for whatever reason, than the fraction of a second that a car gives you.
    - jdavis703 5y ago
      Then the automation needs to be more conservative in its ability and request intervention sooner.
  - merrywhether 5y ago
    This is a misrepresentation of the dumpster fire that was the WMATA train situation. Yes, the fatal crash was the last straw, but the root problem was not the automation system but rather the complete lack of maintenance that led to its inability to work properly. Congress refusing to fund maintenance and then falling behind 10-15 years on it lead to all kinds of systems failing. The fatal fire in the blue line tunnel under the river occurred with a human at the controls, but we’re similarly not blaming that incident on the perils of human operation.
    - jdavis703 5y ago
      I don’t blame the operator for the crash. The other train was behind a blind curve and she hit the emergency brake within a reasonable amount of time given what she could see. However the speeds of the system were set too high for the operator to safely stop because they assumed the ATC would work perfectly.
Waterluvian 5y ago
I have a Subaru Forester base model with lane keeping and adaptive cruise control. I need to be touching the wheel and applying some force to it or it begins yelling at me and eventually brings me slowly to a stop. I’ve had it for a year now and I cannot perceive of a way, without physically altering the system (like hanging a weight from the wheel maybe?) that would allow me to stop being an active participant. I think the opposite is true: Tesla’s move fast and kill people approach is the mistake. Incremental mastering of autonomous capabilities is the way to go.
- tgsovlerkhgsel 5y ago
  Tesla had a similar system, and > physically altering the system (like hanging a weight from the wheel maybe?) was exactly what people were doing. But it's also possible to be physically present, applying force, but being "zoned out", even without malicious intent.
  - johnnyApplePRNG 5y ago
    >But it's also possible to be physically present, applying force, but being "zoned out", even without malicious intent. I've occasionally noticed myself zoning out behind the wheel of my non-self-driving car as well. It's actually very common. [0] [0] https://www.actuarialpost.co.uk/article/quarter-of-fatal-crashes-due-to-falling-asleep-at-the-wheel-15162.htm https://www.actuarialpost.co.uk/article/quarter-of-fatal-cra...
  - FireBeyond 5y ago
    It also previously required only 15 minutes between steering wheel contacts.
- shakna 5y ago
  > I need to be touching the wheel and applying some force to it or it begins yelling at me and eventually brings me slowly to a stop. > I’ve had it for a year now and I cannot perceive of a way, without physically altering the system (like hanging a weight from the wheel maybe?) that would allow me to stop being an active participant. That's exactly what people were doing with the Tesla. Hanging a weight to ensure the safety system doesn't kick in. [0][1] [0] https://edition.cnn.com/2021/04/28/cars/tesla-texas-crash-autopilot/index.html https://edition.cnn.com/2021/04/28/cars/tesla-texas-crash-au... [1] https://twitter.com/ItsKimJava/status/1388240600491859968/photo/2 https://twitter.com/ItsKimJava/status/1388240600491859968/ph...
tgsovlerkhgsel 5y ago
It depends. As long as the resulting package (flawed self driving system + the average driver) isn't significantly more dangerous than the average unassisted human driver, I don't consider it irresponsible to deploy it. "The average driver" includes everyone, ranging from drivers using it as intended with close supervision, drivers who become inattentive because nothing is happening, and drivers who think it's a reasonable idea to climb into the back seat with a water bottle duct taped to the steering wheel to bypass the sensor. OTOH, the average driver for the unassisted scenario also includes the driver who thinks they're able to drive a car while texting.
- TacticalCoder 5y ago
  > As long as the resulting package (flawed self driving system + the average driver) isn't significantly more dangerous than the average unassisted human driver... Shouldn't that compared to "average driver + myriad of modern little safety features" instead of "average unassisted driver"? The one who has the means to drive a Tesla with the "full driving" mode certain has the means to buy, say, a Toyota full of assistance/safety features (lane change assist, unwanted lane change warning and whatnots).
  - politician 5y ago
    Why isn’t defeating the self-driving attention controls a crime like reckless driving? Isn’t that the obvious solution?
    - rcxdude 5y ago
      Gonna be pretty difficult to enforce. Many US states don't even enforce a minimum roadworthiness of cars on the roads.
      - politician 5y ago
        Does that even matter? If the state doesn’t care to enforce its laws against reckless driving, why should the manufacturer be encumbered with that responsibility?
Robotbeat 5y ago
I actually disagree. (And before you respond, please read my post because it’s not a trivial point.) The fact that an huge formal investigation happened with just a single casualty is proof that it may actually be superior for safety in the long-term (when combined with feedback from regulators and government investigators). One death in conventional vehicles is irrelevant. But because of the high profile of Tesla’s technology, it garners a bunch of attention from the public and therefore regulators. This is PRECISELY the dynamic that led to the ridiculously safe airline record. The safer it is, the more that rare deaths will be investigated and the causes sussed out and fixed by industry and regulators together. Perhaps industry/Tesla/whoever hates the regulators and investigations. But I think they are precisely what will cause self driving to become ever safer, and eventually become as safe as industry/Tesla claims, safer than human drivers while also being cheap and ubiquitous. Just like airline travel today. A remarkable combination of safety and affordability. This might be the only way to ever do it. I don’t think the airline industry could’ve ever gotten to current levels of safety by testing everything on closed airfields and over empty land for hundreds of millions of flight hours before they had sufficient statistics to be equal to today. It can’t happen without regulators and enforcement, either.
- 3pt14159 5y ago
  I largely agree with you, but I just wish regulators would start by only allowing these assist programs for people that are already known to be poor drivers. The elderly and convicted drunk drivers, for example. That way we could have the best of both worlds.
  - lastofthemojito 5y ago
    Incentivizing drunk driving seems dangerous.
    - joshgrib 5y ago
      Requiring people to buy a special car/system to be able to drive doesn't seem like an incentive - it seems similar to the interlock system we currently require drunk drivers to purchase to be able to drive. If anything a driver monitoring system seems even better than the interlock system, for example you couldn't have your kids/friends blow for you to bypass it.
fzzzy 5y ago
Do you hate regular cruise control? How is that not partial self driving?
- ghaff 5y ago
  To tell you the truth, I generally do and haven't used it for ages. Where I drive, the roads have some amount of traffic. I find (traditional) cruise control encourages driving at a constant speed to a degree that I wouldn't as a driver with a foot on the gas. So I don't "hate" regular cruise control but I basically never use it.
  - hcurtiss 5y ago
    I think you are in a distinct minority.
    - ghaff 5y ago
      Maybe a fairly small sample size but I don't know the last time I've been in a car where the driver has turned on cruise control. But it probably varies by area of the country. In the Northeast, there's just enough traffic in general that it's not worth it for me.
      - minhazm 5y ago
        In traffic is where traffic aware cruise control is most useful. A lot of people I knew who bought Tesla's in the bay area specifically bought it so their commutes would be less stressful with the bumper to bumper traffic. I drove 3000+ miles across the country last year with > 90% of it on AP and I was way less tired with AP on vs off and it allowed me to just stay focused on the road and look out for any issues.
        ghaff 5y ago
        Yes. I was (explicitly) talking about traditional "dumb" cruise control. I haven't used adaptive cruise control but I agree it sounds more useful than traditional cruise control once you get above minimal traffic.
helsinkiandrew 5y ago
> you need to always pay attention That is the fatal flaw in anything but a perfect system - any kind of system that is taking the decisions about steering from the driver is going to result in the driver at best thinking about other things and worse getting into the back seat to change. If you had to develop a system to make sure someone was paying attention, you wouldn't make them sit in a warm comfy seat looking at a screen - you would make them actively engage with what they were looking at - like steering. And ultimately it doesn't matter how many hundreds of thousands of hours of driving you teach your system with, it may eventually be able to learn about parked cars, kerbs and road signs, but there won't be enough examples of different accidents and how emergency vehicles behave to ever make it behave safely. Humans can cope with driving emergencies fairly well (not perfectly admittedly) no matter how many they've been involved in using logic and higher level reasoning.
swiley 5y ago
We have an education problem. People have no idea what computers do because they're illiterate (literacy would mean knowing at least one language well enough to read and write in it) so they just take other people's word that they can do some magical thing with software updates. The most extreme examples of this were the iPhone hoaxes telling people that software updates provided waterproofing or microwave charging.
backtoyoujim 5y ago
I mean there are videos of a vehicle's occupant sitting in the rear seats making food and drinks while the vehicles are tricked into operating off of the vehicles sensors. It is not solely the trust and dependence but inclusive is the group of idiots with access to wealth without regard to human life.
bishoprook2 5y ago
I expect that to design self-driving you need to push the limits (with some accidents) a bit with a bunch of telemetry. Going from not-much to full-self-driving requires a lot of design increments.
antattack 5y ago
All Level 2 systems need to better integrate with the driver. Upon engagement driver and driver assist are team where communication and predictability is crucial.
dalbasal 5y ago
I think the fundamental flaw is indisputable. Everyone is aware of in-between stage problems. I don't think it's an insurmountable flaw. These things are on the road already. They have issues, but so do human only cars. Tweaks probably get made, like some special handling of emergency vehicle scenarios. But, it's not enough to stop it. Meanwhile, it's not a permanent state. Self driving technology is advancing, becoming more common on roads. Procedures, as well as infrastructure, is growing around the existence of self driven cars. Human supervisor or not, the way these things use the road affects the design of roads. If your emergency speed sign isn't being headed by self driven cars, your emergency speed sign has a bug.
api 5y ago
Full self driving is one of those things where getting 80% of the way there will take 20% of the effort and getting the remaining 20% of the way there will take 80% of the effort. Tesla auto-drive seems like it's about 80% of the way there.
Baeocystin 5y ago
I remember reading Donald Norman's books decades ago, and one of the prime examples of the dangers of automation in cars was adaptive cruise control- which would then suddenly accelerate forward in a now-clear off-ramp, surprising the heck out of the previously-complacent driver, and leading to accidents. We've known for a very long time that this sort of automation/manual control handoff failure is a very big deal, and yet there seems to be an almost willful blindness from the manufacturers to address it in a meaningful way.
dheera 5y ago
Is it? Tesla is still alive because they're selling cars. It's just that the companies that are NOT doing incremental approaches are largely at the mercy of some investors who don't know a thing about self-driving, and they may die at any time. I agree with you that it is technically flawed, but it may still be viable in the end. At least their existence is not dependent on the mercy of some fools who don't get it, they just sell cars to stay alive. That's one of the major problems of today's version of capitalism -- it encourages technically flawed ways to achieve scientific advancement.
- yawaworht1978 5y ago
  Would be interesting to know how many buy based on the fsd hype(including the ones who don't pay for the package) and how many buy because of the "green" factor. However many there are who buy because of the fsd promise, all that revenue is coming from vaporware (beta ware at best) and is possible due to lack of regulatory enforcement. History shows that the longer the self regulatory entities take the p, the harder the regulatory hammer comes down eventually.
deleted 5y ago
[deleted]
supperburg 5y ago
Lex Fridman said they studied this and found that people don’t become “lulled” even after using the system for a long period of time.
slg 5y ago
One problem that is often ignored in these debates is that people already don't always pay attention while driving. Spend some time looking at other drivers next time you are a passenger in slow traffic. The number of drivers on their phones, eating, doing makeup, shaving, or even reading a book is scary. It therefore isn't a clean swap of a human paying attention to a human who isn't. It becomes a complicated equation that we can't just dismiss with "people won't pay attention". It is possible that a 90%/10% split of drivers paying attention to not paying attention is more dangerous when they are all driving manually than a 70%/30% split if those drivers are all using self-driving tech to cover for them. Wouldn't you feel safer if the driver behind you who is answering texts was using this incremental self-driving tech rather than driving manually? No one has enough data on the performance of these systems or how the population of drivers use them to say definitively that they are either safer or more dangerous on the whole. But it is definitely something that needs to be investigated and researched.
MR4D 5y ago
A car that beeps when I drift out of lane, or beeps when I go too fast before a curve, or beeps like hell if I cross over the center median would be hugely useful, because a record of every warning would be there, whether correct or not. Conversely, if it didn't warn me right before an accident, then the absence of that warning would be useful too. All of that information should be put back into the model based on crash reporting. Everything else can be ignored. I would argue that the information should be available to all automakers (perhaps using the NHTSA as a conduit), so that each of them have the same safety information, but can still develop their own models. The FAA actually does this already with the FAA Accident and Incident Data Systems [0] and it has worked pretty darn well. [0] - https://www.asias.faa.gov/apex/f?p=100:2:::NO https://www.asias.faa.gov/apex/f?p=100:2:::NO:::
- oceanghost 5y ago
  The new Toyota RAV4's have this feature-- if you go out of bounds in your lane they beep and the steer5ing wheel gives a bit of resistance. It also reads the speed limit signs and places a reminder in the display. I think it can brake if it detects something in front of it, but I'm not certain.
  - MR4D 5y ago
    Many other cars do as well. My main point (perhaps buried more than it should have been) is that by centralizing accident data along with whether an alert went off (or not), and sharing that with all automobile manufacturers can help this process proceed better. Right now the data is highly fragmented and there is not really a common objective metric by which to make decisions to improve models.
    - oceanghost 5y ago
      I'm sorry, I entirely missed your point. I agree that would be reasonable and desirable :-)

by gundmc 5y ago

judge2020 5y ago

Your reasoning doesn’t apply to the incremental improvements to self-driving, rather Tesla’s decision to allow all cars to use TACC/auto-steer. They haven’t even given people “the button” to enroll in FSD beta, likely because they know it would be extremely bad PR when a bunch of people use it without paying attention.

nathias 5y ago

that's not human nature, that's user stupidity

weird-eye-issue 5y ago
What's the... Difference?
- nathias 5y ago
  One is a permanent property of our nature the other a choice.
  - weird-eye-issue 5y ago
    No, one is a fact and the other is a consequence
formerly_proven 5y ago
The design is fine, it's all the users who are idiots. P.S. /s. Obviously, Mr. Poe.
- rvz 5y ago
  Well they did not 'pay' attention. They 'paid' for the "Fools Self Driving" package. This is why 'attention' and 'driver monitoring' was not included.
- salawat 5y ago
  T. Every maligned designer when someone points out a flaw It's okay. I do it too. Really need to work on seeing yourself making that argument as a starting point and not an endpoint.
deleted 5y ago
[deleted]

mnmmn123456 5y ago

Today, there is at least one of the most advanced Neural Networks entering each car: A human being. If we could just implement the AI to add to this person and not replace it...

salawat 5y ago
This is the bit nobody likes to realize. FSD at it's best...is still about as fallible as a human driver. Minus free will (it is hoped). I will be amused, intrigued, and possibly a bit horrified if by the time FSD hits level 5, and they stick with the Neural Net of Neural Nets architecture if there isn't a rash of system induced variance in behavior as emergent phenomena take shape. Imagined news: All Tesla's on I-95 engaged in creating patterns whereby all non-Teala traffic was bordered by a Tesla on each side. Almost like a game of Go, says expert. Researchers stumped. Then again, that's imply you had an NN capable of retraining itself on the fly to some limited degree, which I assume no one sane would put into service... Hopefully this comment doesn't suffer a date of not aging well.
ben_w 5y ago
What would such an AI even look like? If it spots every real danger but also hallucinates even a few dangers that aren’t really there, it gets ignored or switched off for needlessly slowing the traveler down (false positives, apparently an issue with early Google examples [0]); if it only spots real dangers but misses most of them, it is not helping (false negatives, even worse if a human is blindly assuming the machine knows best and what happened with e.g. Uber [1]); if it’s about the same as humans overall but makes different types of mistake, people rely on it right up until it crashes then go apoplectic because it didn’t see something any human would consider obvious (e.g. Tesla, which gets slightly safer when the AI is active, but people keep showing the AI getting confused about things that they consider obvious [2]). [0] https://theoatmeal.com/blog/google_self_driving_car https://theoatmeal.com/blog/google_self_driving_car [1] https://en.wikipedia.org/wiki/Death_of_Elaine_Herzberg https://en.wikipedia.org/wiki/Death_of_Elaine_Herzberg [2] https://youtube.com/watch?v=Wz6Ins1D9ak https://youtube.com/watch?v=Wz6Ins1D9ak

cmpb 5y ago

I disagree. One feature my car has is to pull me back into the lane when I veer out of it (Subaru's lane keep assist). That is still incremental improvement towards "full self driving". I agree, however, that Tesla's Autopilot is not functional enough, and any tool designed to allow humans to remove their hands from the wheel should not require their immediate attention in any way.

tapoxi 5y ago
I think people just assume Tesla's Autopilot is more capable than it really is. My car has adaptive cruise control and lane keep assist, but I'm not relying on either for anything more complex than sipping a drink while on the highway.
- rmckayfleming 5y ago
  Yep, if anything they’re just a way to make long drives or stop and go highway traffic more tolerable. When I got my first car with those features it seemed like a gimmick, but they really help to reduce fatigue.
Robotbeat 5y ago
Tesla’s autopilot does not allow you to remove your hands from the wheel. You must keep them on, and apply torque occasionally, to keep it engaged.
- Karunamon 5y ago
  In reality, it'll let you get away with going handsfree for upwards of 30 seconds. That's more than long enough to lose your attention.
darkerside 5y ago
I think that's actually a step towards a local maximum that makes us less likely to achieve actual FSD. The safer we can make AI-guided driving where a person is still in control, the higher the bar becomes for a solo AI to be significantly safer than the alternatives.

mrfusion 5y ago

I’d be curious if there are studies out there on how to do automated assists in machines that require vigilance that don’t have this problem.

jdavis703 5y ago
Both airplanes and trains have automated “assist.” At least in the case of WMATA they give up on automatic train control after a fatal crash.
- userbinator 5y ago
  In a plane you also have far more time to react once the autopilot disconnects for whatever reason, than the fraction of a second that a car gives you.
  - jdavis703 5y ago
    Then the automation needs to be more conservative in its ability and request intervention sooner.
- merrywhether 5y ago
  This is a misrepresentation of the dumpster fire that was the WMATA train situation. Yes, the fatal crash was the last straw, but the root problem was not the automation system but rather the complete lack of maintenance that led to its inability to work properly. Congress refusing to fund maintenance and then falling behind 10-15 years on it lead to all kinds of systems failing. The fatal fire in the blue line tunnel under the river occurred with a human at the controls, but we’re similarly not blaming that incident on the perils of human operation.
  - jdavis703 5y ago
    I don’t blame the operator for the crash. The other train was behind a blind curve and she hit the emergency brake within a reasonable amount of time given what she could see. However the speeds of the system were set too high for the operator to safely stop because they assumed the ATC would work perfectly.

Waterluvian 5y ago

I have a Subaru Forester base model with lane keeping and adaptive cruise control. I need to be touching the wheel and applying some force to it or it begins yelling at me and eventually brings me slowly to a stop. I’ve had it for a year now and I cannot perceive of a way, without physically altering the system (like hanging a weight from the wheel maybe?) that would allow me to stop being an active participant. I think the opposite is true: Tesla’s move fast and kill people approach is the mistake. Incremental mastering of autonomous capabilities is the way to go.

tgsovlerkhgsel 5y ago
Tesla had a similar system, and > physically altering the system (like hanging a weight from the wheel maybe?) was exactly what people were doing. But it's also possible to be physically present, applying force, but being "zoned out", even without malicious intent.
- johnnyApplePRNG 5y ago
  >But it's also possible to be physically present, applying force, but being "zoned out", even without malicious intent. I've occasionally noticed myself zoning out behind the wheel of my non-self-driving car as well. It's actually very common. [0] [0] https://www.actuarialpost.co.uk/article/quarter-of-fatal-crashes-due-to-falling-asleep-at-the-wheel-15162.htm https://www.actuarialpost.co.uk/article/quarter-of-fatal-cra...
- FireBeyond 5y ago
  It also previously required only 15 minutes between steering wheel contacts.
shakna 5y ago
> I need to be touching the wheel and applying some force to it or it begins yelling at me and eventually brings me slowly to a stop. > I’ve had it for a year now and I cannot perceive of a way, without physically altering the system (like hanging a weight from the wheel maybe?) that would allow me to stop being an active participant. That's exactly what people were doing with the Tesla. Hanging a weight to ensure the safety system doesn't kick in. [0][1] [0] https://edition.cnn.com/2021/04/28/cars/tesla-texas-crash-autopilot/index.html https://edition.cnn.com/2021/04/28/cars/tesla-texas-crash-au... [1] https://twitter.com/ItsKimJava/status/1388240600491859968/photo/2 https://twitter.com/ItsKimJava/status/1388240600491859968/ph...

tgsovlerkhgsel 5y ago

It depends. As long as the resulting package (flawed self driving system + the average driver) isn't significantly more dangerous than the average unassisted human driver, I don't consider it irresponsible to deploy it. "The average driver" includes everyone, ranging from drivers using it as intended with close supervision, drivers who become inattentive because nothing is happening, and drivers who think it's a reasonable idea to climb into the back seat with a water bottle duct taped to the steering wheel to bypass the sensor. OTOH, the average driver for the unassisted scenario also includes the driver who thinks they're able to drive a car while texting.

TacticalCoder 5y ago
> As long as the resulting package (flawed self driving system + the average driver) isn't significantly more dangerous than the average unassisted human driver... Shouldn't that compared to "average driver + myriad of modern little safety features" instead of "average unassisted driver"? The one who has the means to drive a Tesla with the "full driving" mode certain has the means to buy, say, a Toyota full of assistance/safety features (lane change assist, unwanted lane change warning and whatnots).
- politician 5y ago
  Why isn’t defeating the self-driving attention controls a crime like reckless driving? Isn’t that the obvious solution?
  - rcxdude 5y ago
    Gonna be pretty difficult to enforce. Many US states don't even enforce a minimum roadworthiness of cars on the roads.
    - politician 5y ago
      Does that even matter? If the state doesn’t care to enforce its laws against reckless driving, why should the manufacturer be encumbered with that responsibility?

Robotbeat 5y ago

I actually disagree. (And before you respond, please read my post because it’s not a trivial point.) The fact that an huge formal investigation happened with just a single casualty is proof that it may actually be superior for safety in the long-term (when combined with feedback from regulators and government investigators). One death in conventional vehicles is irrelevant. But because of the high profile of Tesla’s technology, it garners a bunch of attention from the public and therefore regulators. This is PRECISELY the dynamic that led to the ridiculously safe airline record. The safer it is, the more that rare deaths will be investigated and the causes sussed out and fixed by industry and regulators together. Perhaps industry/Tesla/whoever hates the regulators and investigations. But I think they are precisely what will cause self driving to become ever safer, and eventually become as safe as industry/Tesla claims, safer than human drivers while also being cheap and ubiquitous. Just like airline travel today. A remarkable combination of safety and affordability. This might be the only way to ever do it. I don’t think the airline industry could’ve ever gotten to current levels of safety by testing everything on closed airfields and over empty land for hundreds of millions of flight hours before they had sufficient statistics to be equal to today. It can’t happen without regulators and enforcement, either.

3pt14159 5y ago
I largely agree with you, but I just wish regulators would start by only allowing these assist programs for people that are already known to be poor drivers. The elderly and convicted drunk drivers, for example. That way we could have the best of both worlds.
- lastofthemojito 5y ago
  Incentivizing drunk driving seems dangerous.
  - joshgrib 5y ago
    Requiring people to buy a special car/system to be able to drive doesn't seem like an incentive - it seems similar to the interlock system we currently require drunk drivers to purchase to be able to drive. If anything a driver monitoring system seems even better than the interlock system, for example you couldn't have your kids/friends blow for you to bypass it.

fzzzy 5y ago

Do you hate regular cruise control? How is that not partial self driving?

ghaff 5y ago
To tell you the truth, I generally do and haven't used it for ages. Where I drive, the roads have some amount of traffic. I find (traditional) cruise control encourages driving at a constant speed to a degree that I wouldn't as a driver with a foot on the gas. So I don't "hate" regular cruise control but I basically never use it.
- hcurtiss 5y ago
  I think you are in a distinct minority.
  - ghaff 5y ago
    Maybe a fairly small sample size but I don't know the last time I've been in a car where the driver has turned on cruise control. But it probably varies by area of the country. In the Northeast, there's just enough traffic in general that it's not worth it for me.
    - minhazm 5y ago
      In traffic is where traffic aware cruise control is most useful. A lot of people I knew who bought Tesla's in the bay area specifically bought it so their commutes would be less stressful with the bumper to bumper traffic. I drove 3000+ miles across the country last year with > 90% of it on AP and I was way less tired with AP on vs off and it allowed me to just stay focused on the road and look out for any issues.
      - ghaff 5y ago
        Yes. I was (explicitly) talking about traditional "dumb" cruise control. I haven't used adaptive cruise control but I agree it sounds more useful than traditional cruise control once you get above minimal traffic.

helsinkiandrew 5y ago

> you need to always pay attention That is the fatal flaw in anything but a perfect system - any kind of system that is taking the decisions about steering from the driver is going to result in the driver at best thinking about other things and worse getting into the back seat to change. If you had to develop a system to make sure someone was paying attention, you wouldn't make them sit in a warm comfy seat looking at a screen - you would make them actively engage with what they were looking at - like steering. And ultimately it doesn't matter how many hundreds of thousands of hours of driving you teach your system with, it may eventually be able to learn about parked cars, kerbs and road signs, but there won't be enough examples of different accidents and how emergency vehicles behave to ever make it behave safely. Humans can cope with driving emergencies fairly well (not perfectly admittedly) no matter how many they've been involved in using logic and higher level reasoning.

swiley 5y ago

We have an education problem. People have no idea what computers do because they're illiterate (literacy would mean knowing at least one language well enough to read and write in it) so they just take other people's word that they can do some magical thing with software updates. The most extreme examples of this were the iPhone hoaxes telling people that software updates provided waterproofing or microwave charging.

backtoyoujim 5y ago

I mean there are videos of a vehicle's occupant sitting in the rear seats making food and drinks while the vehicles are tricked into operating off of the vehicles sensors. It is not solely the trust and dependence but inclusive is the group of idiots with access to wealth without regard to human life.

bishoprook2 5y ago

I expect that to design self-driving you need to push the limits (with some accidents) a bit with a bunch of telemetry. Going from not-much to full-self-driving requires a lot of design increments.

antattack 5y ago

All Level 2 systems need to better integrate with the driver. Upon engagement driver and driver assist are team where communication and predictability is crucial.

dalbasal 5y ago

I think the fundamental flaw is indisputable. Everyone is aware of in-between stage problems. I don't think it's an insurmountable flaw. These things are on the road already. They have issues, but so do human only cars. Tweaks probably get made, like some special handling of emergency vehicle scenarios. But, it's not enough to stop it. Meanwhile, it's not a permanent state. Self driving technology is advancing, becoming more common on roads. Procedures, as well as infrastructure, is growing around the existence of self driven cars. Human supervisor or not, the way these things use the road affects the design of roads. If your emergency speed sign isn't being headed by self driven cars, your emergency speed sign has a bug.

api 5y ago

Full self driving is one of those things where getting 80% of the way there will take 20% of the effort and getting the remaining 20% of the way there will take 80% of the effort. Tesla auto-drive seems like it's about 80% of the way there.

Baeocystin 5y ago

I remember reading Donald Norman's books decades ago, and one of the prime examples of the dangers of automation in cars was adaptive cruise control- which would then suddenly accelerate forward in a now-clear off-ramp, surprising the heck out of the previously-complacent driver, and leading to accidents. We've known for a very long time that this sort of automation/manual control handoff failure is a very big deal, and yet there seems to be an almost willful blindness from the manufacturers to address it in a meaningful way.

dheera 5y ago

Is it? Tesla is still alive because they're selling cars. It's just that the companies that are NOT doing incremental approaches are largely at the mercy of some investors who don't know a thing about self-driving, and they may die at any time. I agree with you that it is technically flawed, but it may still be viable in the end. At least their existence is not dependent on the mercy of some fools who don't get it, they just sell cars to stay alive. That's one of the major problems of today's version of capitalism -- it encourages technically flawed ways to achieve scientific advancement.

yawaworht1978 5y ago
Would be interesting to know how many buy based on the fsd hype(including the ones who don't pay for the package) and how many buy because of the "green" factor. However many there are who buy because of the fsd promise, all that revenue is coming from vaporware (beta ware at best) and is possible due to lack of regulatory enforcement. History shows that the longer the self regulatory entities take the p, the harder the regulatory hammer comes down eventually.

deleted 5y ago

[deleted]

supperburg 5y ago

Lex Fridman said they studied this and found that people don’t become “lulled” even after using the system for a long period of time.

slg 5y ago

One problem that is often ignored in these debates is that people already don't always pay attention while driving. Spend some time looking at other drivers next time you are a passenger in slow traffic. The number of drivers on their phones, eating, doing makeup, shaving, or even reading a book is scary. It therefore isn't a clean swap of a human paying attention to a human who isn't. It becomes a complicated equation that we can't just dismiss with "people won't pay attention". It is possible that a 90%/10% split of drivers paying attention to not paying attention is more dangerous when they are all driving manually than a 70%/30% split if those drivers are all using self-driving tech to cover for them. Wouldn't you feel safer if the driver behind you who is answering texts was using this incremental self-driving tech rather than driving manually? No one has enough data on the performance of these systems or how the population of drivers use them to say definitively that they are either safer or more dangerous on the whole. But it is definitely something that needs to be investigated and researched.

MR4D 5y ago

A car that beeps when I drift out of lane, or beeps when I go too fast before a curve, or beeps like hell if I cross over the center median would be hugely useful, because a record of every warning would be there, whether correct or not. Conversely, if it didn't warn me right before an accident, then the absence of that warning would be useful too. All of that information should be put back into the model based on crash reporting. Everything else can be ignored. I would argue that the information should be available to all automakers (perhaps using the NHTSA as a conduit), so that each of them have the same safety information, but can still develop their own models. The FAA actually does this already with the FAA Accident and Incident Data Systems [0] and it has worked pretty darn well. [0] - https://www.asias.faa.gov/apex/f?p=100:2:::NO https://www.asias.faa.gov/apex/f?p=100:2:::NO:::

oceanghost 5y ago
The new Toyota RAV4's have this feature-- if you go out of bounds in your lane they beep and the steer5ing wheel gives a bit of resistance. It also reads the speed limit signs and places a reminder in the display. I think it can brake if it detects something in front of it, but I'm not certain.
- MR4D 5y ago
  Many other cars do as well. My main point (perhaps buried more than it should have been) is that by centralizing accident data along with whether an alert went off (or not), and sharing that with all automobile manufacturers can help this process proceed better. Right now the data is highly fragmented and there is not really a common objective metric by which to make decisions to improve models.
  - oceanghost 5y ago
    I'm sorry, I entirely missed your point. I agree that would be reasonable and desirable :-)