7 ms·

From my perspective, everything went right here. The software screwed up, it alerted the passengers that a collision was going to happen, and a human driver qui

From my perspective, everything went right here. The software screwed up, it alerted the passengers that a collision was going to happen, and a human driver quickly took over and avoided the accident. Is this not what we should expect from a beta? The responsibility is still on the driver even with driver assist / full self driving technology turned on. All this talk of regulating the software when we just saw a worst case scenario resolve with a favorable outcome. "But what if the driver wasn't paying attention?" Then the driver will face the consequences as if he had been driving the car himself, because it's his car and it's his responsibility.

wonnage 5y ago
Beta self-driving software shouldn't be on the road in the hands of random drivers. It's crazy to me that people think this is ok because "it's just a beta!" The stakes are vastly different between a bug in a photo sharing app and a 2 ton hunk of metal targeting bicyclists on accident
jdlshore 5y ago
What you're missing is that it isn't reasonable to expect people to be paying enough attention to "quickly take over."
- nexuist 5y ago
  Nobody should be using driver assist technology if they're not ready to accept the consequences it may get them in. We let teenagers buy sports cars despite being totally incapable of understanding the risks that 300+ bhp brings. I don't see the difference between that and FSD.
  - FireBeyond 5y ago
    It blows my mind that the US doesn't have any power-to-weight ratios for new drivers. In Australia until you were off your probationary license (which was a minimum of three years after your learners permit, which was either 1 or 2 years, it's been a while), there were power-to-weight restrictions on the vehicles you were allowed to drive. Here? Saw a YouTube video of a seventeen year old doing his driver's license test in a 1500hp Bugatti. That'll end well, no doubt.
PragmaticPulp 5y ago
> From my perspective, everything went right here. Did you miss the part where the autopilot steered straight toward a clearly visible cyclist? If the autopilot is so unreliable that it requires lightning-quick reflexes to override, it's going to lead to a lot of accidents from people who don't happen to be paying perfect attention at the time. I know the whole story is that drivers are supposed to be 100% engaged, but realistically that's just not going to happen when you literally market something as "full self-driving"
- nexuist 5y ago
  The name is stupid and should be changed, but I also don't want people to be absolved of responsibility because they bought into marketing. "But they said the ship couldn't sink!" Doesn't matter, you're still on the hook for the hundreds of people who drowned. Due diligence is a part of responsible adulthood. The airlines that got fucked by Boeing's MCAS still had to send payouts to families who lost loved ones, because it was ultimately their decision to buy and operate a 787 MAX. They can (and certainly should) sue Boeing for the losses, but it doesn't change the fact that the crashes are theirs to own. You don't get a get out of jail free card for being gullible.
  - Calloutman 5y ago
    This comment is factually wrong. The airlines did not pay compensation to victims families, Boeing did. Boeing accepted full responsibility for the crash, for providing an unsafe system. Ethiopian airlines did their "due diligence" by purchasing an FAA approved aircraft, by Boeing. They weren't to know that a serious case of regulatory capture was happening. https://www.google.com/url?q=https://www.theguardian.com/business/2021/nov/11/boeing-full-responsibility-737-max-plane-crash-ethiopia-compensation&sa=U&ved=2ahUKEwi5t6m6vfH1AhXPesAKHZf7Dv0QFXoECAQQAg&usg=AOvVaw2a8fBu46r9DS46F3XhfCYU https://www.google.com/url?q=https://www.theguardian.com/bus...
    - nexuist 5y ago
      Huh, TIL (and relevant username). Thank you for the correction. I noticed the article says they only admitted fault for the second crash, do you know what happened with the first? Did Boeing admit fault for that too?
willidiots 5y ago
I've been driving a rental Model 3 with FSD for the past day - for the first time - and totally agree. And after a day's experience, it's obvious it needs to be watched. It's an amazing capability but there are clearly still corner cases, and it makes you very aware of them. I liken Autopilot / FSD more to a horse at its current state. You keep a firm but gentle hand on the rein and let it find its way. Sometime you have to check it. If you get complacent (which the screen explicitly warns you against), yeah, things could go wrong. That's also 100% true when driving a normal car. Edit to add: I think a lot of criticism focuses on the term "Full Self-Driving" which implies "Full Self-Driving". That's fair, but I think it misses an important point. ("Autopilot" made a lot more sense to me: real autopilots can't do everything either. They can't even taxi.) When you're actually IN THE CAR, enabling this functionality for the first time, you have to enable three separate toggle switches. Each one pops up a progressively longer, more dire WARNING dialog, emphasizing the software's primitive state, the importance of paying attention and keeping your hand on the wheel. It also demands your constant attention while engaged.
- throwaway0a5e 5y ago
  It's like a teenage driver. You have to keep an eye out for developing situations you know it's too stupid to handle elegantly.
- ModernMech 5y ago
  > It's an amazing capability but there are clearly still corner cases, and it makes you very aware of them. The thing that most of these companies are failing to grasp is that the whole endeavor is like 99% corner cases. It's corner cases all the way down. The hope was that there was this big chunk of the problem space that could be handled in a tractable way, and that would hopefully account for ~80% of driving conditions. Then corner cases could be handled on an as-needed basis to fill in the gaps, and slowly you could handle more and more scenarios. The problem is that the reason for so many corner cases in this domain is that driving is a social activity. The rules of the road actually belie how much communication between humans goes on when they are driving. Driving has its own norms, its own language, culture, emergent behaviors, and traffic exists as a crucial force that must be balanced in a city ecosystem. But autonomous cars aim to solve the driving problem by ignoring the social, and reducing it to an optimization problem. Autonomous cars are blind to the larger world around them except to recognize colors and the nearness of physical bodies. They miss social cues, they lack higher order reasoning about traffic patterns, they can't negotiate with pedestrians or other drivers, they are unable to weigh the ethical considerations between when following a law is actually worse that breaking it. As of now I consider them menaces on the road, and that pains me as someone who got to play with some of the first driverless car technology in the lab. I had such high hopes for it, and what I hoped was not... this.
sho_hn 5y ago
The training we give drivers on how to operate cars and the failure modes the training prepares for are designed for cars without FSD, though, and I'm not sure it adequately prepares drivers to compensate for the possible failure modes of FSD. Especially when those can change from release to release, making it hard to get used in any meaningful sense to the system. This is also a lot more stressful than regular driving and may require better driver condition and more frequent rest. Unfinished vehicles on the road are nothing new; all companies developing cars move pre-prod hardware across public roads. However it usually requires several levels of additional licenses and special safety driving training and instruction before you're allowed to operate a car that may surprise you by behaving very different from a certified production road car. And regular renewal thereafter. I think what's unprecedented here is the scale of the pre-prod testing, the type of experimentation being done and how the drivers are selected. And also the degree to which it's designed into the approach of the experimental engineering being done that it must fail to improve (by widening the training set). There's enough new here it should be discussed imho. Side note: German driver's license training now includes some material on assistance systems and how they may behave unreliably and how to prepare for their failure modes ... but of course this doesn't retroactively expose existing license holders. The rate of change in driver-car interaction being so high is also a novum.
- nexuist 5y ago
  It is for unprecedented and should be discussed. I just don't buy into the kneejerk reaction of getting this off the roads ASAP when we know drivers use FSD to drive millions of miles a year and accidents are not common at all. It shows that the drivers seem to be responsible enough to keep the tech in check.
  - sho_hn 5y ago
    It may show that for the drivers who currently have access, but Tesla selects/limits the beta participants based on proprietary "safe driver" scoring that isn't publically documented. So it's the product manufacturer who can turn the accident probability screw here, or could also get it wrong. And I think that gets to the bottom of what makes many people feel uneasy.
klyrs 5y ago
> Is this not what we should expect from a beta? No, what we should expect from a beta is for it to not be deployed at scale, to non-professional drivers, in a live environment. It's clearly still not ready to go live, much less under the name "full self driving"
paxys 5y ago
> and a human driver quickly took over and avoided the accident Can we be sure that the average driver who has received no special training in the matter will be able to take over from an autopilot in a split second when driving at high speeds? Heck people can't pay attention and keep their eyes on the road even when they are the ones in control. > The responsibility is still on the driver even with driver assist / full self driving technology turned on. If/when the system is rolled out at scale will drivers actually comprehend that? > Then the driver will face the consequences as if he had been driving the car himself Assigning blame after the fact is fine, but no one can bring a dead pedestrian back to life.
- nexuist 5y ago
  > Can we be sure that the average driver who has received no special training in the matter will be able to take over from an autopilot in a split second when driving at high speeds? The beta system controls for this by only allowing in people who score 98 or above in Tesla's safety surveillance program. The restrictions are tight enough that you have to actively try to be a safe driver to get anywhere close to the cutoff (some people even got pissed off because their score went down because someone brake checked them and they had to decelerate too quickly). Not everyone who buys FSD gets into the beta. > If/when the system is rolled out at scale will drivers actually comprehend that? The only ethical way to roll this out at a wider scale than the scale allowed by the safety surveillance program is if it's good enough to avoid incidents like the one depicted here. I think we can all conclude that Tesla deserves the fat lawsuits if they were to green light FSD in the state it is in today. > but no one can bring a dead pedestrian back to life. An unfortunate fact of life, hopefully made irrelevant by the comeuppance of autonomous transportation technology (including but not limited to automobiles) that makes motor vehicle accidents a thing of the past. Until we get there, though, someone has to keep trying.
  - Karrot_Kream 5y ago
    > An unfortunate fact of life, hopefully made irrelevant by the comeuppance of autonomous transportation technology (including but not limited to automobiles) that makes motor vehicle accidents a thing of the past. Until we get there, though, someone has to keep trying. Autonomous transportation already exists, it already works, and it's _already safe_! It just isn't deployed for private automobiles, it's deployed on rail-based transit systems. The technology is widely deployed across Europe and Asia. How many more people need to die to let Americans have a more comfortable ride in their private vehicles?
CTmystery 5y ago
> From my perspective, everything went right here. Holy smokes. Did we watch the same video? > Is this not what we should expect from a beta? Is it OK for a beta software product to nearly kill another person that has no agreement with the company putting the product on the road? People have the right to do whatever they want with their own stuff, fine. But putting others at risk comes with huge responsibility, and must be weighed against the desire to do what you want. In this case the balance clearly falls IMO to Tesla being selfish and negligent by employing their drivers to be the safety backstop in a clearly buggy system.
- judge2020 5y ago
  Many car systems can fail. Think power steering going out with tires rotated 50 degrees to the right a driver that doesn't know how to properly steer without it.
KennyBlanken 5y ago
> Then the driver will face the consequences as if he had been driving the car himself What consequences? Drivers almost never face any consequences whatsoever for manslaughter or murder with a car. Even if they did face "consequences", they would not be anywhere near the consequences a completely random, innocent, and highly vulnerable road user will.
dogmatism 5y ago
> Then the driver will face the consequences as if he had been driving the car himself, because it's his car and it's his responsibility. You must be unfamiliar with the usual consequences of a driver killing cyclist(s)
- wcarron 5y ago
  Yep. Cyclists are murdered/manslaughtered with regularity by drivers who recieve zero punishment, and the cyclists and their families receive zero justice. Not just average schmucks in the US, either. Professional cyclists like Michele Scarponi are killed in the EU, too, by a driver using their phone. I forget the results of that case. But cyclists rarely receive the protection of laws that supposedly exist for that purpose.
Jeff_Brown 5y ago
Expecting every buyer to have the ability and discipline to be perpetually watchful for something terrible when almost all of the time there's nothing to worry about is unreasonable. For most humans, standing guard (well) is really hard work, whereas for most drivers, driving is not. This is part of the reason being a cop or a soldier is so stressful.
andbberger 5y ago
Looking forward to getting killed by a tesla and then having a bunch of tesla stans argue that my preventable death was actually OK and an example of the Great Progress being made towards reducing traffic violence!
- vanilla_nut 5y ago
  So basically the Uber thing from a year or two ago?
randoglando 5y ago
> Is this not what we should expect from a beta? The issue here is that the person on the street does not know they are the ones likely to get impacted by a bug/failure of a software in beta. Beta products are usually not released to the public, or are opt-in.

wonnage 5y ago

Beta self-driving software shouldn't be on the road in the hands of random drivers. It's crazy to me that people think this is ok because "it's just a beta!" The stakes are vastly different between a bug in a photo sharing app and a 2 ton hunk of metal targeting bicyclists on accident

jdlshore 5y ago

What you're missing is that it isn't reasonable to expect people to be paying enough attention to "quickly take over."

nexuist 5y ago
Nobody should be using driver assist technology if they're not ready to accept the consequences it may get them in. We let teenagers buy sports cars despite being totally incapable of understanding the risks that 300+ bhp brings. I don't see the difference between that and FSD.
- FireBeyond 5y ago
  It blows my mind that the US doesn't have any power-to-weight ratios for new drivers. In Australia until you were off your probationary license (which was a minimum of three years after your learners permit, which was either 1 or 2 years, it's been a while), there were power-to-weight restrictions on the vehicles you were allowed to drive. Here? Saw a YouTube video of a seventeen year old doing his driver's license test in a 1500hp Bugatti. That'll end well, no doubt.

PragmaticPulp 5y ago

> From my perspective, everything went right here. Did you miss the part where the autopilot steered straight toward a clearly visible cyclist? If the autopilot is so unreliable that it requires lightning-quick reflexes to override, it's going to lead to a lot of accidents from people who don't happen to be paying perfect attention at the time. I know the whole story is that drivers are supposed to be 100% engaged, but realistically that's just not going to happen when you literally market something as "full self-driving"

nexuist 5y ago
The name is stupid and should be changed, but I also don't want people to be absolved of responsibility because they bought into marketing. "But they said the ship couldn't sink!" Doesn't matter, you're still on the hook for the hundreds of people who drowned. Due diligence is a part of responsible adulthood. The airlines that got fucked by Boeing's MCAS still had to send payouts to families who lost loved ones, because it was ultimately their decision to buy and operate a 787 MAX. They can (and certainly should) sue Boeing for the losses, but it doesn't change the fact that the crashes are theirs to own. You don't get a get out of jail free card for being gullible.
- Calloutman 5y ago
  This comment is factually wrong. The airlines did not pay compensation to victims families, Boeing did. Boeing accepted full responsibility for the crash, for providing an unsafe system. Ethiopian airlines did their "due diligence" by purchasing an FAA approved aircraft, by Boeing. They weren't to know that a serious case of regulatory capture was happening. https://www.google.com/url?q=https://www.theguardian.com/business/2021/nov/11/boeing-full-responsibility-737-max-plane-crash-ethiopia-compensation&sa=U&ved=2ahUKEwi5t6m6vfH1AhXPesAKHZf7Dv0QFXoECAQQAg&usg=AOvVaw2a8fBu46r9DS46F3XhfCYU https://www.google.com/url?q=https://www.theguardian.com/bus...
  - nexuist 5y ago
    Huh, TIL (and relevant username). Thank you for the correction. I noticed the article says they only admitted fault for the second crash, do you know what happened with the first? Did Boeing admit fault for that too?

willidiots 5y ago

I've been driving a rental Model 3 with FSD for the past day - for the first time - and totally agree. And after a day's experience, it's obvious it needs to be watched. It's an amazing capability but there are clearly still corner cases, and it makes you very aware of them. I liken Autopilot / FSD more to a horse at its current state. You keep a firm but gentle hand on the rein and let it find its way. Sometime you have to check it. If you get complacent (which the screen explicitly warns you against), yeah, things could go wrong. That's also 100% true when driving a normal car. Edit to add: I think a lot of criticism focuses on the term "Full Self-Driving" which implies "Full Self-Driving". That's fair, but I think it misses an important point. ("Autopilot" made a lot more sense to me: real autopilots can't do everything either. They can't even taxi.) When you're actually IN THE CAR, enabling this functionality for the first time, you have to enable three separate toggle switches. Each one pops up a progressively longer, more dire WARNING dialog, emphasizing the software's primitive state, the importance of paying attention and keeping your hand on the wheel. It also demands your constant attention while engaged.

throwaway0a5e 5y ago
It's like a teenage driver. You have to keep an eye out for developing situations you know it's too stupid to handle elegantly.
ModernMech 5y ago
> It's an amazing capability but there are clearly still corner cases, and it makes you very aware of them. The thing that most of these companies are failing to grasp is that the whole endeavor is like 99% corner cases. It's corner cases all the way down. The hope was that there was this big chunk of the problem space that could be handled in a tractable way, and that would hopefully account for ~80% of driving conditions. Then corner cases could be handled on an as-needed basis to fill in the gaps, and slowly you could handle more and more scenarios. The problem is that the reason for so many corner cases in this domain is that driving is a social activity. The rules of the road actually belie how much communication between humans goes on when they are driving. Driving has its own norms, its own language, culture, emergent behaviors, and traffic exists as a crucial force that must be balanced in a city ecosystem. But autonomous cars aim to solve the driving problem by ignoring the social, and reducing it to an optimization problem. Autonomous cars are blind to the larger world around them except to recognize colors and the nearness of physical bodies. They miss social cues, they lack higher order reasoning about traffic patterns, they can't negotiate with pedestrians or other drivers, they are unable to weigh the ethical considerations between when following a law is actually worse that breaking it. As of now I consider them menaces on the road, and that pains me as someone who got to play with some of the first driverless car technology in the lab. I had such high hopes for it, and what I hoped was not... this.

sho_hn 5y ago

The training we give drivers on how to operate cars and the failure modes the training prepares for are designed for cars without FSD, though, and I'm not sure it adequately prepares drivers to compensate for the possible failure modes of FSD. Especially when those can change from release to release, making it hard to get used in any meaningful sense to the system. This is also a lot more stressful than regular driving and may require better driver condition and more frequent rest. Unfinished vehicles on the road are nothing new; all companies developing cars move pre-prod hardware across public roads. However it usually requires several levels of additional licenses and special safety driving training and instruction before you're allowed to operate a car that may surprise you by behaving very different from a certified production road car. And regular renewal thereafter. I think what's unprecedented here is the scale of the pre-prod testing, the type of experimentation being done and how the drivers are selected. And also the degree to which it's designed into the approach of the experimental engineering being done that it must fail to improve (by widening the training set). There's enough new here it should be discussed imho. Side note: German driver's license training now includes some material on assistance systems and how they may behave unreliably and how to prepare for their failure modes ... but of course this doesn't retroactively expose existing license holders. The rate of change in driver-car interaction being so high is also a novum.

nexuist 5y ago
It is for unprecedented and should be discussed. I just don't buy into the kneejerk reaction of getting this off the roads ASAP when we know drivers use FSD to drive millions of miles a year and accidents are not common at all. It shows that the drivers seem to be responsible enough to keep the tech in check.
- sho_hn 5y ago
  It may show that for the drivers who currently have access, but Tesla selects/limits the beta participants based on proprietary "safe driver" scoring that isn't publically documented. So it's the product manufacturer who can turn the accident probability screw here, or could also get it wrong. And I think that gets to the bottom of what makes many people feel uneasy.

klyrs 5y ago

> Is this not what we should expect from a beta? No, what we should expect from a beta is for it to not be deployed at scale, to non-professional drivers, in a live environment. It's clearly still not ready to go live, much less under the name "full self driving"

paxys 5y ago

> and a human driver quickly took over and avoided the accident Can we be sure that the average driver who has received no special training in the matter will be able to take over from an autopilot in a split second when driving at high speeds? Heck people can't pay attention and keep their eyes on the road even when they are the ones in control. > The responsibility is still on the driver even with driver assist / full self driving technology turned on. If/when the system is rolled out at scale will drivers actually comprehend that? > Then the driver will face the consequences as if he had been driving the car himself Assigning blame after the fact is fine, but no one can bring a dead pedestrian back to life.

nexuist 5y ago
> Can we be sure that the average driver who has received no special training in the matter will be able to take over from an autopilot in a split second when driving at high speeds? The beta system controls for this by only allowing in people who score 98 or above in Tesla's safety surveillance program. The restrictions are tight enough that you have to actively try to be a safe driver to get anywhere close to the cutoff (some people even got pissed off because their score went down because someone brake checked them and they had to decelerate too quickly). Not everyone who buys FSD gets into the beta. > If/when the system is rolled out at scale will drivers actually comprehend that? The only ethical way to roll this out at a wider scale than the scale allowed by the safety surveillance program is if it's good enough to avoid incidents like the one depicted here. I think we can all conclude that Tesla deserves the fat lawsuits if they were to green light FSD in the state it is in today. > but no one can bring a dead pedestrian back to life. An unfortunate fact of life, hopefully made irrelevant by the comeuppance of autonomous transportation technology (including but not limited to automobiles) that makes motor vehicle accidents a thing of the past. Until we get there, though, someone has to keep trying.
- Karrot_Kream 5y ago
  > An unfortunate fact of life, hopefully made irrelevant by the comeuppance of autonomous transportation technology (including but not limited to automobiles) that makes motor vehicle accidents a thing of the past. Until we get there, though, someone has to keep trying. Autonomous transportation already exists, it already works, and it's _already safe_! It just isn't deployed for private automobiles, it's deployed on rail-based transit systems. The technology is widely deployed across Europe and Asia. How many more people need to die to let Americans have a more comfortable ride in their private vehicles?

CTmystery 5y ago

> From my perspective, everything went right here. Holy smokes. Did we watch the same video? > Is this not what we should expect from a beta? Is it OK for a beta software product to nearly kill another person that has no agreement with the company putting the product on the road? People have the right to do whatever they want with their own stuff, fine. But putting others at risk comes with huge responsibility, and must be weighed against the desire to do what you want. In this case the balance clearly falls IMO to Tesla being selfish and negligent by employing their drivers to be the safety backstop in a clearly buggy system.

judge2020 5y ago
Many car systems can fail. Think power steering going out with tires rotated 50 degrees to the right a driver that doesn't know how to properly steer without it.

KennyBlanken 5y ago

> Then the driver will face the consequences as if he had been driving the car himself What consequences? Drivers almost never face any consequences whatsoever for manslaughter or murder with a car. Even if they did face "consequences", they would not be anywhere near the consequences a completely random, innocent, and highly vulnerable road user will.

dogmatism 5y ago

> Then the driver will face the consequences as if he had been driving the car himself, because it's his car and it's his responsibility. You must be unfamiliar with the usual consequences of a driver killing cyclist(s)

wcarron 5y ago
Yep. Cyclists are murdered/manslaughtered with regularity by drivers who recieve zero punishment, and the cyclists and their families receive zero justice. Not just average schmucks in the US, either. Professional cyclists like Michele Scarponi are killed in the EU, too, by a driver using their phone. I forget the results of that case. But cyclists rarely receive the protection of laws that supposedly exist for that purpose.

Jeff_Brown 5y ago

Expecting every buyer to have the ability and discipline to be perpetually watchful for something terrible when almost all of the time there's nothing to worry about is unreasonable. For most humans, standing guard (well) is really hard work, whereas for most drivers, driving is not. This is part of the reason being a cop or a soldier is so stressful.

andbberger 5y ago

Looking forward to getting killed by a tesla and then having a bunch of tesla stans argue that my preventable death was actually OK and an example of the Great Progress being made towards reducing traffic violence!

vanilla_nut 5y ago
So basically the Uber thing from a year or two ago?

randoglando 5y ago

> Is this not what we should expect from a beta? The issue here is that the person on the street does not know they are the ones likely to get impacted by a bug/failure of a software in beta. Beta products are usually not released to the public, or are opt-in.