26 ms·

Scenarios in which Tesla FSD Beta 9.0 fails

281 points by giacaglia 5y ago

gamblor956 5y ago
The 4th scenario (going the wrong way down a one-way road because FSD misinterpreted the sign) is especially scary, as that has the highest likelihood of causing a fatality.
nikhizzle 5y ago
Does anyone have any insight as to why regulators allow a beta like this on the road with drivers who are not trained to specifically respond to it's mistakes? IMHO this puts everybody and their children at risk so Tesla can beta test earlier, but I would love to corrected.
- croes 5y ago
  They fear Elon's Twitter wrath
- darknavi 5y ago
  > drivers who are not trained to specifically respond to it's mistakes What would this entail? Perhaps some sort of "license" which allows the user to operator a motor vehicle?
  - ethbr0 5y ago
    Safely operating a motor vehicle via a steering wheel, accelerator, and brake (and sometimes clutch and gearshift) is a completely different skillset than monitoring an automated system in realtime. Novel skill requirements include: interpreting FSD UI, anticipating common errors, recognizing intent behind FSD actions, & remaining vigilant even during periods of autonomous success.
    - Syonyk 5y ago
      The history of human machine interactions makes it clear that if you have the human as the primary control (in the main decision loop, operating things) and the automation watching over their shoulders, it works pretty well. The opposite - where the automation handles the common case, with the human watching, simply does not work. It's not worked in aviation, it's not worked in industrial systems, it's not worked in self driving cars. This isn't a novel bit of research, this is well understood. "I'll do 99.90% of it correctly, you watch and catch the 0.1% when I screw up!" style systems that rely on the human to catch the mistakes shouldn't be permitted to operate in any sort of safety critical environment. It simply doesn't work.
      - antisthenes 5y ago
        > "I'll do 99.90% of it correctly, you watch and catch the 0.1% when I screw up!" style systems that rely on the human to catch the mistakes shouldn't be permitted to operate in any sort of safety critical environment. It simply doesn't work. Right, because it actually relies on the human to catch 100% of mistakes _anyway_. The human can already do the 99.90% of it correctly with a reasonable degree of safety. I wouldn't be surprised if the increased cognitive load on the human in such a system is an actual overall decrease in safety and strictly worse than a human operator alone. At least with an unreliable system that Tesla demonstrated here.
      - ethbr0 5y ago
        That's a great way to put it! It should be indicative of the kind and volume of licensing and training we require of commercial pilots, in order to use a more limited and situationally simple autopilot. And we still get Boeing-style @&$#-ups at the UX interface.
        Syonyk 5y ago
        The use of the term "autopilot" is particularly cute here, because any pilot who's ever used one knows that autopilots are dumb as rocks, can't really be trusted, and don't do nearly as much as non-pilots tend to assume they do. "You go that way, at this altitude, until I tell you to do something different" is what most of them manage, and they do it well enough. Some of the high end ones have navigational features you can use ("Follow this GPS route or VOR radial"), and in airliners, they can do quite a bit, but you still are very closely monitoring during anything important. "In the middle of the sky," yeah, you've got some time to figure things out if George goes wonky. During a Cat III ILS approach down to minimums, you have two trained pilots, paying full attention to just what this bit of computer is doing, both prepared to abort the approach and head back into the middle of the sky with an awful lot of thrust behind them if anything goes wrong. But in cruise, there's just not too much that requires an instant response.
    - mbreese 5y ago
      I'm going to posit that none of those novel skill requirements are actually requirements. You don't need to interpret a UI, or recognize intent to know that the car shouldn't turn the wrong way on a one-way street or hit a planter. Anticipating common errors and remaining vigilant while driving is a requirement for all driving, not when dealing with FSD. If you are capable of driving a car safely to begin with, then you're capable of recognizing then the car is going to do something wrong and turning the wheel and/or hitting the brakes. I wouldn't personally trust a FSD beta, but only if it is properly supervised, I think it could be tested safely. But, there's the problem -- how can you really properly supervise this? How many drivers are going to try this and just let the car drive w/o paying attention? Or, using it under controlled conditions, but then giving it too much trust in more adverse conditions?
      - ethbr0 5y ago
        What makes them required is the speed that a reaction is expected. In both the quick and velocity senses of the word. Unless the user is able to understand and anticipate common reactions, the entire system is unsafe at >25 mph or so. Or, to put it another way, I'd love to see the statistics on how many drivers can react quickly enough to avert an unexpected turn into an exit divider at 60 mph, after a number of days of interruption free FSD.
        mbreese 5y ago
        We’re largely talking about an update where the primary benefit is low-speed automation — driving in areas where the speed limits are low because the obstacles are many.
    - tshaddox 5y ago
      No, it’s just unsafe (and presumably illegal in most places) to operate a motor vehicle without maintaining constant awareness of the road and control of the vehicle. There’s no additional training or license that permits you to stop maintaining that awareness and control, just like there’s no additional training that permits you to drive under the influence of alcohol.
      - throwaway0a5e 5y ago
        >There’s no additional training or license that permits you to stop maintaining that awareness and control Being a cop on duty often exempts you from distracted driving laws. I'm not sure how they can do stuff on a laptop while driving safely but I assume the state knows best. /s
      - kemitche 5y ago
        The theoretical "additional license" would be in addition to maintaining awareness, not a substitution. So all the normal road awareness, plus being informed of likely FSD failure conditions, and anticipating them / being ready and capable to intercede.
        tshaddox 5y ago
        If you maintain control at all times, you shouldn’t need additional training for any of the car’s features. We don’t require additional training for cruise control, lane-keep assistance, anti-lock brakes, etc.
      - ectopod 5y ago
        I agree, but if you have to maintain constant control of the vehicle then it isn't self-driving.
  - simion314 5y ago
    Maybe people that are trained with such failure videos that show what can go wrong and NOT with propaganda videos that only show the good parts, causing the driver not to pay attention.
  - vkou 5y ago
    A license that allows you to operate a motor vehicle with a beta self-driving feature. It's very similar to a regular motor vehicle, but has different failure modes. A motorcycle is similar to an automobile, but has different failure modes, and needs a special license. A >5 tonne truck is very similar to an automobile, but has different failure modes, and needs a special license. An automobile that usually drives itself, but sometimes tries to crash itself has different failure modes from an automobile that does not try to drive itself.
  - AlotOfReading 5y ago
    California at least has a specific license for this called the Autonomous Vehicle Operator license. It enforces some minimal amount of training beyond simply having a regular driver's license.
  - bestouff 5y ago
    I guess any driving instructor worth its salt would have the required skills to correct the vehicle if it attempts to do something weird. After all FSD is a still-in-training (maybe for an unusual long time) driver.
    - Syonyk 5y ago
      Probably not. It fails in ways that are totally different from how humans typically fail at driving. Student drivers have the ability to comprehend and understand the 3D world around them - it's just a matter of learning the proper control manipulations to make the car do what you want, and then how to interact with other traffic in safe and expected ways. Tesla's FSD system seems to fail at basic 3D world comprehension and reading, often enough.
  - visarga 5y ago
    Perhaps operating a motor vehicle is different from supervising an AI doing that same task.
  - bigtex 5y ago
    Giant obxonious flashing lights and blinking signs stating "Student Self Driving tech on board"
    - kyleee 5y ago
      "Student Full Self Driver"
  - bobsomers 5y ago
    Safety drivers at proper AV companies usually go through several weeks of rigorous classroom instruction and test track driving with a professional driving instructor, including lots of practice with the real AVs on closed courses with simulated system faults, takeover situations, etc. Anecdotally I've seen trained safety drivers take over from unexpected system failures at speeds near the floor of human reaction time. They are some of the best and most attentive drivers on the road. Average folks beta testing Tesla's software for them are woefully under-prepared to safely mitigate the risks these vehicles pose. In several cases they've meaninglessly died so that Tesla could learn a lesson everyone else saw coming months in advance.
  - paxys 5y ago
    What part of the driving test covers taking over control from an automated vehicle with a split second notice?
- avalys 5y ago
  Because America does not yet require that its citizens ask mommy and daddy for permission for every thing they do.
  - Barrin92 5y ago
    this isn't going to the ice cream store, this is driving two tons of steel with experimental software that glitches constantly through inner city traffic. If you look at the footage, at times these cars behave as erratically as a drunk driver, and they're controlled by what appears to be random 'beta testers' with no qualifications to operate a car under those conditions.
  - mrRandomGuy 5y ago
    It would be a glorious honor to have my 'FSD' Tesla drive into a giant concrete pillar, causing my car to explode and erupt into flames.
    - kyleee 5y ago
      Basically Silicon Valley's version of a Viking burial
  - ASalazarMX 5y ago
    Such crude trolling. Build a new house without a permit and see how that freedom feels.
    - SirZimzim 5y ago
      Yes, you have to get a driver's license and also register your car. If that isn't asking for permission then I don't know. Although it's a stretch to assume this permission includes self driving cars.
- nrjames 5y ago
  I used to work with the USG. There was "Beta" software absolutely everywhere, including on some very sensitive systems, because it was not required to go through security approval until it was out of Beta. In some instances, these applications had been in place for > 10 years. That was a number of years ago, so I hope the situation has changed. In general, the USG doesn't have sophisticated infrastructure and policy to deal with software that is in development. With Tesla, my guess is that it is not that they are allowing it to happen, but that they lack the regulatory infrastructure to prevent it from happening.
  - verelo 5y ago
    USG? I'm not sure what that means, i did a google search and i assume it's not "United States Gypsum" or the University System or Georgia...?
    - darknavi 5y ago
      US government I assume.
      - verelo 5y ago
        Thanks, that makes sense.
- whoknowswhat11 5y ago
  It’s a think of the children on HN :)
- optimiz3 5y ago
  > puts everybody and their children at risk Oh won't somebody think of the children? https://en.m.wikipedia.org/wiki/Think_of_the_children https://en.m.wikipedia.org/wiki/Think_of_the_children
  - mrRandomGuy 5y ago
    Out of all the times to mock someone saying that, I don't think "self-driving-but-not-really-car nearly crashes into things and puts the occupants and bystanders at risk for serious bodily harm as evident by the videos posted above" is one of those times.
    - optimiz3 5y ago
      Wait till you read the stats and see videos of what humans driving cars regularly do!
      - ASalazarMX 5y ago
        Still, an autopilot should aim to be a peer with the best drivers, not with mediocre or bad drivers.
        throwaway0a5e 5y ago
        A peer with average drivers would probably be fine. A peer with average drivers and doing completely nonsensical things every now and then is basically an average driver on their cell phone though and the verdict on that is that it's not fine.
        optimiz3 5y ago
        It's a limited beta with the goal of discovering error cases.
        barbazoo 5y ago
        In this case I'd prefer if "discovering error cases" was done without risking actual lives. Injuring someone would not be a valid way discover an error case.
- SloopJon 5y ago
  The Twitter thread is lacking in attributions, but I saw most of these some months back after a post about scary FSD behavior. I watched with morbid curiosity, and a rising level of anger. The guy with the white ball cap repeatedly attempted a left turn across two lanes of fast-moving traffic, with a drone providing overhead views. He seemed smart, aware, and even a bit conservative in intervening. Nonetheless, I couldn't help but thinking that none of the oncoming vehicles consented to the experiments racking up YouTube views. If he doesn't jump on the brakes at the right time, he potentially causes a head-on collision with a good chance of fatalities. Yes, I do agree that beta drivers should get extra training. However, I'm not sure I agree with the premise of beta testing an aluminum rocket on wheels on public roads in the first place.
  - misiti3780 5y ago
    How do you know the beta drivers did not get training?
    - SloopJon 5y ago
      I said that they should. I don't know whether they do. A March 2021 tweet indicated that beta access will be as simple as pressing a "Download Beta" button in the car. A subsequent tweet said that the program had 2,000 owners, some of whom have had their access revoked for not paying enough attention to the road.
  - aeternum 5y ago
    Do people sign up to be beta testers whenever a new 15 year-old with a permit gets behind the wheel? It's even crazier that we allow that. The cars that student drivers learn in typically do not have dual control so it's quite difficult to intervene. Every highschool has a story about how a student slammed the accelerator instead of the brake.
    - nexuist 5y ago
      People in this thread seem to have overly high expectations about what is required to drive on US public roads. Not even considering that each state has its own licensing and registration requirements, all states except one (Georgia) accept foreign driver's licenses, which means it doesn't even matter what you regulate if countries X, Y, and Z can just not care about it and refuse to enforce it. Regulating self driving is a good idea but it's a tall order to cast beta software as the menace when there are more idiots driving on the roads at this moment than there will ever be Tesla products.
- TheParkShark 5y ago
  Everyone and their children seems a bit hyperbolic. There hasn’t been an accident involving FSD beta 9, but I’m sure someone has been killed at the hands of a drunk driver today. I am failing to find any comments from you arguing for alcohol to be removed from shelves across the country? Why aren’t you pushing your regulators for that?
  - studentrob 5y ago
    It's not hard to see the connection between FSD driving towards concrete pillars and the AP accidents on highways driving towards concrete barriers. None of this is hyperbole. FSD isn't a whole new system, it's based on AP. As Elon often says, they have been going for incremental improvements rather than the all-or-nothing approach taken by Waymo and others.
  - esperent 5y ago
    Drunk driving is already illegal. Claiming that we should ban all alcohol because people use it to commit illegal acts makes about as much sense as claiming we should ban kitchen knives because people occasionally use them for stabbing. Besides that, FSD being less dangerous than drunk driving is a terrible benchmark.
- renewiltord 5y ago
  Because Tesla FSD accidents don’t kill people who were not party to the agreement. That seems intuitively fair to me.
  - snet0 5y ago
    I'm sure you don't actually think this, if you think about it for a second longer. "What's wrong with drunk drivers? They're only harming themselves?"
    - renewiltord 5y ago
      Well, the problem with drunk driving is that it kills other people. We’ve witnessed it. We know it happens. Because we have seen it occur. Tesla FSD driving assist has never killed someone not in the car. Considering that rampant safetyism is par for the course here, I’m hardly convinced of the “he could’ve killed someone” school of thought.
croes 5y ago
I doubt that autonomous driving will work in the next couple of years without infrastructure adjustments. We built roads for cars to replace horses, we will need to built roads for autonmous cars to replace human driven cars.
- dangoor 5y ago
  I live in a place with fairly snowy winters, which seems like a huge problem for self-driving cars. I completely agree with this thought that we should work on making our roads help the self-driving cars. If we had started that effort several years back, we would likely have a lot of road miles that could safely handle self-driving vehicles. If we start now, in several years we'll have a bunch of road miles ready for them. Granted that the US has a lot of road-miles, but doing this work incrementally is the way to go. I have a feeling it will be faster to outfit the roads than to build reliable self-driving software. I will also grant that this is a "feeling" and I'm sure someone has done actual research :) If this capability could be used to help hasten the move to electric vehicles, we could even get a climate change win in at the same time.
  - w0m 5y ago
    North East in winter has ~0% chance of getting FSD capable in the next 20 years I think. 80% of the population of my hometown can't leave the house for months at a time safely today; let alone automation taking over for that.
    - jazzyjackson 5y ago
      Great argument for more rail. Yes there are still conditions that freeze switches but that’s when you run gas lines under the tracks to prevent ice from forming: https://wgntv.com/news/when-its-this-cold-chicago-sets-its-train-tracks-on-fire/ https://wgntv.com/news/when-its-this-cold-chicago-sets-its-t... Weirdest thing that prevents trains from running tho is wet leaves on the tracks, of all things, it’s too slippery.
  - jazzyjackson 5y ago
Syonyk 5y ago
I don't understand how something this broken is allowed to operate on public roads. If I drove like Tesla's FSD seems to based on the videos I've seen, I'd be pulled out of my car and arrested on (well founded) suspicions of "driving while hammered." After a decade of work, it's not capable of doing much beyond "blundering around a city mostly without hitting stuff, but pay attention, because it'll try to hit the most nonsensical thing it doesn't understand around the next corner." It drives in a weirdly non-human way - I've seen videos of it failing to navigate things I'm sure I could get my 6 year old to drive through competently. Except, I don't actually let her drive a car on the road. I'm out in a rural area, and while "staying in the lane" is perfectly functional (if there are lanes, which isn't at all the case everywhere on public roads), there's a lot of stuff I do on a regular basis that I've not seen any evidence of. If there's a cyclist on a two lane road, I typically get over well over the center line if there's no oncoming traffic to make room. If there is oncoming traffic, typically one person will slow down to allow the other car to get over, or the lanes just collectively "shift over" - the oncoming traffic edges the side of the road so I can get myself on or slightly over the centerline to leave room for the bike. And that's without considering things like trailers that are a foot or two into the road, passing a tractor with a sprayer (they don't have turn signals, so be careful where you try to pass), etc. If they've got any of this functionality, I'd love to see it, because I've not seen anything that shows it off. At this point, I think we can reasonably say that it's easier to land people on the moon than teach a car to drive.
- toomuchtodo 5y ago
  > I don't understand how something this broken is allowed to operate on public roads. Also out in a rural area. Running out to pickup lunch a few minutes ago, a young man flipped their old pickup truck on its side in an intersection, having hit the median for some reason. I too don't understand how humans are allowed to operate on public roads. Most of them are terrible at it. About 35k people a year die in motor vehicle incidents [1], and millions more are injured [2]. Total deaths while Tesla Autopilot was active is 7 [3]. I believe the argument is the software will improve to eventually be as good or better than humans, and I have a hard time not believing that, not because the software is good but because we are very bad in aggregate. [1] https://www.iihs.org/topics/fatality-statistics/detail/state-by-state https://www.iihs.org/topics/fatality-statistics/detail/state... [2] https://www.cdc.gov/winnablebattles/report/motor.html https://www.cdc.gov/winnablebattles/report/motor.html [3] https://www.tesladeaths.com/ https://www.tesladeaths.com/
zyang 5y ago
It appears Tesla FSD just ignores things it doesn't understand, which is really dangerous in a production vehicle.
- mrRandomGuy 5y ago
  Why are you getting down-voted? There's literal videos depicting what you state. Is the Musk Fanboy Brigade behind this?
  - darknavi 5y ago
    You guys can downvote?!
    - jacobkranz 5y ago
      You can after you get a certain amount of upvotes (I can't remember the exact number though. 30? 50?)
    - reallydontask 5y ago
      I think you need 500 karma before you can downvote
      - Tempest1981 5y ago
        And only within 24 hours
        darknavi 5y ago
        Wow! I had no idea. That's a pretty cool way to stop troll brigading. Thanks for the info!
- heavyset_go 5y ago
  This is what the automated Uber vehicle[1] that struck and killed a pedestrian did, as well. Despite picking her up via sensors and the ML model, it was programmed to ignore them. [1] https://www.nytimes.com/2018/03/19/technology/uber-driverless-fatality.html https://www.nytimes.com/2018/03/19/technology/uber-driverles...
simion314 5y ago
Do we know what is the process and who decides that an update is ready? This is a big decision so I am wondering what process or personality is needed to decide that all is safe, let's update things.
moojah 5y ago
This really isn't beta grade software, as it isn't feature complete as the failure scenarios in the video clearly show. I'd call it alpha grade, and it has been that for a while. It's not 2 weeks or whatever unrealistic timeline away from being done, as Elon has claimed for ever. 2 perhaps years if we're lucky, but given human and driving complexity probably way more before even the whole of the USA is reliably supported beyond L2.
- H8crilA 5y ago
  Like Donald Trump, but for nerds: http://elonmusk.today/ http://elonmusk.today/ FSD would be equivalent to the Mexican border wall, I guess?
  - lostmsu 5y ago
    And the tax hike on cap gains.
- w0m 5y ago
  >This really isn't beta grade software, as it isn't feature complete as the failure scenarios in the video clearly show. I think it depends what they actually are trying to accomplish. This is Beta for a glorified cruise control overhaul; not a beta for promised RoboTaxi. Musk/Tesla tend to talk about RoboTaxi then slip seemlessly into/out of 'but today we have low engagement cruise control!'. Fair bit of hucksterism.
  - barbazoo 5y ago
    > I think it depends what they actually are trying to accomplish Good point. "Full Self Driving" in my mind paints a picture beyond "a better cruise control". But maybe they meant that and just named it wrong.
    - dragontamer 5y ago
      From Tesla's webpage: > Full Self-Driving Capability > > All new Tesla cars have the hardware needed in the future for full self-driving in almost all circumstances. The system is designed to be able to conduct short and long distance trips with no action required by the person in the driver’s seat.
wedn3sday 5y ago
I watched a human driver in a BMW weave at 90 mph through traffic yesterday, swerving back and forth between lanes inches from other peoples cars. If Tesla can make a car that's even 1% less dangerous than human drivers, that could be thousands of deaths avoided every year. Humans should not be allowed to drive cars on public streets.
- carlmr 5y ago
  But that surely is the description of 1% of the drivers. It should be better than the 1% worst drivers. It should be demonstrably safer than humans on average with a good margin.
  - minhazm 5y ago
    Even if they accomplish that, people will move the goal posts. People are illogical and will see some edge case that self driving might perform worse at, but on an aggregate data basis it might be 10x safer than human drivers. People will see that 1 edge case and say that self driving cars are worse than humans.
    - mikestew 5y ago
      That's an awful lot of presumption of "people"'s behavior toward a system that has failed nearly every promise made. There's no need to move the goal posts when the player isn't anywhere near it. I mean, if we're going to talk about someone moving goal posts around, how about the one claiming "full self-driving" for a system that isn't anything of the sort?
- Syonyk 5y ago
  > Humans should not be allowed to drive cars on public streets. Because the alternative right now is... this abomination that doesn't even grasp what a concrete pillar is?
  - jacquesm 5y ago
    Maybe it does and it is trying to put itself out of its misery?
stsmwg 5y ago
One of the questions I've repeatedly had regarding FSD (and Tesla's approach in particular) is the notion of memory. While a lot of these scenarios are disturbing, I've seen people wavering on lanes, exits and attempting to turn the wrong way onto one-way streets. People have memory, however. If we go through the same confusing intersection a few times, we'll learn how to deal with that specific intersection. It seems like a connected group of FSD cars could perform that learning even faster since it could report that interaction with any car rather than driver-by-driver. Are any of the FSD implementations taking this into account?
- H8crilA 5y ago
  That's kind of the "selling point" of running this experiment on non-consenting public, that it will learn over time and something working will come out of that in the end.
- Syonyk 5y ago
  This has been a common assertion about Tesla's "leadership" in the field - that they can learn from all the cars, push updates, and obviously not have to experience the same issue repeatedly. It's far from clear, in practice, if they're actually doing this. If they have, it would have to be fairly recent, because the list of "Oh, yeah, Autopilot always screws up at this highway split..." is more or less endless. GM's Supercruise relies on fairly solid maps of the areas of operation (mostly limited access highways), so it has an understanding of "what should be there" it can work off and it seems to handle the mapped areas competently. But the problem here is that the learning requires humans taking over, and telling the automation, "No, you're wrong." And then being able to distill that into something useful for other cars - because the human who took over may not have really done the correct thing, just the "Oh FFS, this car is being stupid, no, THAT lane!" thing. And FSD doesn't get that kind of feedback anyway. It's only with a human in the loop that you can learn from how humans handle stuff.
  - stsmwg 5y ago
    Great, thanks for that info. I'm remembering the fatal crash of a Tesla on 101 where the family said the guy driving had complained about the site of the accident before. It's interesting to know that there's at least a mental list of places like this even now. Disengagements should at least prompt a review of that interaction to try and understand why the human didn't like the driving. Though at Tesla's scale that has already become something that has to be automated itself.
killjoywashere 5y ago
I work in ML (medical diagnostics) and own a 2020 Tesla w/ FSD and do not trust it at all. I also happen to be a cyclist, currently living in SV. My regular road ride takes me past Tesla HQ. The more of these FSD fails I see, the more I wonder how many Teslas passing me are running in FSD mode. Scary.
deregulateMed 5y ago
Originally when Tesla was on the verge of collapse, the self driving technology waa high risk, high reward tech that will kill someone but the company was in the negative. If that killed Enough people, declare bankruptcy and investors lose money. However today, isn't Tesla profitable now? They already have created a Veblen good, what benefit does Tesla get out of a high risk feature?
deleted 5y ago
[deleted]
bsagdiyev 5y ago
Why are there so many comments from seemingly different posters all saying the same thing on this, "I don't understand why humans are allowed to drive cars"? It feels kinda... culty? Or too similar to be a coincidence. It honestly probably is but these Tesla posts always bring out those types and it confuses me. Humans kill humans, do we want machines to start doing it on the road automatically now too?
- dekhn 5y ago
  I think the idea is, if humans were held to the standards machines are, we wouldn't let them drive. If you offered me a car that drove itself, and statistically, it killed people at the same rate as humans, but let me not have to drive, I'd take that. Nobody everybody agrees.
  - w0m 5y ago
    Self determination > logic. Look at the American Gun debate as an example; carrying for self defense makes you drastically less safe by most metrics. But people prefer to have the modicum of self determinism over more consistent statistical safety.
    - ggreer 5y ago
      > carrying for self defense makes you drastically less safe by most metrics You're probably referencing studies such as Investigating the Link Between Gun Possession and Gun Assault[1], which looked at whether people who were shot were carrying a gun at the time. It did not differentiate between criminals who were carrying illegally and law-abiding citizens who had concealed carry permits. The selection effect in such studies is clear: Many criminals who carry guns are involved in shootings. Also, many people carry guns because they are at a higher risk of being a victim of violent crime. I have yet to find a study that corrects for these confounders. 1. https://sci-hub.st/10.2105/AJPH.2008.143099 https://sci-hub.st/10.2105/AJPH.2008.143099
  - notahacker 5y ago
    > I think the idea is, if humans were held to the standards machines are, we wouldn't let them drive. But the reverse is true. Humans receive driving bans and even criminal penalties for the sort of driving errors autonomous systems make without penalty.
yssrn 5y ago
Scary set of videos. Is Tesla using Marylanders to train FSD?
creato 5y ago
Some of the issues shown in these videos make me wonder about Tesla's strategy of using non-professional driver (customer) data to train FSD. Things like changing lanes at the last second is a thing that (obnoxious) humans do, and would be a bad example to learn from. There might be a lot of subtle garbage in Tesla's dataset.
- rcMgD2BwE72F 5y ago
  Tesla can easily filter these cases out, automatically. They have triggers to catch things they want (e.g sudden lane changes) and campaigns have conditions too (only from prudent drivers, if lane change is forced, etc). FSD does not learn from all drivers all the time.
  - rcMgD2BwE72F 5y ago
    Sources: https://youtu.be/hx7BXih7zx8?t=639 https://youtu.be/hx7BXih7zx8?t=639 and https://youtu.be/gZ2SeiLjaEc?t=941 https://youtu.be/gZ2SeiLjaEc?t=941
Geee 5y ago
It seems to estimate the surroundings pretty well in the new version, but the path planner still has issues. I'm not sure if they already do this, but I think Tesla should run FSD on all cars in shadow mode, and then use the FSD vector data and driver actions to train the path planner. Basically using FSD vector data as input and driver actions as output of the neural net. If the network can't reach confidence in certain situations, then information is missing from the FSD vector data and more raw video data is needed. It would be possible to measure the difference between human drivers vs. FSD and automatically send data from errors to Tesla for training.
- Syonyk 5y ago
  I've been hearing for years now that this is a Tesla advantage, that they can run in shadow mode, learn from humans, and big data learn all the quirks of the roads so they don't have to learn them in the self driving system. If this has been the case, evidence here is certainly lacking. Turning down a street one presumably hasn't seen a car drive down in this direction? Failing basic stationary obstacle avoidance because they're in the center of the road? Screwing up basic turn lane behavior? Either they aren't taking the "years of learning" data into account, or, worse, they are - and even with all that correction, it's still this bad.
  - Geee 5y ago
    I think they've been focused on solving the "environment mapping and object detection" problem, and if that isn't solved then they can't have the proper inputs to solve the "path planning / decision making" problem. It seems that they're close to solving the first problem, and are ready to move on to the next. As far as I know they've been using the fleet to collect raw camera data for their training set but I don't think they've used the fleet to learn driving behaviour.
  - visarga 5y ago
    It's a cherry picked set of examples you're drawing generalisations from. On any day I can see more reckless driving from humans than I've ever seen online with Tesla.
dreyfan 5y ago
Is "The car doesn't recognize a truck crossing in front of it so it drives under and decapitates the driver" still an open issue? What about the multiple instances where FSD drives the car forcibly into a firetruck?
cs702 5y ago
Many of the comments here seem a bit... unfair to me, considering that these clips were handpicked. I watched (and fast-forwarded) through a few of the original, full-length videos from which these clips were taken. The full-length videos show long drives (in some cases, hours long) almost entirely without human intervention, under conditions explicitly meant to be difficult for self-driving vehicles. One thing I really liked seeing in the full-length videos is that FSD 9 is much better than previous efforts at requesting human intervention in advance, with plenty of time to react, when the software is confused by upcoming road situations. The handpicked clips are exceptions. For BETA software, FSD 9 is doing remarkably well, in my view. I mean, it's clearly NOT yet ready for wide release, but it's much closer than all previous versions of Tesla FSD I've seen before, and acceptable for a closed-to-the-public Beta program.
- richwater 5y ago
  It literally doesn't matter how well it does in 90% of situations when the other 10% can injure or kill people in relatively basic scenarios like the Tweets presented. I mean the car almost ran into a concrete pillar like it wasn't even there. > For BETA software, FSD 9 is doing remarkably well If this was a React website, that'd be great. But it's a production $40,000 multi ton automobile.
  - bob33212 5y ago
    People will die today because a driver was drunk/distracted/suicidal/rode raged or had a medical problem. Are you OK with that or do you think we should attempt to fix that with software? If you do think you should attempt to fix that, do you understand that software engineering is a iterative process? It gets safer overtime.
    - foepys 5y ago
      > People will die today because a driver was drunk/distracted/suicidal/rode raged or had a medical problem. That's a straw man argument. Mandatory breathalyzer tests before every engine start and yearly medical and psychological check-ups would solve a lot of these problems as well. > It gets safer overtime. There is no evidence of this happening yet. It's still trying to drive people into concrete pillars. Remember that you only need to drive into a pillar once to die. Passing it 10,000 times before doesn't help when an update suddenly decides that the pillar is now a lane. I'd be totally fine with Tesla testing this with trained drivers in suitable areas but giving this into the hands of amateurs is pure "move fast and break things", with "break" meaning ending lives. The worst thing is: all of these videos feature sunny weather. How bad would it be in the rain? Ever tried to take picture in the dark with even light rain? Cameras are very bad at this.
mritun 5y ago
Question: Would it be awesome if FDA allowed Pharma to conduct drug-tests like this? Put a "beta-2.9" on the vial and let people try it out... Some may die, be disabled or may linger in vegetative state life-long, but it was their choice afterall and the it can argued that medicinal side-effects are very small cause of mortality and hence long-term the "public beta tests" will make drugs more effective and save more lives!
- CreepGin 5y ago
  Depends on how much the drug costs. If it costs 100k a pop, then I don't see the general public being affected by it too much. RIP those brave rich souls.
  - mritun 5y ago
    General public is affected. A Tesla can cause accident on a public road with other drivers on the road, or cause a cascade or a pile-on - innocent bystanders can die or be injured.
- colinmhayes 5y ago
  I think the difference is releasing self driving provides the data Tesla needs to improve it. Releasing a half baked drug doesn't help the pharma company improve it. If you're asking whether that would be awesome if it lead to pharmaceutical innovations I think it would.
  - flutas 5y ago
    > I think the difference is releasing self driving provides the data Tesla needs to improve it. Releasing a half baked drug doesn't help the pharma company improve it. The pharma company could see results (self-driving car data) and figure out what caused the issues (details of the deaths in pharma, accidents in Tesla) and use that to make the next beta version. I don't really see how that isn't an apt analogy. It's called clinical trials for pharma the key point being it's opt in and doesn't affect anyone around the subject, unlike Tesla's autopilot beta.
  - 5y ago
jacquesm 5y ago
Some regulator somewhere will take this junk off the road with the stroke of a pen and I'll feel that much safer on the road when it happens. And we'll have Elon Musk to thank for electric cars and the self driving winter. It's actually pretty simple: have FSD do a regular driving test. If it can pass that it's good to go, if not it fails the test and will not be allowed to control a vehicle.
j7ake 5y ago
These spectacular fails weaken the rationalist's arguments that "as long as FSD achieves lower deaths per km driven (or any other metric) than humans" then FSD should be accepted in favor of human driving. Even if "on aggregate" FSD performs safer (by some metric) than humans, as long as FSD continues to fail in a way that would have been easily preventable had a human been at the wheel, FSD will not be accepted into society.
- manmal 5y ago
  I think this is already true though for highway driving. Highways are long and tiring, and the surroundings model is easy to get right, so computers have an advantage. Most manufacturers offer a usable cruise control which is safe and can probably be active 90% of the time spent on highways. I often switch it on in my 3yo Hyundai as an extra safety measure in case the car in front of me unexpectedly brakes while I‘m not looking there. Add to that a lane keeping assistant and lane change assistant, and you don‘t need to do much. Except for when the radar doesn’t see an obstacle in front of you, eg because the car in front of you just changed lanes. That needs to be looked out for.
  - deleted 5y ago
    [deleted]
- cptskippy 5y ago
  > spectacular There was nothing spectacular about those failures, I would say because the driver was attentive and caught/corrected the car. That's not to say some of these fails could not have ended in catastrophe, but to call them spectacular is quite the exaggeration. One of those "spectacular fails" was displaying two stop signs in the UI on top of each other while properly treating it as one stop. Using hyperbole like this only makes people ignore or dismiss your otherwise valid point.
- cogman10 5y ago
  Yeah, I've brought this point up in other locations. It does not matter that any autonomous driving tech is safer than human drivers. They MUST be perfect for the general public to accept them. The only accidents they'd be allowed to get into are ones that are beyond their control. Algorithmic accidents, no matter how rare they are, won't be tolerated by the general public. Nobody will accept a self driving car running over a cat or rear ending a bus even if regular humans do that all day long. The expectation for self driving cars is a perfectly attentive driver making correct decisions. Because, that's what you theoretically have. The computer's mind doesn't "wander" and it can't be distracted. There's no excuse for it to drive worse than the best human driver.
manmal 5y ago
Is FSD still operating on a frame-by-frame basis? I remember it was discussed on Autonomy day that the ideal implementation would operate on video feeds, and not just the last frame, to improve accuracy. When you look at the dashboard visualizations of the cars‘ surroundings, the model that is built up looks quirky and inconsistent. Other cars flicker into view for one frame and disappear again; lane markings come and go. I saw a video where a car in front of the Tesla indicated, and the traffic light in the visualization (wrongly) started switching from red to green and back, in sync with the indicator blinking. How could a car behave correctly as long as its surroundings model is so flawed? As long as the dashboard viz isn’t a perfect mirror of what’s outside, this simply cannot work.
- cptskippy 5y ago
  I would think the flickering objects in the UI is a result of objects hovering around the confidence threshold of the model. But... I have a Model 3 and the flickering happens when stationary and nothing around you is moving.
- joakleaf 5y ago
  There also seem to be general problems with objects disappearing when they are obscured by other objects, and then reappear later, when no longer obscured. It is ridiculous, that the model doesn’t keep track of objects, and assume they continue with current velocity when they become obscured. It seems like a relatively simple thing to add depending on how they represent objects. You could even determine when the objects are obscured by other objects. In 8.x videos I noticed cars shifting and rotating a lot over fractions of a second, so it seemed like they needed a Kalman filter for objects and roads. Objects in 9.0 look more stable, but I still see lanes, curbs, and entire intersections shifting noticeably from frame to frame. So if they added time (multiple frames) to the model, it is still not working that well.
- EForEndeavour 5y ago
  You've nicely articulated what was bothering me about the jittery dashboard visualizations. Why on earth is everything flickering in and out of existence, and why is the car's own planned trajectory also flickering with discontinuities?? It seems like they aren't modeling the dimension of time, thus throwing away crucial information about speed and needlessly re-fitting the model to static snapshots of dynamic scenes. It's like the ML system needs its inferences constrained by rules like "objects don't teleport" or "acceleration is never infinite."
DrBenCarson 5y ago
Their current sensor and camera lineup have made this impossible on models already on the road. Good luck to their customers
jdofaz 5y ago
I love my Tesla but watching these videos made it an easy decision not to spend $10k on the FSD option.
okareaman 5y ago
I had a girlfriend that I didn't trust driving my car, especially with me in it. That's how I feel about Elon Musk driving my car.
yawaworht1978 5y ago
Any human driver driving like that would have his driver's licence revoked, might risk prison for endangerment of society and perhaps be subject to a psychological evaluation (not joking, this stuff happens in Germany). Why are there no consequences for neither Tesla nor the lawmakers who are looking away? Sure, test autonomous driving off the roads or on dedicated test environments, but do not permit this on public roads, i do not wanna be part of any beta test.
gumby 5y ago
One case that's perhaps not a bug: > 4th: Tesla doesn't recognize a one-way street and the one-way sign in the street, and it drives towards the wrong way If the car is in Boston this is OK.
selimnairb 5y ago
“Full Self-Driving Beta” should be an oxymoron. Why is this being allowed?
foobarbazetc 5y ago
This thing is like 10 years away from being actually usable, if it ever gets there.
cromwellian 5y ago
All of the Tesla fanboys saying you don't need LIDAR, RADAR, or Maps as a kind of "ground truth" and you can get by on video only should take this as a huge repudiation of brag by Musk. Look, Radar, Lidar, and Maps aren't perfect. A map can be wrong, Radar can have false echos, Lidar can have issues in bad weather, etc. But if you're depending on visual recognition of one-way signs, having a potentially out-of-date backup map to check your prediction again, and err on the side of caution, is better than having no Map at all. Ditto for radar and lidar. Radar enabled cars have collided with semis, remember, because they're aimed low (underneath the semi). But in the case of these planters, or road cones, or monorail poles, the radar is going to be a strong signal to the rest of the system that it isn't seeing something. There's no way FSD should be allowed on public roads, it's not even CLOSE to ready. The other car companies are behaving far more responsibility: full sensor suite, multi-sensor fusion, comparison with detailed road maps, and limiting operation to safe areas, and expanding as safety is proven. Who the hell cares if it's not FSD driving if its so dangerous? I'll take 75% FSD that only operates on highways and long commutes or road trips, over 100% FSD that has so many disengagements and "woah" moments as to be unusable.
- TheParkShark 5y ago
  I see this argument a lot. This is a small beta test and I have yet to see anything better on a larger scale. Most drivers shouldn’t be allowed on public roads yet here we are. Why are you so full of fear and anger over this? Isn’t this a place for open discussion?
  - nullc 5y ago
    > I have yet to see anything better on a larger scale Waymo self-driving is a relatively large scale. It also has blunders but they appear to be far less concerning than Tesla's.
    - TheParkShark 5y ago
      Geo-fenced isn’t large scale. There’s no clear route to allow them to expand that tech.
nullc 5y ago
These aren't even the worst examples I've seen by far-- but I suppose they're the worst people have posted with the latest software. As someone in another thread said: Driving with Tesla FSD is like being a drive instructor for a narcoleptic student driver who is currently tripping balls. I don't want to share a road with these things. I try to stay as far away from every tesla on the road as I can.
- TheParkShark 5y ago
  Please share them if you have. As someone in another thread said: Tesla has the most advanced self driving software currently in development in the world. If there’s something better you can feel free to share that with your other videos.
  - ahahahahah 5y ago
    > Tesla has the most advanced self driving software currently in development in the world This is just nonsense. These videos show them at a disengagement every couple miles. Waymo was at that level 10 years ago and is 10000s of times better now.
    - TheParkShark 5y ago
      Waymo only works in a 50 sq mi area [1] and has it’s own issues [2]. So no, this is not nonsense. [1] https://arstechnica.com/cars/2021/05/why-hasnt-waymo-expanded-its-driverless-service-heres-my-theory/ https://arstechnica.com/cars/2021/05/why-hasnt-waymo-expande... [2] https://www.bbc.com/news/technology-57144755 https://www.bbc.com/news/technology-57144755
      - nullc 5y ago
        And Tesla FSD is available to what... two dozen parties outside of the company? AFAIK both have extremely limited availability. Waymo doesn't require frequent human intervention to prevent life threatening situations.
deegles 5y ago
All the other FSD car companies should sue to get these off the roads since a big enough fuck-up by Tesla will set back the industry for a decade.
sharkmerry 5y ago
Can someone explain what is happening in the first video? are the planters on the left after the turn and it was trying too tight of a turn?
reader_mode 5y ago
Every time I see these kinds of videos and read about Tesla valuation and messaging I can't make up my mind what to think about it. On one hand I find it ridiculous that these kinds of results so far into the development are still making Tesla the most valuable car maker, selling this as FSD with no proof they will ever reach it etc. On the other hand they secured billions of $ for R&D into FSD and BEV design and brought EV production to scale. Their messaging is way to optimistic (borderline fraudulent) but they are doing better things with the money than the alternatives. Traditional automakers have shown to be completely impotent in this regard. Money is flowing into all kinds of wasteful bullshit (that Tesla is backing BTC makes them less trustworthy in my eyes)
- BrissyCoder 5y ago
  Highly recommend listening to the 3 episode miniseries TrueAnon did on Musk and Tesla. Really eye opening. The dude should be in prison.
MobileVet 5y ago
For a company that exists because of technological and economy of scale gains, primarily in battery tech, it was always odd to me the way Elon would say LIDAR was too expensive or not fast enough. LIDAR et al have come a long way in the last 5 years and will continue to evolve at an exponential rate due to the efforts being put into them. Yes, I get it, humans use 2 eyes only, but sensor fusion of complementary technologies is so powerful. When lives are on the line, having more than one type of sensor involved is just prudent engineering.
GhostVII 5y ago
I don't know why Tesla is so focused on self driving in cities. Seems like a far better use of time would be to get to level 3 self driving on highways (which is what other automakers seem to be trying to do). That gets you most of the benefits of self driving (not having to pay attention for long trips), with probably 1% of the complexity of city driving. Getting city driving working just seems so impossibly hard, there is no way you are going to be able to use ML to determine road rules since they vary by jurisdiction and over time, and there are so many edge cases you won't have enough data for. But how do you write a program that can read a street sign, interpret it's meaning, and act upon it safely? Even if Tesla just got autopilot to the point where it could be level 3, where drivers don't have to pay attention, that would be a massive improvement, and so much easier than FSD. Maybe they painted themselves into a corner though with all Elon's blatant lies about FSD, can't stop development without a lawsuit. They seem kind of screwed anyways, looks like they are running into the limits of the current camera positioning and onboard processing power, might need to do another iteration of replacing every FSD owners hardware.
villgax 5y ago
This is what is needed to pit against human drivers & other FSD software from manufacturers.
MarkMc 5y ago
Here's an idea, although I'm not sure if it would work: The government allows Tesla to conduct a randomised control trial: When you get in a Tesla you enter your destination, then there's a 50% chance of having to drive the car manually (the 'control') and 50% chance of having the car drive you (the 'treatment'). After 10 million trips, if the treatment group had fewer accidents than the control group, then Tesla is allowed to roll out self driving to everyone. But on the other hand, if treatment has more accidents than the control group then Tesla must pay, say, $10 million for each excess fatality.

281 points by giacaglia 5y ago

The 4th scenario (going the wrong way down a one-way road because FSD misinterpreted the sign) is especially scary, as that has the highest likelihood of causing a fatality.

nikhizzle 5y ago

Does anyone have any insight as to why regulators allow a beta like this on the road with drivers who are not trained to specifically respond to it's mistakes? IMHO this puts everybody and their children at risk so Tesla can beta test earlier, but I would love to corrected.

croes 5y ago
They fear Elon's Twitter wrath
darknavi 5y ago
> drivers who are not trained to specifically respond to it's mistakes What would this entail? Perhaps some sort of "license" which allows the user to operator a motor vehicle?
- ethbr0 5y ago
  Safely operating a motor vehicle via a steering wheel, accelerator, and brake (and sometimes clutch and gearshift) is a completely different skillset than monitoring an automated system in realtime. Novel skill requirements include: interpreting FSD UI, anticipating common errors, recognizing intent behind FSD actions, & remaining vigilant even during periods of autonomous success.
  - Syonyk 5y ago
    The history of human machine interactions makes it clear that if you have the human as the primary control (in the main decision loop, operating things) and the automation watching over their shoulders, it works pretty well. The opposite - where the automation handles the common case, with the human watching, simply does not work. It's not worked in aviation, it's not worked in industrial systems, it's not worked in self driving cars. This isn't a novel bit of research, this is well understood. "I'll do 99.90% of it correctly, you watch and catch the 0.1% when I screw up!" style systems that rely on the human to catch the mistakes shouldn't be permitted to operate in any sort of safety critical environment. It simply doesn't work.
    - antisthenes 5y ago
      > "I'll do 99.90% of it correctly, you watch and catch the 0.1% when I screw up!" style systems that rely on the human to catch the mistakes shouldn't be permitted to operate in any sort of safety critical environment. It simply doesn't work. Right, because it actually relies on the human to catch 100% of mistakes _anyway_. The human can already do the 99.90% of it correctly with a reasonable degree of safety. I wouldn't be surprised if the increased cognitive load on the human in such a system is an actual overall decrease in safety and strictly worse than a human operator alone. At least with an unreliable system that Tesla demonstrated here.
    - ethbr0 5y ago
      That's a great way to put it! It should be indicative of the kind and volume of licensing and training we require of commercial pilots, in order to use a more limited and situationally simple autopilot. And we still get Boeing-style @&$#-ups at the UX interface.
      - Syonyk 5y ago
        The use of the term "autopilot" is particularly cute here, because any pilot who's ever used one knows that autopilots are dumb as rocks, can't really be trusted, and don't do nearly as much as non-pilots tend to assume they do. "You go that way, at this altitude, until I tell you to do something different" is what most of them manage, and they do it well enough. Some of the high end ones have navigational features you can use ("Follow this GPS route or VOR radial"), and in airliners, they can do quite a bit, but you still are very closely monitoring during anything important. "In the middle of the sky," yeah, you've got some time to figure things out if George goes wonky. During a Cat III ILS approach down to minimums, you have two trained pilots, paying full attention to just what this bit of computer is doing, both prepared to abort the approach and head back into the middle of the sky with an awful lot of thrust behind them if anything goes wrong. But in cruise, there's just not too much that requires an instant response.
  - mbreese 5y ago
    I'm going to posit that none of those novel skill requirements are actually requirements. You don't need to interpret a UI, or recognize intent to know that the car shouldn't turn the wrong way on a one-way street or hit a planter. Anticipating common errors and remaining vigilant while driving is a requirement for all driving, not when dealing with FSD. If you are capable of driving a car safely to begin with, then you're capable of recognizing then the car is going to do something wrong and turning the wheel and/or hitting the brakes. I wouldn't personally trust a FSD beta, but only if it is properly supervised, I think it could be tested safely. But, there's the problem -- how can you really properly supervise this? How many drivers are going to try this and just let the car drive w/o paying attention? Or, using it under controlled conditions, but then giving it too much trust in more adverse conditions?
    - ethbr0 5y ago
      What makes them required is the speed that a reaction is expected. In both the quick and velocity senses of the word. Unless the user is able to understand and anticipate common reactions, the entire system is unsafe at >25 mph or so. Or, to put it another way, I'd love to see the statistics on how many drivers can react quickly enough to avert an unexpected turn into an exit divider at 60 mph, after a number of days of interruption free FSD.
      - mbreese 5y ago
        We’re largely talking about an update where the primary benefit is low-speed automation — driving in areas where the speed limits are low because the obstacles are many.
  - tshaddox 5y ago
    No, it’s just unsafe (and presumably illegal in most places) to operate a motor vehicle without maintaining constant awareness of the road and control of the vehicle. There’s no additional training or license that permits you to stop maintaining that awareness and control, just like there’s no additional training that permits you to drive under the influence of alcohol.
    - throwaway0a5e 5y ago
      >There’s no additional training or license that permits you to stop maintaining that awareness and control Being a cop on duty often exempts you from distracted driving laws. I'm not sure how they can do stuff on a laptop while driving safely but I assume the state knows best. /s
    - kemitche 5y ago
      The theoretical "additional license" would be in addition to maintaining awareness, not a substitution. So all the normal road awareness, plus being informed of likely FSD failure conditions, and anticipating them / being ready and capable to intercede.
      - tshaddox 5y ago
        If you maintain control at all times, you shouldn’t need additional training for any of the car’s features. We don’t require additional training for cruise control, lane-keep assistance, anti-lock brakes, etc.
    - ectopod 5y ago
      I agree, but if you have to maintain constant control of the vehicle then it isn't self-driving.
- simion314 5y ago
  Maybe people that are trained with such failure videos that show what can go wrong and NOT with propaganda videos that only show the good parts, causing the driver not to pay attention.
- vkou 5y ago
  A license that allows you to operate a motor vehicle with a beta self-driving feature. It's very similar to a regular motor vehicle, but has different failure modes. A motorcycle is similar to an automobile, but has different failure modes, and needs a special license. A >5 tonne truck is very similar to an automobile, but has different failure modes, and needs a special license. An automobile that usually drives itself, but sometimes tries to crash itself has different failure modes from an automobile that does not try to drive itself.
- AlotOfReading 5y ago
  California at least has a specific license for this called the Autonomous Vehicle Operator license. It enforces some minimal amount of training beyond simply having a regular driver's license.
- bestouff 5y ago
  I guess any driving instructor worth its salt would have the required skills to correct the vehicle if it attempts to do something weird. After all FSD is a still-in-training (maybe for an unusual long time) driver.
  - Syonyk 5y ago
    Probably not. It fails in ways that are totally different from how humans typically fail at driving. Student drivers have the ability to comprehend and understand the 3D world around them - it's just a matter of learning the proper control manipulations to make the car do what you want, and then how to interact with other traffic in safe and expected ways. Tesla's FSD system seems to fail at basic 3D world comprehension and reading, often enough.
- visarga 5y ago
  Perhaps operating a motor vehicle is different from supervising an AI doing that same task.
- bigtex 5y ago
  Giant obxonious flashing lights and blinking signs stating "Student Self Driving tech on board"
  - kyleee 5y ago
    "Student Full Self Driver"
- bobsomers 5y ago
  Safety drivers at proper AV companies usually go through several weeks of rigorous classroom instruction and test track driving with a professional driving instructor, including lots of practice with the real AVs on closed courses with simulated system faults, takeover situations, etc. Anecdotally I've seen trained safety drivers take over from unexpected system failures at speeds near the floor of human reaction time. They are some of the best and most attentive drivers on the road. Average folks beta testing Tesla's software for them are woefully under-prepared to safely mitigate the risks these vehicles pose. In several cases they've meaninglessly died so that Tesla could learn a lesson everyone else saw coming months in advance.
- paxys 5y ago
  What part of the driving test covers taking over control from an automated vehicle with a split second notice?
avalys 5y ago
Because America does not yet require that its citizens ask mommy and daddy for permission for every thing they do.
- Barrin92 5y ago
  this isn't going to the ice cream store, this is driving two tons of steel with experimental software that glitches constantly through inner city traffic. If you look at the footage, at times these cars behave as erratically as a drunk driver, and they're controlled by what appears to be random 'beta testers' with no qualifications to operate a car under those conditions.
- mrRandomGuy 5y ago
  It would be a glorious honor to have my 'FSD' Tesla drive into a giant concrete pillar, causing my car to explode and erupt into flames.
  - kyleee 5y ago
    Basically Silicon Valley's version of a Viking burial
- ASalazarMX 5y ago
  Such crude trolling. Build a new house without a permit and see how that freedom feels.
  - SirZimzim 5y ago
    Yes, you have to get a driver's license and also register your car. If that isn't asking for permission then I don't know. Although it's a stretch to assume this permission includes self driving cars.
nrjames 5y ago
I used to work with the USG. There was "Beta" software absolutely everywhere, including on some very sensitive systems, because it was not required to go through security approval until it was out of Beta. In some instances, these applications had been in place for > 10 years. That was a number of years ago, so I hope the situation has changed. In general, the USG doesn't have sophisticated infrastructure and policy to deal with software that is in development. With Tesla, my guess is that it is not that they are allowing it to happen, but that they lack the regulatory infrastructure to prevent it from happening.
- verelo 5y ago
  USG? I'm not sure what that means, i did a google search and i assume it's not "United States Gypsum" or the University System or Georgia...?
  - darknavi 5y ago
    US government I assume.
    - verelo 5y ago
      Thanks, that makes sense.
whoknowswhat11 5y ago
It’s a think of the children on HN :)
optimiz3 5y ago
> puts everybody and their children at risk Oh won't somebody think of the children? https://en.m.wikipedia.org/wiki/Think_of_the_children https://en.m.wikipedia.org/wiki/Think_of_the_children
- mrRandomGuy 5y ago
  Out of all the times to mock someone saying that, I don't think "self-driving-but-not-really-car nearly crashes into things and puts the occupants and bystanders at risk for serious bodily harm as evident by the videos posted above" is one of those times.
  - optimiz3 5y ago
    Wait till you read the stats and see videos of what humans driving cars regularly do!
    - ASalazarMX 5y ago
      Still, an autopilot should aim to be a peer with the best drivers, not with mediocre or bad drivers.
      - throwaway0a5e 5y ago
        A peer with average drivers would probably be fine. A peer with average drivers and doing completely nonsensical things every now and then is basically an average driver on their cell phone though and the verdict on that is that it's not fine.
      - optimiz3 5y ago
        It's a limited beta with the goal of discovering error cases.
        barbazoo 5y ago
        In this case I'd prefer if "discovering error cases" was done without risking actual lives. Injuring someone would not be a valid way discover an error case.
SloopJon 5y ago
The Twitter thread is lacking in attributions, but I saw most of these some months back after a post about scary FSD behavior. I watched with morbid curiosity, and a rising level of anger. The guy with the white ball cap repeatedly attempted a left turn across two lanes of fast-moving traffic, with a drone providing overhead views. He seemed smart, aware, and even a bit conservative in intervening. Nonetheless, I couldn't help but thinking that none of the oncoming vehicles consented to the experiments racking up YouTube views. If he doesn't jump on the brakes at the right time, he potentially causes a head-on collision with a good chance of fatalities. Yes, I do agree that beta drivers should get extra training. However, I'm not sure I agree with the premise of beta testing an aluminum rocket on wheels on public roads in the first place.
- misiti3780 5y ago
  How do you know the beta drivers did not get training?
  - SloopJon 5y ago
    I said that they should. I don't know whether they do. A March 2021 tweet indicated that beta access will be as simple as pressing a "Download Beta" button in the car. A subsequent tweet said that the program had 2,000 owners, some of whom have had their access revoked for not paying enough attention to the road.
- aeternum 5y ago
  Do people sign up to be beta testers whenever a new 15 year-old with a permit gets behind the wheel? It's even crazier that we allow that. The cars that student drivers learn in typically do not have dual control so it's quite difficult to intervene. Every highschool has a story about how a student slammed the accelerator instead of the brake.
  - nexuist 5y ago
    People in this thread seem to have overly high expectations about what is required to drive on US public roads. Not even considering that each state has its own licensing and registration requirements, all states except one (Georgia) accept foreign driver's licenses, which means it doesn't even matter what you regulate if countries X, Y, and Z can just not care about it and refuse to enforce it. Regulating self driving is a good idea but it's a tall order to cast beta software as the menace when there are more idiots driving on the roads at this moment than there will ever be Tesla products.
TheParkShark 5y ago
Everyone and their children seems a bit hyperbolic. There hasn’t been an accident involving FSD beta 9, but I’m sure someone has been killed at the hands of a drunk driver today. I am failing to find any comments from you arguing for alcohol to be removed from shelves across the country? Why aren’t you pushing your regulators for that?
- studentrob 5y ago
  It's not hard to see the connection between FSD driving towards concrete pillars and the AP accidents on highways driving towards concrete barriers. None of this is hyperbole. FSD isn't a whole new system, it's based on AP. As Elon often says, they have been going for incremental improvements rather than the all-or-nothing approach taken by Waymo and others.
- esperent 5y ago
  Drunk driving is already illegal. Claiming that we should ban all alcohol because people use it to commit illegal acts makes about as much sense as claiming we should ban kitchen knives because people occasionally use them for stabbing. Besides that, FSD being less dangerous than drunk driving is a terrible benchmark.
renewiltord 5y ago
Because Tesla FSD accidents don’t kill people who were not party to the agreement. That seems intuitively fair to me.
- snet0 5y ago
  I'm sure you don't actually think this, if you think about it for a second longer. "What's wrong with drunk drivers? They're only harming themselves?"
  - renewiltord 5y ago
    Well, the problem with drunk driving is that it kills other people. We’ve witnessed it. We know it happens. Because we have seen it occur. Tesla FSD driving assist has never killed someone not in the car. Considering that rampant safetyism is par for the course here, I’m hardly convinced of the “he could’ve killed someone” school of thought.

croes 5y ago

I doubt that autonomous driving will work in the next couple of years without infrastructure adjustments. We built roads for cars to replace horses, we will need to built roads for autonmous cars to replace human driven cars.

dangoor 5y ago
I live in a place with fairly snowy winters, which seems like a huge problem for self-driving cars. I completely agree with this thought that we should work on making our roads help the self-driving cars. If we had started that effort several years back, we would likely have a lot of road miles that could safely handle self-driving vehicles. If we start now, in several years we'll have a bunch of road miles ready for them. Granted that the US has a lot of road-miles, but doing this work incrementally is the way to go. I have a feeling it will be faster to outfit the roads than to build reliable self-driving software. I will also grant that this is a "feeling" and I'm sure someone has done actual research :) If this capability could be used to help hasten the move to electric vehicles, we could even get a climate change win in at the same time.
- w0m 5y ago
  North East in winter has ~0% chance of getting FSD capable in the next 20 years I think. 80% of the population of my hometown can't leave the house for months at a time safely today; let alone automation taking over for that.
  - jazzyjackson 5y ago
    Great argument for more rail. Yes there are still conditions that freeze switches but that’s when you run gas lines under the tracks to prevent ice from forming: https://wgntv.com/news/when-its-this-cold-chicago-sets-its-train-tracks-on-fire/ https://wgntv.com/news/when-its-this-cold-chicago-sets-its-t... Weirdest thing that prevents trains from running tho is wet leaves on the tracks, of all things, it’s too slippery.
- jazzyjackson 5y ago

Syonyk 5y ago

I don't understand how something this broken is allowed to operate on public roads. If I drove like Tesla's FSD seems to based on the videos I've seen, I'd be pulled out of my car and arrested on (well founded) suspicions of "driving while hammered." After a decade of work, it's not capable of doing much beyond "blundering around a city mostly without hitting stuff, but pay attention, because it'll try to hit the most nonsensical thing it doesn't understand around the next corner." It drives in a weirdly non-human way - I've seen videos of it failing to navigate things I'm sure I could get my 6 year old to drive through competently. Except, I don't actually let her drive a car on the road. I'm out in a rural area, and while "staying in the lane" is perfectly functional (if there are lanes, which isn't at all the case everywhere on public roads), there's a lot of stuff I do on a regular basis that I've not seen any evidence of. If there's a cyclist on a two lane road, I typically get over well over the center line if there's no oncoming traffic to make room. If there is oncoming traffic, typically one person will slow down to allow the other car to get over, or the lanes just collectively "shift over" - the oncoming traffic edges the side of the road so I can get myself on or slightly over the centerline to leave room for the bike. And that's without considering things like trailers that are a foot or two into the road, passing a tractor with a sprayer (they don't have turn signals, so be careful where you try to pass), etc. If they've got any of this functionality, I'd love to see it, because I've not seen anything that shows it off. At this point, I think we can reasonably say that it's easier to land people on the moon than teach a car to drive.

toomuchtodo 5y ago
> I don't understand how something this broken is allowed to operate on public roads. Also out in a rural area. Running out to pickup lunch a few minutes ago, a young man flipped their old pickup truck on its side in an intersection, having hit the median for some reason. I too don't understand how humans are allowed to operate on public roads. Most of them are terrible at it. About 35k people a year die in motor vehicle incidents [1], and millions more are injured [2]. Total deaths while Tesla Autopilot was active is 7 [3]. I believe the argument is the software will improve to eventually be as good or better than humans, and I have a hard time not believing that, not because the software is good but because we are very bad in aggregate. [1] https://www.iihs.org/topics/fatality-statistics/detail/state-by-state https://www.iihs.org/topics/fatality-statistics/detail/state... [2] https://www.cdc.gov/winnablebattles/report/motor.html https://www.cdc.gov/winnablebattles/report/motor.html [3] https://www.tesladeaths.com/ https://www.tesladeaths.com/

zyang 5y ago

It appears Tesla FSD just ignores things it doesn't understand, which is really dangerous in a production vehicle.

mrRandomGuy 5y ago
Why are you getting down-voted? There's literal videos depicting what you state. Is the Musk Fanboy Brigade behind this?
- darknavi 5y ago
  You guys can downvote?!
  - jacobkranz 5y ago
    You can after you get a certain amount of upvotes (I can't remember the exact number though. 30? 50?)
  - reallydontask 5y ago
    I think you need 500 karma before you can downvote
    - Tempest1981 5y ago
      And only within 24 hours
      - darknavi 5y ago
        Wow! I had no idea. That's a pretty cool way to stop troll brigading. Thanks for the info!
heavyset_go 5y ago
This is what the automated Uber vehicle[1] that struck and killed a pedestrian did, as well. Despite picking her up via sensors and the ML model, it was programmed to ignore them. [1] https://www.nytimes.com/2018/03/19/technology/uber-driverless-fatality.html https://www.nytimes.com/2018/03/19/technology/uber-driverles...

simion314 5y ago

Do we know what is the process and who decides that an update is ready? This is a big decision so I am wondering what process or personality is needed to decide that all is safe, let's update things.

moojah 5y ago

This really isn't beta grade software, as it isn't feature complete as the failure scenarios in the video clearly show. I'd call it alpha grade, and it has been that for a while. It's not 2 weeks or whatever unrealistic timeline away from being done, as Elon has claimed for ever. 2 perhaps years if we're lucky, but given human and driving complexity probably way more before even the whole of the USA is reliably supported beyond L2.

H8crilA 5y ago
Like Donald Trump, but for nerds: http://elonmusk.today/ http://elonmusk.today/ FSD would be equivalent to the Mexican border wall, I guess?
- lostmsu 5y ago
  And the tax hike on cap gains.
w0m 5y ago
>This really isn't beta grade software, as it isn't feature complete as the failure scenarios in the video clearly show. I think it depends what they actually are trying to accomplish. This is Beta for a glorified cruise control overhaul; not a beta for promised RoboTaxi. Musk/Tesla tend to talk about RoboTaxi then slip seemlessly into/out of 'but today we have low engagement cruise control!'. Fair bit of hucksterism.
- barbazoo 5y ago
  > I think it depends what they actually are trying to accomplish Good point. "Full Self Driving" in my mind paints a picture beyond "a better cruise control". But maybe they meant that and just named it wrong.
  - dragontamer 5y ago
    From Tesla's webpage: > Full Self-Driving Capability > > All new Tesla cars have the hardware needed in the future for full self-driving in almost all circumstances. The system is designed to be able to conduct short and long distance trips with no action required by the person in the driver’s seat.

wedn3sday 5y ago

I watched a human driver in a BMW weave at 90 mph through traffic yesterday, swerving back and forth between lanes inches from other peoples cars. If Tesla can make a car that's even 1% less dangerous than human drivers, that could be thousands of deaths avoided every year. Humans should not be allowed to drive cars on public streets.

carlmr 5y ago
But that surely is the description of 1% of the drivers. It should be better than the 1% worst drivers. It should be demonstrably safer than humans on average with a good margin.
- minhazm 5y ago
  Even if they accomplish that, people will move the goal posts. People are illogical and will see some edge case that self driving might perform worse at, but on an aggregate data basis it might be 10x safer than human drivers. People will see that 1 edge case and say that self driving cars are worse than humans.
  - mikestew 5y ago
    That's an awful lot of presumption of "people"'s behavior toward a system that has failed nearly every promise made. There's no need to move the goal posts when the player isn't anywhere near it. I mean, if we're going to talk about someone moving goal posts around, how about the one claiming "full self-driving" for a system that isn't anything of the sort?
Syonyk 5y ago
> Humans should not be allowed to drive cars on public streets. Because the alternative right now is... this abomination that doesn't even grasp what a concrete pillar is?
- jacquesm 5y ago
  Maybe it does and it is trying to put itself out of its misery?

stsmwg 5y ago

One of the questions I've repeatedly had regarding FSD (and Tesla's approach in particular) is the notion of memory. While a lot of these scenarios are disturbing, I've seen people wavering on lanes, exits and attempting to turn the wrong way onto one-way streets. People have memory, however. If we go through the same confusing intersection a few times, we'll learn how to deal with that specific intersection. It seems like a connected group of FSD cars could perform that learning even faster since it could report that interaction with any car rather than driver-by-driver. Are any of the FSD implementations taking this into account?

H8crilA 5y ago
That's kind of the "selling point" of running this experiment on non-consenting public, that it will learn over time and something working will come out of that in the end.
Syonyk 5y ago
This has been a common assertion about Tesla's "leadership" in the field - that they can learn from all the cars, push updates, and obviously not have to experience the same issue repeatedly. It's far from clear, in practice, if they're actually doing this. If they have, it would have to be fairly recent, because the list of "Oh, yeah, Autopilot always screws up at this highway split..." is more or less endless. GM's Supercruise relies on fairly solid maps of the areas of operation (mostly limited access highways), so it has an understanding of "what should be there" it can work off and it seems to handle the mapped areas competently. But the problem here is that the learning requires humans taking over, and telling the automation, "No, you're wrong." And then being able to distill that into something useful for other cars - because the human who took over may not have really done the correct thing, just the "Oh FFS, this car is being stupid, no, THAT lane!" thing. And FSD doesn't get that kind of feedback anyway. It's only with a human in the loop that you can learn from how humans handle stuff.
- stsmwg 5y ago
  Great, thanks for that info. I'm remembering the fatal crash of a Tesla on 101 where the family said the guy driving had complained about the site of the accident before. It's interesting to know that there's at least a mental list of places like this even now. Disengagements should at least prompt a review of that interaction to try and understand why the human didn't like the driving. Though at Tesla's scale that has already become something that has to be automated itself.

killjoywashere 5y ago

I work in ML (medical diagnostics) and own a 2020 Tesla w/ FSD and do not trust it at all. I also happen to be a cyclist, currently living in SV. My regular road ride takes me past Tesla HQ. The more of these FSD fails I see, the more I wonder how many Teslas passing me are running in FSD mode. Scary.

deregulateMed 5y ago

Originally when Tesla was on the verge of collapse, the self driving technology waa high risk, high reward tech that will kill someone but the company was in the negative. If that killed Enough people, declare bankruptcy and investors lose money. However today, isn't Tesla profitable now? They already have created a Veblen good, what benefit does Tesla get out of a high risk feature?

deleted 5y ago

[deleted]

bsagdiyev 5y ago

Why are there so many comments from seemingly different posters all saying the same thing on this, "I don't understand why humans are allowed to drive cars"? It feels kinda... culty? Or too similar to be a coincidence. It honestly probably is but these Tesla posts always bring out those types and it confuses me. Humans kill humans, do we want machines to start doing it on the road automatically now too?

dekhn 5y ago
I think the idea is, if humans were held to the standards machines are, we wouldn't let them drive. If you offered me a car that drove itself, and statistically, it killed people at the same rate as humans, but let me not have to drive, I'd take that. Nobody everybody agrees.
- w0m 5y ago
  Self determination > logic. Look at the American Gun debate as an example; carrying for self defense makes you drastically less safe by most metrics. But people prefer to have the modicum of self determinism over more consistent statistical safety.
  - ggreer 5y ago
    > carrying for self defense makes you drastically less safe by most metrics You're probably referencing studies such as Investigating the Link Between Gun Possession and Gun Assault[1], which looked at whether people who were shot were carrying a gun at the time. It did not differentiate between criminals who were carrying illegally and law-abiding citizens who had concealed carry permits. The selection effect in such studies is clear: Many criminals who carry guns are involved in shootings. Also, many people carry guns because they are at a higher risk of being a victim of violent crime. I have yet to find a study that corrects for these confounders. 1. https://sci-hub.st/10.2105/AJPH.2008.143099 https://sci-hub.st/10.2105/AJPH.2008.143099
- notahacker 5y ago
  > I think the idea is, if humans were held to the standards machines are, we wouldn't let them drive. But the reverse is true. Humans receive driving bans and even criminal penalties for the sort of driving errors autonomous systems make without penalty.

yssrn 5y ago

Scary set of videos. Is Tesla using Marylanders to train FSD?

creato 5y ago

Some of the issues shown in these videos make me wonder about Tesla's strategy of using non-professional driver (customer) data to train FSD. Things like changing lanes at the last second is a thing that (obnoxious) humans do, and would be a bad example to learn from. There might be a lot of subtle garbage in Tesla's dataset.

rcMgD2BwE72F 5y ago
Tesla can easily filter these cases out, automatically. They have triggers to catch things they want (e.g sudden lane changes) and campaigns have conditions too (only from prudent drivers, if lane change is forced, etc). FSD does not learn from all drivers all the time.
- rcMgD2BwE72F 5y ago
  Sources: https://youtu.be/hx7BXih7zx8?t=639 https://youtu.be/hx7BXih7zx8?t=639 and https://youtu.be/gZ2SeiLjaEc?t=941 https://youtu.be/gZ2SeiLjaEc?t=941

Geee 5y ago

It seems to estimate the surroundings pretty well in the new version, but the path planner still has issues. I'm not sure if they already do this, but I think Tesla should run FSD on all cars in shadow mode, and then use the FSD vector data and driver actions to train the path planner. Basically using FSD vector data as input and driver actions as output of the neural net. If the network can't reach confidence in certain situations, then information is missing from the FSD vector data and more raw video data is needed. It would be possible to measure the difference between human drivers vs. FSD and automatically send data from errors to Tesla for training.

Syonyk 5y ago
I've been hearing for years now that this is a Tesla advantage, that they can run in shadow mode, learn from humans, and big data learn all the quirks of the roads so they don't have to learn them in the self driving system. If this has been the case, evidence here is certainly lacking. Turning down a street one presumably hasn't seen a car drive down in this direction? Failing basic stationary obstacle avoidance because they're in the center of the road? Screwing up basic turn lane behavior? Either they aren't taking the "years of learning" data into account, or, worse, they are - and even with all that correction, it's still this bad.
- Geee 5y ago
  I think they've been focused on solving the "environment mapping and object detection" problem, and if that isn't solved then they can't have the proper inputs to solve the "path planning / decision making" problem. It seems that they're close to solving the first problem, and are ready to move on to the next. As far as I know they've been using the fleet to collect raw camera data for their training set but I don't think they've used the fleet to learn driving behaviour.
- visarga 5y ago
  It's a cherry picked set of examples you're drawing generalisations from. On any day I can see more reckless driving from humans than I've ever seen online with Tesla.

dreyfan 5y ago

Is "The car doesn't recognize a truck crossing in front of it so it drives under and decapitates the driver" still an open issue? What about the multiple instances where FSD drives the car forcibly into a firetruck?

cs702 5y ago

Many of the comments here seem a bit... unfair to me, considering that these clips were handpicked. I watched (and fast-forwarded) through a few of the original, full-length videos from which these clips were taken. The full-length videos show long drives (in some cases, hours long) almost entirely without human intervention, under conditions explicitly meant to be difficult for self-driving vehicles. One thing I really liked seeing in the full-length videos is that FSD 9 is much better than previous efforts at requesting human intervention in advance, with plenty of time to react, when the software is confused by upcoming road situations. The handpicked clips are exceptions. For BETA software, FSD 9 is doing remarkably well, in my view. I mean, it's clearly NOT yet ready for wide release, but it's much closer than all previous versions of Tesla FSD I've seen before, and acceptable for a closed-to-the-public Beta program.

richwater 5y ago
It literally doesn't matter how well it does in 90% of situations when the other 10% can injure or kill people in relatively basic scenarios like the Tweets presented. I mean the car almost ran into a concrete pillar like it wasn't even there. > For BETA software, FSD 9 is doing remarkably well If this was a React website, that'd be great. But it's a production $40,000 multi ton automobile.
- bob33212 5y ago
  People will die today because a driver was drunk/distracted/suicidal/rode raged or had a medical problem. Are you OK with that or do you think we should attempt to fix that with software? If you do think you should attempt to fix that, do you understand that software engineering is a iterative process? It gets safer overtime.
  - foepys 5y ago
    > People will die today because a driver was drunk/distracted/suicidal/rode raged or had a medical problem. That's a straw man argument. Mandatory breathalyzer tests before every engine start and yearly medical and psychological check-ups would solve a lot of these problems as well. > It gets safer overtime. There is no evidence of this happening yet. It's still trying to drive people into concrete pillars. Remember that you only need to drive into a pillar once to die. Passing it 10,000 times before doesn't help when an update suddenly decides that the pillar is now a lane. I'd be totally fine with Tesla testing this with trained drivers in suitable areas but giving this into the hands of amateurs is pure "move fast and break things", with "break" meaning ending lives. The worst thing is: all of these videos feature sunny weather. How bad would it be in the rain? Ever tried to take picture in the dark with even light rain? Cameras are very bad at this.

mritun 5y ago

Question: Would it be awesome if FDA allowed Pharma to conduct drug-tests like this? Put a "beta-2.9" on the vial and let people try it out... Some may die, be disabled or may linger in vegetative state life-long, but it was their choice afterall and the it can argued that medicinal side-effects are very small cause of mortality and hence long-term the "public beta tests" will make drugs more effective and save more lives!

CreepGin 5y ago
Depends on how much the drug costs. If it costs 100k a pop, then I don't see the general public being affected by it too much. RIP those brave rich souls.
- mritun 5y ago
  General public is affected. A Tesla can cause accident on a public road with other drivers on the road, or cause a cascade or a pile-on - innocent bystanders can die or be injured.
colinmhayes 5y ago
I think the difference is releasing self driving provides the data Tesla needs to improve it. Releasing a half baked drug doesn't help the pharma company improve it. If you're asking whether that would be awesome if it lead to pharmaceutical innovations I think it would.
- flutas 5y ago
  > I think the difference is releasing self driving provides the data Tesla needs to improve it. Releasing a half baked drug doesn't help the pharma company improve it. The pharma company could see results (self-driving car data) and figure out what caused the issues (details of the deaths in pharma, accidents in Tesla) and use that to make the next beta version. I don't really see how that isn't an apt analogy. It's called clinical trials for pharma the key point being it's opt in and doesn't affect anyone around the subject, unlike Tesla's autopilot beta.
- 5y ago

jacquesm 5y ago

Some regulator somewhere will take this junk off the road with the stroke of a pen and I'll feel that much safer on the road when it happens. And we'll have Elon Musk to thank for electric cars and the self driving winter. It's actually pretty simple: have FSD do a regular driving test. If it can pass that it's good to go, if not it fails the test and will not be allowed to control a vehicle.

j7ake 5y ago

These spectacular fails weaken the rationalist's arguments that "as long as FSD achieves lower deaths per km driven (or any other metric) than humans" then FSD should be accepted in favor of human driving. Even if "on aggregate" FSD performs safer (by some metric) than humans, as long as FSD continues to fail in a way that would have been easily preventable had a human been at the wheel, FSD will not be accepted into society.

manmal 5y ago
I think this is already true though for highway driving. Highways are long and tiring, and the surroundings model is easy to get right, so computers have an advantage. Most manufacturers offer a usable cruise control which is safe and can probably be active 90% of the time spent on highways. I often switch it on in my 3yo Hyundai as an extra safety measure in case the car in front of me unexpectedly brakes while I‘m not looking there. Add to that a lane keeping assistant and lane change assistant, and you don‘t need to do much. Except for when the radar doesn’t see an obstacle in front of you, eg because the car in front of you just changed lanes. That needs to be looked out for.
- deleted 5y ago
  [deleted]
cptskippy 5y ago
> spectacular There was nothing spectacular about those failures, I would say because the driver was attentive and caught/corrected the car. That's not to say some of these fails could not have ended in catastrophe, but to call them spectacular is quite the exaggeration. One of those "spectacular fails" was displaying two stop signs in the UI on top of each other while properly treating it as one stop. Using hyperbole like this only makes people ignore or dismiss your otherwise valid point.
cogman10 5y ago
Yeah, I've brought this point up in other locations. It does not matter that any autonomous driving tech is safer than human drivers. They MUST be perfect for the general public to accept them. The only accidents they'd be allowed to get into are ones that are beyond their control. Algorithmic accidents, no matter how rare they are, won't be tolerated by the general public. Nobody will accept a self driving car running over a cat or rear ending a bus even if regular humans do that all day long. The expectation for self driving cars is a perfectly attentive driver making correct decisions. Because, that's what you theoretically have. The computer's mind doesn't "wander" and it can't be distracted. There's no excuse for it to drive worse than the best human driver.

manmal 5y ago

Is FSD still operating on a frame-by-frame basis? I remember it was discussed on Autonomy day that the ideal implementation would operate on video feeds, and not just the last frame, to improve accuracy. When you look at the dashboard visualizations of the cars‘ surroundings, the model that is built up looks quirky and inconsistent. Other cars flicker into view for one frame and disappear again; lane markings come and go. I saw a video where a car in front of the Tesla indicated, and the traffic light in the visualization (wrongly) started switching from red to green and back, in sync with the indicator blinking. How could a car behave correctly as long as its surroundings model is so flawed? As long as the dashboard viz isn’t a perfect mirror of what’s outside, this simply cannot work.

cptskippy 5y ago
I would think the flickering objects in the UI is a result of objects hovering around the confidence threshold of the model. But... I have a Model 3 and the flickering happens when stationary and nothing around you is moving.
joakleaf 5y ago
There also seem to be general problems with objects disappearing when they are obscured by other objects, and then reappear later, when no longer obscured. It is ridiculous, that the model doesn’t keep track of objects, and assume they continue with current velocity when they become obscured. It seems like a relatively simple thing to add depending on how they represent objects. You could even determine when the objects are obscured by other objects. In 8.x videos I noticed cars shifting and rotating a lot over fractions of a second, so it seemed like they needed a Kalman filter for objects and roads. Objects in 9.0 look more stable, but I still see lanes, curbs, and entire intersections shifting noticeably from frame to frame. So if they added time (multiple frames) to the model, it is still not working that well.
EForEndeavour 5y ago
You've nicely articulated what was bothering me about the jittery dashboard visualizations. Why on earth is everything flickering in and out of existence, and why is the car's own planned trajectory also flickering with discontinuities?? It seems like they aren't modeling the dimension of time, thus throwing away crucial information about speed and needlessly re-fitting the model to static snapshots of dynamic scenes. It's like the ML system needs its inferences constrained by rules like "objects don't teleport" or "acceleration is never infinite."

DrBenCarson 5y ago

Their current sensor and camera lineup have made this impossible on models already on the road. Good luck to their customers

jdofaz 5y ago

I love my Tesla but watching these videos made it an easy decision not to spend $10k on the FSD option.

okareaman 5y ago

I had a girlfriend that I didn't trust driving my car, especially with me in it. That's how I feel about Elon Musk driving my car.

yawaworht1978 5y ago

Any human driver driving like that would have his driver's licence revoked, might risk prison for endangerment of society and perhaps be subject to a psychological evaluation (not joking, this stuff happens in Germany). Why are there no consequences for neither Tesla nor the lawmakers who are looking away? Sure, test autonomous driving off the roads or on dedicated test environments, but do not permit this on public roads, i do not wanna be part of any beta test.

gumby 5y ago

One case that's perhaps not a bug: > 4th: Tesla doesn't recognize a one-way street and the one-way sign in the street, and it drives towards the wrong way If the car is in Boston this is OK.

selimnairb 5y ago

“Full Self-Driving Beta” should be an oxymoron. Why is this being allowed?

foobarbazetc 5y ago

This thing is like 10 years away from being actually usable, if it ever gets there.

cromwellian 5y ago

All of the Tesla fanboys saying you don't need LIDAR, RADAR, or Maps as a kind of "ground truth" and you can get by on video only should take this as a huge repudiation of brag by Musk. Look, Radar, Lidar, and Maps aren't perfect. A map can be wrong, Radar can have false echos, Lidar can have issues in bad weather, etc. But if you're depending on visual recognition of one-way signs, having a potentially out-of-date backup map to check your prediction again, and err on the side of caution, is better than having no Map at all. Ditto for radar and lidar. Radar enabled cars have collided with semis, remember, because they're aimed low (underneath the semi). But in the case of these planters, or road cones, or monorail poles, the radar is going to be a strong signal to the rest of the system that it isn't seeing something. There's no way FSD should be allowed on public roads, it's not even CLOSE to ready. The other car companies are behaving far more responsibility: full sensor suite, multi-sensor fusion, comparison with detailed road maps, and limiting operation to safe areas, and expanding as safety is proven. Who the hell cares if it's not FSD driving if its so dangerous? I'll take 75% FSD that only operates on highways and long commutes or road trips, over 100% FSD that has so many disengagements and "woah" moments as to be unusable.

TheParkShark 5y ago
I see this argument a lot. This is a small beta test and I have yet to see anything better on a larger scale. Most drivers shouldn’t be allowed on public roads yet here we are. Why are you so full of fear and anger over this? Isn’t this a place for open discussion?
- nullc 5y ago
  > I have yet to see anything better on a larger scale Waymo self-driving is a relatively large scale. It also has blunders but they appear to be far less concerning than Tesla's.
  - TheParkShark 5y ago
    Geo-fenced isn’t large scale. There’s no clear route to allow them to expand that tech.

nullc 5y ago

These aren't even the worst examples I've seen by far-- but I suppose they're the worst people have posted with the latest software. As someone in another thread said: Driving with Tesla FSD is like being a drive instructor for a narcoleptic student driver who is currently tripping balls. I don't want to share a road with these things. I try to stay as far away from every tesla on the road as I can.

TheParkShark 5y ago
Please share them if you have. As someone in another thread said: Tesla has the most advanced self driving software currently in development in the world. If there’s something better you can feel free to share that with your other videos.
- ahahahahah 5y ago
  > Tesla has the most advanced self driving software currently in development in the world This is just nonsense. These videos show them at a disengagement every couple miles. Waymo was at that level 10 years ago and is 10000s of times better now.
  - TheParkShark 5y ago
    Waymo only works in a 50 sq mi area [1] and has it’s own issues [2]. So no, this is not nonsense. [1] https://arstechnica.com/cars/2021/05/why-hasnt-waymo-expanded-its-driverless-service-heres-my-theory/ https://arstechnica.com/cars/2021/05/why-hasnt-waymo-expande... [2] https://www.bbc.com/news/technology-57144755 https://www.bbc.com/news/technology-57144755
    - nullc 5y ago
      And Tesla FSD is available to what... two dozen parties outside of the company? AFAIK both have extremely limited availability. Waymo doesn't require frequent human intervention to prevent life threatening situations.

deegles 5y ago

All the other FSD car companies should sue to get these off the roads since a big enough fuck-up by Tesla will set back the industry for a decade.

sharkmerry 5y ago

Can someone explain what is happening in the first video? are the planters on the left after the turn and it was trying too tight of a turn?

reader_mode 5y ago

Every time I see these kinds of videos and read about Tesla valuation and messaging I can't make up my mind what to think about it. On one hand I find it ridiculous that these kinds of results so far into the development are still making Tesla the most valuable car maker, selling this as FSD with no proof they will ever reach it etc. On the other hand they secured billions of $ for R&D into FSD and BEV design and brought EV production to scale. Their messaging is way to optimistic (borderline fraudulent) but they are doing better things with the money than the alternatives. Traditional automakers have shown to be completely impotent in this regard. Money is flowing into all kinds of wasteful bullshit (that Tesla is backing BTC makes them less trustworthy in my eyes)

BrissyCoder 5y ago
Highly recommend listening to the 3 episode miniseries TrueAnon did on Musk and Tesla. Really eye opening. The dude should be in prison.

MobileVet 5y ago

For a company that exists because of technological and economy of scale gains, primarily in battery tech, it was always odd to me the way Elon would say LIDAR was too expensive or not fast enough. LIDAR et al have come a long way in the last 5 years and will continue to evolve at an exponential rate due to the efforts being put into them. Yes, I get it, humans use 2 eyes only, but sensor fusion of complementary technologies is so powerful. When lives are on the line, having more than one type of sensor involved is just prudent engineering.

GhostVII 5y ago

I don't know why Tesla is so focused on self driving in cities. Seems like a far better use of time would be to get to level 3 self driving on highways (which is what other automakers seem to be trying to do). That gets you most of the benefits of self driving (not having to pay attention for long trips), with probably 1% of the complexity of city driving. Getting city driving working just seems so impossibly hard, there is no way you are going to be able to use ML to determine road rules since they vary by jurisdiction and over time, and there are so many edge cases you won't have enough data for. But how do you write a program that can read a street sign, interpret it's meaning, and act upon it safely? Even if Tesla just got autopilot to the point where it could be level 3, where drivers don't have to pay attention, that would be a massive improvement, and so much easier than FSD. Maybe they painted themselves into a corner though with all Elon's blatant lies about FSD, can't stop development without a lawsuit. They seem kind of screwed anyways, looks like they are running into the limits of the current camera positioning and onboard processing power, might need to do another iteration of replacing every FSD owners hardware.

villgax 5y ago

This is what is needed to pit against human drivers & other FSD software from manufacturers.

MarkMc 5y ago

Here's an idea, although I'm not sure if it would work: The government allows Tesla to conduct a randomised control trial: When you get in a Tesla you enter your destination, then there's a 50% chance of having to drive the car manually (the 'control') and 50% chance of having the car drive you (the 'treatment'). After 10 million trips, if the treatment group had fewer accidents than the control group, then Tesla is allowed to roll out self driving to everyone. But on the other hand, if treatment has more accidents than the control group then Tesla must pay, say, $10 million for each excess fatality.