21 ms·

After 6 months of working fine, Tesla software update drives at barriers again

339 points by SamuelAdams 8y ago

x38iq84n 8y ago
I have always wondered... Does the AP have some higher-level notion of object permanence, continuity (road behind a horizon or after a curve) and things like that? Does it track a pedestrian that is momentarily hidden behind an obstacle and will probably reemerge in a second or two on the other side? Does it expect that kids may run after that ball that just flew from behind a car? Does it continuously track and improve classification of all objects in the field of vision, with their trajectories and speeds if they are moving? Personally I don't think it does, otherwise it would not erratically slam into clearly visible and marked large objects in its way, or it would be aware of a truck moving in a perpendicular way and so on. I am of opinion that without such higher-level awareness it can never succeed, hope to learn about the state of the self-driving art.
- dragontamer 8y ago
  I don't think neural networks are wired to "remember" things. In theory, they could be hooked up that way. But your typical convolutional neural network is looking at things frame-by-frame. In theory, ANNs could have an output layer that passes data from one frame to another frame to assist things. But there's no real programming to "hardcode" something like object permanence into an ANN. You pretty much throw a bunch of data into the system and hope for the best.
  - hemogloben 8y ago
    NNs are just the first step in the pipeline. Their outputs (detected objects, segmentation, etc) will be piped into other software that builds higher level models. Considering the path-planning requirements I would be absolutely shocked if Autopilot wasn't build history models and estimated paths for objects around the vehicle (other cars etc).
    - JanSolo 8y ago
      Agreed; I imagine they use neural networks to detect and classify objects which are then saved into a scene-graph for use in pathing. I expect what happened was that they trained their NNs for improved detection in one area but unknowingly reduced it in another. Perhaps now it can detect tricycles 99% but road barriers went down to only 30%. Having worked with NNs it's very common to see gains in one domain which come at a cost of reduced performance in another.
      - stevenjohns 8y ago
        They must have known. I haven’t worked with NNs for a few years but I don’t believe the methodology has changed where you would stop testing it over different sets of training data. Barriers are a pretty big part of driving on roads and highways and the only reason it would have been unknowingly reduced would be if they just weren’t testing the NN against data with them.
  - robrenaud 8y ago
    There are architectures that use CNN on image inputs, and LSTMs across frames of the video to keep memory. https://arxiv.org/pdf/1609.06377.pdf https://arxiv.org/pdf/1609.06377.pdf
    - x38iq84n 8y ago
      Thanks (to all in this subthread). I've skimmed through this doc, I have watched https://vimeo.com/274274744 https://vimeo.com/274274744 linked from original reddit discussion and I have lost all remaining faith in AP as it is today. This should be in a closed alpha version, not anywhere near paying customers and marketed as FSD ready/feature ready as it's none of that and won't be for many years. I expect there will be at least a generation of Tesla cars sold with FSD-readiness package that will never see FSD in their lifetime.
- deleted 8y ago
  [deleted]
- eclipxe 8y ago
  https://www.youtube.com/watch?v=rACZACXgreQ https://www.youtube.com/watch?v=rACZACXgreQ
- automathematics 8y ago
  In these videos, "Autopilot" is mentioned as the culprit, which seems to be a subset of the features Tesla has. It seems to me there are 3 layers: (level 0: Just Adaptive Cruise Control (use radar to adjust speed up to a max). Human still steers. - 1: Adaptive Cruise Control (use radar to adjust speed up to a max) + Autosteer (camera's watch the lane markers and follow them). This is referred to as "Autopilot" NOTE: This is where the accidents happen. The car isn't driving towards a barrier, it's following the lane markers and hits an error state. This is also NOT self driving. - 2: "Nav on Autopilot". This is an additional function you turn on where the car has more intelligent (use this word loosely) capabilities on highways. The car will still do everything on level 1 combined with lane changes (using cameras to detect objects and trajectories differentiating cars from trucks from pedestrians from bikes from motorcycles etc). It will still follow lane lines, but with a lot of additional information (is there an object? am I merging? am I exiting? etc) - 3: "Full Self Driving". This is an additional package that doesn't exist to the public. Internally I'm sure they're testing the functionality but this is using all of the sensors and algorithms and likely neural networks to decide what to do. A cool point though is that all Tesla's are likely running this code in "shadow mode" where data can be collected and assumptions can be tested without endangering any actual drivers. (see here for some cool data on this: https://electrek.co/2019/03/05/tesla-autopilot-detects-stop-lines-intersections/ https://electrek.co/2019/03/05/tesla-autopilot-detects-stop-...). "Hey, I think the car, if full self driving SHOULD take action X. *compare to what the driver actually does and log the data" over BILLIONS of miles So when a Tesla hits the barrier or gets in an accident, we're actually running "#1" and people start freaking out. But when we get to the capabilities of #3 a lot of the "object permanence and continuity" stuff starts to come into play. Full Disclosure: - I drive a Model 3 every day on Autopilot 75% of the time - I only 75% think self driving is capability under the current Tesla software suite but I bought the package anyways.
SamuelAdams 8y ago
Additional context: on March 23, 2018, a Tesla Model X, while using autopilot, drove into a barrier and killed the driver. This was fixed in a software update, but the issue seems to have resurfaced. NTSB analysis of the March 23 2018 collision: https://www.ntsb.gov/investigations/AccidentReports/Pages/HWY18FH011-preliminary.aspx https://www.ntsb.gov/investigations/AccidentReports/Pages/HW... Tesla's statement of the March 23 2018 collision: https://www.tesla.com/blog/update-last-week%E2%80%99s-accident https://www.tesla.com/blog/update-last-week%E2%80%99s-accide...
- MagicPropmaker 8y ago
  Well, it wasn't really "fixed" -- the driver is still dead.
  - craftyguy 8y ago
    Sure, if you construe the meaning of 'fixed' to mean 'roll back all events and time', which I don't think anyone expects. They probably fixed the software bug.
    - falcolas 8y ago
      I think that it's important to note that "we'll fix bugs via live patches" is a deadly decision when it comes to self driving software. Just discussing the incident in terms of a "software bug" does a disservice to the severity of the issue.
      - craftyguy 8y ago
        No, it does not. While I agree that testing patches on users with no testing beforehand is horrible, 1) I doubt Tesla does this, and 2) the person died before they fixed it, which GP seemed to be confused about.
      - gwbas1c 8y ago
        Remember, autopilot and autosteer are not self-driving. Tesla is very explicit that the driver must remain alert, and supervise, at all times. That being said, drowsy driving is a thing, and it's very easy to fall asleep behind the wheel. The car really needs a better strategy to handle this situation.
sschueller 8y ago
Eventually we need the NTSB to certify updates before they are pushed out over the air. Similar to what the FAA does. These OTA updates are not ok for large machinery and endagers not only the Tesla driver but all others on the road.
- x38iq84n 8y ago
  Indeed. I have never understood the cheering for OTA of something that can literally kill you. Were they limited to infotainment and the useless easter eggs I would not care, however AP OTA is downright scary, especially pushed by vendor instead of being initiated by an informed user.
  - 0815test 8y ago
    The AP can only kill you if you're distracted/asleep at the wheel and not paying attention. You're always in charge of driving the car appropriately, the "AP" can only provide tentative assistance of a very rough sort. It will try to guess at the control inputs you might want to provide (which does make things a bit more comfortable in the best case) but the actual control always rests with the human driver.
    - saagarjha 8y ago
      The issue is that the way the feature is designed and marketed makes it very easy for the driver to be distracted and not paying attention.
    - craftyguy 8y ago
      It shouldn't be marketed as 'Autopilot' then. Less informed people will expect it to do more than it can. It's (IMHO, criminally) misleading to market something as being capable of doing something it cannot do.
    - lisper 8y ago
      > The AP can only kill you if you're distracted/asleep at the wheel and not paying attention. That's not true. There are many situations where the AP could make a sudden turn and kill you long before you had time to react even if you were paying attention. And in fact, the situation in the video seem to be not so far away from that.
FreedomToCreate 8y ago
Tesla can only really gather depth data from there radars, while the cameras operate a DNN to detect features and make adjustments for steering and speed based on fusing those two pieces of data together. An error in the radar or the DNN detecting features incorrectly because of lighting, road color change or object on the road which the model was not trained on, can cause problems like this.
- notfromhere 8y ago
  if it came back after an update, wouldn't it be the issue of the algorithm training itself incorrectly?
- zmarty 8y ago
  "Tesla can only really gather depth data from there radars" - they can also do it visually through depth from motion
SilasX 8y ago
I would be a lot more forgiving of these screwups if Tesla didn't constantly swear up and down that they've solved self-driving cars. As far back as 2016 they were claiming they had full SDC capability above human driver safety [1], and their recent Model Y announcement suggests that the only thing holding it up is regulatory approval, and not failure to achieve the desired spec. >Model Y will have Full Self-Driving capability, enabling automatic driving on city streets and highways pending regulatory approval, as well as the ability to come find you anywhere in a parking lot. [1] https://www.tesla.com/blog/all-tesla-cars-being-produced-now-have-full-self-driving-hardware https://www.tesla.com/blog/all-tesla-cars-being-produced-now... [2] https://news.ycombinator.com/item?id=19397942 https://news.ycombinator.com/item?id=19397942 (linking HN because it's hard to find the text in the page with their UI)
- Scoundreller 8y ago
  > As far back as 2016 they were claiming they had full SDC capability above human driver safety One crash/incident doesn’t mean it’s less safe than a human driver on average, even if it’s something a human driver might have avoided. We should expect different failure modes from a machine, but adopt it anyway if it avoids enough human failure modes to make up for it. Source: my thermostat thought 33C was an acceptable temp once, but I still didn’t switch to full-manual HVAC control.
  - tptacek 8y ago
    Don't Teslas fare significantly worse than cars in similar class in terms of driver fatalities?
    - lordalch 8y ago
      I don't have the fatalities stats, but their safety ratings have been perfect. [0] https://www.cnbc.com/2018/09/20/tesla-model-3-earns-perfect-5-star-nhtsa-safety-rating.html https://www.cnbc.com/2018/09/20/tesla-model-3-earns-perfect-...
    - Scoundreller 8y ago
sabareesh 8y ago
Since ModelX incident I have made sure not to drive on first lane. I always drive on the 2nd lane and it is much better.
- bdcravens 8y ago
  For a $700 car I might consider crazy compromises, but $70k+? Absolutely not.
  - warp_factor 8y ago
    That's what I find so interesting with Tesla owner. They spend a fortune on a car, then they minimize every single issue they have with the car. My explanation for this is that for a lot of owners, the car is a way to be part of a hyped group more than being a utilitarian object (what a car should be).
    - gojomo 8y ago
      https://en.wikipedia.org/wiki/Choice-supportive_bias https://en.wikipedia.org/wiki/Choice-supportive_bias
    - mikestew 8y ago
      Happens with a lot of stuff, certain American-made motorcycles, for instance. You're in the club now, and the only way to stay in the club is to carry the manufacturer's water. In no case are we to admit that we spent a bunch of money on something that does a poor job of fulfilling its advertised purpose.
      - bdcravens 8y ago
        Or more relevant to the HN audience, Apple.
        mikestew 8y ago
        What, Tesla wasn't relevant enough for you to not take an "obligatory" dig at Apple product owners?
anth_anm 8y ago
Said it before, saying it again. OTA updates to safety critical things is not a feature. It's a bug.
bsaul 8y ago
I wonder how do unit tests work with NN (or if they're even a relevant concept at all). You could replay some testing video frames and make sure the objects are correctly identified, but i suppose that's already what training is about... If an issue like that resurface, does it mean that the original frames leading to the 2018 accident aren't part of the training (or at least frames from someone driving in this kind of scenario) ?
- inertiatic 8y ago
  Well, testing in the strict sense would be measuring accuracy on data not in the training set, and even accuracy on the training set data (which isn't guaranteed to be 100% I believe)
- bluGill 8y ago
  The frames from the 2018 should not be part of the training. They should be part of the test set to prove whatever training they do works.
- neilalexander 8y ago
  Yes, the problem is indeed that there's no real way to "look into" a neural network and understand how it has been trained. All you can do is observe that the given inputs generate the desired outputs. Even if there was training based on the 2018 frames, that doesn't mean that you have verifiably fixed the problem. It's difficult to train a neural network selectively – every time you "train" the network with additional data, you are increasing the chance that you are also teaching it something you didn't intend which then can have a side-effect in some seemingly unrelated scenario. You can see this in real life with image recognition networks. Teach them too much and they gradually become less effective at identifying anything.
- jononor 8y ago
  Unit tests are not sufficient for neural networks. Say the network takes 2D images of typical 224x224 pixels with 3 channels (RGB) of 8 bit values, this input space has 256^(224x244x3) = 3.5×10^362507 possibilities. Billions of years to test them all. This is before we consider stereo vision, 3D images and state over time. How to know which subset of these inputs are necessary to give reasonable coverage? Right now I don't think we have very good answers to these things. Of course one can always add a regression test (with some K mutations) when someone crashes. It is better than nothing, but hardly good assurance that something like this will never happen again. The entire area of safety and quality assurance with neural networks is still in active research actively, from multiple angles. For some examples of how chaotic neural networks can be, look up 'adversarial examples'.
Scoundreller 8y ago
That’s a very strange way to close a highway. Where are the blinkenlights? The words? The flashing arrows? The plastic jersey barriers? The buckets full of sand or water? The repainted lines directing you to the right? Can’t tell if this is temporary or long-term closure, but if it’s a multi month thing, I would expect more “Don’t you dare fork to the left” signalling. Not everyone is a local.
- blahyawnblah 8y ago
  It's a lane of traffic that can go either way depending on demand. Common in larger cities in California and Washington.
  - deleted 8y ago
    [deleted]
- jakeogh 8y ago
  Why would any of that be relevant?
  - Scoundreller 8y ago
    Because the same things that confuse humans can confuse computers that are largely trained with human data.
    - sgt101 8y ago
      Upsetting as it is, it also turns out that different things also confuse the computers. Sometimes unexpectedly and suddenly.
    - deleted 8y ago
      [deleted]
    - dymk 8y ago
      Millions of drivers per year use roads like this in California and Washington. They’re not very confusing for a human driver who’s paying attention.
paul7986 8y ago
Scary... software developers at these robotic cars and their mistakes/bugs aren’t just going to bring down a business application(lose money) but kill their customers and innocent drivers. Progress to where it’s safer is going to be a killer and we the drivers on the road are unwilling guinea pigs to billionaires’ dreams/goals.
- nathanaldensr 8y ago
  Unwilling? Don't buy a Tesla. Don't believe the hype. It's that simple. Granted, there is nothing stopping any ECU from killing you, but I'd trust a company like Honda way before Tesla.
  - Tomte 8y ago
    I generally have zero interest in cars and don't follow the new models, but my impression from articles I've seen in the last years is that Volvo is actually a top contender for driver assistant systems (when you don't fool yourself into thinking you have an autopilot, but you really want sensible safety augmentation features). Is that impression accurate?
    - mongol 8y ago
      I have no idea. Volvo certainly has the culture to do something good out of it. But do they have the money and resources required? Today they are owned by Chinese Geely. I don't know what partnerships and capital they can work with to compete with the top contenders (who I assume have Silicon Valley capital behind them)
      - sangnoir 8y ago
        Geely has invested a lot into Volvo[1], and Volvo are innovating in interesting ways[2] . I would choose the electric Polestar 2 over a Tesla in the same price-bracket due to Volvo's culture of safety. Hopefully the cheaper versions will be released soon. 1. https://www.bloomberg.com/news/features/2018-05-24/volvo-is-better-than-ever-thanks-to-this-chinese-billionaire https://www.bloomberg.com/news/features/2018-05-24/volvo-is-... 2. https://arstechnica.com/cars/2019/02/volvo-spinoff-polestar-reveals-its-battery-ev-the-polestar-2/ https://arstechnica.com/cars/2019/02/volvo-spinoff-polestar-...
mslev 8y ago
Watching the video, '2019.5.15 - Try 2' is interesting. You can see the car moving normally, then it starts to follow the black crack in the road and moves to the right- at this moment, the white Nissan in front is blocking the white lines ahead where the lanes actually split. Does AP use other cars as reference points, or just the road? Ideally in this situation it would be both: "The line has disappeared, and there's a new one now, but that car went over it". Instead it seems to just be following whatever lines it can see. Does that make sense? Note- not at all defending the AP behavior here. Just thinking out loud.
- treis 8y ago
  >Watching the video, '2019.5.15 - Try 2' is interesting. You can see the car moving normally, then it starts to follow the black crack in the road and moves to the right- at this moment, the white Nissan in front is blocking the white lines ahead where the lanes actually split. It seems like it's failing in different ways: Try 1- Toughest to tell, but it looks like it failed to recognize any lines. Kept going straight which was at the barrier. Hard to tell if the car would have recovered. Try 2- Looks like the car tried to go left into the closed lane. Seems like an error in detecting the barriers closing the road. I'd guess that it would have avoided the concrete barrier and driven down the closed lane Try 3 - This one looks like it picked the wrong lane marker to be the left side of the road. In that it thought the right lane marker of the closed lane was actually the left lane marker. This one probably ends up with a smashed car and dead driver.
- deleted 8y ago
  [deleted]
Tomte 8y ago
The comments are scary. People defending Tesla's misleading advertisement re: full self-driving hardware etc. Because clearly "having full self-driving hardware" only means the hardware is there, not that the car can actually self-drive. I bet those people find themselves misunderstood in pretty much every discussion with people outside their nerdy circle.
- mikejb 8y ago
  On the earnings call, Musk even claimed that the Navigate on Autopilot feature set is "full self-driving" [1] [1] https://youtu.be/oB0sAGykWT8?t=2895 https://youtu.be/oB0sAGykWT8?t=2895
tbabb 8y ago
I have been yelling about this for a long time: Tesla is not going to be able to deliver full self-driving as promised. They don't have the hardware, for one (ranging is terrible; they need stereo cameras), and second, their software strategy seems to be a dead end. They need sensor fusion. The system needs to make maximum use of all the information available to it: Where is the road striping? Where are the other cars going? Where are the road signs and signals? (If there's one in your path, you certainly shouldn't drive into it!) Are there camera-visible obstructions? What were the interpretations and actions of previous Tesla trips along the same route? In these problem cases, all data except the left and right lane striping seems to be completely ignored. There was even more information at the fatal offramp location (cross-striping over the lane separation zone), which the vehicle drove straight over. The system is not making maximum use of the information available to it, in fact it is using hardly any of it at all, and fixating on what it thinks is a single most salient piece of data. Sensor fusion algorithms tend to behave the opposite way-- each additional piece of data informs the interpretation of all the other data. You can have very poor-quality data, but if it is even moderately over-constrained, your state estimate can be very good in spite of it. I think it would be completely reasonable to have a neural net in the loop of a sensor fusion algorithm, with fusion constraints informing the NN's interpretation, and the NN's estimates feeding back into the fusion algorithm as uncertain data. IMO Tesla will do at least one of: * Very expensively retract their promise of full self driving for delivered vehicles * Completely overhaul/redesign their driving software and start again nearly from scratch * Get into a regulatory/legal tangle with the NTSA/courts/DOJ over all the dead people their system is making.
- Dumblydorr 8y ago
  Where is your evidence for these claims?
- eclipxe 8y ago
  They do have stereo cameras and sensor fusing, as well as detecting more than just the lines in the road. Here is what the camera sees: https://www.youtube.com/watch?v=rACZACXgreQ https://www.youtube.com/watch?v=rACZACXgreQ
throwaway789214 8y ago
Crazy how people are blatantly ignoring that even on autopilot drivers need to be attentive and keep situational awareness at all times. I can't remember that there was ever a case in which autopilot caused a situation that an attentive driver couldn't have recovered from. Yet it is blamed on the system when people ignore this requirement and end up in an accident. Throw-away for obvious reasons...
- rhino369 8y ago
  So what’s the point of it? Not steering but needing to able to steer immediately seems much more mentally taxing than just steering.
  - throwaway789214 8y ago
    Do not use it if it is net negative for you but if you do, follow the instructions. You can't have the cake and eat it. There are a lot of people at Tesla working fulltime on auto-pilot and some drivers think they know better than them by disregarding their instructions how to use it? Frankly, that's just crazy.
    - jessaustin 8y ago
      Perhaps those drivers don't have your inside view of Tesla R&D?
      - throwaway789214 8y ago
        Inside view not required, self-awareness of users should suffice to come to that conclusion (i.e. "I know nothing of the inner workings of this system, so my judgement has to be worse than that of the ones who designed it.").
        rohit2412 8y ago
        Maybe you can tell Elon musk to be more self aware too. https://www.businessinsider.com/elon-musk-breaks-tesla-autopilot-rule-2018-12 https://www.businessinsider.com/elon-musk-breaks-tesla-autop...
systemspeed 8y ago
With demand for self-driving vehicles as high as it is, yet with the lag in advancement of computer vision, I think it's about time for a serious discussion about smart roads. I realize this isn't entirely relevant, but I can't be the only one thinking that we're missing the forest for the trees by trying to solve a transportation problem while simultaneously solving a vision problem.
- progfix 8y ago
  Might as well put them on rails (Smart-Rails(TM)) and make a railway network for smart people. Transportation problems solved!
- vkou 8y ago
  So, trains?
ummonk 8y ago
Maybe it's merely that I'm used to cones rather than barriers of that sort, but that stretch of road looks rather confusing to me as a human as well.
syntaxing 8y ago
Tesla should just partner or acquire a LiDAR startup already like Baraja [1]...They can let the computer vision do all the magic they want. Just have one scanning LiDAR as a redundant system so that it doesn't run into anything in front of the car. [1] https://www.baraja.com/ https://www.baraja.com/
- ip26 8y ago
  I don't know why, but supposedly they've made "no LiDAR" their hill to die on.
  - syntaxing 8y ago
    Yeah, it's such a weird mentality. Their new patent on a "CNN" ASIC is pretty neat but still doesn't solve a lot of their problems. Are they banking on a magical depth CNN architecture to be released or something?!
  - edshiro 8y ago
    Costs. Installing LIDAR will significantly hike up the price of their vehicles.
newnewpdro 8y ago
I find it absurd how the NHTSA doesn't treat Tesla similarly to how it treated Toyota ~10 years ago with the sudden unintended acceleration controversy. [1] Tesla has demonstrably flawed and dangerous vehicles operating on our roads. These persistent autopilot bugs are killing people. If we treated the Tesla autopilot as a licensed driver, it'd have its license suspended. (Not that it could have passed a driving test in the first place) [1] https://en.wikipedia.org/wiki/2009%E2%80%932011_Toyota_vehicle_recalls#Investigations https://en.wikipedia.org/wiki/2009%E2%80%932011_Toyota_vehic...
- DarmokJalad1701 8y ago
  > These persistent autopilot bugs are killing people. Are there any numbers on this?
GoToRO 8y ago
That's one way to keep drivers alert I guess... Add a little bit of randomness into your boring drive.
lgleason 8y ago
Tongue in cheek......this is planned obsolescence. New model with upgraded features is announced so you start to brick the old ones by making them run into things. :) Of course there is that pesky issue of it potentially injuring/killing the driver and passengers.... In all seriousness, these things are getting better, but Skynet is not here yet. Just because a company, governmental agency etc. says something is safe you always need to evaluate things yourself..and it's probably not a good idea to rely on these auto pilots unless you want to win a Darwin award. I love the cars, but the hype is a bit annoying around stuff like this.
perfunctory 8y ago
It troubles me that we use the term "software bug" to describe this.
mcguire 8y ago
One might suspect, given the reintroduction of the bug, that Tesla doesn't understand their code.
erobbins 8y ago
I think Tesla eventually has a catastrophic accident and is sued and/or criminally prosecuted into oblivion. I feel sorry for the people who are going to have to die for this to happen. This trope that humans are bad drivers is, in general, crap. Humans are very good drivers. The US has 7.3 deaths per billion km driven. This means if you drive 50km a day, every day, you are (essentially) guaranteed to die... after 7500 YEARS. You have less than a 1 in 10 million chance of dying on any given trip you take. That is NOT risky, and is NOT dangerous.
- Joe-Z 8y ago
  Cars have gotten so safe though. There's all this measures put in place to ensure your survival _in case of a crash_. Wouldn't it be more relevant to calculate the chances of getting injured / end up in a crash at all? I mean just being caught up in a traffic accident and have no bodily harm done to you can be a traumatic event.
  - erobbins 8y ago
    You're not wrong. There are plenty of accidents caused by inattention and poor driving that aren't fatal, but that do result in injuries... a quick search didn't give me much information but crashes with injuries appear to be about 100x more common than fatal crashes. This doesn't take into account how serious the injuries are, however. So if we assume that serious injuries are 10x more common than fatalities, that raises the odds to one incident per 750 years, or 1 incident per 75 years of any accident at all. You're still talking about things that happen to people once or twice in their entire lives. Of course there will be outliers (I've been in 5 accidents myself, only 1 with injuries) but the odds of being in a crash don't seem to be worth the risk of trying to automate mousetraps... we should concentrate on replacing car-based transportation instead of trying to use magical black boxes to make it theoretically safer.
    - FireBeyond 8y ago
      As an aside, as of 2019, you are more likely to die of an opiate overdose in the US than as a result of a car accident.
cmurf 8y ago
So far Tesla automation seems to handle edge cases. I have never rear ended or side swiped anyone, getting into such an accident or avoiding one is an edge case for me. Useful but not really automation. For it to even have a chance during lane keeping of doing something only a sleeping or suicidal driver would do? It's utter bullshit. It's less safe than a human. Autonomous driving is overhyped vaporware. Autonomous airplanes would be easier than cars, and we still are no where near that.
jtaft 8y ago
Can they simulate their sensors input pretty well? Do they use car driving simulators (video-game like) to see what would happen in similar scenarios? Of course, real world testing would still be necessary.
- jaimex2 8y ago
  When a customer disengages or corrects AP that periods sensor data is recorded and uploaded to Tesla. The uploads are curated and submitted into the neural network training data-set. They basically have an infinite amount of real world data coming in.
alanh 8y ago
My Model 3 suddenly changed lanes today for no discernible reason. I think it considered my lane to jump over into the next lane, for some reason. I should have hit 'Record' to save the footage from TeslaCam. (Notably, AutoPilot is not supposed to change lanes without explicit confirmation from the driver, which is clearly illustrated on the dashboard/panel.) It's a stretch of road on which I have previously used AutoPilot many times.
- gpm 8y ago
  Recently I accidentally changed lanes in an intersection. The road had 3 lanes in each direction (+ separate streetcar tracks down the center), including the right lane which was required to turn. I was in the center lane. After the intersection the road was still a 3 lane road - which didn't quite register on my brain. I moved from the center lane to the right lane (figuring it exited so I was supposed to be in the now-right lane) when I should have remained in the center lane. While I am a novice driver, I've gone through that intersection before without blinking or doing the wrong thing. It's not a particularly complicated intersection. Anyways, point is, driving is surprisingly hard. I think counting anecdotes on the internet probably gives you a sample heavily biased against Tesla, because most people don't go post "so I did this stupid thing" but they do post "so my car did this stupid thing".
  - alanh 8y ago
    My car and I are stupid in different ways. It has perfect attention and will always brake in time to avoid a fender-bender in traffic. It’s also a lot worse than I am about reading and communicating intent with other drivers or simply recognizing the objects around us.
gwbas1c 8y ago
The video really doesn't convey what happened. It's not really clear when the driver took over. My Model 3 is very odd when going through forks like this. It just has trouble picking one side of the lane or the other. Yesterday I got a little scared when it swerved back and forth while trying to figure out how to take an exit.
jaimex2 8y ago
Warning: Autosteer is intended for use only on highways and limited-access roads with a fully attentive driver. When using Autosteer, hold the steering wheel and be mindful of road conditions and surrounding traffic. Do not use Autosteer on city streets, in construction zones, or in areas where bicyclists or pedestrians may be present. Never depend on Autosteer to determine an appropriate driving path. Always be prepared to take immediate action. Failure to follow these instructions could cause damage, serious injury or death. https://www.tesla.com/content/dam/tesla/Ownership/Own/Model%203%20Owners%20Manual.pdf#page=75 https://www.tesla.com/content/dam/tesla/Ownership/Own/Model%...
- kurtisc 8y ago
  And then... https://www.businessinsider.com/elon-musk-breaks-tesla-autopilot-rule-2018-12?r=US&IR=T https://www.businessinsider.com/elon-musk-breaks-tesla-autop...
steelframe 8y ago
The day I took delivery of my 2016 Model X with AP 1.0, Tesla announced AP 2.0. A friend of mine immediately ordered a Model X with AP 2.0 and rubbed it in my face. For the entire next year, my AP 1.0 (which is non-Tesla technology -- Mobileye rocks) had no trouble doing adaptive cruise control and lane assist. Meanwhile his AP 2.0 would brake suddenly and swerve all over the place. It took a full year of OTA updates before his AP 2.0 was finally on-par with the functionality that I had the whole time. Of course, by then Tesla pulled a "we're sorry, but the princess is in another castle" and came out with AP 2.5. Now this kind of stuff doesn't matter to me. I got tired of that company's shit and have pulled out of the Teslasphere entirely. I'm now driving a non-Tesla EV, and I'll never look back. I'm also letting my government representatives know that they should support a common EV charging standard and keep Tesla so-called "self-driving" shit off public roads.
dreamcompiler 8y ago
I used to be a big supporter of Tesla, but I now have to apply Internet-of-Shit Rule #1 to them: No software updates ever without explicit owner approval. And "owner" is me, not Elon Musk.
101001001001 8y ago
I still want a model 3. I’ll just not use the autopilot.
sandos 8y ago
What I dont understand is, that yellow sign is actually very visible compared to other things the neural nets can detect. Why hasn't it been done yet? Detecting things as visible as that sign should not be a huge problem.

339 points by SamuelAdams 8y ago

I have always wondered... Does the AP have some higher-level notion of object permanence, continuity (road behind a horizon or after a curve) and things like that? Does it track a pedestrian that is momentarily hidden behind an obstacle and will probably reemerge in a second or two on the other side? Does it expect that kids may run after that ball that just flew from behind a car? Does it continuously track and improve classification of all objects in the field of vision, with their trajectories and speeds if they are moving? Personally I don't think it does, otherwise it would not erratically slam into clearly visible and marked large objects in its way, or it would be aware of a truck moving in a perpendicular way and so on. I am of opinion that without such higher-level awareness it can never succeed, hope to learn about the state of the self-driving art.

dragontamer 8y ago
I don't think neural networks are wired to "remember" things. In theory, they could be hooked up that way. But your typical convolutional neural network is looking at things frame-by-frame. In theory, ANNs could have an output layer that passes data from one frame to another frame to assist things. But there's no real programming to "hardcode" something like object permanence into an ANN. You pretty much throw a bunch of data into the system and hope for the best.
- hemogloben 8y ago
  NNs are just the first step in the pipeline. Their outputs (detected objects, segmentation, etc) will be piped into other software that builds higher level models. Considering the path-planning requirements I would be absolutely shocked if Autopilot wasn't build history models and estimated paths for objects around the vehicle (other cars etc).
  - JanSolo 8y ago
    Agreed; I imagine they use neural networks to detect and classify objects which are then saved into a scene-graph for use in pathing. I expect what happened was that they trained their NNs for improved detection in one area but unknowingly reduced it in another. Perhaps now it can detect tricycles 99% but road barriers went down to only 30%. Having worked with NNs it's very common to see gains in one domain which come at a cost of reduced performance in another.
    - stevenjohns 8y ago
      They must have known. I haven’t worked with NNs for a few years but I don’t believe the methodology has changed where you would stop testing it over different sets of training data. Barriers are a pretty big part of driving on roads and highways and the only reason it would have been unknowingly reduced would be if they just weren’t testing the NN against data with them.
- robrenaud 8y ago
  There are architectures that use CNN on image inputs, and LSTMs across frames of the video to keep memory. https://arxiv.org/pdf/1609.06377.pdf https://arxiv.org/pdf/1609.06377.pdf
  - x38iq84n 8y ago
    Thanks (to all in this subthread). I've skimmed through this doc, I have watched https://vimeo.com/274274744 https://vimeo.com/274274744 linked from original reddit discussion and I have lost all remaining faith in AP as it is today. This should be in a closed alpha version, not anywhere near paying customers and marketed as FSD ready/feature ready as it's none of that and won't be for many years. I expect there will be at least a generation of Tesla cars sold with FSD-readiness package that will never see FSD in their lifetime.
deleted 8y ago
[deleted]
eclipxe 8y ago
https://www.youtube.com/watch?v=rACZACXgreQ https://www.youtube.com/watch?v=rACZACXgreQ
automathematics 8y ago
In these videos, "Autopilot" is mentioned as the culprit, which seems to be a subset of the features Tesla has. It seems to me there are 3 layers: (level 0: Just Adaptive Cruise Control (use radar to adjust speed up to a max). Human still steers. - 1: Adaptive Cruise Control (use radar to adjust speed up to a max) + Autosteer (camera's watch the lane markers and follow them). This is referred to as "Autopilot" NOTE: This is where the accidents happen. The car isn't driving towards a barrier, it's following the lane markers and hits an error state. This is also NOT self driving. - 2: "Nav on Autopilot". This is an additional function you turn on where the car has more intelligent (use this word loosely) capabilities on highways. The car will still do everything on level 1 combined with lane changes (using cameras to detect objects and trajectories differentiating cars from trucks from pedestrians from bikes from motorcycles etc). It will still follow lane lines, but with a lot of additional information (is there an object? am I merging? am I exiting? etc) - 3: "Full Self Driving". This is an additional package that doesn't exist to the public. Internally I'm sure they're testing the functionality but this is using all of the sensors and algorithms and likely neural networks to decide what to do. A cool point though is that all Tesla's are likely running this code in "shadow mode" where data can be collected and assumptions can be tested without endangering any actual drivers. (see here for some cool data on this: https://electrek.co/2019/03/05/tesla-autopilot-detects-stop-lines-intersections/ https://electrek.co/2019/03/05/tesla-autopilot-detects-stop-...). "Hey, I think the car, if full self driving SHOULD take action X. *compare to what the driver actually does and log the data" over BILLIONS of miles So when a Tesla hits the barrier or gets in an accident, we're actually running "#1" and people start freaking out. But when we get to the capabilities of #3 a lot of the "object permanence and continuity" stuff starts to come into play. Full Disclosure: - I drive a Model 3 every day on Autopilot 75% of the time - I only 75% think self driving is capability under the current Tesla software suite but I bought the package anyways.

SamuelAdams 8y ago

Additional context: on March 23, 2018, a Tesla Model X, while using autopilot, drove into a barrier and killed the driver. This was fixed in a software update, but the issue seems to have resurfaced. NTSB analysis of the March 23 2018 collision: https://www.ntsb.gov/investigations/AccidentReports/Pages/HWY18FH011-preliminary.aspx https://www.ntsb.gov/investigations/AccidentReports/Pages/HW... Tesla's statement of the March 23 2018 collision: https://www.tesla.com/blog/update-last-week%E2%80%99s-accident https://www.tesla.com/blog/update-last-week%E2%80%99s-accide...

MagicPropmaker 8y ago
Well, it wasn't really "fixed" -- the driver is still dead.
- craftyguy 8y ago
  Sure, if you construe the meaning of 'fixed' to mean 'roll back all events and time', which I don't think anyone expects. They probably fixed the software bug.
  - falcolas 8y ago
    I think that it's important to note that "we'll fix bugs via live patches" is a deadly decision when it comes to self driving software. Just discussing the incident in terms of a "software bug" does a disservice to the severity of the issue.
    - craftyguy 8y ago
      No, it does not. While I agree that testing patches on users with no testing beforehand is horrible, 1) I doubt Tesla does this, and 2) the person died before they fixed it, which GP seemed to be confused about.
    - gwbas1c 8y ago
      Remember, autopilot and autosteer are not self-driving. Tesla is very explicit that the driver must remain alert, and supervise, at all times. That being said, drowsy driving is a thing, and it's very easy to fall asleep behind the wheel. The car really needs a better strategy to handle this situation.

sschueller 8y ago

Eventually we need the NTSB to certify updates before they are pushed out over the air. Similar to what the FAA does. These OTA updates are not ok for large machinery and endagers not only the Tesla driver but all others on the road.

x38iq84n 8y ago
Indeed. I have never understood the cheering for OTA of something that can literally kill you. Were they limited to infotainment and the useless easter eggs I would not care, however AP OTA is downright scary, especially pushed by vendor instead of being initiated by an informed user.
- 0815test 8y ago
  The AP can only kill you if you're distracted/asleep at the wheel and not paying attention. You're always in charge of driving the car appropriately, the "AP" can only provide tentative assistance of a very rough sort. It will try to guess at the control inputs you might want to provide (which does make things a bit more comfortable in the best case) but the actual control always rests with the human driver.
  - saagarjha 8y ago
    The issue is that the way the feature is designed and marketed makes it very easy for the driver to be distracted and not paying attention.
  - craftyguy 8y ago
    It shouldn't be marketed as 'Autopilot' then. Less informed people will expect it to do more than it can. It's (IMHO, criminally) misleading to market something as being capable of doing something it cannot do.
  - lisper 8y ago
    > The AP can only kill you if you're distracted/asleep at the wheel and not paying attention. That's not true. There are many situations where the AP could make a sudden turn and kill you long before you had time to react even if you were paying attention. And in fact, the situation in the video seem to be not so far away from that.

FreedomToCreate 8y ago

Tesla can only really gather depth data from there radars, while the cameras operate a DNN to detect features and make adjustments for steering and speed based on fusing those two pieces of data together. An error in the radar or the DNN detecting features incorrectly because of lighting, road color change or object on the road which the model was not trained on, can cause problems like this.

notfromhere 8y ago
if it came back after an update, wouldn't it be the issue of the algorithm training itself incorrectly?
zmarty 8y ago
"Tesla can only really gather depth data from there radars" - they can also do it visually through depth from motion

SilasX 8y ago

I would be a lot more forgiving of these screwups if Tesla didn't constantly swear up and down that they've solved self-driving cars. As far back as 2016 they were claiming they had full SDC capability above human driver safety [1], and their recent Model Y announcement suggests that the only thing holding it up is regulatory approval, and not failure to achieve the desired spec. >Model Y will have Full Self-Driving capability, enabling automatic driving on city streets and highways pending regulatory approval, as well as the ability to come find you anywhere in a parking lot. [1] https://www.tesla.com/blog/all-tesla-cars-being-produced-now-have-full-self-driving-hardware https://www.tesla.com/blog/all-tesla-cars-being-produced-now... [2] https://news.ycombinator.com/item?id=19397942 https://news.ycombinator.com/item?id=19397942 (linking HN because it's hard to find the text in the page with their UI)

Scoundreller 8y ago
> As far back as 2016 they were claiming they had full SDC capability above human driver safety One crash/incident doesn’t mean it’s less safe than a human driver on average, even if it’s something a human driver might have avoided. We should expect different failure modes from a machine, but adopt it anyway if it avoids enough human failure modes to make up for it. Source: my thermostat thought 33C was an acceptable temp once, but I still didn’t switch to full-manual HVAC control.
- tptacek 8y ago
  Don't Teslas fare significantly worse than cars in similar class in terms of driver fatalities?
  - lordalch 8y ago
    I don't have the fatalities stats, but their safety ratings have been perfect. [0] https://www.cnbc.com/2018/09/20/tesla-model-3-earns-perfect-5-star-nhtsa-safety-rating.html https://www.cnbc.com/2018/09/20/tesla-model-3-earns-perfect-...
  - Scoundreller 8y ago

sabareesh 8y ago

Since ModelX incident I have made sure not to drive on first lane. I always drive on the 2nd lane and it is much better.

bdcravens 8y ago
For a $700 car I might consider crazy compromises, but $70k+? Absolutely not.
- warp_factor 8y ago
  That's what I find so interesting with Tesla owner. They spend a fortune on a car, then they minimize every single issue they have with the car. My explanation for this is that for a lot of owners, the car is a way to be part of a hyped group more than being a utilitarian object (what a car should be).
  - gojomo 8y ago
    https://en.wikipedia.org/wiki/Choice-supportive_bias https://en.wikipedia.org/wiki/Choice-supportive_bias
  - mikestew 8y ago
    Happens with a lot of stuff, certain American-made motorcycles, for instance. You're in the club now, and the only way to stay in the club is to carry the manufacturer's water. In no case are we to admit that we spent a bunch of money on something that does a poor job of fulfilling its advertised purpose.
    - bdcravens 8y ago
      Or more relevant to the HN audience, Apple.
      - mikestew 8y ago
        What, Tesla wasn't relevant enough for you to not take an "obligatory" dig at Apple product owners?

anth_anm 8y ago

Said it before, saying it again. OTA updates to safety critical things is not a feature. It's a bug.

bsaul 8y ago

I wonder how do unit tests work with NN (or if they're even a relevant concept at all). You could replay some testing video frames and make sure the objects are correctly identified, but i suppose that's already what training is about... If an issue like that resurface, does it mean that the original frames leading to the 2018 accident aren't part of the training (or at least frames from someone driving in this kind of scenario) ?

inertiatic 8y ago
Well, testing in the strict sense would be measuring accuracy on data not in the training set, and even accuracy on the training set data (which isn't guaranteed to be 100% I believe)
bluGill 8y ago
The frames from the 2018 should not be part of the training. They should be part of the test set to prove whatever training they do works.
neilalexander 8y ago
Yes, the problem is indeed that there's no real way to "look into" a neural network and understand how it has been trained. All you can do is observe that the given inputs generate the desired outputs. Even if there was training based on the 2018 frames, that doesn't mean that you have verifiably fixed the problem. It's difficult to train a neural network selectively – every time you "train" the network with additional data, you are increasing the chance that you are also teaching it something you didn't intend which then can have a side-effect in some seemingly unrelated scenario. You can see this in real life with image recognition networks. Teach them too much and they gradually become less effective at identifying anything.
jononor 8y ago
Unit tests are not sufficient for neural networks. Say the network takes 2D images of typical 224x224 pixels with 3 channels (RGB) of 8 bit values, this input space has 256^(224x244x3) = 3.5×10^362507 possibilities. Billions of years to test them all. This is before we consider stereo vision, 3D images and state over time. How to know which subset of these inputs are necessary to give reasonable coverage? Right now I don't think we have very good answers to these things. Of course one can always add a regression test (with some K mutations) when someone crashes. It is better than nothing, but hardly good assurance that something like this will never happen again. The entire area of safety and quality assurance with neural networks is still in active research actively, from multiple angles. For some examples of how chaotic neural networks can be, look up 'adversarial examples'.

Scoundreller 8y ago

That’s a very strange way to close a highway. Where are the blinkenlights? The words? The flashing arrows? The plastic jersey barriers? The buckets full of sand or water? The repainted lines directing you to the right? Can’t tell if this is temporary or long-term closure, but if it’s a multi month thing, I would expect more “Don’t you dare fork to the left” signalling. Not everyone is a local.

blahyawnblah 8y ago
It's a lane of traffic that can go either way depending on demand. Common in larger cities in California and Washington.
- deleted 8y ago
  [deleted]
jakeogh 8y ago
Why would any of that be relevant?
- Scoundreller 8y ago
  Because the same things that confuse humans can confuse computers that are largely trained with human data.
  - sgt101 8y ago
    Upsetting as it is, it also turns out that different things also confuse the computers. Sometimes unexpectedly and suddenly.
  - deleted 8y ago
    [deleted]
  - dymk 8y ago
    Millions of drivers per year use roads like this in California and Washington. They’re not very confusing for a human driver who’s paying attention.

paul7986 8y ago

Scary... software developers at these robotic cars and their mistakes/bugs aren’t just going to bring down a business application(lose money) but kill their customers and innocent drivers. Progress to where it’s safer is going to be a killer and we the drivers on the road are unwilling guinea pigs to billionaires’ dreams/goals.

nathanaldensr 8y ago
Unwilling? Don't buy a Tesla. Don't believe the hype. It's that simple. Granted, there is nothing stopping any ECU from killing you, but I'd trust a company like Honda way before Tesla.
- Tomte 8y ago
  I generally have zero interest in cars and don't follow the new models, but my impression from articles I've seen in the last years is that Volvo is actually a top contender for driver assistant systems (when you don't fool yourself into thinking you have an autopilot, but you really want sensible safety augmentation features). Is that impression accurate?
  - mongol 8y ago
    I have no idea. Volvo certainly has the culture to do something good out of it. But do they have the money and resources required? Today they are owned by Chinese Geely. I don't know what partnerships and capital they can work with to compete with the top contenders (who I assume have Silicon Valley capital behind them)
    - sangnoir 8y ago
      Geely has invested a lot into Volvo[1], and Volvo are innovating in interesting ways[2] . I would choose the electric Polestar 2 over a Tesla in the same price-bracket due to Volvo's culture of safety. Hopefully the cheaper versions will be released soon. 1. https://www.bloomberg.com/news/features/2018-05-24/volvo-is-better-than-ever-thanks-to-this-chinese-billionaire https://www.bloomberg.com/news/features/2018-05-24/volvo-is-... 2. https://arstechnica.com/cars/2019/02/volvo-spinoff-polestar-reveals-its-battery-ev-the-polestar-2/ https://arstechnica.com/cars/2019/02/volvo-spinoff-polestar-...

mslev 8y ago

Watching the video, '2019.5.15 - Try 2' is interesting. You can see the car moving normally, then it starts to follow the black crack in the road and moves to the right- at this moment, the white Nissan in front is blocking the white lines ahead where the lanes actually split. Does AP use other cars as reference points, or just the road? Ideally in this situation it would be both: "The line has disappeared, and there's a new one now, but that car went over it". Instead it seems to just be following whatever lines it can see. Does that make sense? Note- not at all defending the AP behavior here. Just thinking out loud.

treis 8y ago
>Watching the video, '2019.5.15 - Try 2' is interesting. You can see the car moving normally, then it starts to follow the black crack in the road and moves to the right- at this moment, the white Nissan in front is blocking the white lines ahead where the lanes actually split. It seems like it's failing in different ways: Try 1- Toughest to tell, but it looks like it failed to recognize any lines. Kept going straight which was at the barrier. Hard to tell if the car would have recovered. Try 2- Looks like the car tried to go left into the closed lane. Seems like an error in detecting the barriers closing the road. I'd guess that it would have avoided the concrete barrier and driven down the closed lane Try 3 - This one looks like it picked the wrong lane marker to be the left side of the road. In that it thought the right lane marker of the closed lane was actually the left lane marker. This one probably ends up with a smashed car and dead driver.
deleted 8y ago
[deleted]

Tomte 8y ago

The comments are scary. People defending Tesla's misleading advertisement re: full self-driving hardware etc. Because clearly "having full self-driving hardware" only means the hardware is there, not that the car can actually self-drive. I bet those people find themselves misunderstood in pretty much every discussion with people outside their nerdy circle.

mikejb 8y ago
On the earnings call, Musk even claimed that the Navigate on Autopilot feature set is "full self-driving" [1] [1] https://youtu.be/oB0sAGykWT8?t=2895 https://youtu.be/oB0sAGykWT8?t=2895

tbabb 8y ago

I have been yelling about this for a long time: Tesla is not going to be able to deliver full self-driving as promised. They don't have the hardware, for one (ranging is terrible; they need stereo cameras), and second, their software strategy seems to be a dead end. They need sensor fusion. The system needs to make maximum use of all the information available to it: Where is the road striping? Where are the other cars going? Where are the road signs and signals? (If there's one in your path, you certainly shouldn't drive into it!) Are there camera-visible obstructions? What were the interpretations and actions of previous Tesla trips along the same route? In these problem cases, all data except the left and right lane striping seems to be completely ignored. There was even more information at the fatal offramp location (cross-striping over the lane separation zone), which the vehicle drove straight over. The system is not making maximum use of the information available to it, in fact it is using hardly any of it at all, and fixating on what it thinks is a single most salient piece of data. Sensor fusion algorithms tend to behave the opposite way-- each additional piece of data informs the interpretation of all the other data. You can have very poor-quality data, but if it is even moderately over-constrained, your state estimate can be very good in spite of it. I think it would be completely reasonable to have a neural net in the loop of a sensor fusion algorithm, with fusion constraints informing the NN's interpretation, and the NN's estimates feeding back into the fusion algorithm as uncertain data. IMO Tesla will do at least one of: * Very expensively retract their promise of full self driving for delivered vehicles * Completely overhaul/redesign their driving software and start again nearly from scratch * Get into a regulatory/legal tangle with the NTSA/courts/DOJ over all the dead people their system is making.

Dumblydorr 8y ago
Where is your evidence for these claims?
eclipxe 8y ago
They do have stereo cameras and sensor fusing, as well as detecting more than just the lines in the road. Here is what the camera sees: https://www.youtube.com/watch?v=rACZACXgreQ https://www.youtube.com/watch?v=rACZACXgreQ

throwaway789214 8y ago

Crazy how people are blatantly ignoring that even on autopilot drivers need to be attentive and keep situational awareness at all times. I can't remember that there was ever a case in which autopilot caused a situation that an attentive driver couldn't have recovered from. Yet it is blamed on the system when people ignore this requirement and end up in an accident. Throw-away for obvious reasons...

rhino369 8y ago
So what’s the point of it? Not steering but needing to able to steer immediately seems much more mentally taxing than just steering.
- throwaway789214 8y ago
  Do not use it if it is net negative for you but if you do, follow the instructions. You can't have the cake and eat it. There are a lot of people at Tesla working fulltime on auto-pilot and some drivers think they know better than them by disregarding their instructions how to use it? Frankly, that's just crazy.
  - jessaustin 8y ago
    Perhaps those drivers don't have your inside view of Tesla R&D?
    - throwaway789214 8y ago
      Inside view not required, self-awareness of users should suffice to come to that conclusion (i.e. "I know nothing of the inner workings of this system, so my judgement has to be worse than that of the ones who designed it.").
      - rohit2412 8y ago
        Maybe you can tell Elon musk to be more self aware too. https://www.businessinsider.com/elon-musk-breaks-tesla-autopilot-rule-2018-12 https://www.businessinsider.com/elon-musk-breaks-tesla-autop...

systemspeed 8y ago

With demand for self-driving vehicles as high as it is, yet with the lag in advancement of computer vision, I think it's about time for a serious discussion about smart roads. I realize this isn't entirely relevant, but I can't be the only one thinking that we're missing the forest for the trees by trying to solve a transportation problem while simultaneously solving a vision problem.

progfix 8y ago
Might as well put them on rails (Smart-Rails(TM)) and make a railway network for smart people. Transportation problems solved!
vkou 8y ago
So, trains?

ummonk 8y ago

Maybe it's merely that I'm used to cones rather than barriers of that sort, but that stretch of road looks rather confusing to me as a human as well.

syntaxing 8y ago

Tesla should just partner or acquire a LiDAR startup already like Baraja [1]...They can let the computer vision do all the magic they want. Just have one scanning LiDAR as a redundant system so that it doesn't run into anything in front of the car. [1] https://www.baraja.com/ https://www.baraja.com/

ip26 8y ago
I don't know why, but supposedly they've made "no LiDAR" their hill to die on.
- syntaxing 8y ago
  Yeah, it's such a weird mentality. Their new patent on a "CNN" ASIC is pretty neat but still doesn't solve a lot of their problems. Are they banking on a magical depth CNN architecture to be released or something?!
- edshiro 8y ago
  Costs. Installing LIDAR will significantly hike up the price of their vehicles.

newnewpdro 8y ago

I find it absurd how the NHTSA doesn't treat Tesla similarly to how it treated Toyota ~10 years ago with the sudden unintended acceleration controversy. [1] Tesla has demonstrably flawed and dangerous vehicles operating on our roads. These persistent autopilot bugs are killing people. If we treated the Tesla autopilot as a licensed driver, it'd have its license suspended. (Not that it could have passed a driving test in the first place) [1] https://en.wikipedia.org/wiki/2009%E2%80%932011_Toyota_vehicle_recalls#Investigations https://en.wikipedia.org/wiki/2009%E2%80%932011_Toyota_vehic...

DarmokJalad1701 8y ago
> These persistent autopilot bugs are killing people. Are there any numbers on this?

GoToRO 8y ago

That's one way to keep drivers alert I guess... Add a little bit of randomness into your boring drive.

lgleason 8y ago

Tongue in cheek......this is planned obsolescence. New model with upgraded features is announced so you start to brick the old ones by making them run into things. :) Of course there is that pesky issue of it potentially injuring/killing the driver and passengers.... In all seriousness, these things are getting better, but Skynet is not here yet. Just because a company, governmental agency etc. says something is safe you always need to evaluate things yourself..and it's probably not a good idea to rely on these auto pilots unless you want to win a Darwin award. I love the cars, but the hype is a bit annoying around stuff like this.

perfunctory 8y ago

It troubles me that we use the term "software bug" to describe this.

mcguire 8y ago

One might suspect, given the reintroduction of the bug, that Tesla doesn't understand their code.

erobbins 8y ago

I think Tesla eventually has a catastrophic accident and is sued and/or criminally prosecuted into oblivion. I feel sorry for the people who are going to have to die for this to happen. This trope that humans are bad drivers is, in general, crap. Humans are very good drivers. The US has 7.3 deaths per billion km driven. This means if you drive 50km a day, every day, you are (essentially) guaranteed to die... after 7500 YEARS. You have less than a 1 in 10 million chance of dying on any given trip you take. That is NOT risky, and is NOT dangerous.

Joe-Z 8y ago
Cars have gotten so safe though. There's all this measures put in place to ensure your survival _in case of a crash_. Wouldn't it be more relevant to calculate the chances of getting injured / end up in a crash at all? I mean just being caught up in a traffic accident and have no bodily harm done to you can be a traumatic event.
- erobbins 8y ago
  You're not wrong. There are plenty of accidents caused by inattention and poor driving that aren't fatal, but that do result in injuries... a quick search didn't give me much information but crashes with injuries appear to be about 100x more common than fatal crashes. This doesn't take into account how serious the injuries are, however. So if we assume that serious injuries are 10x more common than fatalities, that raises the odds to one incident per 750 years, or 1 incident per 75 years of any accident at all. You're still talking about things that happen to people once or twice in their entire lives. Of course there will be outliers (I've been in 5 accidents myself, only 1 with injuries) but the odds of being in a crash don't seem to be worth the risk of trying to automate mousetraps... we should concentrate on replacing car-based transportation instead of trying to use magical black boxes to make it theoretically safer.
  - FireBeyond 8y ago
    As an aside, as of 2019, you are more likely to die of an opiate overdose in the US than as a result of a car accident.

cmurf 8y ago

So far Tesla automation seems to handle edge cases. I have never rear ended or side swiped anyone, getting into such an accident or avoiding one is an edge case for me. Useful but not really automation. For it to even have a chance during lane keeping of doing something only a sleeping or suicidal driver would do? It's utter bullshit. It's less safe than a human. Autonomous driving is overhyped vaporware. Autonomous airplanes would be easier than cars, and we still are no where near that.

jtaft 8y ago

Can they simulate their sensors input pretty well? Do they use car driving simulators (video-game like) to see what would happen in similar scenarios? Of course, real world testing would still be necessary.

jaimex2 8y ago
When a customer disengages or corrects AP that periods sensor data is recorded and uploaded to Tesla. The uploads are curated and submitted into the neural network training data-set. They basically have an infinite amount of real world data coming in.

alanh 8y ago

My Model 3 suddenly changed lanes today for no discernible reason. I think it considered my lane to jump over into the next lane, for some reason. I should have hit 'Record' to save the footage from TeslaCam. (Notably, AutoPilot is not supposed to change lanes without explicit confirmation from the driver, which is clearly illustrated on the dashboard/panel.) It's a stretch of road on which I have previously used AutoPilot many times.

gpm 8y ago
Recently I accidentally changed lanes in an intersection. The road had 3 lanes in each direction (+ separate streetcar tracks down the center), including the right lane which was required to turn. I was in the center lane. After the intersection the road was still a 3 lane road - which didn't quite register on my brain. I moved from the center lane to the right lane (figuring it exited so I was supposed to be in the now-right lane) when I should have remained in the center lane. While I am a novice driver, I've gone through that intersection before without blinking or doing the wrong thing. It's not a particularly complicated intersection. Anyways, point is, driving is surprisingly hard. I think counting anecdotes on the internet probably gives you a sample heavily biased against Tesla, because most people don't go post "so I did this stupid thing" but they do post "so my car did this stupid thing".
- alanh 8y ago
  My car and I are stupid in different ways. It has perfect attention and will always brake in time to avoid a fender-bender in traffic. It’s also a lot worse than I am about reading and communicating intent with other drivers or simply recognizing the objects around us.

gwbas1c 8y ago

The video really doesn't convey what happened. It's not really clear when the driver took over. My Model 3 is very odd when going through forks like this. It just has trouble picking one side of the lane or the other. Yesterday I got a little scared when it swerved back and forth while trying to figure out how to take an exit.

jaimex2 8y ago

Warning: Autosteer is intended for use only on highways and limited-access roads with a fully attentive driver. When using Autosteer, hold the steering wheel and be mindful of road conditions and surrounding traffic. Do not use Autosteer on city streets, in construction zones, or in areas where bicyclists or pedestrians may be present. Never depend on Autosteer to determine an appropriate driving path. Always be prepared to take immediate action. Failure to follow these instructions could cause damage, serious injury or death. https://www.tesla.com/content/dam/tesla/Ownership/Own/Model%203%20Owners%20Manual.pdf#page=75 https://www.tesla.com/content/dam/tesla/Ownership/Own/Model%...

kurtisc 8y ago
And then... https://www.businessinsider.com/elon-musk-breaks-tesla-autopilot-rule-2018-12?r=US&IR=T https://www.businessinsider.com/elon-musk-breaks-tesla-autop...

steelframe 8y ago

The day I took delivery of my 2016 Model X with AP 1.0, Tesla announced AP 2.0. A friend of mine immediately ordered a Model X with AP 2.0 and rubbed it in my face. For the entire next year, my AP 1.0 (which is non-Tesla technology -- Mobileye rocks) had no trouble doing adaptive cruise control and lane assist. Meanwhile his AP 2.0 would brake suddenly and swerve all over the place. It took a full year of OTA updates before his AP 2.0 was finally on-par with the functionality that I had the whole time. Of course, by then Tesla pulled a "we're sorry, but the princess is in another castle" and came out with AP 2.5. Now this kind of stuff doesn't matter to me. I got tired of that company's shit and have pulled out of the Teslasphere entirely. I'm now driving a non-Tesla EV, and I'll never look back. I'm also letting my government representatives know that they should support a common EV charging standard and keep Tesla so-called "self-driving" shit off public roads.

dreamcompiler 8y ago

I used to be a big supporter of Tesla, but I now have to apply Internet-of-Shit Rule #1 to them: No software updates ever without explicit owner approval. And "owner" is me, not Elon Musk.

101001001001 8y ago

I still want a model 3. I’ll just not use the autopilot.

sandos 8y ago

What I dont understand is, that yellow sign is actually very visible compared to other things the neural nets can detect. Why hasn't it been done yet? Detecting things as visible as that sign should not be a huge problem.