22 ms·

Maximise the video and watch the left hand HUD pane from 0:10 to 0:11. The dotted black line coming from the front of the car (which I am assuming is the inten

Maximise the video and watch the left hand HUD pane from 0:10 to 0:11. The dotted black line coming from the front of the car (which I am assuming is the intended route) quickly snaps from straight ahead, to a hair pin right, to a normal right turn. Ignoring the fact that the right turn happened to be into a pedestrian crossing with people on it - what was the car even trying to do? The sat-nav shows it should have just continued forwards. I am astounded that software capable of these outputs is allowed on the roads. When could crossing a junction then taking a hairpin right back across it, ever be the correct thing to do?

andreyk 5y ago
Good catch! That's definitely quite concerning...
taneq 5y ago
On the one hand I applaud Tesla for being so open about what their system is thinking with their visualisations. That could be interpreted to show a deep belief in their system's capabilities. On the other hand, it's always terrified me how jittery any version of AutoPilot's perception of the world is. Would you let your car be driven by someone with zero object permanence, 10/20 vision and only a vague idea of the existence or properties of any object other than lane markings and number plates?
- macNchz 5y ago
  I test drove a Model 3 yesterday and this was something that really jumped out at me. I didn't try any of the automatic driving features, but driving around Brooklyn watching the way the car was perceiving the world around it did not inspire confidence at all. Tesla's over the top marketing and hype seems at once to have been a key ingredient in their success, but also so frustrating because their product is genuinely awesome. I've long been kind of a skeptic but I could not have been more impressed with the test drive I took. It had me ready to buy in about 90 seconds. I wish there were actually-competitive competitors with similar range and power that aren't weird looking SUVs, from brands that don't lean into pure hype.
  - zaptrem 5y ago
    I think the guy above meant to reply to you: > Presumably you are aware that the visualization you see in any retail car is old software, and several iterations behind what the linked video is about (FSD Beta v10). Plus the visualization is quite a bit different (and in many ways simplified) versus what's used for the actual piloting in retail vehicles.
- epistasis 5y ago
  The extremely poor performance of the visualizations are disturbing to me. It's also completely wasted space on the display that I wish were devoted to driving directions instead of telling me what I can already see outside of the car.
  - taneq 5y ago
    I think whether it's wasted space or not depends entirely on the reliability of the system. For a many-9's system which for all intents and purposes isn't going to fail dangerously, I agree, the visualisation is just peacocking. For a beta-quality system, knowing what the car is thinking is an important driver feedback which gives additional warning before it does something really stupid.
    - lanstin 5y ago
      Yeah, you know to pay attention when the lane edges drop out of the console.
- bhauer 5y ago
  Presumably you are aware that the visualization you see in any retail car is old software, and several iterations behind what the linked video is about (FSD Beta v10). Plus the visualization is quite a bit different (and in many ways simplified) versus what's used for the actual piloting in retail vehicles.
- wolverine876 5y ago
  > I applaud Tesla for being so open about what their system is thinking with their visualisations How do you know that what is on the screen matches what the system is 'thinking'? What reason or obligation would Tesla have to engineer accurate visualizations for the real thing and show them to you (remember the definition of accuracy: correct, complete, consistent)? Would Tesla show its customers something that would make them uncomfortable or show Tesla in a bad light, or even question the excitement Tesla hopes to generate? I think it's likely that the display is marketing, not engineering.
- lanstin 5y ago
  And orange cones - the orange cone detection team at Tesla is outstanding. Garbage cans are also pretty accurate. Not sure why they detect garbage cans but not parked cars, but that's just me.
sydd 5y ago
> what was the car even trying to do? I guess it thought that the road ahead is too narrow to continue. But if this is the case why did it simply not stop?
mwint 5y ago
Keep watching the HUD pane - before the driver takes over, it corrects and decides to go straight. In fact, it's turning the wheel to the left before the driver stops the correction. I think this is a navigation issue. This is exactly what I would have done if I had a passenger yell "WAIT TURN RIGHT TURN RIGHT oh nevermind GO STRAIGHT"
- fabatka 5y ago
  What do you mean by "nagivation issue"? I don't see the navigation system changing momentarily and then back (that would be analogous to your example). If in your phrasing the navigation system includes object recognition then if it istructs the car to suddenly steer right without any real reason, how could we trust that it stops before hitting pedestrians? Even if in this situation it would've corrected itself, I wouldn't say that this is a minor issue.
- rwcarlsen 5y ago
  So you're saying the tesla self driving capability is on par with humans driving at near their worst in a panic'd sort of circumstance. Great!
  - mwint 5y ago
    Right, nothing should ever be built unless it can be 100% correct immediately. I’m stunned that it hasn’t, so far as we know, actually hit anything yet. I’m not sure how, but clearly they’re doing something right between choosing drivers and writing software.
    - freeone3000 5y ago
      There were several high publicity collisions with a Tesla colliding into stationary objects -- a highway barricade [1], a blue truck [2], a parked police car[3], two instances of a parked ambulance[4]... Teslas driving themselves hit objects all the time. 1. https://www.google.com/amp/s/abcnews.go.com/amp/Business/tesla-autopilot-steered-driver-barrier-fatal-crash-ntsb/story%3fid=68936725 https://www.google.com/amp/s/abcnews.go.com/amp/Business/tes... 2. Ibid 3. https://www.google.com/amp/s/www.mercurynews.com/2020/07/14/tesla-on-autopilot-hits-police-vehicle-which-hits-ambulance-driver-possibly-drunk-police/amp/ https://www.google.com/amp/s/www.mercurynews.com/2020/07/14/... 4. https://www.google.com/amp/s/amp.cnn.com/cnn/2021/08/30/business/tesla-crash-police-car/index.html https://www.google.com/amp/s/amp.cnn.com/cnn/2021/08/30/busi...
oefrha 5y ago
From the admittedly not much footage of Tesla FSD (1-2 hours total maybe) I’ve watched, it seems to be roughly on par with a student driver who occasionally panics for no apparent reason.
- CameronNemo 5y ago
  Petition to label all Teslas with FSD as student drivers. They need a little LED panel that tells you when the driver is using the AI.
  - misiti3780 5y ago
    I agree - but do you think if they did that the haters would stop hating ?
    - refulgentis 5y ago
      Assuming by hater you mean "someone who is scared by Full Self Driving including cars veering into pedestrians", it'd help me: I'm in an urban area and don't have a car, and I'd be lying if I claimed I wasn't always scared when I see a Tesla approach when I'm in a crosswalk, I'm never sure if Full Self Driving is on or not.
      - misiti3780 5y ago
        "I'd be lying if I claimed I wasn't always scared when I see a Tesla approach when I'm in a crosswalk, I'm never sure if Full Self Driving is on or not." Sure - which is why I said I agreed there should be some indication of it being in FSD or not, originally.
        refulgentis 5y ago
        Sure - which is why I'm answering your question regarding how haters will feel under this proposed intervention, that yes, you've agreed with
      - 5y ago
300bps 5y ago
Speaking as someone who believed all the hype about having fully autonomous cars by 2020, I think the truth is that it is orders of magnitude harder than we thought it was and that we are orders of magnitude less capable than we thought we were.
- sidlls 5y ago
  Something every software engineer should learn before their insufferable egos solidify.
  - lanstin 5y ago
    Maybe a good interview question for experienced developers should be "tell us about a time when you attempted something in software and failed at it." In my case it was being given a 10 or 20 KLOC C server that had memory leaks all over and segfaults all over the place (written by a friend) and told to make it prod ready.
    - sidlls 5y ago
      That's a rough one. Mine was fixing some physics/flight aerodynamics software that had been generated by a Fortran-to-C compiler (e.g., http://www.netlib.org/f2c/ http://www.netlib.org/f2c/). It was in inscrutable mess and couple-years-out-of-academic-physics-turned-SE me thought I'd be able to do it easily since I had so much Fortran and C experience. I failed quite hard at that.
knicholes 5y ago
Maybe their reinforcement learning algorithm allows for a bit of exploration, and that was one of the very unlikely actions for it to take.
- josefresco 5y ago
  Google Maps will update (prompt first) a route in progress depending on traffic. If you don't notice right away it can be very surprising! At least it's not actually driving the car.
- ohgodplsno 5y ago
  You do not run reinforcement learning algorithms with a two ton car on the road, unless you are an absolute psychopath.
  - zaptrem 5y ago
    Or your name is Wayve https://wayve.ai/ https://wayve.ai/
antattack 5y ago
Looking at the video Tesla was probably not closer than 25 feet from any pedestrians. If pedestrians were closer I think their proximity would make the car stop as in other videos. As to abrupt maneuver, EU limits rate of change for steering control and I think it would be wise for US to adopt something similar for driver assist systems.
- zaptrem 5y ago
  This makes evasive maneuvers impossible. EU regulations make it impossible for ADAS to even change lanes half the time. Let’s not go down the rabbit hole of government specifying how machines we haven’t even built yet should work.
  - antattack 5y ago
    Because they are sudden and unpredictable, evasive maneuvers cannot be supervised. It would be unfair to rely on driver to supervise such scenarios. If evasive maneuver is required it would have to be done in fully autonomous way. For the time of the maneuver system would have to be responsible for driving.
- phkahler 5y ago
  >> As to abrupt maneuver, EU limits rate of change for steering control and I think it would be wise for US to adopt something similar for driver assist systems. That's a terrible idea. First it's a bandaid over an underlying problem. Second it's a legislative answer to a technical problem, which is IMHO never a good idea.
  - antattack 5y ago
    Rate limiting is a very valid engineering solution. It's implemented in all kinds of controls. Above everything, the car's driver assist should not be making moves that human does not understand if system requires human to supervise it.
- theslurmmustflo 5y ago
  like this? https://twitter.com/tripgore/status/1311661887533314048?s=21 https://twitter.com/tripgore/status/1311661887533314048?s=21
ggrrhh_ta 5y ago
There is a "no right turn" sign just before it decides to turn right. Could that have anything to do with it?
- Imnimo 5y ago
  There's also a "One way ->" sign pointing to the right, maybe it thinks it's only allowed to go that way?
  - ggrrhh_ta 5y ago
    I see.
  - CyanBird 5y ago
    There are actually two of these, one besides the traffic lights and other one on the traffic lights post where the people where crossing the street Id say that yeah, it seems that the car detected these signs and read them as "you can only go that way" even when the street was open
- notJim 5y ago
  That sign only applies to the left lane. It's to stop people from turning right from the left lane.
GaylordTuring 5y ago
> I am astounded that software capable of these outputs is allowed on the roads. Me too. However, in my encounter with software like this, it usually runs on meat brains.
- belter 5y ago
  I know ...its hard to believe... "They're Made Out of Meat" https://youtu.be/7tScAyNaRdQ https://youtu.be/7tScAyNaRdQ
ChicagoBoy11 5y ago
I've made this argument here a few times and am always shot down, but I think its important to highlight that the airline industry has an extremely robust history of automation and HCI in critical transportation scenarios and it seems to me that all the lessons that we have learned have been chucked out the window with self-driving cars. Being able to effectively reason about what the automation is doing is such an important part of why these technologies have been so successful in flight, and examples like this illustrate how far off we are to something like that in cars. The issue of response time, too, is one we cant ignore, and it is certainly a far greater challenge in automobiles. I don't have answers, but it does seem to me like we are not placing a premium enough on structuring this tech to optimize driver supervision over the driving behavior. Granted, the whole point is to one day NOT HAVE to supervise it all, but at this rate we're going to kill a lot of people until we get there.
- darkmarmot 5y ago
  I recently avoided a startup prospect because they were looking to build a car OS that wasn't hard real-time. The very idea that they're trying to develop such things that might intermittently pause to do some garbage collection is freaking terrifying.
  - spywaregorilla 5y ago
    That doesn't seem especially bad. The car could, for instance, predict whether or not it was safe to do garbage collection. Humans do the same when they decide to look away while driving.
    - toxik 5y ago
      Heck, some humans even do literal garbage collection while driving!
    - giantrobot 5y ago
      A human that looks around while driving is still taking in a lot of real-time input from the environment. Assuming they're not a terrible driver they examined the upcoming environment and made a prediction that at their current speed there were no turns, obstacles, or apparent dangers before looking away. If they didn't fully turn their head they can quickly bring their eyes back to the road in the middle of their task to update their situation. If a GC has to pause the world to do its job there's none of that background processing happening while it's "looking away".
Philip-J-Fry 5y ago
I've watched quite a few FSD videos and in almost every single one the car barely knows what it wants. The route jumps all over the place. I'm pretty sure it's just the nature of their system using cameras and the noisy data that can generate. The sat nav didn't update the route to go that way. The FSD system decided to. It probably completely lost track of the road ahead because of the unmarked section of the road and just locked on to the nearest road it could find, which was that right turn. I've seen videos where it cross over unmarked road and then wants to go into the wrong lane on the other side because it saw it first. It seems like it will just panic search for a road to lock onto because the greatest contributor to their navigation algorithm is just the white lines it can follow.
- angelzen 5y ago
  That could be selection bias. The [edit, was '99%'] vast majority of the time the car does the boring thing are not click-worthy. Have you driven a Tesla for a reasonably long period of time? There is a bigger lesson in there: click-driven video selection creates a very warped view of the world. The video stream is dominated by 'interesting' events, ignoring the boringly mundane that overwhelmingly dominates real life. A few recent panics come to mind.
  - margalabargala 5y ago
    99% perfect is not good enough, it's horrifyingly bad for a car on public roads. When I drive, I don't spend 1 minute plowing into pedestrians for every 1 hour and 39 minutes I spend driving normally. If a FSD Tesla spends just 99% of its time doing boring, non-click-worthy things, that is itself interesting in how dangerous it is. To your point, I'm definitely interested in knowing how many minutes of boring driving tend to elapse between these events. The quantity of these sorts of videos that have been publicized recently gives me the impression that one of these cars would not be able to spend 100 minutes of intervention-free driving around a complex urban environment with pedestrians and construction without probable damage to either a human or an inanimate object.
    - 5y ago
notJim 5y ago
> I am astounded that software capable of these outputs is allowed on the roads. Well, great news for you then! Elon Musk just announced they're adding a button to allow a lot more people to get access to this beta software. https://twitter.com/elonmusk/status/1438751064765906945 https://twitter.com/elonmusk/status/1438751064765906945
modeless 5y ago
The car almost certainly decided that the road ahead was blocked. For context, this release of FSD is the first to use a new obstacle detection technique, and as a result it is the first one that doesn't drive directly into the pillars on that street without perceiving them at all. So it's very likely that this new obstacle detection system glitched out here.
- TheJoeMan 5y ago
  Yeah I’d have to say just stop using FSD underneath a monorail. I haven’t seen as much failures in suburbs
  - modeless 5y ago
    The guy in the video is intentionally stress testing the system by repeatedly driving in an area he knows is poorly handled. But it's totally fair IMO, this is a real road and while it confuses humans too (as evidenced by all the people here claiming that it's illegal to turn right here), FSD must handle it better than this.
    - cptskippy 5y ago
      Considering this data is probably fed back to Tesla, it's probably safe to say this guy is actually helping.
tommymachine 5y ago
Seems like the hairpin portion may have been an animation between the two paths, to smooth the UX? Also, it would give the driver a chance to correct the motion, as happened in the video.
joakleaf 5y ago
As I understood the Tesla AI presentations the path is determined using a Monte Carlo Beamsearch which looks for a feasible path while optimizing an objective that includes minimizing sideways g-forces and keeping the paths derivatives low (smooth path). Form the videos I have seen this fails often (in that the path doesn’t go straight even if it can). Knowing a bit about randomized met heuristics myself, I am not surprised. I think, they need to perform some post processing on these paths (a low iteration local search). I think, they should also start with a guess (like just go straight here, or do a standard turn), and then check if the guess is ok. I think, that could help with a lot of the problems they have.
- joakleaf 5y ago
  For anyone interested the Monto Carlo Tree Search used for the planning (that the tentacle shows) is described here (from the AI day video, at the 1h21m50s): https://youtu.be/j0z4FweCy4M?t=4910 https://youtu.be/j0z4FweCy4M?t=4910
Geee 5y ago
Tesla's system is real time, which means that it makes routing decisions on every frame, and the decisions are independent from previous decisions. It doesn't have memory. It just makes decisions based on what it actually sees with its cameras. It doesn't trust maps, like most other systems. For some reason, it thinks that it must turn right. Situations like these are sent to Tesla to be analyzed, the corresponding data is added to the training data, and the error is corrected in the next versions. This is how the system improves. After this situation is fixed, there will be an another edge case that the HN crowd panics over.
- ra7 5y ago
  > It doesn't have memory. It just makes decisions based on what it actually sees with its cameras. It doesn't trust maps, like most other systems. This is incorrect. Tesla also relies on maps for traffic signs, intersections, stop signs etc. They just don't have additional details like the others do. > After this situation is fixed, there will be an another edge case that the HN crowd panics over. Is anything Tesla FSD can't handle an "edge case" now? It's literally an everyday driving scenario.
  - Geee 5y ago
    It's a specific edge case that the driver was testing. It has issues around those monorail pillars.
    - ra7 5y ago
      Monorail pillars are also not an edge case. Plenty of cities have monorails. Just because FSD doesn't work there doesn't mean it's an edge case.
- tsimionescu 5y ago
  You're probably right, but from your tone you seem to be implying that this is in any way an acceptable way to develop safety critical software, which is a bit baffling.
masklinn 5y ago
> I am astounded that software capable of these outputs is allowed on the roads. 'murica No, seriously, when California decided to increase reporting requirements a few years back Arizona reminded everyone you could run any shitbox with no requirements. That same year, Elaine Herzberg would be run over by an uber self-driving shitbox, in Arizona.
TrainedMonkey 5y ago
Maybe it thought it was a roundabout. Aside from getting a scare, pedestrian risk was likely quite low - there is a separate emergency stop circuit.
sh1mmer 5y ago
My Audi e-tron has this habit of switching the adaptive cruise control to the on-ramp speed limit even when I’m in the middle of the freeway. It’s something I’ve learned to deal with but the sudden attempt to break from 70 to 55 is pretty bad especially as it’s unexpected for other drivers around you. While I’m sure Audi are much worse at updating their software to fix known issues than Tesla I find myself skeptical that the mix of hacks cars use to implement these features scale well, especially in construction zones. Hence I’m pretty content with radar based cruise control and some basic lane maintenance and then doing the rest of the driving myself. I can imagine if my car were slightly better at some of these things I’d be a significantly worse safety driver as I’d start to be lulled into a false sense of security.
seniorgarcia 5y ago
Just to point to another possibility, since the drivers right hand is not in shot, even though his left hand is carefully poised over the steering wheel for the whole shot, it also looks like he pulled hard on the steering wheel with his right hand. That might explain the car trying to re-route on the display, although I'm not even sure it does. To me it looks like the black line disappears into compression artefacts at just the right time. Mostly I think we don't have enough evidence either way and speculating just expresses our desire for the technology to work or not.
- beprogrammed 5y ago
  You can see his right hand reflected in the display, it was just idle during the event.
  - seniorgarcia 5y ago
    Gotta give you that. He still might have moved the steering wheel with his thigh though ;-) Crazy how it handled a shitty situation like around 6:55 in https://streamable.com/grihhc https://streamable.com/grihhc but failed there. Guess that shows what most of the ADAS engineers already know, the really hard part are those last few percentage points to make it actually reliable.
belter 5y ago
The problem shown here: https://twitter.com/robinivski/status/1438580718813261833/photo/1 https://twitter.com/robinivski/status/1438580718813261833/ph... Is the reason why applying Silicon Valley hubris to self driving type problems is not going to work. It's not an exaggeration to say, in 2050 ...We still be beta testing FSD on Mars roads :-)
eloff 5y ago
It seems like basic sanity checking on the proposed actions is not being done. Why not? With safety systems you always want multiple levels of overlapping checks.
Arnavion 5y ago
Computer-vision-based self-driving is a mistake. Self-driving cars should drive on their own roads with no non-self-driving vehicles. Other vehicles, traffic lights, road hazards, route changes, etc should be signals that are directly sent to the vehicles rather than the vehicles relying on seeing and identifying cars and traffic lights and road signs. We know how to make computers navigate maps perfectly. We don't know how to make computers see perfectly. The other uncontrollable humans on the map just make it worse. Yes this'll not work in existing cities. That's fine. That's the point. An inscrutable black box "ML model" that can be fooled by things that humans aren't fooled by should not be in a situation where it can harm humans. I as a pedestrian did not consent to being in danger of being run over by an algorithm that nobody understands and can only tweak. Build new cities that are self-driving first or even self-driving only, where signaling is set up as I wrote in the first paragraph so that a car reliably knows where to drive and how to not drive over people. Take the opportunity to fix the other problems that old cities have like thin roads and not enough walking paths.
ncr100 5y ago
I wonder if it's an upgrade bug related to calibration? There was a video by one Tesla user over the past week that talked about how their Tesla v10 FSD software would attempt to turn into driveways, repeatedly, when driving down a straight road. The user did a recalibration of their cameras, rebooted the car, and the problem went away. https://www.youtube.com/watch?v=A5sbargRd3g https://www.youtube.com/watch?v=A5sbargRd3g
DrammBA 5y ago
At the end of the video (0:32 to 0:33) you can see it quickly snap to right turn again. Why is the car attempting multiple right turns while the map indicates a straight line?
ASalazarMX 5y ago
I don't own or plan to own a Tesla, but if i did, I'm sure I wouldn't use the self driving feature because it seems more inconvenient than simply driving yourself. It's not even a driving assistant, you become the car's assistant. Edit: to be clear, I plan to buy an EV in the future, but I'll drive it myself. Most of my driving is in a 15Km radius, urban, and babysitting the self-drive seems like more trouble than it's worth.
macinjosh 5y ago
Hairpin turn usually denotes a 180deg turn. Not 90deg.
sdrawkcabmai 5y ago
I wonder if gps tracking got messed up. Isn’t he under some type of bridge? That causes gps reflections and other forms of ghosting.

andreyk 5y ago

Good catch! That's definitely quite concerning...

taneq 5y ago

On the one hand I applaud Tesla for being so open about what their system is thinking with their visualisations. That could be interpreted to show a deep belief in their system's capabilities. On the other hand, it's always terrified me how jittery any version of AutoPilot's perception of the world is. Would you let your car be driven by someone with zero object permanence, 10/20 vision and only a vague idea of the existence or properties of any object other than lane markings and number plates?

macNchz 5y ago
I test drove a Model 3 yesterday and this was something that really jumped out at me. I didn't try any of the automatic driving features, but driving around Brooklyn watching the way the car was perceiving the world around it did not inspire confidence at all. Tesla's over the top marketing and hype seems at once to have been a key ingredient in their success, but also so frustrating because their product is genuinely awesome. I've long been kind of a skeptic but I could not have been more impressed with the test drive I took. It had me ready to buy in about 90 seconds. I wish there were actually-competitive competitors with similar range and power that aren't weird looking SUVs, from brands that don't lean into pure hype.
- zaptrem 5y ago
  I think the guy above meant to reply to you: > Presumably you are aware that the visualization you see in any retail car is old software, and several iterations behind what the linked video is about (FSD Beta v10). Plus the visualization is quite a bit different (and in many ways simplified) versus what's used for the actual piloting in retail vehicles.
epistasis 5y ago
The extremely poor performance of the visualizations are disturbing to me. It's also completely wasted space on the display that I wish were devoted to driving directions instead of telling me what I can already see outside of the car.
- taneq 5y ago
  I think whether it's wasted space or not depends entirely on the reliability of the system. For a many-9's system which for all intents and purposes isn't going to fail dangerously, I agree, the visualisation is just peacocking. For a beta-quality system, knowing what the car is thinking is an important driver feedback which gives additional warning before it does something really stupid.
  - lanstin 5y ago
    Yeah, you know to pay attention when the lane edges drop out of the console.
bhauer 5y ago
Presumably you are aware that the visualization you see in any retail car is old software, and several iterations behind what the linked video is about (FSD Beta v10). Plus the visualization is quite a bit different (and in many ways simplified) versus what's used for the actual piloting in retail vehicles.
wolverine876 5y ago
> I applaud Tesla for being so open about what their system is thinking with their visualisations How do you know that what is on the screen matches what the system is 'thinking'? What reason or obligation would Tesla have to engineer accurate visualizations for the real thing and show them to you (remember the definition of accuracy: correct, complete, consistent)? Would Tesla show its customers something that would make them uncomfortable or show Tesla in a bad light, or even question the excitement Tesla hopes to generate? I think it's likely that the display is marketing, not engineering.
lanstin 5y ago
And orange cones - the orange cone detection team at Tesla is outstanding. Garbage cans are also pretty accurate. Not sure why they detect garbage cans but not parked cars, but that's just me.

sydd 5y ago

> what was the car even trying to do? I guess it thought that the road ahead is too narrow to continue. But if this is the case why did it simply not stop?

mwint 5y ago

Keep watching the HUD pane - before the driver takes over, it corrects and decides to go straight. In fact, it's turning the wheel to the left before the driver stops the correction. I think this is a navigation issue. This is exactly what I would have done if I had a passenger yell "WAIT TURN RIGHT TURN RIGHT oh nevermind GO STRAIGHT"

fabatka 5y ago
What do you mean by "nagivation issue"? I don't see the navigation system changing momentarily and then back (that would be analogous to your example). If in your phrasing the navigation system includes object recognition then if it istructs the car to suddenly steer right without any real reason, how could we trust that it stops before hitting pedestrians? Even if in this situation it would've corrected itself, I wouldn't say that this is a minor issue.
rwcarlsen 5y ago
So you're saying the tesla self driving capability is on par with humans driving at near their worst in a panic'd sort of circumstance. Great!
- mwint 5y ago
  Right, nothing should ever be built unless it can be 100% correct immediately. I’m stunned that it hasn’t, so far as we know, actually hit anything yet. I’m not sure how, but clearly they’re doing something right between choosing drivers and writing software.
  - freeone3000 5y ago
    There were several high publicity collisions with a Tesla colliding into stationary objects -- a highway barricade [1], a blue truck [2], a parked police car[3], two instances of a parked ambulance[4]... Teslas driving themselves hit objects all the time. 1. https://www.google.com/amp/s/abcnews.go.com/amp/Business/tesla-autopilot-steered-driver-barrier-fatal-crash-ntsb/story%3fid=68936725 https://www.google.com/amp/s/abcnews.go.com/amp/Business/tes... 2. Ibid 3. https://www.google.com/amp/s/www.mercurynews.com/2020/07/14/tesla-on-autopilot-hits-police-vehicle-which-hits-ambulance-driver-possibly-drunk-police/amp/ https://www.google.com/amp/s/www.mercurynews.com/2020/07/14/... 4. https://www.google.com/amp/s/amp.cnn.com/cnn/2021/08/30/business/tesla-crash-police-car/index.html https://www.google.com/amp/s/amp.cnn.com/cnn/2021/08/30/busi...

oefrha 5y ago

From the admittedly not much footage of Tesla FSD (1-2 hours total maybe) I’ve watched, it seems to be roughly on par with a student driver who occasionally panics for no apparent reason.

CameronNemo 5y ago
Petition to label all Teslas with FSD as student drivers. They need a little LED panel that tells you when the driver is using the AI.
- misiti3780 5y ago
  I agree - but do you think if they did that the haters would stop hating ?
  - refulgentis 5y ago
    Assuming by hater you mean "someone who is scared by Full Self Driving including cars veering into pedestrians", it'd help me: I'm in an urban area and don't have a car, and I'd be lying if I claimed I wasn't always scared when I see a Tesla approach when I'm in a crosswalk, I'm never sure if Full Self Driving is on or not.
    - misiti3780 5y ago
      "I'd be lying if I claimed I wasn't always scared when I see a Tesla approach when I'm in a crosswalk, I'm never sure if Full Self Driving is on or not." Sure - which is why I said I agreed there should be some indication of it being in FSD or not, originally.
      - refulgentis 5y ago
        Sure - which is why I'm answering your question regarding how haters will feel under this proposed intervention, that yes, you've agreed with
    - 5y ago

300bps 5y ago

Speaking as someone who believed all the hype about having fully autonomous cars by 2020, I think the truth is that it is orders of magnitude harder than we thought it was and that we are orders of magnitude less capable than we thought we were.

sidlls 5y ago
Something every software engineer should learn before their insufferable egos solidify.
- lanstin 5y ago
  Maybe a good interview question for experienced developers should be "tell us about a time when you attempted something in software and failed at it." In my case it was being given a 10 or 20 KLOC C server that had memory leaks all over and segfaults all over the place (written by a friend) and told to make it prod ready.
  - sidlls 5y ago
    That's a rough one. Mine was fixing some physics/flight aerodynamics software that had been generated by a Fortran-to-C compiler (e.g., http://www.netlib.org/f2c/ http://www.netlib.org/f2c/). It was in inscrutable mess and couple-years-out-of-academic-physics-turned-SE me thought I'd be able to do it easily since I had so much Fortran and C experience. I failed quite hard at that.

knicholes 5y ago

Maybe their reinforcement learning algorithm allows for a bit of exploration, and that was one of the very unlikely actions for it to take.

josefresco 5y ago
Google Maps will update (prompt first) a route in progress depending on traffic. If you don't notice right away it can be very surprising! At least it's not actually driving the car.
ohgodplsno 5y ago
You do not run reinforcement learning algorithms with a two ton car on the road, unless you are an absolute psychopath.
- zaptrem 5y ago
  Or your name is Wayve https://wayve.ai/ https://wayve.ai/

antattack 5y ago

Looking at the video Tesla was probably not closer than 25 feet from any pedestrians. If pedestrians were closer I think their proximity would make the car stop as in other videos. As to abrupt maneuver, EU limits rate of change for steering control and I think it would be wise for US to adopt something similar for driver assist systems.

zaptrem 5y ago
This makes evasive maneuvers impossible. EU regulations make it impossible for ADAS to even change lanes half the time. Let’s not go down the rabbit hole of government specifying how machines we haven’t even built yet should work.
- antattack 5y ago
  Because they are sudden and unpredictable, evasive maneuvers cannot be supervised. It would be unfair to rely on driver to supervise such scenarios. If evasive maneuver is required it would have to be done in fully autonomous way. For the time of the maneuver system would have to be responsible for driving.
phkahler 5y ago
>> As to abrupt maneuver, EU limits rate of change for steering control and I think it would be wise for US to adopt something similar for driver assist systems. That's a terrible idea. First it's a bandaid over an underlying problem. Second it's a legislative answer to a technical problem, which is IMHO never a good idea.
- antattack 5y ago
  Rate limiting is a very valid engineering solution. It's implemented in all kinds of controls. Above everything, the car's driver assist should not be making moves that human does not understand if system requires human to supervise it.
theslurmmustflo 5y ago
like this? https://twitter.com/tripgore/status/1311661887533314048?s=21 https://twitter.com/tripgore/status/1311661887533314048?s=21

ggrrhh_ta 5y ago

There is a "no right turn" sign just before it decides to turn right. Could that have anything to do with it?

Imnimo 5y ago
There's also a "One way ->" sign pointing to the right, maybe it thinks it's only allowed to go that way?
- ggrrhh_ta 5y ago
  I see.
- CyanBird 5y ago
  There are actually two of these, one besides the traffic lights and other one on the traffic lights post where the people where crossing the street Id say that yeah, it seems that the car detected these signs and read them as "you can only go that way" even when the street was open
notJim 5y ago
That sign only applies to the left lane. It's to stop people from turning right from the left lane.

GaylordTuring 5y ago

> I am astounded that software capable of these outputs is allowed on the roads. Me too. However, in my encounter with software like this, it usually runs on meat brains.

belter 5y ago
I know ...its hard to believe... "They're Made Out of Meat" https://youtu.be/7tScAyNaRdQ https://youtu.be/7tScAyNaRdQ

ChicagoBoy11 5y ago

I've made this argument here a few times and am always shot down, but I think its important to highlight that the airline industry has an extremely robust history of automation and HCI in critical transportation scenarios and it seems to me that all the lessons that we have learned have been chucked out the window with self-driving cars. Being able to effectively reason about what the automation is doing is such an important part of why these technologies have been so successful in flight, and examples like this illustrate how far off we are to something like that in cars. The issue of response time, too, is one we cant ignore, and it is certainly a far greater challenge in automobiles. I don't have answers, but it does seem to me like we are not placing a premium enough on structuring this tech to optimize driver supervision over the driving behavior. Granted, the whole point is to one day NOT HAVE to supervise it all, but at this rate we're going to kill a lot of people until we get there.

darkmarmot 5y ago
I recently avoided a startup prospect because they were looking to build a car OS that wasn't hard real-time. The very idea that they're trying to develop such things that might intermittently pause to do some garbage collection is freaking terrifying.
- spywaregorilla 5y ago
  That doesn't seem especially bad. The car could, for instance, predict whether or not it was safe to do garbage collection. Humans do the same when they decide to look away while driving.
  - toxik 5y ago
    Heck, some humans even do literal garbage collection while driving!
  - giantrobot 5y ago
    A human that looks around while driving is still taking in a lot of real-time input from the environment. Assuming they're not a terrible driver they examined the upcoming environment and made a prediction that at their current speed there were no turns, obstacles, or apparent dangers before looking away. If they didn't fully turn their head they can quickly bring their eyes back to the road in the middle of their task to update their situation. If a GC has to pause the world to do its job there's none of that background processing happening while it's "looking away".

Philip-J-Fry 5y ago

I've watched quite a few FSD videos and in almost every single one the car barely knows what it wants. The route jumps all over the place. I'm pretty sure it's just the nature of their system using cameras and the noisy data that can generate. The sat nav didn't update the route to go that way. The FSD system decided to. It probably completely lost track of the road ahead because of the unmarked section of the road and just locked on to the nearest road it could find, which was that right turn. I've seen videos where it cross over unmarked road and then wants to go into the wrong lane on the other side because it saw it first. It seems like it will just panic search for a road to lock onto because the greatest contributor to their navigation algorithm is just the white lines it can follow.

angelzen 5y ago
That could be selection bias. The [edit, was '99%'] vast majority of the time the car does the boring thing are not click-worthy. Have you driven a Tesla for a reasonably long period of time? There is a bigger lesson in there: click-driven video selection creates a very warped view of the world. The video stream is dominated by 'interesting' events, ignoring the boringly mundane that overwhelmingly dominates real life. A few recent panics come to mind.
- margalabargala 5y ago
  99% perfect is not good enough, it's horrifyingly bad for a car on public roads. When I drive, I don't spend 1 minute plowing into pedestrians for every 1 hour and 39 minutes I spend driving normally. If a FSD Tesla spends just 99% of its time doing boring, non-click-worthy things, that is itself interesting in how dangerous it is. To your point, I'm definitely interested in knowing how many minutes of boring driving tend to elapse between these events. The quantity of these sorts of videos that have been publicized recently gives me the impression that one of these cars would not be able to spend 100 minutes of intervention-free driving around a complex urban environment with pedestrians and construction without probable damage to either a human or an inanimate object.
  - 5y ago

notJim 5y ago

> I am astounded that software capable of these outputs is allowed on the roads. Well, great news for you then! Elon Musk just announced they're adding a button to allow a lot more people to get access to this beta software. https://twitter.com/elonmusk/status/1438751064765906945 https://twitter.com/elonmusk/status/1438751064765906945

modeless 5y ago

The car almost certainly decided that the road ahead was blocked. For context, this release of FSD is the first to use a new obstacle detection technique, and as a result it is the first one that doesn't drive directly into the pillars on that street without perceiving them at all. So it's very likely that this new obstacle detection system glitched out here.

TheJoeMan 5y ago
Yeah I’d have to say just stop using FSD underneath a monorail. I haven’t seen as much failures in suburbs
- modeless 5y ago
  The guy in the video is intentionally stress testing the system by repeatedly driving in an area he knows is poorly handled. But it's totally fair IMO, this is a real road and while it confuses humans too (as evidenced by all the people here claiming that it's illegal to turn right here), FSD must handle it better than this.
  - cptskippy 5y ago
    Considering this data is probably fed back to Tesla, it's probably safe to say this guy is actually helping.

tommymachine 5y ago

Seems like the hairpin portion may have been an animation between the two paths, to smooth the UX? Also, it would give the driver a chance to correct the motion, as happened in the video.

joakleaf 5y ago

As I understood the Tesla AI presentations the path is determined using a Monte Carlo Beamsearch which looks for a feasible path while optimizing an objective that includes minimizing sideways g-forces and keeping the paths derivatives low (smooth path). Form the videos I have seen this fails often (in that the path doesn’t go straight even if it can). Knowing a bit about randomized met heuristics myself, I am not surprised. I think, they need to perform some post processing on these paths (a low iteration local search). I think, they should also start with a guess (like just go straight here, or do a standard turn), and then check if the guess is ok. I think, that could help with a lot of the problems they have.

joakleaf 5y ago
For anyone interested the Monto Carlo Tree Search used for the planning (that the tentacle shows) is described here (from the AI day video, at the 1h21m50s): https://youtu.be/j0z4FweCy4M?t=4910 https://youtu.be/j0z4FweCy4M?t=4910

Geee 5y ago

Tesla's system is real time, which means that it makes routing decisions on every frame, and the decisions are independent from previous decisions. It doesn't have memory. It just makes decisions based on what it actually sees with its cameras. It doesn't trust maps, like most other systems. For some reason, it thinks that it must turn right. Situations like these are sent to Tesla to be analyzed, the corresponding data is added to the training data, and the error is corrected in the next versions. This is how the system improves. After this situation is fixed, there will be an another edge case that the HN crowd panics over.

ra7 5y ago
> It doesn't have memory. It just makes decisions based on what it actually sees with its cameras. It doesn't trust maps, like most other systems. This is incorrect. Tesla also relies on maps for traffic signs, intersections, stop signs etc. They just don't have additional details like the others do. > After this situation is fixed, there will be an another edge case that the HN crowd panics over. Is anything Tesla FSD can't handle an "edge case" now? It's literally an everyday driving scenario.
- Geee 5y ago
  It's a specific edge case that the driver was testing. It has issues around those monorail pillars.
  - ra7 5y ago
    Monorail pillars are also not an edge case. Plenty of cities have monorails. Just because FSD doesn't work there doesn't mean it's an edge case.
tsimionescu 5y ago
You're probably right, but from your tone you seem to be implying that this is in any way an acceptable way to develop safety critical software, which is a bit baffling.

masklinn 5y ago

> I am astounded that software capable of these outputs is allowed on the roads. 'murica No, seriously, when California decided to increase reporting requirements a few years back Arizona reminded everyone you could run any shitbox with no requirements. That same year, Elaine Herzberg would be run over by an uber self-driving shitbox, in Arizona.

TrainedMonkey 5y ago

Maybe it thought it was a roundabout. Aside from getting a scare, pedestrian risk was likely quite low - there is a separate emergency stop circuit.

sh1mmer 5y ago

My Audi e-tron has this habit of switching the adaptive cruise control to the on-ramp speed limit even when I’m in the middle of the freeway. It’s something I’ve learned to deal with but the sudden attempt to break from 70 to 55 is pretty bad especially as it’s unexpected for other drivers around you. While I’m sure Audi are much worse at updating their software to fix known issues than Tesla I find myself skeptical that the mix of hacks cars use to implement these features scale well, especially in construction zones. Hence I’m pretty content with radar based cruise control and some basic lane maintenance and then doing the rest of the driving myself. I can imagine if my car were slightly better at some of these things I’d be a significantly worse safety driver as I’d start to be lulled into a false sense of security.

seniorgarcia 5y ago

Just to point to another possibility, since the drivers right hand is not in shot, even though his left hand is carefully poised over the steering wheel for the whole shot, it also looks like he pulled hard on the steering wheel with his right hand. That might explain the car trying to re-route on the display, although I'm not even sure it does. To me it looks like the black line disappears into compression artefacts at just the right time. Mostly I think we don't have enough evidence either way and speculating just expresses our desire for the technology to work or not.

beprogrammed 5y ago
You can see his right hand reflected in the display, it was just idle during the event.
- seniorgarcia 5y ago
  Gotta give you that. He still might have moved the steering wheel with his thigh though ;-) Crazy how it handled a shitty situation like around 6:55 in https://streamable.com/grihhc https://streamable.com/grihhc but failed there. Guess that shows what most of the ADAS engineers already know, the really hard part are those last few percentage points to make it actually reliable.

belter 5y ago

The problem shown here: https://twitter.com/robinivski/status/1438580718813261833/photo/1 https://twitter.com/robinivski/status/1438580718813261833/ph... Is the reason why applying Silicon Valley hubris to self driving type problems is not going to work. It's not an exaggeration to say, in 2050 ...We still be beta testing FSD on Mars roads :-)

eloff 5y ago

It seems like basic sanity checking on the proposed actions is not being done. Why not? With safety systems you always want multiple levels of overlapping checks.

Arnavion 5y ago

Computer-vision-based self-driving is a mistake. Self-driving cars should drive on their own roads with no non-self-driving vehicles. Other vehicles, traffic lights, road hazards, route changes, etc should be signals that are directly sent to the vehicles rather than the vehicles relying on seeing and identifying cars and traffic lights and road signs. We know how to make computers navigate maps perfectly. We don't know how to make computers see perfectly. The other uncontrollable humans on the map just make it worse. Yes this'll not work in existing cities. That's fine. That's the point. An inscrutable black box "ML model" that can be fooled by things that humans aren't fooled by should not be in a situation where it can harm humans. I as a pedestrian did not consent to being in danger of being run over by an algorithm that nobody understands and can only tweak. Build new cities that are self-driving first or even self-driving only, where signaling is set up as I wrote in the first paragraph so that a car reliably knows where to drive and how to not drive over people. Take the opportunity to fix the other problems that old cities have like thin roads and not enough walking paths.

ncr100 5y ago

I wonder if it's an upgrade bug related to calibration? There was a video by one Tesla user over the past week that talked about how their Tesla v10 FSD software would attempt to turn into driveways, repeatedly, when driving down a straight road. The user did a recalibration of their cameras, rebooted the car, and the problem went away. https://www.youtube.com/watch?v=A5sbargRd3g https://www.youtube.com/watch?v=A5sbargRd3g

DrammBA 5y ago

At the end of the video (0:32 to 0:33) you can see it quickly snap to right turn again. Why is the car attempting multiple right turns while the map indicates a straight line?

ASalazarMX 5y ago

I don't own or plan to own a Tesla, but if i did, I'm sure I wouldn't use the self driving feature because it seems more inconvenient than simply driving yourself. It's not even a driving assistant, you become the car's assistant. Edit: to be clear, I plan to buy an EV in the future, but I'll drive it myself. Most of my driving is in a 15Km radius, urban, and babysitting the self-drive seems like more trouble than it's worth.

macinjosh 5y ago

Hairpin turn usually denotes a 180deg turn. Not 90deg.

sdrawkcabmai 5y ago

I wonder if gps tracking got messed up. Isn’t he under some type of bridge? That causes gps reflections and other forms of ghosting.