3 ms·

I have always wondered... Does the AP have some higher-level notion of object permanence, continuity (road behind a horizon or after a curve) and things like th

I have always wondered... Does the AP have some higher-level notion of object permanence, continuity (road behind a horizon or after a curve) and things like that? Does it track a pedestrian that is momentarily hidden behind an obstacle and will probably reemerge in a second or two on the other side? Does it expect that kids may run after that ball that just flew from behind a car? Does it continuously track and improve classification of all objects in the field of vision, with their trajectories and speeds if they are moving? Personally I don't think it does, otherwise it would not erratically slam into clearly visible and marked large objects in its way, or it would be aware of a truck moving in a perpendicular way and so on. I am of opinion that without such higher-level awareness it can never succeed, hope to learn about the state of the self-driving art.

dragontamer 8y ago
I don't think neural networks are wired to "remember" things. In theory, they could be hooked up that way. But your typical convolutional neural network is looking at things frame-by-frame. In theory, ANNs could have an output layer that passes data from one frame to another frame to assist things. But there's no real programming to "hardcode" something like object permanence into an ANN. You pretty much throw a bunch of data into the system and hope for the best.
- hemogloben 8y ago
  NNs are just the first step in the pipeline. Their outputs (detected objects, segmentation, etc) will be piped into other software that builds higher level models. Considering the path-planning requirements I would be absolutely shocked if Autopilot wasn't build history models and estimated paths for objects around the vehicle (other cars etc).
  - JanSolo 8y ago
    Agreed; I imagine they use neural networks to detect and classify objects which are then saved into a scene-graph for use in pathing. I expect what happened was that they trained their NNs for improved detection in one area but unknowingly reduced it in another. Perhaps now it can detect tricycles 99% but road barriers went down to only 30%. Having worked with NNs it's very common to see gains in one domain which come at a cost of reduced performance in another.
    - stevenjohns 8y ago
      They must have known. I haven’t worked with NNs for a few years but I don’t believe the methodology has changed where you would stop testing it over different sets of training data. Barriers are a pretty big part of driving on roads and highways and the only reason it would have been unknowingly reduced would be if they just weren’t testing the NN against data with them.
- robrenaud 8y ago
  There are architectures that use CNN on image inputs, and LSTMs across frames of the video to keep memory. https://arxiv.org/pdf/1609.06377.pdf https://arxiv.org/pdf/1609.06377.pdf
  - x38iq84n 8y ago
    Thanks (to all in this subthread). I've skimmed through this doc, I have watched https://vimeo.com/274274744 https://vimeo.com/274274744 linked from original reddit discussion and I have lost all remaining faith in AP as it is today. This should be in a closed alpha version, not anywhere near paying customers and marketed as FSD ready/feature ready as it's none of that and won't be for many years. I expect there will be at least a generation of Tesla cars sold with FSD-readiness package that will never see FSD in their lifetime.
deleted 8y ago
[deleted]
eclipxe 8y ago
https://www.youtube.com/watch?v=rACZACXgreQ https://www.youtube.com/watch?v=rACZACXgreQ
automathematics 8y ago
In these videos, "Autopilot" is mentioned as the culprit, which seems to be a subset of the features Tesla has. It seems to me there are 3 layers: (level 0: Just Adaptive Cruise Control (use radar to adjust speed up to a max). Human still steers. - 1: Adaptive Cruise Control (use radar to adjust speed up to a max) + Autosteer (camera's watch the lane markers and follow them). This is referred to as "Autopilot" NOTE: This is where the accidents happen. The car isn't driving towards a barrier, it's following the lane markers and hits an error state. This is also NOT self driving. - 2: "Nav on Autopilot". This is an additional function you turn on where the car has more intelligent (use this word loosely) capabilities on highways. The car will still do everything on level 1 combined with lane changes (using cameras to detect objects and trajectories differentiating cars from trucks from pedestrians from bikes from motorcycles etc). It will still follow lane lines, but with a lot of additional information (is there an object? am I merging? am I exiting? etc) - 3: "Full Self Driving". This is an additional package that doesn't exist to the public. Internally I'm sure they're testing the functionality but this is using all of the sensors and algorithms and likely neural networks to decide what to do. A cool point though is that all Tesla's are likely running this code in "shadow mode" where data can be collected and assumptions can be tested without endangering any actual drivers. (see here for some cool data on this: https://electrek.co/2019/03/05/tesla-autopilot-detects-stop-lines-intersections/ https://electrek.co/2019/03/05/tesla-autopilot-detects-stop-...). "Hey, I think the car, if full self driving SHOULD take action X. *compare to what the driver actually does and log the data" over BILLIONS of miles So when a Tesla hits the barrier or gets in an accident, we're actually running "#1" and people start freaking out. But when we get to the capabilities of #3 a lot of the "object permanence and continuity" stuff starts to come into play. Full Disclosure: - I drive a Model 3 every day on Autopilot 75% of the time - I only 75% think self driving is capability under the current Tesla software suite but I bought the package anyways.

dragontamer 8y ago

I don't think neural networks are wired to "remember" things. In theory, they could be hooked up that way. But your typical convolutional neural network is looking at things frame-by-frame. In theory, ANNs could have an output layer that passes data from one frame to another frame to assist things. But there's no real programming to "hardcode" something like object permanence into an ANN. You pretty much throw a bunch of data into the system and hope for the best.

hemogloben 8y ago
NNs are just the first step in the pipeline. Their outputs (detected objects, segmentation, etc) will be piped into other software that builds higher level models. Considering the path-planning requirements I would be absolutely shocked if Autopilot wasn't build history models and estimated paths for objects around the vehicle (other cars etc).
- JanSolo 8y ago
  Agreed; I imagine they use neural networks to detect and classify objects which are then saved into a scene-graph for use in pathing. I expect what happened was that they trained their NNs for improved detection in one area but unknowingly reduced it in another. Perhaps now it can detect tricycles 99% but road barriers went down to only 30%. Having worked with NNs it's very common to see gains in one domain which come at a cost of reduced performance in another.
  - stevenjohns 8y ago
    They must have known. I haven’t worked with NNs for a few years but I don’t believe the methodology has changed where you would stop testing it over different sets of training data. Barriers are a pretty big part of driving on roads and highways and the only reason it would have been unknowingly reduced would be if they just weren’t testing the NN against data with them.
robrenaud 8y ago
There are architectures that use CNN on image inputs, and LSTMs across frames of the video to keep memory. https://arxiv.org/pdf/1609.06377.pdf https://arxiv.org/pdf/1609.06377.pdf
- x38iq84n 8y ago
  Thanks (to all in this subthread). I've skimmed through this doc, I have watched https://vimeo.com/274274744 https://vimeo.com/274274744 linked from original reddit discussion and I have lost all remaining faith in AP as it is today. This should be in a closed alpha version, not anywhere near paying customers and marketed as FSD ready/feature ready as it's none of that and won't be for many years. I expect there will be at least a generation of Tesla cars sold with FSD-readiness package that will never see FSD in their lifetime.

deleted 8y ago

[deleted]

eclipxe 8y ago

https://www.youtube.com/watch?v=rACZACXgreQ https://www.youtube.com/watch?v=rACZACXgreQ

automathematics 8y ago

In these videos, "Autopilot" is mentioned as the culprit, which seems to be a subset of the features Tesla has. It seems to me there are 3 layers: (level 0: Just Adaptive Cruise Control (use radar to adjust speed up to a max). Human still steers. - 1: Adaptive Cruise Control (use radar to adjust speed up to a max) + Autosteer (camera's watch the lane markers and follow them). This is referred to as "Autopilot" NOTE: This is where the accidents happen. The car isn't driving towards a barrier, it's following the lane markers and hits an error state. This is also NOT self driving. - 2: "Nav on Autopilot". This is an additional function you turn on where the car has more intelligent (use this word loosely) capabilities on highways. The car will still do everything on level 1 combined with lane changes (using cameras to detect objects and trajectories differentiating cars from trucks from pedestrians from bikes from motorcycles etc). It will still follow lane lines, but with a lot of additional information (is there an object? am I merging? am I exiting? etc) - 3: "Full Self Driving". This is an additional package that doesn't exist to the public. Internally I'm sure they're testing the functionality but this is using all of the sensors and algorithms and likely neural networks to decide what to do. A cool point though is that all Tesla's are likely running this code in "shadow mode" where data can be collected and assumptions can be tested without endangering any actual drivers. (see here for some cool data on this: https://electrek.co/2019/03/05/tesla-autopilot-detects-stop-lines-intersections/ https://electrek.co/2019/03/05/tesla-autopilot-detects-stop-...). "Hey, I think the car, if full self driving SHOULD take action X. *compare to what the driver actually does and log the data" over BILLIONS of miles So when a Tesla hits the barrier or gets in an accident, we're actually running "#1" and people start freaking out. But when we get to the capabilities of #3 a lot of the "object permanence and continuity" stuff starts to come into play. Full Disclosure: - I drive a Model 3 every day on Autopilot 75% of the time - I only 75% think self driving is capability under the current Tesla software suite but I bought the package anyways.