4 ms·

This is the equivalent to "let's test it in production, what could go wrong?". Also, why wouldn't they be able to test it in a simulator?

deleted 3y ago
[deleted]
lolinder 3y ago
You can't test it in a simulator because half of what you're testing is the interaction with the sensors. You can't simulate sensors with high enough fidelity to be meaningful. And this is more like a staging environment than it is production—production is when you put it in the hands of customers, this is an internal release available only to employees. I'm firmly opposed to FSD beta, but this video doesn't concern me at all.
- tantalor 3y ago
  > can't simulate sensors with high enough fidelity Why not? What's the limiting factor? Video bandwidth? We can run video games at 4k 60fps. Is that not enough?
  - deleted 3y ago
    [deleted]
  - andsoitis 3y ago
    > What's the limiting factor? Even if you could perfectly simulate conditions for the sensors to pick up (incl. precipitation, light, dust) you cannot possibly simulate all possible scenarios that can happen in the real world.
    - tantalor 3y ago
      Most software systems are not tested on "all possible scenarios". Instead you run some basic must-pass scenarios, like the one in the video, where there is a red light in front of the car, and a left turn green, in clear conditions and daylight. If the model doesn't pass, then don't ship it!
- manuelabeledo 3y ago
  > You can't test it in a simulator because half of what you're testing is the interaction with the sensors. Yeah, that's what integration testing is for. I would be surprised if they couldn't test the brains of FSD independently in a simulator. That's just QA 101.
- ra7 3y ago
  > You can't simulate sensors with high enough fidelity to be meaningful. You can if you invest in good simulators. See: https://waymo.com/blog/2021/06/SimulationCity.html https://waymo.com/blog/2021/06/SimulationCity.html > One way we ensure statistical realism in our simulated world is by creating realistic conditions for our autonomous driving technology to experience. So, if the Waymo Driver is going through spring showers in Detroit at sunset in SimulationCity, we can recreate raindrops on our sensors and even simulate other minute details such as the dimming light and solar glare. If they can simulate rain drops, it's definitely possible to simulate traffic lights. They are a basic scenario for a self driving car. What this video highlights is the extreme lack of validation culture within Tesla FSD.
  - lolinder 3y ago
    My point isn't that you can't simulate it at all, my point is that at some point you have to take it out of the simulation and it will behave differently in the real world than it did in the simulated one, because your simulation will never be good enough to account for everything. We can't expect people to never test self-driving cars on public, populated roads unless we are just acknowledging that we don't believe self-driving cars should be a thing, because I sure as hell am not comfortable with the idea of cars going straight from simulation to production.
    - bluefishinit 3y ago
      > I sure as hell am not comfortable with the idea of cars going straight from simulation to production. This is what you're advocating for by saying people should test these things on public roads. That is production, none of the other drivers have consented to be part of the "test".
    - ra7 3y ago
      FSD makes basic errors, the kind that can easily be identified in simulation rather than requiring 450k cars on the road. They don't have to go straight from simulation to production either. That's why all the serious players have closed course testing and safety driver supervised testing.
    - 3y ago
- rob74 3y ago
  Has Tesla built an entire "staging town" where no unsuspecting public is allowed to enter in which they test their cars? No? Then I would definitely call it "testing in production"...
  - tantalor 3y ago
    It's not open to the public, but Aurora has a testing track in Pittsburgh: https://goo.gl/maps/pqvXWptAEWjC918z5 https://goo.gl/maps/pqvXWptAEWjC918z5
biosboiii 3y ago
They already do, it suceeds in most cases, the edge cases are what messes things up. See https://carla.org/ https://carla.org/

deleted 3y ago

[deleted]

lolinder 3y ago

You can't test it in a simulator because half of what you're testing is the interaction with the sensors. You can't simulate sensors with high enough fidelity to be meaningful. And this is more like a staging environment than it is production—production is when you put it in the hands of customers, this is an internal release available only to employees. I'm firmly opposed to FSD beta, but this video doesn't concern me at all.

tantalor 3y ago
> can't simulate sensors with high enough fidelity Why not? What's the limiting factor? Video bandwidth? We can run video games at 4k 60fps. Is that not enough?
- deleted 3y ago
  [deleted]
- andsoitis 3y ago
  > What's the limiting factor? Even if you could perfectly simulate conditions for the sensors to pick up (incl. precipitation, light, dust) you cannot possibly simulate all possible scenarios that can happen in the real world.
  - tantalor 3y ago
    Most software systems are not tested on "all possible scenarios". Instead you run some basic must-pass scenarios, like the one in the video, where there is a red light in front of the car, and a left turn green, in clear conditions and daylight. If the model doesn't pass, then don't ship it!
manuelabeledo 3y ago
> You can't test it in a simulator because half of what you're testing is the interaction with the sensors. Yeah, that's what integration testing is for. I would be surprised if they couldn't test the brains of FSD independently in a simulator. That's just QA 101.
ra7 3y ago
> You can't simulate sensors with high enough fidelity to be meaningful. You can if you invest in good simulators. See: https://waymo.com/blog/2021/06/SimulationCity.html https://waymo.com/blog/2021/06/SimulationCity.html > One way we ensure statistical realism in our simulated world is by creating realistic conditions for our autonomous driving technology to experience. So, if the Waymo Driver is going through spring showers in Detroit at sunset in SimulationCity, we can recreate raindrops on our sensors and even simulate other minute details such as the dimming light and solar glare. If they can simulate rain drops, it's definitely possible to simulate traffic lights. They are a basic scenario for a self driving car. What this video highlights is the extreme lack of validation culture within Tesla FSD.
- lolinder 3y ago
  My point isn't that you can't simulate it at all, my point is that at some point you have to take it out of the simulation and it will behave differently in the real world than it did in the simulated one, because your simulation will never be good enough to account for everything. We can't expect people to never test self-driving cars on public, populated roads unless we are just acknowledging that we don't believe self-driving cars should be a thing, because I sure as hell am not comfortable with the idea of cars going straight from simulation to production.
  - bluefishinit 3y ago
    > I sure as hell am not comfortable with the idea of cars going straight from simulation to production. This is what you're advocating for by saying people should test these things on public roads. That is production, none of the other drivers have consented to be part of the "test".
  - ra7 3y ago
    FSD makes basic errors, the kind that can easily be identified in simulation rather than requiring 450k cars on the road. They don't have to go straight from simulation to production either. That's why all the serious players have closed course testing and safety driver supervised testing.
  - 3y ago
rob74 3y ago
Has Tesla built an entire "staging town" where no unsuspecting public is allowed to enter in which they test their cars? No? Then I would definitely call it "testing in production"...
- tantalor 3y ago
  It's not open to the public, but Aurora has a testing track in Pittsburgh: https://goo.gl/maps/pqvXWptAEWjC918z5 https://goo.gl/maps/pqvXWptAEWjC918z5

biosboiii 3y ago

They already do, it suceeds in most cases, the edge cases are what messes things up. See https://carla.org/ https://carla.org/