8 ms·

> Does anyone have insights on what QA looks like at Tesla for FSD work? Because all of these seem table-stakes before even thinking about releasing the BETA FS

> Does anyone have insights on what QA looks like at Tesla for FSD work? Because all of these seem table-stakes before even thinking about releasing the BETA FSD. Tesla is not exactly in love with QA. Especially for FSD. FSD is mainly 2 things: 1. (By far most important) shareholder value creating promise, that's been solved for 6 years according to their CEO. 2. Software engineering research project What FSD is not is a safety critical systems (which it should be). They focus on cool ML stuff and getting features, with any disregard for how to design, build and test safety critical systems. Validation and QA is basically non-existent.

panick21_ 4y ago
Do you have actual knowlage of Tesla internal QA processes, any kind of source at all? Based on there presentation, they for sure have a whole load of tests, many built directly from real world situation that the car has to handle. They simulate sensor input based on the simulation and check the car does the right thing. They very likely have some internal test drivers and before the software goes public it goes to the cars of the engineers. Those are just some of things we know about. I have no source on their approach to testing safety critical systems, but we do know that they have a lot of software that has based all test by all the major governments. They are one of the few (or only) car maker fully compliant to a number of standards on automated breaking in the US. We have many real world example of videos where other cars would have killed somebody and the Tesla stopped based on image recognition. So they do clearly have some idea of how to do this stuff. So when making these claims I would like to know what they are based on. It might very well be true that their processes are insufficient but I would actual know some real data. Part of what a government could do, is forcing car maker to open their QA processes. Or the government could (should) have its own open test suit that a car needs to be able to handle, but clearly we are not there yet.
- julianlam 4y ago
  > We have many real world example of videos where other cars would have killed somebody and the Tesla stopped based on image recognition. I think you and I must've watched a different video.
  - panick21_ 4y ago
    Yes I have also seen many videos where it makes mistakes. But also many where it prevented them.
- justapassenger 4y ago
  2 sources. 1. I know people working at Tesla. 2. Much more important one - Elon's Twitter feed. They're doing last minute changes, and once it compiles and passes some automated tests, it's tested internally only over few days before it's released to the customers. Even if they had world class internal testing (they don't), for something having to work in such a diverse environment like self driving system without any geo-fencing, those timelines are all you need to know.
  - esalman 4y ago
    Some manufacturers hold off on newer, untested tech for years before adding that to their vehicles. This is what happens when safely is a priority. That's why I bought/will keep buying Toyota/Lexus.
    - vardump 4y ago
      Euro NCAP etc. seem to classify Teslas as (some of) the safest vehicles on roads. https://www.euroncap.com/en/results/tesla/model+y/46618 https://www.euroncap.com/en/results/tesla/model+y/46618 Same for NHTSA: https://www.nhtsa.gov/vehicle/2022/TESLA/MODEL%2525203/4%252520DR/RWD https://www.nhtsa.gov/vehicle/2022/TESLA/MODEL%2525203/4%252...
      - rootusrootus 4y ago
        Because they have a lower center of gravity and good crash structure. These are good things. But avoiding the accident in the first place significantly reduces the need to test that crashworthiness.
        vardump 4y ago
        Tesla was the top one also in avoiding accidents. See Euro NCAP test.
      - deleted 4y ago
        [deleted]
- whamlastxmas 4y ago
  The person above you has no idea what they’re talking about. There’s literally hundreds of people at Tesla whose job is QA and tools to support QA
  - justapassenger 4y ago
    And how does that change anything about my statements? Yeah, they have QA. But for the problem they claim they’re solving (robotaxis) and speed of pushing stuff to customers (on the order of days) it vastly, vastly insufficient. And it lacks any safety lifecycle process regards - again, just look at the timelines. Even if you’re super efficient, you cannot possibly claim you can even such a basic things like proper change management (no, commit message isn’t that) or validation.
    - dylan604 4y ago
      > and speed of pushing stuff to customers (on the order of days) well, if you don't get the software pushed to the QA team (the customers), how else are they going to get it tested?
      - 93po 4y ago
        can we please stop with this disinformation? the customers are not the QA team.
        dylan604 4y ago
        what do you call them? there's no way possible that they can make changes to the software and have them thoroughly vetted before the OTA push. Tesla does not have enough cars owned by the company driving on public roads to vet these changes. The QA team at best can analyze the data received from the customers. That makes the customers the testers in my book.
        93po 4y ago
        the fact that you don't know how tesla vets these changes, very extensively, prior to any physical car receiving the update reinforces that you have no idea what you're talking about tesla does extensive, meaningful vetting of these updates. i'll let you do the research yourself so that maybe you can quit spreading misinformation
- JoshCole 4y ago
  I strongly feel people ought to have these discussions while consistently citing actual data sources relevant to the discussion. For example, did you predict, based on the speculation of Tesla being incompetent with regard to safety, that they have the lowest probability of injury scores of any car manufacturer? Because they do. Did you predict, based on speculation about Elon Musk's incompetence in predicting that self-driving would happen, that there are millions of self-driving miles each quarter? Because there are. Did you predict, based on speculation about Tesla incompetence in full self-driving, that the probability of accident per mile is lower rather than higher in cars that have self-driving capabilities? Because they do. I know this sort of view is very controversial on Hacker News, but I still think it is worth stating, because I think people are actually advocating for policies which kill people because they don't actually know the data disagrees with their assumptions. https://www.tesla.com/VehicleSafetyReport https://www.tesla.com/VehicleSafetyReport
  - freejazz 4y ago
    Please ignore all the times I'm wrong in favor of all the times I'm right!
    - JoshCole 4y ago
      I agree that people who don't cite the evidence are ignoring the evidence? Are you trying to say I'm doing that by pointing to relevant datasets which track the number of accidents and the probability of injury? If so, why are there accidents tracked in the datasets such that the rate can be calculated? This kind of contradicts the claim that I'm asking to ignore, but I definitely agree that other people are ignoring the data if that is what you are trying to say.
      - freejazz 4y ago
        No, your argument is just ridiculous. The standard isn't and shouldn't be how much they get right. It should be what they get wrong and how they do that. I completely disagree with your point, and phrasing it obtusely just makes you obnoxious from a conversational standpoint.

panick21_ 4y ago

Do you have actual knowlage of Tesla internal QA processes, any kind of source at all? Based on there presentation, they for sure have a whole load of tests, many built directly from real world situation that the car has to handle. They simulate sensor input based on the simulation and check the car does the right thing. They very likely have some internal test drivers and before the software goes public it goes to the cars of the engineers. Those are just some of things we know about. I have no source on their approach to testing safety critical systems, but we do know that they have a lot of software that has based all test by all the major governments. They are one of the few (or only) car maker fully compliant to a number of standards on automated breaking in the US. We have many real world example of videos where other cars would have killed somebody and the Tesla stopped based on image recognition. So they do clearly have some idea of how to do this stuff. So when making these claims I would like to know what they are based on. It might very well be true that their processes are insufficient but I would actual know some real data. Part of what a government could do, is forcing car maker to open their QA processes. Or the government could (should) have its own open test suit that a car needs to be able to handle, but clearly we are not there yet.

julianlam 4y ago
> We have many real world example of videos where other cars would have killed somebody and the Tesla stopped based on image recognition. I think you and I must've watched a different video.
- panick21_ 4y ago
  Yes I have also seen many videos where it makes mistakes. But also many where it prevented them.
justapassenger 4y ago
2 sources. 1. I know people working at Tesla. 2. Much more important one - Elon's Twitter feed. They're doing last minute changes, and once it compiles and passes some automated tests, it's tested internally only over few days before it's released to the customers. Even if they had world class internal testing (they don't), for something having to work in such a diverse environment like self driving system without any geo-fencing, those timelines are all you need to know.
- esalman 4y ago
  Some manufacturers hold off on newer, untested tech for years before adding that to their vehicles. This is what happens when safely is a priority. That's why I bought/will keep buying Toyota/Lexus.
  - vardump 4y ago
    Euro NCAP etc. seem to classify Teslas as (some of) the safest vehicles on roads. https://www.euroncap.com/en/results/tesla/model+y/46618 https://www.euroncap.com/en/results/tesla/model+y/46618 Same for NHTSA: https://www.nhtsa.gov/vehicle/2022/TESLA/MODEL%2525203/4%252520DR/RWD https://www.nhtsa.gov/vehicle/2022/TESLA/MODEL%2525203/4%252...
    - rootusrootus 4y ago
      Because they have a lower center of gravity and good crash structure. These are good things. But avoiding the accident in the first place significantly reduces the need to test that crashworthiness.
      - vardump 4y ago
        Tesla was the top one also in avoiding accidents. See Euro NCAP test.
    - deleted 4y ago
      [deleted]
whamlastxmas 4y ago
The person above you has no idea what they’re talking about. There’s literally hundreds of people at Tesla whose job is QA and tools to support QA
- justapassenger 4y ago
  And how does that change anything about my statements? Yeah, they have QA. But for the problem they claim they’re solving (robotaxis) and speed of pushing stuff to customers (on the order of days) it vastly, vastly insufficient. And it lacks any safety lifecycle process regards - again, just look at the timelines. Even if you’re super efficient, you cannot possibly claim you can even such a basic things like proper change management (no, commit message isn’t that) or validation.
  - dylan604 4y ago
    > and speed of pushing stuff to customers (on the order of days) well, if you don't get the software pushed to the QA team (the customers), how else are they going to get it tested?
    - 93po 4y ago
      can we please stop with this disinformation? the customers are not the QA team.
      - dylan604 4y ago
        what do you call them? there's no way possible that they can make changes to the software and have them thoroughly vetted before the OTA push. Tesla does not have enough cars owned by the company driving on public roads to vet these changes. The QA team at best can analyze the data received from the customers. That makes the customers the testers in my book.
        93po 4y ago
        the fact that you don't know how tesla vets these changes, very extensively, prior to any physical car receiving the update reinforces that you have no idea what you're talking about tesla does extensive, meaningful vetting of these updates. i'll let you do the research yourself so that maybe you can quit spreading misinformation
JoshCole 4y ago
I strongly feel people ought to have these discussions while consistently citing actual data sources relevant to the discussion. For example, did you predict, based on the speculation of Tesla being incompetent with regard to safety, that they have the lowest probability of injury scores of any car manufacturer? Because they do. Did you predict, based on speculation about Elon Musk's incompetence in predicting that self-driving would happen, that there are millions of self-driving miles each quarter? Because there are. Did you predict, based on speculation about Tesla incompetence in full self-driving, that the probability of accident per mile is lower rather than higher in cars that have self-driving capabilities? Because they do. I know this sort of view is very controversial on Hacker News, but I still think it is worth stating, because I think people are actually advocating for policies which kill people because they don't actually know the data disagrees with their assumptions. https://www.tesla.com/VehicleSafetyReport https://www.tesla.com/VehicleSafetyReport
- freejazz 4y ago
  Please ignore all the times I'm wrong in favor of all the times I'm right!
  - JoshCole 4y ago
    I agree that people who don't cite the evidence are ignoring the evidence? Are you trying to say I'm doing that by pointing to relevant datasets which track the number of accidents and the probability of injury? If so, why are there accidents tracked in the datasets such that the rate can be calculated? This kind of contradicts the claim that I'm asking to ignore, but I definitely agree that other people are ignoring the data if that is what you are trying to say.
    - freejazz 4y ago
      No, your argument is just ridiculous. The standard isn't and shouldn't be how much they get right. It should be what they get wrong and how they do that. I completely disagree with your point, and phrasing it obtusely just makes you obnoxious from a conversational standpoint.