5 ms·

Full disclosure, I founded a somewhat competing company in the space, comma.ai Why is this progress? Evaluating progress in self driving cars is extremely comp

Full disclosure, I founded a somewhat competing company in the space, comma.ai Why is this progress? Evaluating progress in self driving cars is extremely complex. Why do people trust that the California government is able to do so? A milestone is a working product with many user videos on YouTube. This is a press release from the California DMV.

AlotOfReading 5y ago
Seems like it might be a bit difficult to have user videos on YouTube if you can't legally have users. Getting the permits issued is a good first step.
- georgehotz 5y ago
  Tesla has tons of videos on YouTube from users showcasing similar technology, including in San Francisco. While FSD beta may still have issues, I see no reason to believe that this doesn't also have similar issues.
  - AlotOfReading 5y ago
    I wasn't aware that Tesla was operating commercial autonomous rideshare in SF. Can you link something? I've seen the FSD beta, but that's neither "L4" (whatever you think of the SAE levels) nor commercial. This announcement is specifically about safety drivers and commercial operation, because both Cruise and Waymo have been operating non-commercial fleets with safety drivers for awhile now.
    - gitfan86 5y ago
      Tesla is currently in beta everywhere, not just SF. Tesla is taking the path of improving the technology with driver monitoring, rather than focusing on getting a single city to label them as 'L4'
      - AlotOfReading 5y ago
        Tesla FSD can't operate without a driver and isn't doing commercial rideshare, so it's not relevant in this discussion of a permit to operate a driverless commercial fleet in SF.
        gitfan86 5y ago
        George brought it up in the thread and you responded to him. Tesla and Comma.AI are building the exact same technology. Are you saying that we shouldn't discuss these other companies until they have a permit for SF driving?
        AlotOfReading 5y ago
        George asked why this announcement of the first permit to operate a driverless, commercial fleet is progress, citing Tesla. Unless Tesla has a commercial, driverless fleet somewhere, I'm not sure I understand the objection here. Having the permits to do those things in SF is literally the progress.
        gitfan86 5y ago
        There isn't an objection. There is a question. Can the public see videos of this? If not, why not?
        AlotOfReading 5y ago
        Waymo has been doing commercial driverless ops in Chandler for 4-5 years now, so you can absolutely see videos of that. You just can't see it in SF because until this announcement no one has had the permit to do it commercially, Tesla included.
        gitfan86 5y ago
        There must have been some sort of testing period and approval process that involved showing the cars are safe? Why can't you see that until after this official announcement?
  - throwawaysdcman 5y ago
    I'm sure there are still issues, but I'd bet they're dramatically more rare than what we see with the FSD beta.
    - gitfan86 5y ago
      Why would you think that? If they have much better technology, that they are deploying on public roads, why wouldn't they let the public know with easily accessible proof?
      - jowday 5y ago
        https://youtu.be/sliYTyRpRB8 https://youtu.be/sliYTyRpRB8 Obviously this is cherry picked, but you don’t think a car equipped with LIDAR, imaging radar, more cameras, more compute, and temporally dense HD maps that has focused on the same area for its entire existence would perform better than a Tesla?
        BluSyn 5y ago
        I would expect a strictly mapped system would perform significantly worse in an area like SF, where roads and obstacles frequently change. Human drivers are not rigid, they are adaptable, so building rigid AV systems is doomed to fail outside perfect conditions.
        throwawaysdcman 5y ago
        They're not rigid - they use the map and fall back to onboard system when it's not accurate. That happens plenty of times in the video I linked.
  - ahahahahah 5y ago
    That's like saying the old motorola razr and an iphone 6 are similar technology. While true, it's hiding a huge difference in that "similar".
  - wutbrodo 5y ago
    > Tesla has tons of videos on YouTube from users showcasing similar technology As you know, Tesla's technology 1) lacks lidar and 2) is not targeted towards driverless operation (yet). The AV industry isn't in a place where it's building light AGI that elegantly handles the driving task through a sophisticated understanding of the world. The systems out there are still fit to their fairly near-term objectives; it's ridiculous to assume that Tesla's camera-only, hands-on-the-wheel system tells you which issues (if any) are faced by systems with different objectives built on dramatically different hardware.
rubicon33 5y ago
It's not technical progress, but it's progress nonetheless. There are significant legal / legislative barriers to get through aren't there? This would, at least in my mind, be progress on that front...
- georgehotz 5y ago
  Once software capable of driving at a superhuman level (say 10x the average) exists, the regulatory side will be trivial. What regulator wouldn't want to save 30,000 lives? Sadly, that software is still a ways away. Don't fall for the trap believing that self driving is at all a regulatory problem. It is only a software one.
  - rubicon33 5y ago
    Interesting. I was under the impression that self driving cars are not THAT far away from being a thing. Sounds like it's further than I am thinking.
    - deleted 5y ago
      [deleted]
  - yazaddaruvala 5y ago
    The disconnect seems to be your mental model of self-driving cars might be biased by your current focus. The way I feel most people see it, self-driving cars means three things: 1. As a developer, the hardware and software to accomplish it. 2. As a rider, the improved UX (i.e. safe, comfortable, and fast). 3. As a current-driver, taking over the liability. Based on the Comma 3 launch event, it seems clear that you and Comma.ai are hyper-focused on 1 and 2, but have no plan for 3 anytime soon. I think this is the big disconnect: Cruise and Waymo are focused on 1, 3, and 2 (in that order). DMV approval is entirely in the bucket of 'transferring liability', and this announcement is progress on that front. The chicken and egg problem of 'publicly facing data driven technical viability to then get transfer of liability' and 'transfer of liability to then get data at scale' is a fundamental problem of self-driving. Comma.ai and Tesla are tackling this problem with 100% 'publicly facing data driven technical viability' and 0% 'transfer of liability'. Then one day Comma.ai and Tesla may also try to tackle the 'transfer of liability'. However, why limit the competition? Cruise and Waymo want to start with 'transfer of liability'. Its useful work that needs to eventually be done. Might as well have some companies start early. Hopefully even if Cruise and/or Waymo mess up, the DMV will use those learnings to improve their validation strategies for future players looking to transfer liability (i.e. Comma.ai and Tesla when they choose to be ready).
Forge36 5y ago
It's a regulatory hurdle. Having passed it field tests and product viability can continue
jacquesm 5y ago
I really respect your skills as a hacker but it is actually quite scary that you would not understand that DMV issuing these permits is actually a strong sign of progress, and that 'user videos on YouTube' would somehow be the yardstick by which you believe we should measure such progress. Does the same hold for the FAA? The FDA?
- georgehotz 5y ago
  It depends what your opinions of those agencies are, and if you believe they have technical competence or not. After the 737 MAX fiasco, I've begun to seriously doubt it. "you would not understand that DMV issuing these permits is actually a strong sign of progress" explain to me why you believe this. Videos on YouTube convince me something is real and exists. A government press release convinces me that some process was followed, but perhaps one having nothing to do with the solution to the problem. Self driving is almost entirely a software problem. Testing that software is equally as complex. Do you believe the CA DMV has an army of good software engineers? Data scientists? If not, how did they evaluate it?
  - throwawaysdcman 5y ago
    Assuming your argument is that all you need is cameras, at what point does it become entirely a software problem? Once you have cameras that cover a sufficient FoV around the car? Or can pivot so that they can, like human eyes? How do you know that the cameras you have meet that requirement? Even then, I'm not sure how you can say its always just a software problem when no one has accomplished it yet.
  - jacquesm 5y ago
    Agreed that the 737 MAX did not help, but this is also a failure of all of the other regulatory agencies the world over that rubber stamped the FAA approval instead of doing their own. It certainly looks as though this served as a wake up call - it should! - and that a re-occurrence is relatively unlikely in the near future (until everybody becomes complacent again). Self driving may be a software problem, but that does not necessarily mean that we have the means to make that software a reality today. Given that fact that we know that intelligent beings make plenty of errors and that they are much smarter than your average self driving system, the enormous range of conditions and inputs and the relatively limited amount of computing power that can be thrown at this problem I highly doubt that it is a 'mere matter of software engineering' at this point, taking into account the state of the art. Agreed that testing it is super complex, in fact, I think that testing is the key: without a fully representative test equivalent to a driving test none of this software should be on the street. That's what we agreed upon: pass the driving test and you are allowed to drive. American driving tests are ridiculously simple, so it should be easy to pass such a test and yet that hasn't even been made a mandatory requirement for certifying a self driving solution. Until then I don't think any of this stuff should be on public roads. I don't think the CA DMV needs an army of sofware engineers or data scientists, they are first and foremost a regulatory body that can hire external experts to give them guidance, and I suspect that they genuinely believe that self driving systems from these manufacturers are now solid enough to be given wider latitude during this experimental phase. I personally disagree with the decision, but I do think it is progress, that a regulatory body would dare to attach their name to such a decision rather than to play it safe, this is a clear shift in perception. Time will tell whether or not it was the right decision, if it turns out it wasn't there will be a bigger set-back than this was a step forward.
machinelearning 5y ago
This isn’t technical progress but is regulatory progress. Technical progress is the current limiting factor so I agree that it is more important to emphasize right now. On the other hand, I disagree that user videos on YouTube is a sign of progress. 1. There is an obvious selection bias as only a minority of people with access to the software actually upload videos. 2. The videos are qualitative in nature and it is not feasible for a single human to go through all of them and form an objective opinion on the state of the art. So it’s prone to confirmation bias too. I’d argue that there isn’t a common yardstick of technical progress for game theoretic reasons but I’d love to see one.
ehsankia 5y ago
> A milestone is a working product with many user videos on YouTube So Waymo then? https://www.youtube.com/results?search_query=waymo https://www.youtube.com/results?search_query=waymo

AlotOfReading 5y ago

Seems like it might be a bit difficult to have user videos on YouTube if you can't legally have users. Getting the permits issued is a good first step.

georgehotz 5y ago
Tesla has tons of videos on YouTube from users showcasing similar technology, including in San Francisco. While FSD beta may still have issues, I see no reason to believe that this doesn't also have similar issues.
- AlotOfReading 5y ago
  I wasn't aware that Tesla was operating commercial autonomous rideshare in SF. Can you link something? I've seen the FSD beta, but that's neither "L4" (whatever you think of the SAE levels) nor commercial. This announcement is specifically about safety drivers and commercial operation, because both Cruise and Waymo have been operating non-commercial fleets with safety drivers for awhile now.
  - gitfan86 5y ago
    Tesla is currently in beta everywhere, not just SF. Tesla is taking the path of improving the technology with driver monitoring, rather than focusing on getting a single city to label them as 'L4'
    - AlotOfReading 5y ago
      Tesla FSD can't operate without a driver and isn't doing commercial rideshare, so it's not relevant in this discussion of a permit to operate a driverless commercial fleet in SF.
      - gitfan86 5y ago
        George brought it up in the thread and you responded to him. Tesla and Comma.AI are building the exact same technology. Are you saying that we shouldn't discuss these other companies until they have a permit for SF driving?
        AlotOfReading 5y ago
        George asked why this announcement of the first permit to operate a driverless, commercial fleet is progress, citing Tesla. Unless Tesla has a commercial, driverless fleet somewhere, I'm not sure I understand the objection here. Having the permits to do those things in SF is literally the progress.
        gitfan86 5y ago
        There isn't an objection. There is a question. Can the public see videos of this? If not, why not?
        AlotOfReading 5y ago
        Waymo has been doing commercial driverless ops in Chandler for 4-5 years now, so you can absolutely see videos of that. You just can't see it in SF because until this announcement no one has had the permit to do it commercially, Tesla included.
        gitfan86 5y ago
        There must have been some sort of testing period and approval process that involved showing the cars are safe? Why can't you see that until after this official announcement?
- throwawaysdcman 5y ago
  I'm sure there are still issues, but I'd bet they're dramatically more rare than what we see with the FSD beta.
  - gitfan86 5y ago
    Why would you think that? If they have much better technology, that they are deploying on public roads, why wouldn't they let the public know with easily accessible proof?
    - jowday 5y ago
      https://youtu.be/sliYTyRpRB8 https://youtu.be/sliYTyRpRB8 Obviously this is cherry picked, but you don’t think a car equipped with LIDAR, imaging radar, more cameras, more compute, and temporally dense HD maps that has focused on the same area for its entire existence would perform better than a Tesla?
      - BluSyn 5y ago
        I would expect a strictly mapped system would perform significantly worse in an area like SF, where roads and obstacles frequently change. Human drivers are not rigid, they are adaptable, so building rigid AV systems is doomed to fail outside perfect conditions.
        throwawaysdcman 5y ago
        They're not rigid - they use the map and fall back to onboard system when it's not accurate. That happens plenty of times in the video I linked.
- ahahahahah 5y ago
  That's like saying the old motorola razr and an iphone 6 are similar technology. While true, it's hiding a huge difference in that "similar".
- wutbrodo 5y ago
  > Tesla has tons of videos on YouTube from users showcasing similar technology As you know, Tesla's technology 1) lacks lidar and 2) is not targeted towards driverless operation (yet). The AV industry isn't in a place where it's building light AGI that elegantly handles the driving task through a sophisticated understanding of the world. The systems out there are still fit to their fairly near-term objectives; it's ridiculous to assume that Tesla's camera-only, hands-on-the-wheel system tells you which issues (if any) are faced by systems with different objectives built on dramatically different hardware.

rubicon33 5y ago

It's not technical progress, but it's progress nonetheless. There are significant legal / legislative barriers to get through aren't there? This would, at least in my mind, be progress on that front...

georgehotz 5y ago
Once software capable of driving at a superhuman level (say 10x the average) exists, the regulatory side will be trivial. What regulator wouldn't want to save 30,000 lives? Sadly, that software is still a ways away. Don't fall for the trap believing that self driving is at all a regulatory problem. It is only a software one.
- rubicon33 5y ago
  Interesting. I was under the impression that self driving cars are not THAT far away from being a thing. Sounds like it's further than I am thinking.
  - deleted 5y ago
    [deleted]
- yazaddaruvala 5y ago
  The disconnect seems to be your mental model of self-driving cars might be biased by your current focus. The way I feel most people see it, self-driving cars means three things: 1. As a developer, the hardware and software to accomplish it. 2. As a rider, the improved UX (i.e. safe, comfortable, and fast). 3. As a current-driver, taking over the liability. Based on the Comma 3 launch event, it seems clear that you and Comma.ai are hyper-focused on 1 and 2, but have no plan for 3 anytime soon. I think this is the big disconnect: Cruise and Waymo are focused on 1, 3, and 2 (in that order). DMV approval is entirely in the bucket of 'transferring liability', and this announcement is progress on that front. The chicken and egg problem of 'publicly facing data driven technical viability to then get transfer of liability' and 'transfer of liability to then get data at scale' is a fundamental problem of self-driving. Comma.ai and Tesla are tackling this problem with 100% 'publicly facing data driven technical viability' and 0% 'transfer of liability'. Then one day Comma.ai and Tesla may also try to tackle the 'transfer of liability'. However, why limit the competition? Cruise and Waymo want to start with 'transfer of liability'. Its useful work that needs to eventually be done. Might as well have some companies start early. Hopefully even if Cruise and/or Waymo mess up, the DMV will use those learnings to improve their validation strategies for future players looking to transfer liability (i.e. Comma.ai and Tesla when they choose to be ready).

Forge36 5y ago

It's a regulatory hurdle. Having passed it field tests and product viability can continue

jacquesm 5y ago

I really respect your skills as a hacker but it is actually quite scary that you would not understand that DMV issuing these permits is actually a strong sign of progress, and that 'user videos on YouTube' would somehow be the yardstick by which you believe we should measure such progress. Does the same hold for the FAA? The FDA?

georgehotz 5y ago
It depends what your opinions of those agencies are, and if you believe they have technical competence or not. After the 737 MAX fiasco, I've begun to seriously doubt it. "you would not understand that DMV issuing these permits is actually a strong sign of progress" explain to me why you believe this. Videos on YouTube convince me something is real and exists. A government press release convinces me that some process was followed, but perhaps one having nothing to do with the solution to the problem. Self driving is almost entirely a software problem. Testing that software is equally as complex. Do you believe the CA DMV has an army of good software engineers? Data scientists? If not, how did they evaluate it?
- throwawaysdcman 5y ago
  Assuming your argument is that all you need is cameras, at what point does it become entirely a software problem? Once you have cameras that cover a sufficient FoV around the car? Or can pivot so that they can, like human eyes? How do you know that the cameras you have meet that requirement? Even then, I'm not sure how you can say its always just a software problem when no one has accomplished it yet.
- jacquesm 5y ago
  Agreed that the 737 MAX did not help, but this is also a failure of all of the other regulatory agencies the world over that rubber stamped the FAA approval instead of doing their own. It certainly looks as though this served as a wake up call - it should! - and that a re-occurrence is relatively unlikely in the near future (until everybody becomes complacent again). Self driving may be a software problem, but that does not necessarily mean that we have the means to make that software a reality today. Given that fact that we know that intelligent beings make plenty of errors and that they are much smarter than your average self driving system, the enormous range of conditions and inputs and the relatively limited amount of computing power that can be thrown at this problem I highly doubt that it is a 'mere matter of software engineering' at this point, taking into account the state of the art. Agreed that testing it is super complex, in fact, I think that testing is the key: without a fully representative test equivalent to a driving test none of this software should be on the street. That's what we agreed upon: pass the driving test and you are allowed to drive. American driving tests are ridiculously simple, so it should be easy to pass such a test and yet that hasn't even been made a mandatory requirement for certifying a self driving solution. Until then I don't think any of this stuff should be on public roads. I don't think the CA DMV needs an army of sofware engineers or data scientists, they are first and foremost a regulatory body that can hire external experts to give them guidance, and I suspect that they genuinely believe that self driving systems from these manufacturers are now solid enough to be given wider latitude during this experimental phase. I personally disagree with the decision, but I do think it is progress, that a regulatory body would dare to attach their name to such a decision rather than to play it safe, this is a clear shift in perception. Time will tell whether or not it was the right decision, if it turns out it wasn't there will be a bigger set-back than this was a step forward.

machinelearning 5y ago

This isn’t technical progress but is regulatory progress. Technical progress is the current limiting factor so I agree that it is more important to emphasize right now. On the other hand, I disagree that user videos on YouTube is a sign of progress. 1. There is an obvious selection bias as only a minority of people with access to the software actually upload videos. 2. The videos are qualitative in nature and it is not feasible for a single human to go through all of them and form an objective opinion on the state of the art. So it’s prone to confirmation bias too. I’d argue that there isn’t a common yardstick of technical progress for game theoretic reasons but I’d love to see one.

ehsankia 5y ago

> A milestone is a working product with many user videos on YouTube So Waymo then? https://www.youtube.com/results?search_query=waymo https://www.youtube.com/results?search_query=waymo