5 ms·

I'm genuinely curious why you think Tesla is ahead compared to Waymo and Cruise. Autopilot struggles in my car on some fairly boring roads, while Waymo and Crui

I'm genuinely curious why you think Tesla is ahead compared to Waymo and Cruise. Autopilot struggles in my car on some fairly boring roads, while Waymo and Cruise are both operating real taxi services in vehicles without drivers. I can understand the argument that Tesla has lots of data from real world driving. But Google also has fleets of cars mapping out every road in the world.

cs702 4y ago
Please keep in mind that I'm talking about FSD Beta, not the current production software, which is dozens of versions behind. Are you on Beta 10.69.2.3? If I had to articulate my reasons: * Judging by the videos available online, my perception is that many situations that were impossible for Tesla FSD Beta a year ago have become uneventful in recent weeks. Take a look at Chuck Cook's videos for example (I like the fact that he always highlights the failures). * Judging again by the videos available online, my perception is that Tesla FSD Beta has encountered and had to deal with more crazy edge cases than any other system. A possible explanation for this is that for a long time Tesla FSD Beta hasn't been geofenced or restricted only to certain types of roads, like highways. You can test it anywhere in North America. * Tesla FSD Beta currently has 160,000 individuals testing it without road restrictions. As far as I know, no other system has been exposed to similar open-ended large-scale testing. * Occupancy networks look like a real breakthrough to me -- DNNs that predict whether each voxel in a 3D model is occupied by an object, using only video data as an input. I understood the high-level explanation of these DNNs on AI Day 2. I haven't seen anything like it from anyone else. * Tesla's DOJO also looks like a breakthrough to me. I understood the high-level explanation of it on AI Day 2. IIRC, DOJO cabinets are 6x faster at training existing neural networks than Nvidia rigs, at 6x lower cost, so call it ~36x more efficient.
- dontreact 4y ago
  I'd sum up your points 1,2,3 as "more data". This would be a reason to think they can one day be ahead if they can take advantage of this, but not evidence that they are currently ahead. Occupancy networks: waymo has published research on this before Tesla announced this at AI day (not clear to me who got there first though https://arxiv.org/pdf/2203.03875v1.pdf https://arxiv.org/pdf/2203.03875v1.pdf) Tesla's Dojo -> Waymo has TPUs to train on To me all of this is outweighed by the fact that Waymo has a driverless deployment and Tesla does not. I am pretty biased because as a Tesla owner I am pretty pissed off at this point at how the false positives on the system in detecting close following are stopping my safety score from getting high enough to even be able to access the product I purchased. But it is pretty hard to say one way or another.
  - cs702 4y ago
    > I'd sum up your points 1,2,3 as "more data". This would be a reason to think they can one day be ahead if they can take advantage of this, but not evidence that they are currently ahead. I'd sum up those three points as "more data and more real-world, open-ended, large-scale testing by regular people." Big difference. > Occupancy networks: waymo has published research on this before Tesla announced this at AI day (not clear to me who got there first though https://arxiv.org/pdf/2203.03875v1.pdf https://arxiv.org/pdf/2203.03875v1.pdf) AFAIK, Tesla FSD Beta is the only system that has been using these DNNs for open-ended testing. > Tesla's Dojo -> Waymo has TPUs to train on I've trained AI models on TPUs. They're nowhere near close to 36x more efficient than Nvidia GPUs. > I am pretty biased because as a Tesla owner I am pretty pissed off at this point at how the false positives on the system in detecting close following are stopping my safety score from getting high enough to even be able to access the product I purchased. Oh, I get your frustration... but I also understand why Tesla is being so strict with safety scores at this point. It wouldn't be fair to blame them for that.
    - itsoktocry 4y ago
      You think it's actually "safety score" and not Tesla protecting their brand by restricting who gets to demonstrate the system? Maybe you should consider that when watching YouTube videos of people using it....
    - dontreact 4y ago
      TPUs are hard to use outside of Google, (I have tried in and out of Google). I think the situation is improving, but the efficiency from using a large pod is really remarkable. What topology did you train AI models on? Within Google it's common to train across a whole pod or even across multiple pods 8x16x16 is the largest currently. Also if Tesla actually published numbers on an MlPerf benchmark, I would be more inclined to believe claims about 36x better efficiency. https://mlcommons.org/en/training-normal-20/ https://mlcommons.org/en/training-normal-20/ The fastest times I'm seeing here for image classification and for object detection (not the same, but probably closest proxy out of the tasks benchmarked) are for TPUs. To know who has better training technology I don't think you should be using a cost-efficiency metric, it seems to me the best thing to use would be who can train networks the fastest. Cost metrics are easy to game especially if you are the ones making the chips (Of course them making chips is cheaper than buying Nvidia chips for them once the capital investment is made). To measure who is ahead in technology, I think you have to look at who can train models the fastest, and right now as far as I can tell, TPUs are unbeat for this. (Although practically speaking it's hard to pull off these large topology things externally and there are also other caveats with ML perf related to how the training setups are optimized, but nonetheless, it's a better signal than what Elon says in a presentation :) )
- cypress66 4y ago
  As someone who also likes to watch Chuck Cook, I don't think Tesla is close to waymo. Tesla fsd in its current state will either crash or do some serious fuck up if you let it unattended for a few hours or maybe less (based on the disengagements in those videos). Forget about driverless Tesla with the current fsd. Waymo has been operating driverless since 2019. I do agree that it is progressing very nicely. Imo tesla fsd needs 2 more years and a hardware update and it will be there.
  - cs702 4y ago
    I've yet to see how Waymo and other self-driving systems perform in open-ended testing, outside tightly restricted, geofenced environments. Otherwise, I agree that Tesla FSD Beta has been progressing nicely. I don't know if it will take 1, 2, or 5 years to get FSD Beta to an acceptable rate of graceful failures, but I agree it looks likely to get there before the end of the decade!
    - Veserv 4y ago
      I've yet to see Tesla FSD Beta perform in any meaningful operational environment, no matter how restricted, within a factor of 100x of Waymo or Cruise, and Waymo and Cruise are still at least a factor of 10x off of "equal to human drivers". Superiority can not be claimed if a system is unacceptably bad in all circumstances (i.e. 1,000x worse than the minimum acceptable standard for real commercial usage) and worse in every case than a comparable alternative. It only makes sense if the system is better at something other than being allowed to create unacceptably bad results in a more diverse range of circumstances.
      - panick21_ 4y ago
        Tesla has not focused on any individual area to any large extent so why would you measure them on that? That not the problem they are trying to solve and its not the problem that worth investing 30 billion for. > within a factor of 100x of Waymo What's your evidence for this? Is there anybody who has done a systematic comparison of Tesla performance in Arizon zone of Waymo?
- jeffbee 4y ago
  I really feel that you have been duped here. There is no reason to believe that Tesla Dojo even exists. At Hot Chips last year they showed some 3D renders of their supposed board. At Hot Chips this year they showed the same renders. At "AI Day" last month they showed a retarded humanoid robot. We have no basis to conclude that Dojo does, can, or will exist.
  - mrb 4y ago
    Dojo does exist. There are real-world pictures of it: https://media.datacenterdynamics.com/media/images/training_tiles_III.original.png https://media.datacenterdynamics.com/media/images/training_t... source: https://www.datacenterdynamics.com/en/news/tesla-details-dojo-supercomputer-reveals-dojo-d1-chip-and-training-tile-module/ https://www.datacenterdynamics.com/en/news/tesla-details-doj... However, to my knowledge, Tesla is still in the "bring up" step. I think that was a mistake to pack 25 dies on a single system-on-wafer. Very hard to cool.
    - itsoktocry 4y ago
      That's just a picture of a testing rig set up on a workbench. It's the same picture in both articles. Classic Tesla. Do you think the minisub and Tesla ventilators also existed?
- itsoktocry 4y ago
  >haven't seen anything like it from anyone else. Have you considered that other companies don't make it a priority to market these things? Elon knows his audience: people who will go on message boards and talk about it. Most people don't care about the underlying AI tech. Do you think the other companies aren't making any breakthroughs? How do they have Robotaxis then? Your entire claimed expertise seems to come from YouTube promotional videos. Maybe take a step back from marketing hype.
badpun 4y ago
Compared to general self-driving, it's relatively simple to make self-driving taxis work in just a couple of cities (preferably ones with minimum adversarial weather, too) - you just test the code against that particular dataset until it performes reasonably well, manually fixing the edge cases along the way if need be. It's still a monumental, multi-billion dollar project, but I would be surprised if it wasn't achievable.
- bastawhiz 4y ago
  Do you know that their tech is specifically optimized for those cities and won't work well elsewhere? Or is that speculation? As far as I'm aware, unless you were able to compare the performance of FSD against Waymo on an arbitrary road, you can't really make that argument.
  - badpun 4y ago
    I'm not talking about any specific tech solution. I'm merely pointing out that getting a self-driving taxi to work in a single city of your choice is a vastly simpler problem than general FSD.
Initial_BP 4y ago
Not sure about the current state of the actual ML, but compared to other self driving companies Tesla has a treasure trove of data because they have so many vehicles on the road at all hours of every day. The edge cases are the parts that are hard to identify and solve so having all that drive time data to identify edge cases would seem to give them a big advantage.
- jboy55 4y ago
  If the data doesn't have the details required to build accurate models, then the data is just costing Tesla money. Since Tesla's are just cameras only, with telemetry, they can replay scenarios with existing roads, but what happens when someone cones off half of the road?
  - Initial_BP 4y ago
    I imagine they have a way to replay incidents from their cars in a simulation and even if it's not super accurate, they could likely look at camera data and rebuild a similar situation (Sim or real life to identify and test the edge case) I work at another autonomous car company (as a security engineer not ML related work) and I know we have a Lot of simulated situations that we run the ML against and add more from situations collected from actual driving.
- teyc 4y ago
  Most self-driving is about avoiding collissions, and signalling intent, especially when streets are narrow and there's merging or shared use. The physics of cars, people, bikes and kids around roads are well understood (acceleration, velocity). This can be simulated, and a game engine can generate data for virtual sensors to be trained. There's no reason to require time on the road.
  - cptskippy 4y ago
    But you'll never be able to come up with all of the possible scenarios to simulate. What Tesla has demonstrated is creating virtual scenarios where they can dynamic adjust all factors (light, weather, traffic, etc) and base them off real world situations they've encountered where their Model failed.
Salgat 4y ago
Aren't Waymo and Cruise constrained to specially tailored cities? Autonomous driving that can be activated literally anywhere in the country is far more impressive, even if it has cases where it doesn't perform as well.
- birdyrooster 4y ago
  The more far flung reaches are are served well by traditional vehicles and the people are trained to use driving machines since most of the workers are agricultural. The city centers and suburbs are a big enough win even if it isn't as magical as taking a self driving trip from a small cottage in northern Vermont all the way to a campsite up in Big Sur.
  - kortex 4y ago
    I'm in Upstate New York and despite my penchant for gardening, I'm hardly an agricultural worker. Any urban-geofenced product is useless to me and the half a million folks in the greater capitol area and nearby counties aside from the tricity residents. Which is at least 50% of the folks. Hardly a passing grade, let alone big win.
- bastawhiz 4y ago
  I have a double friends with FSD that rarely use it because it "scares [them]" by occasionally making dangerous decisions. Surely Waymo or Cruise could launch anywhere, but they make the conscious choice not to so they avoid this exact problem.
- itsoktocry 4y ago
  To me actual driverless cars are more impressive than FSD which doesn't allow me to take my hands off the wheel, yes.
  - panick21_ 4y ago
    They are driverless but have a remote operator that can jump in at any time. Strictly speaking driverless cars are not actually hard to make. As long as you have a good data connection you can remote drive a car just fine.
oceanplexian 4y ago
They’re operating Taxi services in San Francisco. A city that doesn’t experience any real-world weather, with an area of like 50 square miles where speeds generally never exceed 25MPH. They also have humans watching cameras that take over when the self driving breaks down. It’s a completely different problem space, like claiming someone built a train and therefore they can easily build self driving cars since they are both “driverless”.
- bastawhiz 4y ago
  Yes, but you're choosing only one metric to evaluate on. Waymo/Cruise are level 4+. Anecdotally (does anyone have comparable data?), they also have a much lower accident rate. Solving a problem partially for all conditions and areas rather than ~completely for a specific but large area and set of conditions doesn't seem like it puts you meaningfully ahead. Edit: and surely Waymo/Cruise could launch everywhere with performance that's lower than their current launch cities, but they choose not to. I don't think there's any compelling reason to assume their tech doesn't work outside of SF or Arizona or wherever, they just don't want to be in the news for their cars plowing someone into a highway divider or running over a pedestrian.

cs702 4y ago

Please keep in mind that I'm talking about FSD Beta, not the current production software, which is dozens of versions behind. Are you on Beta 10.69.2.3? If I had to articulate my reasons: * Judging by the videos available online, my perception is that many situations that were impossible for Tesla FSD Beta a year ago have become uneventful in recent weeks. Take a look at Chuck Cook's videos for example (I like the fact that he always highlights the failures). * Judging again by the videos available online, my perception is that Tesla FSD Beta has encountered and had to deal with more crazy edge cases than any other system. A possible explanation for this is that for a long time Tesla FSD Beta hasn't been geofenced or restricted only to certain types of roads, like highways. You can test it anywhere in North America. * Tesla FSD Beta currently has 160,000 individuals testing it without road restrictions. As far as I know, no other system has been exposed to similar open-ended large-scale testing. * Occupancy networks look like a real breakthrough to me -- DNNs that predict whether each voxel in a 3D model is occupied by an object, using only video data as an input. I understood the high-level explanation of these DNNs on AI Day 2. I haven't seen anything like it from anyone else. * Tesla's DOJO also looks like a breakthrough to me. I understood the high-level explanation of it on AI Day 2. IIRC, DOJO cabinets are 6x faster at training existing neural networks than Nvidia rigs, at 6x lower cost, so call it ~36x more efficient.

dontreact 4y ago
I'd sum up your points 1,2,3 as "more data". This would be a reason to think they can one day be ahead if they can take advantage of this, but not evidence that they are currently ahead. Occupancy networks: waymo has published research on this before Tesla announced this at AI day (not clear to me who got there first though https://arxiv.org/pdf/2203.03875v1.pdf https://arxiv.org/pdf/2203.03875v1.pdf) Tesla's Dojo -> Waymo has TPUs to train on To me all of this is outweighed by the fact that Waymo has a driverless deployment and Tesla does not. I am pretty biased because as a Tesla owner I am pretty pissed off at this point at how the false positives on the system in detecting close following are stopping my safety score from getting high enough to even be able to access the product I purchased. But it is pretty hard to say one way or another.
- cs702 4y ago
  > I'd sum up your points 1,2,3 as "more data". This would be a reason to think they can one day be ahead if they can take advantage of this, but not evidence that they are currently ahead. I'd sum up those three points as "more data and more real-world, open-ended, large-scale testing by regular people." Big difference. > Occupancy networks: waymo has published research on this before Tesla announced this at AI day (not clear to me who got there first though https://arxiv.org/pdf/2203.03875v1.pdf https://arxiv.org/pdf/2203.03875v1.pdf) AFAIK, Tesla FSD Beta is the only system that has been using these DNNs for open-ended testing. > Tesla's Dojo -> Waymo has TPUs to train on I've trained AI models on TPUs. They're nowhere near close to 36x more efficient than Nvidia GPUs. > I am pretty biased because as a Tesla owner I am pretty pissed off at this point at how the false positives on the system in detecting close following are stopping my safety score from getting high enough to even be able to access the product I purchased. Oh, I get your frustration... but I also understand why Tesla is being so strict with safety scores at this point. It wouldn't be fair to blame them for that.
  - itsoktocry 4y ago
    You think it's actually "safety score" and not Tesla protecting their brand by restricting who gets to demonstrate the system? Maybe you should consider that when watching YouTube videos of people using it....
  - dontreact 4y ago
    TPUs are hard to use outside of Google, (I have tried in and out of Google). I think the situation is improving, but the efficiency from using a large pod is really remarkable. What topology did you train AI models on? Within Google it's common to train across a whole pod or even across multiple pods 8x16x16 is the largest currently. Also if Tesla actually published numbers on an MlPerf benchmark, I would be more inclined to believe claims about 36x better efficiency. https://mlcommons.org/en/training-normal-20/ https://mlcommons.org/en/training-normal-20/ The fastest times I'm seeing here for image classification and for object detection (not the same, but probably closest proxy out of the tasks benchmarked) are for TPUs. To know who has better training technology I don't think you should be using a cost-efficiency metric, it seems to me the best thing to use would be who can train networks the fastest. Cost metrics are easy to game especially if you are the ones making the chips (Of course them making chips is cheaper than buying Nvidia chips for them once the capital investment is made). To measure who is ahead in technology, I think you have to look at who can train models the fastest, and right now as far as I can tell, TPUs are unbeat for this. (Although practically speaking it's hard to pull off these large topology things externally and there are also other caveats with ML perf related to how the training setups are optimized, but nonetheless, it's a better signal than what Elon says in a presentation :) )
cypress66 4y ago
As someone who also likes to watch Chuck Cook, I don't think Tesla is close to waymo. Tesla fsd in its current state will either crash or do some serious fuck up if you let it unattended for a few hours or maybe less (based on the disengagements in those videos). Forget about driverless Tesla with the current fsd. Waymo has been operating driverless since 2019. I do agree that it is progressing very nicely. Imo tesla fsd needs 2 more years and a hardware update and it will be there.
- cs702 4y ago
  I've yet to see how Waymo and other self-driving systems perform in open-ended testing, outside tightly restricted, geofenced environments. Otherwise, I agree that Tesla FSD Beta has been progressing nicely. I don't know if it will take 1, 2, or 5 years to get FSD Beta to an acceptable rate of graceful failures, but I agree it looks likely to get there before the end of the decade!
  - Veserv 4y ago
    I've yet to see Tesla FSD Beta perform in any meaningful operational environment, no matter how restricted, within a factor of 100x of Waymo or Cruise, and Waymo and Cruise are still at least a factor of 10x off of "equal to human drivers". Superiority can not be claimed if a system is unacceptably bad in all circumstances (i.e. 1,000x worse than the minimum acceptable standard for real commercial usage) and worse in every case than a comparable alternative. It only makes sense if the system is better at something other than being allowed to create unacceptably bad results in a more diverse range of circumstances.
    - panick21_ 4y ago
      Tesla has not focused on any individual area to any large extent so why would you measure them on that? That not the problem they are trying to solve and its not the problem that worth investing 30 billion for. > within a factor of 100x of Waymo What's your evidence for this? Is there anybody who has done a systematic comparison of Tesla performance in Arizon zone of Waymo?
jeffbee 4y ago
I really feel that you have been duped here. There is no reason to believe that Tesla Dojo even exists. At Hot Chips last year they showed some 3D renders of their supposed board. At Hot Chips this year they showed the same renders. At "AI Day" last month they showed a retarded humanoid robot. We have no basis to conclude that Dojo does, can, or will exist.
- mrb 4y ago
  Dojo does exist. There are real-world pictures of it: https://media.datacenterdynamics.com/media/images/training_tiles_III.original.png https://media.datacenterdynamics.com/media/images/training_t... source: https://www.datacenterdynamics.com/en/news/tesla-details-dojo-supercomputer-reveals-dojo-d1-chip-and-training-tile-module/ https://www.datacenterdynamics.com/en/news/tesla-details-doj... However, to my knowledge, Tesla is still in the "bring up" step. I think that was a mistake to pack 25 dies on a single system-on-wafer. Very hard to cool.
  - itsoktocry 4y ago
    That's just a picture of a testing rig set up on a workbench. It's the same picture in both articles. Classic Tesla. Do you think the minisub and Tesla ventilators also existed?
itsoktocry 4y ago
>haven't seen anything like it from anyone else. Have you considered that other companies don't make it a priority to market these things? Elon knows his audience: people who will go on message boards and talk about it. Most people don't care about the underlying AI tech. Do you think the other companies aren't making any breakthroughs? How do they have Robotaxis then? Your entire claimed expertise seems to come from YouTube promotional videos. Maybe take a step back from marketing hype.

badpun 4y ago

Compared to general self-driving, it's relatively simple to make self-driving taxis work in just a couple of cities (preferably ones with minimum adversarial weather, too) - you just test the code against that particular dataset until it performes reasonably well, manually fixing the edge cases along the way if need be. It's still a monumental, multi-billion dollar project, but I would be surprised if it wasn't achievable.

bastawhiz 4y ago
Do you know that their tech is specifically optimized for those cities and won't work well elsewhere? Or is that speculation? As far as I'm aware, unless you were able to compare the performance of FSD against Waymo on an arbitrary road, you can't really make that argument.
- badpun 4y ago
  I'm not talking about any specific tech solution. I'm merely pointing out that getting a self-driving taxi to work in a single city of your choice is a vastly simpler problem than general FSD.

Initial_BP 4y ago

Not sure about the current state of the actual ML, but compared to other self driving companies Tesla has a treasure trove of data because they have so many vehicles on the road at all hours of every day. The edge cases are the parts that are hard to identify and solve so having all that drive time data to identify edge cases would seem to give them a big advantage.

jboy55 4y ago
If the data doesn't have the details required to build accurate models, then the data is just costing Tesla money. Since Tesla's are just cameras only, with telemetry, they can replay scenarios with existing roads, but what happens when someone cones off half of the road?
- Initial_BP 4y ago
  I imagine they have a way to replay incidents from their cars in a simulation and even if it's not super accurate, they could likely look at camera data and rebuild a similar situation (Sim or real life to identify and test the edge case) I work at another autonomous car company (as a security engineer not ML related work) and I know we have a Lot of simulated situations that we run the ML against and add more from situations collected from actual driving.
teyc 4y ago
Most self-driving is about avoiding collissions, and signalling intent, especially when streets are narrow and there's merging or shared use. The physics of cars, people, bikes and kids around roads are well understood (acceleration, velocity). This can be simulated, and a game engine can generate data for virtual sensors to be trained. There's no reason to require time on the road.
- cptskippy 4y ago
  But you'll never be able to come up with all of the possible scenarios to simulate. What Tesla has demonstrated is creating virtual scenarios where they can dynamic adjust all factors (light, weather, traffic, etc) and base them off real world situations they've encountered where their Model failed.

Salgat 4y ago

Aren't Waymo and Cruise constrained to specially tailored cities? Autonomous driving that can be activated literally anywhere in the country is far more impressive, even if it has cases where it doesn't perform as well.

birdyrooster 4y ago
The more far flung reaches are are served well by traditional vehicles and the people are trained to use driving machines since most of the workers are agricultural. The city centers and suburbs are a big enough win even if it isn't as magical as taking a self driving trip from a small cottage in northern Vermont all the way to a campsite up in Big Sur.
- kortex 4y ago
  I'm in Upstate New York and despite my penchant for gardening, I'm hardly an agricultural worker. Any urban-geofenced product is useless to me and the half a million folks in the greater capitol area and nearby counties aside from the tricity residents. Which is at least 50% of the folks. Hardly a passing grade, let alone big win.
bastawhiz 4y ago
I have a double friends with FSD that rarely use it because it "scares [them]" by occasionally making dangerous decisions. Surely Waymo or Cruise could launch anywhere, but they make the conscious choice not to so they avoid this exact problem.
itsoktocry 4y ago
To me actual driverless cars are more impressive than FSD which doesn't allow me to take my hands off the wheel, yes.
- panick21_ 4y ago
  They are driverless but have a remote operator that can jump in at any time. Strictly speaking driverless cars are not actually hard to make. As long as you have a good data connection you can remote drive a car just fine.

oceanplexian 4y ago

They’re operating Taxi services in San Francisco. A city that doesn’t experience any real-world weather, with an area of like 50 square miles where speeds generally never exceed 25MPH. They also have humans watching cameras that take over when the self driving breaks down. It’s a completely different problem space, like claiming someone built a train and therefore they can easily build self driving cars since they are both “driverless”.

bastawhiz 4y ago
Yes, but you're choosing only one metric to evaluate on. Waymo/Cruise are level 4+. Anecdotally (does anyone have comparable data?), they also have a much lower accident rate. Solving a problem partially for all conditions and areas rather than ~completely for a specific but large area and set of conditions doesn't seem like it puts you meaningfully ahead. Edit: and surely Waymo/Cruise could launch everywhere with performance that's lower than their current launch cities, but they choose not to. I don't think there's any compelling reason to assume their tech doesn't work outside of SF or Arizona or wherever, they just don't want to be in the news for their cars plowing someone into a highway divider or running over a pedestrian.