3 ms·

Every software release is a totally new driver. If they release software at a cadence of say, 6 or 8 weeks, would you feel comfortable riding? You don't know ho

Every software release is a totally new driver. If they release software at a cadence of say, 6 or 8 weeks, would you feel comfortable riding? You don't know how safe the car was during that time -- any part of the driver can regress. You're basically trusting that they know how to do simulation, that they bore the cost of running that simulation, and that their simulator is realistic enough to yield trustworthy statistics. Sadly, our regulatory agencies are currently set up for very delayed decisionmaking. Since every company can release software at any time, you could imagine a regulatory platform that tracks software releases and their corresponding safety statistics in real time.

jessriedel 2y ago
I’m quite confident in it and personally ride it monthly. Happy to bet it will turn out to be no more than twice as dangerous as the median driver today, ie, as dangerous as driving in the 1980s. In the long term it will be vastly safer, obviously.
- kajecounterhack 2y ago
  I'm glad you feel that way (and I mostly feel the same / ride Waymos frequently), but it doesn't change the reality that you're putting all your faith in a self-regulating company that does not cooperate with regulators more than it absolutely needs to, and where a change in leadership / safety culture / process can cause a catastrophe overnight. And this isn't hyperbole; 1 software push is all that separates a car that crashes in scenario X from one where it was found to not crash in that same scenario. Do you trust the code to be bug free? Do you trust the bugs to be discovered before they maybe kill you? I have a vested interest in the success of AVs (worked in the industry for the better part of a decade) but the companies building them are far from perfect in terms of how they conduct themselves wrt safety. There needs to be some development in terms of the supporting tech. The way I think about this is: planes came before the FAA and key technologies that we use to manage flights, such as radar and radio being used in conjunction. AVs are in their "just learned to fly" era.
  - jessriedel 2y ago
    In the next couple years Waymo will have accumulated more than 100M miles driven, which is the number a human drives statistically before causing 1 death on average. Are you still going to be worried when they drive 1B miles and have 2 deaths? What about 100B miles with 30 deaths? At what point will you accept the statistical evidence?
    - kajecounterhack 2y ago
      A new software release has driven zero miles. You do not know what kind of performance it has -- regressions both happen and are expected with every launch. The number of miles a release ends up driving is mostly in simulation + some human-supervised on-road testing, and may be low / influenced by the availability of resources ($$$). There are little to no regulations or 3rd party checks on the process used to clear releases. They do not simulate every release on all X miles driven. They can't, it's super expensive, and that's to speak nothing of challenges like simulation realism or compatibility of the simulator with multiple generations of hardware that isn't backwards compatible. They could have no deaths for years, then suddenly a whole lot after a bad software push. Recently they issued a recall because of a bug where the car could hit a telephone pole. Imagine if the software had a bug when their cars were ubiquitous on highways, slamming cars into concrete medians? Imagine if a fault could bork critical technologies like the lidar at a high rate across the fleet? What happens to Waymo if they kill even 1 child due to a bug or lapse in model quality? I imagine many would wake up to the reality of their safety story and demand a way to audit software releases. Or would you continue to let your kid ride Waymo? This is also to say nothing of sensor failures and other ways the car can mess up. The probability of your dying in a Waymo is not # incidents / # miles, because past performance does not guarantee future performance when it comes to new releases. It's unknown quantity L (likelihood of encountering a dangerous bug per unit time) * T (time spent in car). Without more data about L (which changes based on many factors, like the company's process for certifying releases), L*T could go from 0 to something unacceptably high really fast. You trust the car you drive today because the parts are quality controlled and regulated. Software drivers are a very new part added to this chain of things to trust. Their quality control process is not yet publicly regulated. The reality is there needs to be an advancement in technology to ensure software quality / safety, ideally from the role of an unbiased auditor that can catch faults and ground the fleet asap, then restore public faith by ensuring gaps in the qualification processes are fixed. No such mechanism exists today.

jessriedel 2y ago

I’m quite confident in it and personally ride it monthly. Happy to bet it will turn out to be no more than twice as dangerous as the median driver today, ie, as dangerous as driving in the 1980s. In the long term it will be vastly safer, obviously.

kajecounterhack 2y ago
I'm glad you feel that way (and I mostly feel the same / ride Waymos frequently), but it doesn't change the reality that you're putting all your faith in a self-regulating company that does not cooperate with regulators more than it absolutely needs to, and where a change in leadership / safety culture / process can cause a catastrophe overnight. And this isn't hyperbole; 1 software push is all that separates a car that crashes in scenario X from one where it was found to not crash in that same scenario. Do you trust the code to be bug free? Do you trust the bugs to be discovered before they maybe kill you? I have a vested interest in the success of AVs (worked in the industry for the better part of a decade) but the companies building them are far from perfect in terms of how they conduct themselves wrt safety. There needs to be some development in terms of the supporting tech. The way I think about this is: planes came before the FAA and key technologies that we use to manage flights, such as radar and radio being used in conjunction. AVs are in their "just learned to fly" era.
- jessriedel 2y ago
  In the next couple years Waymo will have accumulated more than 100M miles driven, which is the number a human drives statistically before causing 1 death on average. Are you still going to be worried when they drive 1B miles and have 2 deaths? What about 100B miles with 30 deaths? At what point will you accept the statistical evidence?
  - kajecounterhack 2y ago
    A new software release has driven zero miles. You do not know what kind of performance it has -- regressions both happen and are expected with every launch. The number of miles a release ends up driving is mostly in simulation + some human-supervised on-road testing, and may be low / influenced by the availability of resources ($$$). There are little to no regulations or 3rd party checks on the process used to clear releases. They do not simulate every release on all X miles driven. They can't, it's super expensive, and that's to speak nothing of challenges like simulation realism or compatibility of the simulator with multiple generations of hardware that isn't backwards compatible. They could have no deaths for years, then suddenly a whole lot after a bad software push. Recently they issued a recall because of a bug where the car could hit a telephone pole. Imagine if the software had a bug when their cars were ubiquitous on highways, slamming cars into concrete medians? Imagine if a fault could bork critical technologies like the lidar at a high rate across the fleet? What happens to Waymo if they kill even 1 child due to a bug or lapse in model quality? I imagine many would wake up to the reality of their safety story and demand a way to audit software releases. Or would you continue to let your kid ride Waymo? This is also to say nothing of sensor failures and other ways the car can mess up. The probability of your dying in a Waymo is not # incidents / # miles, because past performance does not guarantee future performance when it comes to new releases. It's unknown quantity L (likelihood of encountering a dangerous bug per unit time) * T (time spent in car). Without more data about L (which changes based on many factors, like the company's process for certifying releases), L*T could go from 0 to something unacceptably high really fast. You trust the car you drive today because the parts are quality controlled and regulated. Software drivers are a very new part added to this chain of things to trust. Their quality control process is not yet publicly regulated. The reality is there needs to be an advancement in technology to ensure software quality / safety, ideally from the role of an unbiased auditor that can catch faults and ground the fleet asap, then restore public faith by ensuring gaps in the qualification processes are fixed. No such mechanism exists today.