7 ms·

Not mission critical, rather life critical. I've worked on several life critical systems as a security researcher, helping to develop controls in tightly contr

Not mission critical, rather life critical. I've worked on several life critical systems as a security researcher, helping to develop controls in tightly controlled environments to keep people safe. Work on software controlled switches in industrial machines mostly. As I've followed these developments and those of Google and Tesla I've been wondering at times if I live in some alternate universe. The level of diligence and care that must go into systems which have the potential to maim an kill are nothing to laugh at. They require serious engineering which takes time. You can find the best and smartest humans on the planet and they'll still fail at building reasonable controls for life critical systems in controlled environments. When you are building these controls for life critical systems which will operate in the diversity of our roadways, honestly the task gets very hard. I'm not against progress and I do think that self-driving cars are the future. However, it may be wise for those trying to march towards that future at breakneck speeds to take a step back and consider that these systems should first be implemented as backups to humans. Develop the solutions through preemptive braking and accident avoidance where humans are still very much engaged in the driving activities or focus on building expertise in controlled environments. On that note, kudos to Google for doing just that. Their first phase has focused on low speed situations and have been restricted very much to lab work whereby they have paid employee's serving as observers and backup.

semi-extrinsic 10y ago
This. There is a huge amount of safety evaluations, precautions, training and failsafes we have to go through just for using a Class 4 laser at work in a controlled environment, where the worst possible outcome is permanently blinding someone working with it. For a machine with the potential to easily maim and kill, to be completely autonomously operated in a chaotic environment where it interacts not just with other computer controlled machines but also with humans, even children, I honestly hope that the amount of safety precautions required will grow very quickly.
jacquesm 10y ago
I've worked on CNC mills and lathes, design of the logic and the hardware driving the servos as well as some fuel estimation software for airplanes. The degree of care that went into the e-stop design of the lathe and mill controller and the number of checks and tests of the fuel estimation software are stuff that I'm (hopefully justified) proud of, that's not stuff to cut any corners with. In both cases that took me longer than Hotz claims to have spent on creating the whole self driving car code, there is no way you can do this safely with that mentality. It is absolutely irresponsible to take that attitude towards systems like this, life is not a video game with 3 more lives if you lose this one (or cause others to lose theirs).
WhitneyLand 10y ago
If 10,000 people were killed on the way to making driverless cars safe, and this wild west approach ended up getting them into mass market use a few years earlier, it would save the lives of thousands of people overall. Is this a bad thing? Is it not justified by utilitarianism?
- jacquesm 10y ago
  That would kill the driverless car concept. See, we already have cars, it's not like we're going from 'horses and buggies' to cars, it is going from one good solution to an even better one and even though there is a risk at being stuck at the present local maximum for a little longer going down in order to find the peak of this particular mount improbable would not be an acceptable path for the majority of the public. And without having a time machine or a functional crystal ball you won't know if it is 10K, 100K or a few million people that get killed before you get it right so the bar has been set at 'it needs to be better than the present level before it is acceptable'.
- throwaway729 10y ago
  > If 10,000 people were killed on the way to making driverless cars safe, and this wild west approach ended up getting them into mass market use a few years earlier, it would save the lives of thousands of people overall. This is a perfect example of the biggest and IMO most compelling critique of utilitarianism -- namely, that it's too susceptible to bullshitting. People decide what they want to do, then completely invent (or, in more sophisticated cases, carefully fabricate) numbers and time-frames in order to hide their unethical opportunism behind the facade of objective moral assessment. This is especially problematic when you have to estimate things that are pretty much impossible to estimate without completely bullshitting. For example, "number of people killed by not following certain safety standards", or "impact on time-to-completing of some particular safety standard". We're talking about a novel technology, so there's no way anyone has even remotely high confidence in a sufficiently bounded estimate of these numbers. It's all just bullshitting. > Is thus a bad thing? Is it not justified by utilitarianism? These are two VERY different questions.
  - 10y ago
jtmarmon 10y ago
i understand where you're coming from, but there is a counter argument to what you're saying, which is that although there may be some dangers to things like the comma one and tesla autopilot, their release may save more lives than they edanger, having a net positive to humans. humans are pretty bad at driving
- abduhl 10y ago
  This is an ends justify the means argument. How many lives have been saved TO DATE with self driving cars? How many lives will be saved IN THE FUTURE with self driving cars? What is the acceptable casualty rate to get from where we are now to where you think we will be in the future. Is your family an acceptable casualty? If not, then why is mine?
  - mikeash 10y ago
    "Is your family an acceptable casualty?" is a pointless question. It can be applied equally well to the opposite argument: what is the acceptable casualty rate for not advancing safety in this way? Is your family an acceptable casualty? Continuing on with the status quo is just as much of an action and a choice as pushing forward with automation. Both choices are going to get people killed. We can either evaluate the choices and choose the one that seems better, or we can ignore the question and possibly have more people die than is necessary, but there's no option where your family isn't at risk.
    - abduhl 10y ago
      This argument is a red herring. We know the current casualty rate and have accepted it. Nobody gets into a car with the illusion that it's 100% safe.
      - mikeash 10y ago
        And if technology reduces that casualty rate, why does the fact that it's new suddenly make it unacceptable?
mikeryan 10y ago
Google I'm totally cool with. They've been at this a long time and still don't seem very close to actually releasing a product. I like Tesla's and what Elon's accomplished has been impressive but he seems hellbent on "winning the race" to release a self driving car and that worries me.
- kyledrake 10y ago
  It's a good concern to have, but I also worry about waiting too long here. At the end of the day, there's going to be problems that the testing would have never found regardless of how much testing there is, and we need to be psychologically prepared for that. It took airplanes a long time to become safe (through a long and scary process), and now they're safer than my front porch. Whatever the outcome, self driving cars will very likely be safer than human drivers, and that means more lives are saved than lost, even with the occasional horrific problem. It is entirely plausible that self driving cars are already safer than human ones.
- dreamcompiler 10y ago
  I wish it were possible to buy a Tesla without any autopilot features except basic cruise control, because I know how damnably difficult the edge cases are for a self-driving car, and I'm perfectly capable of driving a car manually, thank you very much. Tesla's cars are so good I'd almost be willing to pay more for one without autopilot. Almost.
  - prostoalex 10y ago
    While driving, you have to do a double-pull on the lever to activate the autopilot, a single pull just activates the basic cruise control. So you can still buy one and just never activate it. Alternatively, earlier editions of Model S shipped without autopilot, which was then sold as an aftermarket upgrade as late as this spring (when autopilot was an optional $2,500 package on a new Model S).
    - dreamcompiler 10y ago
      Fair enough re: the older Teslas. As to the new ones, even if I never activate autopilot I'm still driving around in a massive autopilot data collection engine for Tesla Corp, which bugs me.

semi-extrinsic 10y ago

This. There is a huge amount of safety evaluations, precautions, training and failsafes we have to go through just for using a Class 4 laser at work in a controlled environment, where the worst possible outcome is permanently blinding someone working with it. For a machine with the potential to easily maim and kill, to be completely autonomously operated in a chaotic environment where it interacts not just with other computer controlled machines but also with humans, even children, I honestly hope that the amount of safety precautions required will grow very quickly.

jacquesm 10y ago

I've worked on CNC mills and lathes, design of the logic and the hardware driving the servos as well as some fuel estimation software for airplanes. The degree of care that went into the e-stop design of the lathe and mill controller and the number of checks and tests of the fuel estimation software are stuff that I'm (hopefully justified) proud of, that's not stuff to cut any corners with. In both cases that took me longer than Hotz claims to have spent on creating the whole self driving car code, there is no way you can do this safely with that mentality. It is absolutely irresponsible to take that attitude towards systems like this, life is not a video game with 3 more lives if you lose this one (or cause others to lose theirs).

WhitneyLand 10y ago

If 10,000 people were killed on the way to making driverless cars safe, and this wild west approach ended up getting them into mass market use a few years earlier, it would save the lives of thousands of people overall. Is this a bad thing? Is it not justified by utilitarianism?

jacquesm 10y ago
That would kill the driverless car concept. See, we already have cars, it's not like we're going from 'horses and buggies' to cars, it is going from one good solution to an even better one and even though there is a risk at being stuck at the present local maximum for a little longer going down in order to find the peak of this particular mount improbable would not be an acceptable path for the majority of the public. And without having a time machine or a functional crystal ball you won't know if it is 10K, 100K or a few million people that get killed before you get it right so the bar has been set at 'it needs to be better than the present level before it is acceptable'.
throwaway729 10y ago
> If 10,000 people were killed on the way to making driverless cars safe, and this wild west approach ended up getting them into mass market use a few years earlier, it would save the lives of thousands of people overall. This is a perfect example of the biggest and IMO most compelling critique of utilitarianism -- namely, that it's too susceptible to bullshitting. People decide what they want to do, then completely invent (or, in more sophisticated cases, carefully fabricate) numbers and time-frames in order to hide their unethical opportunism behind the facade of objective moral assessment. This is especially problematic when you have to estimate things that are pretty much impossible to estimate without completely bullshitting. For example, "number of people killed by not following certain safety standards", or "impact on time-to-completing of some particular safety standard". We're talking about a novel technology, so there's no way anyone has even remotely high confidence in a sufficiently bounded estimate of these numbers. It's all just bullshitting. > Is thus a bad thing? Is it not justified by utilitarianism? These are two VERY different questions.
- 10y ago

jtmarmon 10y ago

i understand where you're coming from, but there is a counter argument to what you're saying, which is that although there may be some dangers to things like the comma one and tesla autopilot, their release may save more lives than they edanger, having a net positive to humans. humans are pretty bad at driving

abduhl 10y ago
This is an ends justify the means argument. How many lives have been saved TO DATE with self driving cars? How many lives will be saved IN THE FUTURE with self driving cars? What is the acceptable casualty rate to get from where we are now to where you think we will be in the future. Is your family an acceptable casualty? If not, then why is mine?
- mikeash 10y ago
  "Is your family an acceptable casualty?" is a pointless question. It can be applied equally well to the opposite argument: what is the acceptable casualty rate for not advancing safety in this way? Is your family an acceptable casualty? Continuing on with the status quo is just as much of an action and a choice as pushing forward with automation. Both choices are going to get people killed. We can either evaluate the choices and choose the one that seems better, or we can ignore the question and possibly have more people die than is necessary, but there's no option where your family isn't at risk.
  - abduhl 10y ago
    This argument is a red herring. We know the current casualty rate and have accepted it. Nobody gets into a car with the illusion that it's 100% safe.
    - mikeash 10y ago
      And if technology reduces that casualty rate, why does the fact that it's new suddenly make it unacceptable?

mikeryan 10y ago

Google I'm totally cool with. They've been at this a long time and still don't seem very close to actually releasing a product. I like Tesla's and what Elon's accomplished has been impressive but he seems hellbent on "winning the race" to release a self driving car and that worries me.

kyledrake 10y ago
It's a good concern to have, but I also worry about waiting too long here. At the end of the day, there's going to be problems that the testing would have never found regardless of how much testing there is, and we need to be psychologically prepared for that. It took airplanes a long time to become safe (through a long and scary process), and now they're safer than my front porch. Whatever the outcome, self driving cars will very likely be safer than human drivers, and that means more lives are saved than lost, even with the occasional horrific problem. It is entirely plausible that self driving cars are already safer than human ones.
dreamcompiler 10y ago
I wish it were possible to buy a Tesla without any autopilot features except basic cruise control, because I know how damnably difficult the edge cases are for a self-driving car, and I'm perfectly capable of driving a car manually, thank you very much. Tesla's cars are so good I'd almost be willing to pay more for one without autopilot. Almost.
- prostoalex 10y ago
  While driving, you have to do a double-pull on the lever to activate the autopilot, a single pull just activates the basic cruise control. So you can still buy one and just never activate it. Alternatively, earlier editions of Model S shipped without autopilot, which was then sold as an aftermarket upgrade as late as this spring (when autopilot was an optional $2,500 package on a new Model S).
  - dreamcompiler 10y ago
    Fair enough re: the older Teslas. As to the new ones, even if I never activate autopilot I'm still driving around in a massive autopilot data collection engine for Tesla Corp, which bugs me.