6 ms·

It's fascinating that so many brilliant people in this thread can talk so casually about any single loss of life for the tech built by the very same people in t

It's fascinating that so many brilliant people in this thread can talk so casually about any single loss of life for the tech built by the very same people in this thread. I would be devastated if I learned my code ended even one person's life. You're in a bubble if you think a world hyper connected to social media that's in the same world scared that autonomy is about to kill millions of jobs in one jab and would NOT over-react to a single accident. From the article it's impossible to know if the cyclist was at fault, so I won't jump to conclusions, but we can't, especially as the creators of the future of autonomous vehicles talk so dissociatively about human life. Either we go out into the public roads with very high certainty that OUR software and OUR hardware won't kill someone, or we wait till we get there. Public opinion on human driving is doing just fine with human drivers right now, so we can and should take our time to get to the future we want or risk a backlash and a guilty conscious.

habosa 9y ago
Thank you for adding this perspective. Honestly I am not sure how to rationalize the dueling thoughts in my head. On one hand, the "just one isn't so bad" argument is rational. In terms of fatalities per mile, self driving cars already seem to be safer than humans as a class. Therefore it would be wrong of me to say that this tech does not belong on the road when I believe I should be allowed to drive. On the other hand, you said it best. If I ever wrote code running on a machine that killed somebody I don't know how I'd continue in my career. Hell, I work at Google very far from Waymo but if one of our cars struck and killed someone I'd feel some personal responsibility. So ... I guess it's complicated.
larkeith 9y ago
> Public opinion on human driving is doing just fine with human drivers right now There were over 40,000 motor vehicle deaths in 2016 in the US [1] - more than a hundred daily. While loss of human life is always a tragedy, individual fatalities are absolutely worth it if we can begin to reduce that count sooner than by proceeding with excessive caution. It would be morally unacceptable to delay development merely to avoid a guilty conscience. [1] http://www.nsc.org/NewsDocuments/2017/Fatality-estimates-June17.pdf http://www.nsc.org/NewsDocuments/2017/Fatality-estimates-Jun...
- InclinedPlane 9y ago
  Currently, Uber self-driving vehicles are clocking in at a fatality rate of about 50x the rate for human drivers. You can bring your hypotheticals to bear all you like, but this is a serious problem.
- jonathanyc 9y ago
  And it’d be morally unacceptable to allow public deployment of software that has not been sufficiently tested. As other comments have pointed out, self-driving cars have undergone ridiculously little testing. In fact, based on only the objective statistics it is very unlikely that they are anywhere near as good at driving as humans. There is no reason self-driving cars can’t be tested in private. Companies can hire pedestrians to interact with the cars, and the software can go through the same certification process that buildings and vehicles currently go through. It’s a false dichotomy to say that you can either have self driving cars or minimally safe and accountable development, but not both. EDIT: Here is a link to a thread replying to a parent which is now auto-collapsed. https://news.ycombinator.com/item?id=16620968 https://news.ycombinator.com/item?id=16620968
  - jamiequint 9y ago
    > "it’d be morally unacceptable to allow public deployment of software that has not been sufficiently tested" Does "sufficiently tested" mean you get the end-product a year later? What if the end-product saves 10,000 lives a year? In that case what cost is morally acceptable to get there a year faster?
    - thorell 9y ago
      You can't project the actual cost without sufficient testing.
    - rootlocus 9y ago
      > What if the end-product saves 10,000 lives a year? You can't have the certainty to make such statements without "sufficiently testing" the product.
DoreenMichele 9y ago
It's fascinating that so many brilliant people in this thread can talk so casually about any single loss of life for the tech built by the very same people in this thread. I would be devastated if I learned my code ended even one person's life. Generally speaking, it is easier to pinpoint X thing led to Y tragedy than to quantify the tragedies that would have happened without X. It is incredibly hard to show what did not happen but "should" have. Even if you can get it straight in your own mind, good luck proving it to the world. People are living longer and child mortality is down. We don't really celebrate that. We just complain about overpopulation and how we are destroying the environment. When Iraq lit oil wells on fire on its way out of Kuwait, they were expected to burn for months and be a global environmental disaster. When crack teams from around the world converged and put them out far faster than expected, it was not celebrated with the same degree of fervor that it had originally been decried as a disaster. Y2K was also predicted to be a global catastrophe. It was quietly prevented and gets remembered as "Those fools who ever thought this was a big deal!" If self driving cars could reduce mortality compared to current rates and we were confident of that, waiting until it is perfected before releasing it means accepting ongoing deaths that don't have to happen. It's not uncommon for people to feel that new tech needs to be perfected while missing the fact that it may be a big improvement over the current status quo without being perfected. I don't really know anything about self driving cars. I have mixed feelings about replying to your comment because I don't actually know how I feel about this particular issue. But the thought process you are putting forth is both common and not very pragmatic. I guess a pithy rebuttal would quote some saying about the perfect being the enemy of the good. But I think that leaves out a lot of important points.
CooIHandLuke 9y ago
What we can't do as the creators of the future of autonomous vehicles is give in for emotional appeals when the relevant thing is statistics about car accidents. If the media does it, bash the media (and hope for decent publications to support the appeal of relevant people in the field). If politicians do it, bash politicians, though nothing really helps when it comes to politics. But don't be nice to bureaucrats and their oblivious supporters who are actually preventing thousands of lives being saved every year by slowing down development and deployment.
rhizome 9y ago
As someone said on Twitter, among software developers, "ethics" is seen as a hobby at best.
- nothrabannosir 9y ago
  While pithy and edgy, this attitude is, unfortunately, also unconstructive and uninsightful. For those out there who do wish to put their money where their mouth is, do you have any advice besides “don’t work at Facebook”? (Or Uber…) Because that doesn’t seem to be cutting it. At least doctors have the Hippocratic oath.
  - rhizome 9y ago
    do you have any advice besides “don’t work at Facebook”? Sure: start by reading Spinoza's TIE.
mipmap04 9y ago
My career is in software, but when I peer out of my cave at those in more traditional engineering backgrounds, I realize they have so many more checkpoints and structures for guaranteeing performance / operability under load. For 99% of us in software, if our software fails, nothing really happens of consequence other than downtime - no one dies. In this case, the software written can have mortal consequences. Maybe there's a need for increased scrutiny on software development - especially as IoT gains more popularity and people are relying more and more on the software that supports or replaces human involvement in critical and/or potentially high risk activities. Raises the question though: will stakeholders be willing to pay the upfront financial costs to obtain the level of quality and assurance that other engineering disciplines provide? We'll see.
gus_massa 9y ago
> I would be devastated if I learned my code ended even one person's life. I agree, but there is already a lot of software in a car that can kill people in case of an error. In particular the airbag have multiple sensors and a complex heuristic to decide when the have to open, which ones, full or partial strength, ... https://en.wikipedia.org/wiki/Airbag#Operation https://en.wikipedia.org/wiki/Airbag#Operation An airbag that doesn't open can kill someone. An airbag that open unnecessary can kill someone. From Wikipedia: > From 1990 to 2000, the United States National Highway Traffic Safety Administration identified 175 fatalities caused by air bags. Most of these (104) have been children, while the rest were adults. About 3.3 million air bag deployments have occurred during that interval, and the agency estimates more than 6,377 lives saved and countless injuries prevented. So, there is already a lot of programmers that have to be careful because an error can kill someone in the car (or the plane, or a medical machine, ...) I guess it's more easy to ignore the people killed by airbags because we are use to them and we imagine that they have a simple sensor (something like a light switch. So in case of a failure nobody is guilty. With a self driving car, the first problem is that it is new and scary, and the second problem is that in an error that creates an accident we are use to blame the driver. But now we don't have anyone to blame (perhaps the programmer).
tooop 9y ago
Crossing a 4 lane road (where speed limit is at least 40mph) outside crosswalk? It's clear that it is cyclists fault. The only thing that is questionable - would a human driver have avoided this or at least hit the cyclist with lesser speed.
neotek 9y ago
>Either we go out into the public roads with very high certainty that OUR software and OUR hardware won't kill someone, or we wait till we get there. As long as it kills fewer people than would otherwise have died due to negligent human drivers I'd say it's already there.

It's fascinating that so many brilliant people in this thread can talk so casually about any single loss of life for the tech built by the very same people in this thread. I would be devastated if I learned my code ended even one person's life. You're in a bubble if you think a world hyper connected to social media that's in the same world scared that autonomy is about to kill millions of jobs in one jab and would NOT over-react to a single accident. From the article it's impossible to know if the cyclist was at fault, so I won't jump to conclusions, but we can't, especially as the creators of the future of autonomous vehicles talk so dissociatively about human life. Either we go out into the public roads with very high certainty that OUR software and OUR hardware won't kill someone, or we wait till we get there. Public opinion on human driving is doing just fine with human drivers right now, so we can and should take our time to get to the future we want or risk a backlash and a guilty conscious.

habosa 9y ago

Thank you for adding this perspective. Honestly I am not sure how to rationalize the dueling thoughts in my head. On one hand, the "just one isn't so bad" argument is rational. In terms of fatalities per mile, self driving cars already seem to be safer than humans as a class. Therefore it would be wrong of me to say that this tech does not belong on the road when I believe I should be allowed to drive. On the other hand, you said it best. If I ever wrote code running on a machine that killed somebody I don't know how I'd continue in my career. Hell, I work at Google very far from Waymo but if one of our cars struck and killed someone I'd feel some personal responsibility. So ... I guess it's complicated.

larkeith 9y ago

> Public opinion on human driving is doing just fine with human drivers right now There were over 40,000 motor vehicle deaths in 2016 in the US [1] - more than a hundred daily. While loss of human life is always a tragedy, individual fatalities are absolutely worth it if we can begin to reduce that count sooner than by proceeding with excessive caution. It would be morally unacceptable to delay development merely to avoid a guilty conscience. [1] http://www.nsc.org/NewsDocuments/2017/Fatality-estimates-June17.pdf http://www.nsc.org/NewsDocuments/2017/Fatality-estimates-Jun...

InclinedPlane 9y ago
Currently, Uber self-driving vehicles are clocking in at a fatality rate of about 50x the rate for human drivers. You can bring your hypotheticals to bear all you like, but this is a serious problem.
jonathanyc 9y ago
And it’d be morally unacceptable to allow public deployment of software that has not been sufficiently tested. As other comments have pointed out, self-driving cars have undergone ridiculously little testing. In fact, based on only the objective statistics it is very unlikely that they are anywhere near as good at driving as humans. There is no reason self-driving cars can’t be tested in private. Companies can hire pedestrians to interact with the cars, and the software can go through the same certification process that buildings and vehicles currently go through. It’s a false dichotomy to say that you can either have self driving cars or minimally safe and accountable development, but not both. EDIT: Here is a link to a thread replying to a parent which is now auto-collapsed. https://news.ycombinator.com/item?id=16620968 https://news.ycombinator.com/item?id=16620968
- jamiequint 9y ago
  > "it’d be morally unacceptable to allow public deployment of software that has not been sufficiently tested" Does "sufficiently tested" mean you get the end-product a year later? What if the end-product saves 10,000 lives a year? In that case what cost is morally acceptable to get there a year faster?
  - thorell 9y ago
    You can't project the actual cost without sufficient testing.
  - rootlocus 9y ago
    > What if the end-product saves 10,000 lives a year? You can't have the certainty to make such statements without "sufficiently testing" the product.

DoreenMichele 9y ago

It's fascinating that so many brilliant people in this thread can talk so casually about any single loss of life for the tech built by the very same people in this thread. I would be devastated if I learned my code ended even one person's life. Generally speaking, it is easier to pinpoint X thing led to Y tragedy than to quantify the tragedies that would have happened without X. It is incredibly hard to show what did not happen but "should" have. Even if you can get it straight in your own mind, good luck proving it to the world. People are living longer and child mortality is down. We don't really celebrate that. We just complain about overpopulation and how we are destroying the environment. When Iraq lit oil wells on fire on its way out of Kuwait, they were expected to burn for months and be a global environmental disaster. When crack teams from around the world converged and put them out far faster than expected, it was not celebrated with the same degree of fervor that it had originally been decried as a disaster. Y2K was also predicted to be a global catastrophe. It was quietly prevented and gets remembered as "Those fools who ever thought this was a big deal!" If self driving cars could reduce mortality compared to current rates and we were confident of that, waiting until it is perfected before releasing it means accepting ongoing deaths that don't have to happen. It's not uncommon for people to feel that new tech needs to be perfected while missing the fact that it may be a big improvement over the current status quo without being perfected. I don't really know anything about self driving cars. I have mixed feelings about replying to your comment because I don't actually know how I feel about this particular issue. But the thought process you are putting forth is both common and not very pragmatic. I guess a pithy rebuttal would quote some saying about the perfect being the enemy of the good. But I think that leaves out a lot of important points.

CooIHandLuke 9y ago

What we can't do as the creators of the future of autonomous vehicles is give in for emotional appeals when the relevant thing is statistics about car accidents. If the media does it, bash the media (and hope for decent publications to support the appeal of relevant people in the field). If politicians do it, bash politicians, though nothing really helps when it comes to politics. But don't be nice to bureaucrats and their oblivious supporters who are actually preventing thousands of lives being saved every year by slowing down development and deployment.

rhizome 9y ago

As someone said on Twitter, among software developers, "ethics" is seen as a hobby at best.

nothrabannosir 9y ago
While pithy and edgy, this attitude is, unfortunately, also unconstructive and uninsightful. For those out there who do wish to put their money where their mouth is, do you have any advice besides “don’t work at Facebook”? (Or Uber…) Because that doesn’t seem to be cutting it. At least doctors have the Hippocratic oath.
- rhizome 9y ago
  do you have any advice besides “don’t work at Facebook”? Sure: start by reading Spinoza's TIE.

mipmap04 9y ago

My career is in software, but when I peer out of my cave at those in more traditional engineering backgrounds, I realize they have so many more checkpoints and structures for guaranteeing performance / operability under load. For 99% of us in software, if our software fails, nothing really happens of consequence other than downtime - no one dies. In this case, the software written can have mortal consequences. Maybe there's a need for increased scrutiny on software development - especially as IoT gains more popularity and people are relying more and more on the software that supports or replaces human involvement in critical and/or potentially high risk activities. Raises the question though: will stakeholders be willing to pay the upfront financial costs to obtain the level of quality and assurance that other engineering disciplines provide? We'll see.

gus_massa 9y ago

> I would be devastated if I learned my code ended even one person's life. I agree, but there is already a lot of software in a car that can kill people in case of an error. In particular the airbag have multiple sensors and a complex heuristic to decide when the have to open, which ones, full or partial strength, ... https://en.wikipedia.org/wiki/Airbag#Operation https://en.wikipedia.org/wiki/Airbag#Operation An airbag that doesn't open can kill someone. An airbag that open unnecessary can kill someone. From Wikipedia: > From 1990 to 2000, the United States National Highway Traffic Safety Administration identified 175 fatalities caused by air bags. Most of these (104) have been children, while the rest were adults. About 3.3 million air bag deployments have occurred during that interval, and the agency estimates more than 6,377 lives saved and countless injuries prevented. So, there is already a lot of programmers that have to be careful because an error can kill someone in the car (or the plane, or a medical machine, ...) I guess it's more easy to ignore the people killed by airbags because we are use to them and we imagine that they have a simple sensor (something like a light switch. So in case of a failure nobody is guilty. With a self driving car, the first problem is that it is new and scary, and the second problem is that in an error that creates an accident we are use to blame the driver. But now we don't have anyone to blame (perhaps the programmer).

tooop 9y ago

Crossing a 4 lane road (where speed limit is at least 40mph) outside crosswalk? It's clear that it is cyclists fault. The only thing that is questionable - would a human driver have avoided this or at least hit the cyclist with lesser speed.

neotek 9y ago

>Either we go out into the public roads with very high certainty that OUR software and OUR hardware won't kill someone, or we wait till we get there. As long as it kills fewer people than would otherwise have died due to negligent human drivers I'd say it's already there.