One of the most dangerous states of mind, is to be pursuing something that you are convinced will save people's lives in the future, but that might hurt people

One of the most dangerous states of mind, is to be pursuing something that you are convinced will save people's lives in the future, but that might hurt people now. This type of thinking is unrivaled in causing previously normal people to do horrendously bad stuff. Add to this the potential for profits, and it becomes a very toxic mix, where a person can be killing people while trying to make a profit while fully convinced that they are a very good person and are doing humanity a great service. We have historical experience with an analogous situation. Medical experiments and drug development. People became so focused on "the greater good for humanity" that they did horrible stuff to study participants (see Tuskegee Syphilis Study). Or drugs were not adequately assessed for safety in pregnant patients and severe birth defects occurred (see thalidomide). This is one of the reasons we have Institutional Review Boards to review medical studies and the FDA to review drugs, because people involved in a project like this are so good at fooling themselves. When developing drugs, pharmacy companies aren't allowed to say "This drug will cure X and benefit so many people. We would like to ignore the side effects happening now because the cure will be so beneficial in the future." We need the equivalent of an Institutional Review Board/FDA for self driving cars. As part of the submission, the engineers have to document what sensors they are using, what the limitations of those sensors are, what algorithms/machine learning techniques they are using, and the limitations of those. They also need to disclose the engineering tradeoffs and design decisions (for example, not having LIDAR). Then there need to be tests for self-driving cars for safety on the unhappy path. What happens if there is a concrete barrier in front of you? What happens if the lane markers are bad? What happens if the car in front of you brakes? What happens if you are driving with the sun facing you? What happens if the truck in front of you drops something in the road? What happens if a child darts in front of you? And so on... Then based on the testing, certify the car for self-driving only in certain conditions that were shown to be safe based on the independent testing. After a self driving car is on the road, all data collected by the self driving cars should be shared with the review board so that there can be ongoing monitoring. In addition, users of those cars should be able to report incidents and near misses to the review board.