3 ms·

> Such a technology, if weaponized, to could kill hundreds of thousands of people. Try and drive a Tesla through a red light or through a person that it clearl

> Such a technology, if weaponized, to could kill hundreds of thousands of people. Try and drive a Tesla through a red light or through a person that it clearly sees. Try and drive a Tesla fast without being in it. It just doesn't work. On the other hand, nothing stops someone from attaching a simple remote control system to a regular truck and driving through a crowd of people. Nothing stops someone from making an aimbot for a machine gun. That can kill a lot of people fast. https://www.youtube.com/watch?v=6QcfZGDvHU8 https://www.youtube.com/watch?v=6QcfZGDvHU8

ChuckMcM 6y ago
Interestingly I avoided injury by a Tesla which was approaching the intersection I was crossing and the driver was distracted. I heard the tire screech and looked over to see a driver clearly panicked both because they realized they nearly hit someone and there phone was now nowhere to be found. That is a great thing. But what if someone inside of TESLA, the folks who write the software and send it over the air to your car, decided to add their own "feature" to the car such that they could override the safety system with a network packet, or a text message to the car. That person would be in a position to tell the car to ignore its safety systems and kill its occupant. It is kind of a staple of dystopian fiction, but now that we're getting closer to having that software out there in real time, the question becomes less fiction and more "What would have to be true for that to happen?" The change here is cars that are always connected to the Internet and have command authority over all of the systems in the car. Before if you wanted to alter the software controlling a car you took it to a dealer, or a mod-shop, but now it just shows up in your car. What sort of insider threat programs does Tesla have? What sort of controls are their on releases? Do third parties have any opportunity to audit everything in the code? What happens if Elon orders an employee to put some code in? Do they do it? Do they report it? And not to pick on Tesla here, the same goes for GM/Cruise and Waymo right? ALL self driving systems that are currently in development have full control of the car, an always connected component, and a dynamic software update capability. Should it be required they also have a mechanical switch that forces manual control without any means of circumventing the switch in software? That would have to come as a government regulation right? And then what happens when someone steals the Waymo van by throwing the switch and driving away with it? Hopefully that gives you a sense of where my head is here in these questions.
- nojvek 6y ago
  Our society is built on Trust. You have to trust your bank to keep your money in your name and allow you to withdraw when you want it. You have to trust that the fire truck will come when you have a fire. You have to trust that your neighbours won’t randomly wake up one day become zombies thirsty for your blood. Same way you ought to trust, but not blind trust. Trust but verify. I think the self driving game will be won with someone who as a transparency mindset. Here’s the car. Here’s the software, here’s the guarantee that it hasn’t been modified and here’s the extensive tests that we’ve run on it, verified by 3rd parties vouching for its safety and quality. Same with any technology company. You win by making solid technology backed with real rigorous testing vouched by trusted 3rd parties and general public.
  - ChuckMcM 6y ago
    I agree with you. For folks reading along, there is an entire discipline around this stuff, it is call vulnerability analysis and surety systems. There are a lot of good papers published by Sandia National Laboratory on these topics. They were responsible for developing the US surety system around access to nuclear weapons[1]. When I worked at Sun they gave a talk at an e-comerce payment processing forum that Sun was participating in to discuss how you approach the problem of securing something in presence of known and unknown bad actors. In particular they discussed the systems around banking which prevent your bank from stealing your money from you. A topic that I found quite interesting. But one of the things that has always stuck with me was the discussion of the trade-off between the "cost of effort" to "actualize" a vulnerability. It is the difference between having something that could be done in theory versus having doing it having a high enough payoff to actually do it. When you look at things like dye packs in money and silent alarms and time locked safes, those are parts of a system that minimize the amount of money you can expect to make off with in a bank robbery. They are part of a surety system that is protecting the money in the bank. And they don't make it impossible to rob the bank, they make the likely-hood that you'll have enough profit from it to risk it low enough that people don't do it. [1] They are designed, in part, to prevent anyone from detonating a US nuclear device without specific authorization from the President.

ChuckMcM 6y ago

Interestingly I avoided injury by a Tesla which was approaching the intersection I was crossing and the driver was distracted. I heard the tire screech and looked over to see a driver clearly panicked both because they realized they nearly hit someone and there phone was now nowhere to be found. That is a great thing. But what if someone inside of TESLA, the folks who write the software and send it over the air to your car, decided to add their own "feature" to the car such that they could override the safety system with a network packet, or a text message to the car. That person would be in a position to tell the car to ignore its safety systems and kill its occupant. It is kind of a staple of dystopian fiction, but now that we're getting closer to having that software out there in real time, the question becomes less fiction and more "What would have to be true for that to happen?" The change here is cars that are always connected to the Internet and have command authority over all of the systems in the car. Before if you wanted to alter the software controlling a car you took it to a dealer, or a mod-shop, but now it just shows up in your car. What sort of insider threat programs does Tesla have? What sort of controls are their on releases? Do third parties have any opportunity to audit everything in the code? What happens if Elon orders an employee to put some code in? Do they do it? Do they report it? And not to pick on Tesla here, the same goes for GM/Cruise and Waymo right? ALL self driving systems that are currently in development have full control of the car, an always connected component, and a dynamic software update capability. Should it be required they also have a mechanical switch that forces manual control without any means of circumventing the switch in software? That would have to come as a government regulation right? And then what happens when someone steals the Waymo van by throwing the switch and driving away with it? Hopefully that gives you a sense of where my head is here in these questions.

nojvek 6y ago
Our society is built on Trust. You have to trust your bank to keep your money in your name and allow you to withdraw when you want it. You have to trust that the fire truck will come when you have a fire. You have to trust that your neighbours won’t randomly wake up one day become zombies thirsty for your blood. Same way you ought to trust, but not blind trust. Trust but verify. I think the self driving game will be won with someone who as a transparency mindset. Here’s the car. Here’s the software, here’s the guarantee that it hasn’t been modified and here’s the extensive tests that we’ve run on it, verified by 3rd parties vouching for its safety and quality. Same with any technology company. You win by making solid technology backed with real rigorous testing vouched by trusted 3rd parties and general public.
- ChuckMcM 6y ago
  I agree with you. For folks reading along, there is an entire discipline around this stuff, it is call vulnerability analysis and surety systems. There are a lot of good papers published by Sandia National Laboratory on these topics. They were responsible for developing the US surety system around access to nuclear weapons[1]. When I worked at Sun they gave a talk at an e-comerce payment processing forum that Sun was participating in to discuss how you approach the problem of securing something in presence of known and unknown bad actors. In particular they discussed the systems around banking which prevent your bank from stealing your money from you. A topic that I found quite interesting. But one of the things that has always stuck with me was the discussion of the trade-off between the "cost of effort" to "actualize" a vulnerability. It is the difference between having something that could be done in theory versus having doing it having a high enough payoff to actually do it. When you look at things like dye packs in money and silent alarms and time locked safes, those are parts of a system that minimize the amount of money you can expect to make off with in a bank robbery. They are part of a surety system that is protecting the money in the bank. And they don't make it impossible to rob the bank, they make the likely-hood that you'll have enough profit from it to risk it low enough that people don't do it. [1] They are designed, in part, to prevent anyone from detonating a US nuclear device without specific authorization from the President.