14 ms·
This is pretty bad from a hardware/software verification perspective. People who care about correctness can just go home, I guess. It's amazing that we're gonna
by titzer 8y ago
This is pretty bad from a hardware/software verification perspective. People who care about correctness can just go home, I guess. It's amazing that we're gonna let these things on the streets. When you weigh almost 2 tons and can go >100mph, it doesn't matter if you have a gun mounted on you or not, you are a weapon.
- ben174 8y agoIt only has to perform better than a human.
- deleted 8y ago[deleted]
- titzer 8y agoHumans are not remotely programmable.
- vntok 8y agoThe advertising industry would beg to differ.
- IMTDb 8y agoEver heard of the concept of "propaganda" ?
- mhh__ 8y agoThere is obviously a difference between hacking a single vehicle and (say) convincing a generation of youth to die for a cause. Also, while I agree that you can "program" a human, you can't program them to jump up and down and touch their toes (so to speak)
- rohit2412 8y agoBetter than a human aided by the best driver assistance technology
- thomaskcr 8y agoI think you're right if you look at things from a statistical perspective but I don't think it's going to end up actually working out like that. A person can explain "why" they did something when an accident happens. I've spent a lot of consulting hours helping unwind the "why" of ML/AI models, and we bill regardless of whether we can actually find that answer. Putting aside by personal bias that I think Tesla is functionally incapable of doing anything well, lets pretend they deploy FSD - I think the fact that in that 1 accident to every 100+ human accidents you can't find "why" will override the fact that there are xx% less accidents. Just from my experience consulting in healthcare and working with actuaries during the Obamacare debates and modeling out the impact of that -- people just honestly aren't open to statistical arguments when it comes to human life or emotional issues. (Not saying they're wrong, me and my wife completely disagree about so many issues where it's emotions vs statistics, I think she has a very valid point on many of those - what I'm trying to say is this is going to be a philosophical debate more than a statistics debate so Tesla should gear up accordingly).
- make3 8y agodoes correctness even work with deep neural networks to begin with? especially with the existence of adversarial examples, out of domain situations, etc
- wetpaws 8y agoAdversarial examples are a thing even for humans, and yet here we are.
- Ultimatt 8y agoYeah because all the verified neural networks made from meat travelling >100mph are legit ok to drive 100% of the time. There are no road accident deaths to speak of.
- nikofeyn 8y agothere is something fundamentally different between a machine and human. how the brain operates is not well known, so to claim our brain is a neural network made of meat is just a guess. the truth is, people have a lot of experience with other people. we can usually predict behavior of other drivers on the road, and can even predict pathological behavior or know how to handle it. that is a major point. when machines and software fail, they can fail in big, unpredictable ways. humans fail in fairly predictable ways. none of us have this experience with neural networks or machines. i barely trust my computer to work on a daily basis, much less a vehicle. and lastly, when a human is driving, fault is usually clearly and easily assignable. how fault is assigned in the case of machines driving is not clear at all.
- Ultimatt 8y agoThe fault hardly matters, the issue is one of safety. Humans are not safe already. Yet you feel entirely fine on the road. Humans are clearly not the same as ANNs but they are a network of neurones and the cognitive ability of a driver is not verified anywhere near frequently enough globally. In the UK post driving test about the only time anyone will check you for ability to drive is when you are declared blind from an eye test. But its not like you need to mandatorily have an eye test regularly, or any other kind of test. Unless you want to dispute a 70 year old human brain is not on average in the same kind of cognitive shape as a 30 year old human brain https://sci-hub.tw/10.1111/j.1532-5415.2012.04059.x https://sci-hub.tw/10.1111/j.1532-5415.2012.04059.x Yet we have no policies saying over 70s cannot drive, or can only drive in good conditions etc. The verifiability of a driver is not something we have ever demanded before. Even when you could have done plenty for safety when the driver was a person.
- nikofeyn 8y ago
- fhood 8y agoWhat do you mean by correctness in this context?
- Nae3Au5x 8y agoIn the same sense that any object that can potentially kill someone (hint: everything) is a weapon. Which is technically correct but a useless observation. If we use a more narrow definition of weapon, e.g. a tool optimized for the primary purpose of injuring or killing people then it certainly is not a weapon.
- titzer 8y ago> any object that can potentially kill someone (hint: everything) is a weapon Any object that can be remotely programmed to drive itself at high velocity into a target, yes.
- TeMPOraL 8y ago> In the same sense that any object that can potentially kill someone (hint: everything) is a weapon. Which is technically correct but a useless observation. It's not useless; the damage that can be caused varies by degree between things. Then you have to look at two cases - suitability for object to be used as a weapon, and the damage it can cause on accident. Unlike knives or hammers, both those factors are very high for cars. The fact how dangerous cars is is very much underappreciated by people in general, as evidenced by the number of morons on the road. We already lose hundreds of people daily in the US alone because of this; now we're trying to add another class of drivers into the mix - algorithms written by greedy optimizers caring primarily for short-term profit and being first to market. This should give us some pause. I'm not saying this technology is not possible or not wonderful, but I think the current ecology of self-driving efforts is unhealthy. We have a for-profit race by companies, many of which can't be trusted with getting software right, and most (all?) of them pursuing self-driving capabilities by means of half-understood brute-force black boxes the neural networks are.
- Nae3Au5x 8y agoI was quibbling about semantics. I am aware that cars can be dangerous. > and the damage it can cause on accident You are conflating (un)safety of a tool used in the way it is intended (kitchen knife = cutting a steak) with accidents (cutting a finger) and with malicious use (stabbing people). Those three categories are not the same for object-that-may-act-as-weapon and object-designed-as-weapon. Conflating them collapses the number useful things we can communicate. So are you concerned about Tesla intentionally building killing instruments? Or potential for accidents? Or the potential for intentional misuse? > The fact how dangerous cars is is very much underappreciated by people in general, as evidenced by the number of morons on the road. Cars also provide immense utility. If all they did were providing the thrill of speeding then they would probably be banned as too dangerous. One of the tradeoffs is the overhead of enabling people to drive. We could drive down the number of morons by requiring astronaut training for vehicle operators but again, that tradeoff seems too harsh and it's more efficient to occasionally let people die in traffic accidents than letting them die because nobody qualified as ambulance driver. > We have a for-profit race by companies, many of which can't be trusted with getting software right In the short term this may cause more deaths than necessary. But on the other hand it might be the quickest way to find a winner and then hold the rest to the same standard. As long as the experimental fleets are small they are just a blip in the statistics. Right now they should be equated to the yearly batch of first-year drivers who have an inherently higher risk profile due to lack of experience. We still accept them on our roads in the expectation that they improve. What is important is to make sure that they are as good as or better than humans once they roll out in large fleets.
- 3pt14159 8y agoI worry less about software / hardware crashes leading to death and more about cyberattack. I've written about it before[0]: We need regulations before a mass breach of autonomous systems happens. Not after. This whole bullshit where corporations claim all sorts of things about their security then get owned trivially has happened for every OS and device out there. The supposedly secure QNX was installed into countless tanks, nuclear power plants, and autonomous devices before getting owned in 2017[1]. I don't buy this "just trust us" attitude from automotive and other autonomous manufacturers. One of them is going to fuck up at some point. [0] https://www.zachaysan.com/cars https://www.zachaysan.com/cars and apologies for the length; it was a very stressful time in my life. [1] http://support.blackberry.com/kb/articleDetail?articleNumber=000046674 http://support.blackberry.com/kb/articleDetail?articleNumber...
- mhneu 8y agoYES. Security compromise is the huge achilles heel of self-driving cars - and anything that is algorithmically controlled. When you have 100,000 self-driving cars on the road, and someone figures out how to hack them, and drive them simultaneously into crowds, that will be a BIG BIG problem. And this problem isn't just about self-driving cars. It is a fundamental issue with all algorithms. It's a lot harder to hack humans to do bad things (although it can be done with sustained messaging and propaganda). But algorithms can be compromised and exploited.
- TeMPOraL 8y ago> It's a lot harder to hack humans to do bad things I'd say, it's probably easier to hack individual humans, but - like everything in computing - hacking algorithms scales well, while human factors generally don't.
- mhneu 8y agoA thoughtful point.
- Shivetya 8y agoLet us not also forget, we also want to prevent governments from having the ability to shut down them down as well. While there may be a need to disable a car in a chase or such I am against the idea of wholesale shutting down a highway full of cars or similar scenarios.
- colordrops 8y agoThis is pure unmitigated FUD. Do you have a comment on the actual content of the article?
- titzer 8y agoFor example, chips have flaws. Hardware designers work incredibly hard to verify that their chip designs are functionally correct, because the cost of a correctness bug can be catastrophic. With the proliferation of custom hardware designs, verification of the whole stack is even more important. It's bad from the perspective that now instead of building on top and reusing a chip company's huge investment in a verified product, there is a custom product to which an unknown level of verification expertise has been applied. Doubly so because these chips are designed for approximate workloads, so they cut corners left and right on correctness for efficiency. People should be more skeptical.
- colordrops 8y agoBeing skeptical is fine, but it would be fair to back such a strong statement up with facts rather than completely ungrounded speculation.
- titzer 8y agoThis is not how verification works, generally. You have to prove your chip is correct.
- joshuamorton 8y agoI think his point is that you aren't backing up the statement that the chips are unverified. Nor does your statement that they are built for approximate workloads imply that the chips are unverified or cutting corners.
- slivym 8y agoNo it isn't. There are already standards in electronics for Functional safety and these chips will have to abide by them. Part of that is that there'll be fail-safe chips that are fully ASIL-D compliant, and the full system will be ASIL-D classified as well. The difficult part of this problem is the hardware, not the functional safety - we've got that pretty much well defined by now.