5 ms·

Python straw man aside, Comma could improve their safety dramatically, today, by documenting their safety systems, requiring a specific, audited checklist for n

by bri3d 6y ago

Python straw man aside, Comma could improve their safety dramatically, today, by documenting their safety systems, requiring a specific, audited checklist for new vehicles to be introduced, producing a real change management and detailed test result report for each release, utilizing a real safety processor in their "Panda" CAN intercept product, and working to reduce their use of bus effects like message-stuffing or first-in-wins racing with CAN control messages where possible. But there's no way I would bring this up in their GitHub, for example, because it isn't the culture of the project - and that's the problem. For example, their "leader" shows up and makes commits to their safety code like "WTF WHY WAS THIS SHIT EVERYWHERE." To Comma's credit, they have made great strides in safety since the first two years of their product - since 2018, they've introduced safety checks that (while undocumented), seem to be somewhat consistent (or trying to be) across vehicles, are written in MISRA C which is at least something that's actually used in the field, and exist at the low level of the system. Commits are signed now, and their founder seems to make fewer non-documenting troll commits to the product. These are all signs that the company is growing up (and, if we look at the history of the company and their leadership, we certainly see some signs as to why this may be the case...). But, these are things that should have been in place since day one, and the founders and company were certainly not unaware. It is absolutely fair to judge products and companies for the decisions they make, ESPECIALLY when their product is purportedly "world-changing," and in the case of Comma, safety does not seem ingrained into the decision-making process. As an aside, this blog post is killing me, between extensive Peter Thiel quotes, random jabs at California's COVID policy, Elon worship, and a lionization of a startup company with an immature founder building safety-related hardware.

23iofj 6y ago
> For example, their "leader" shows up and makes commits to their safety code like "WTF WHY WAS THIS SHIT EVERYWHERE." Every self-driving product that deploys to consumers is going to end up in court eventually. It's inevitable. Imagine trying to defend comma.ai to a lay jury... the prosecution won't even have to try.
- acct776 6y ago
  There's still some culture of personal responsibility in the US.
Hydraulix989 6y ago
I'd pessimistically imagine the real commit trees of actual SDC companies then are no different, given the "move fast and break things" ethos many of these companies have. Meanwhile, taking these extra rote precautions (e.g. cut down on global state, throw unit tests on everything, don't use go-to's) has no real effect when the elephant in the room is that even correctly-implemented state-of-the-art machine learning models only have classification accuracies in the mid 90%s at best. That's (relatively) fine if you're deciding what ad to show to a teenager using TikTok. When we are talking about driving a car, that loosely means having an accident every 100 miles. Based on publicly available disengagement data, this corroborates (only Waymo has surpassed the 1000 mile mark, GM might have recently, too). Meanwhile, verification in this domain only means replaying actual recorded sensor data in a "simulation" using one-off example scenarios, not doing any kind of formal analysis or exhaustive proofs -- which simply don't exist yet for ML. There's no real way currently to take an arbitrary mis-classified example and determine its root cause -- or even correct it within the existing model without reducing accuracy elsewhere! It's literally impossible to avoid being unsafe, so code style rules and procedures amount to mere hoop-jumping
- bri3d 6y ago
  I agree with you overall - in my post, I was trying to accept the premise of the overall safety model: "the ML will break but as long as the driver is paying attention it's OK." This was out of an effort to be fair to Comma, as, to your point, this is the overall state of the industry. I disagree, however that code style rules and procedures amount to hoop-jumping, as not every failure mode is the same (especially if you do accept the driver-attention / driver-in-control premise). An unexpected failure at the control layer (especially the control override layer) is a different class of failure to an unexpected direction command from the ML model, for example.
offtop5 6y ago
I think it's a matter of personal responsibility on one hand, but rest assured when this product get someone killed, Comma will be in court. I don't think they have the money to fight multiple Court challenges, even if you're right you still need to fight. I doubt this company will be around in 5 years, and I at least hope anyone who installs this thing in their car and then gets into an accident loses their license.
- elicash 6y ago
  It's not a matter of personal responsibility when drivers using this software can crash into other drivers. Just like we don't let people drive drunk. I'm actually not anti-Comma, just responding to this specific point.
bri3d 6y ago
I cannot edit my original post any more, but a helpful HNer "DesolationJones" was able to provide a little bit more insight into Comma's safety strategy using information trawled from Reddit comments and Medium posts: https://news.ycombinator.com/item?id=26158158 https://news.ycombinator.com/item?id=26158158 . It seems that Comma are actively working on documenting their safety systems (via a new hire to own this task) and creating an audited checklist. And, there is a "new vehicle" checklist enumerated in a Medium post including some safety constraints, although in reading the Panda "safety" tree, I found no actual acknowledgement that these specific constraints were audited for each vehicle addition. So, it seems that the great strides since 2018 continue - once these promises come true, they will have almost reached what I would consider the baseline level for a safety-related automotive system. Perhaps they will be able to secure an exit this way - I am actually rather surprised they haven't, given the froth level in this space as well as the valuable training data they have collected from their users.
- commaai 6y ago
  We aren't interested in an exit. Our mission is to "solve self driving cars while delivering shippable intermediaries"

by bri3d 6y ago

23iofj 6y ago

> For example, their "leader" shows up and makes commits to their safety code like "WTF WHY WAS THIS SHIT EVERYWHERE." Every self-driving product that deploys to consumers is going to end up in court eventually. It's inevitable. Imagine trying to defend comma.ai to a lay jury... the prosecution won't even have to try.

acct776 6y ago
There's still some culture of personal responsibility in the US.

Hydraulix989 6y ago

I'd pessimistically imagine the real commit trees of actual SDC companies then are no different, given the "move fast and break things" ethos many of these companies have. Meanwhile, taking these extra rote precautions (e.g. cut down on global state, throw unit tests on everything, don't use go-to's) has no real effect when the elephant in the room is that even correctly-implemented state-of-the-art machine learning models only have classification accuracies in the mid 90%s at best. That's (relatively) fine if you're deciding what ad to show to a teenager using TikTok. When we are talking about driving a car, that loosely means having an accident every 100 miles. Based on publicly available disengagement data, this corroborates (only Waymo has surpassed the 1000 mile mark, GM might have recently, too). Meanwhile, verification in this domain only means replaying actual recorded sensor data in a "simulation" using one-off example scenarios, not doing any kind of formal analysis or exhaustive proofs -- which simply don't exist yet for ML. There's no real way currently to take an arbitrary mis-classified example and determine its root cause -- or even correct it within the existing model without reducing accuracy elsewhere! It's literally impossible to avoid being unsafe, so code style rules and procedures amount to mere hoop-jumping

bri3d 6y ago
I agree with you overall - in my post, I was trying to accept the premise of the overall safety model: "the ML will break but as long as the driver is paying attention it's OK." This was out of an effort to be fair to Comma, as, to your point, this is the overall state of the industry. I disagree, however that code style rules and procedures amount to hoop-jumping, as not every failure mode is the same (especially if you do accept the driver-attention / driver-in-control premise). An unexpected failure at the control layer (especially the control override layer) is a different class of failure to an unexpected direction command from the ML model, for example.

offtop5 6y ago

I think it's a matter of personal responsibility on one hand, but rest assured when this product get someone killed, Comma will be in court. I don't think they have the money to fight multiple Court challenges, even if you're right you still need to fight. I doubt this company will be around in 5 years, and I at least hope anyone who installs this thing in their car and then gets into an accident loses their license.

elicash 6y ago
It's not a matter of personal responsibility when drivers using this software can crash into other drivers. Just like we don't let people drive drunk. I'm actually not anti-Comma, just responding to this specific point.

bri3d 6y ago

I cannot edit my original post any more, but a helpful HNer "DesolationJones" was able to provide a little bit more insight into Comma's safety strategy using information trawled from Reddit comments and Medium posts: https://news.ycombinator.com/item?id=26158158 https://news.ycombinator.com/item?id=26158158 . It seems that Comma are actively working on documenting their safety systems (via a new hire to own this task) and creating an audited checklist. And, there is a "new vehicle" checklist enumerated in a Medium post including some safety constraints, although in reading the Panda "safety" tree, I found no actual acknowledgement that these specific constraints were audited for each vehicle addition. So, it seems that the great strides since 2018 continue - once these promises come true, they will have almost reached what I would consider the baseline level for a safety-related automotive system. Perhaps they will be able to secure an exit this way - I am actually rather surprised they haven't, given the froth level in this space as well as the valuable training data they have collected from their users.

commaai 6y ago
We aren't interested in an exit. Our mission is to "solve self driving cars while delivering shippable intermediaries"