5 ms·

anyone care to talk about this part "This feature is currently operating in “shadow mode” in the fleet, which compares our software algorithm to real-world driv

anyone care to talk about this part "This feature is currently operating in “shadow mode” in the fleet, which compares our software algorithm to real-world driver behavior across tens of millions of instances around the world." that seems very juicy? edit: it seems people put a disclaimer on their ownership of tesla stock on hn about tesla threads. i dont own any.

leesec 7y ago
This is a known feature unveiled, I believe, on Tesla's autonomy day. It just means they're able to test Autopilot on cars even if they aren't running autopilot.
- boldslogan 7y ago
  neat. I would say im optomistic on this actual self driving stuff. But coming from almost no machine learning knowledge, doesn't this make tesla way ahead of google/waymo and others? Because they can simply check when they failed and tweak. I'm just curious if it does make them way ahead, and if so how much more ahead? Is this driving a car vs horse wagon levels?
  - leesec 7y ago
    Which company is in the lead is a very contentious topic it seems. This one specific feature doesn't seem to push it one way or the other. But IMO Tesla has the best approach and has a huge lead on the data collection pipeline. No other self driving company has even close to as many cars to test on / gather data from.
    - bllguo 7y ago
      contentious? Genuinely curious for a source here. everything ive ever seen says Waymo is far in the lead, though I dont work in this space.
      - leesec 7y ago
        The contentious part is that Waymo doesn't have a product yet. It's easy to imagine something is better when you can't actually use it. In truth there are only 3 self driving products that I know of available today. Tesla's Autopilot, GM's Supercruise (only on CT6), and commma.ai's openpilot. Of those Autopilot is easily the best. I never know why a source would matter in these cases as the writers of these sources usually have no idea and the answer isn't clear anyways. But if you want you can use me as a source because I'm an engineer who worked in automotive and has driven all 3 of aforementiond products. I also am not sure Waymo will ever release a feasible product other than their LIDAR.
        icelancer 7y ago
        >> It's easy to imagine something is better when you can't actually use it. Yup. Carmack illustrated this very well in his interviews when he would not talk about unreleased games vs. Doom/Quake/whatever. He always said: "It's really easy to be the best graphics engine when it isn't available to the public."
        bllguo 7y ago
        That is a great point. Would you agree that the _narrative_ is that Waymo has the superior tech right now? Or did I misjudge that? I suppose I used the wrong word. Any expert or well-informed opinion is what I was after.
        leesec 7y ago
        Yes I would agree Waymo owns the narrative and a lot of it is based on their disengagement stats. I wouldn't say those tell the whole story. They are also completely different approaches, i.e. Waymo using LIDAR and Tesla relying on cameras as the primary sensor. But the major question is can LIDAR _ever_ be financially feasible, and if so can it become feasible faster than camera gets good enough at vision. Right now the sensors on the Waymo van probably cost more than the van itself. That's either a very expensive purchase for the consumer or a very large amount of cap-ex Waymo would have to put up for a fleet of self-driving taxis. And if it turns out cameras were the right choice, or visa versa, that means the opposite company would have spent a LOT of their R&D funding on the wrong tech, and would be way behind.
    - mrep 7y ago
      > But IMO Tesla has the best approach and has a huge lead on the data collection pipeline. Not only that, but by using cheap hardware that consumers can reasonably afford, they really only need to get to level 3 (eyes off so you can work/entertain but driver still in the seat) for it to be massively beneficial for everyone who commutes to work. Waymo pretty much requires level 4 as it's too expensive to be sold as a consumer car and nobody will be in a taxi when it is driving between customers.
  - Klathmon 7y ago
    I think the real answer is nobody knows. It's obviously got some benefits, and I think most people in ML would agree that real world data from the "average person" (as in "not a Dev or paid to do this") is worlds better than simulations. Still, "ahead" and "behind" isn't really a thing before you reach the finish line here. Many think Tesla will never reach full self driving or won't reach it without additional sensors, others think it's possible but decades or more away, still others think it's right around the corner and can be done with what is on a Tesla right now. Until someone reaches that finish line, we won't have anything to compare it to, so we don't have any idea how close or far anyone really is.
    - jameslevy 7y ago
      Is there any public research or data regarding the value or predicted delta value of long tail "average person" driving data of this kind versus the kind of data captured by driving around the same small portions of SF or MV? I've looked around but haven't found anything. Granted, Tesla and its competitors don't have an incentive to publish this kind of data...but are they even in a position to be able to calculate it currently?
      - Animats 7y ago
        Probably not. Ordinary staying-on-road is pretty much solved. There's years of dashcam imagery available. The last minute of video before each crash would probably be more useful. Training on dashcam data of people doing stupid stuff from Youtube might be useful. You don't need steering, braking, and accel data; you can run a 3D SLAM algorithm to extract the path and get that.
        xbmcuser 7y ago
        My 2 problems with Musk is that he has dismissed lidar in favor of just cameras because lidar is expensive. So far most of the Tesla auto pilot accidents would have been avoided with lidar. Even 2 very basic lidar sensors on the front and back could make a huge difference. And he calls a driver assistance system autopilot when it hasn't reached the level of autonomy which the word autopilot implies.
  - dkonofalski 7y ago
    Probably not way ahead but definitely ahead. Waymo has hundreds, if not thousands of fleet vehicles driving all day, every day around various parts of the country too and they can also check when people took over or when the brakes were hit. I think Tesla's approach is a little neater and nicer but that's just an opinion.
    - leesec 7y ago
      Tesla has hundreds of thousands driving and growing rapidly?
      - dkonofalski 7y ago
        There are a lot of Teslas that do not have the hardware necessary to do the training required. Additionally, Waymo cars are driving all day long while most Teslas are still commuter vehicles.
- 11thEarlOfMar 7y ago
  Or, it means that the feature is running and the neural nets exercising, but not outputting actual control. The results are captured, but not acted on.
  - ceejayoz 7y ago
    That's how I took it. They can then compare to what the human driver actually did, and find false positives (driver continued when the AI would've stopped) and false negatives (driver stopped when the AI would've continued).
    - abledon 7y ago
      ugh! the amount of supervised training data that would generate is staggering.
      - deleted 7y ago
        [deleted]
      - kolbe 7y ago
        Hence why many investors don't mind that Tesla isn't generating wonderful cash flow. If there's a reasonable expectation that this data collection will lead to full autonomy, then Tesla should price their cars at whatever level is necessary to move their maximum production into consumers' hands, even if it were a loss (which it's not). And even if they don't use the data to train, they can use it as an argument for how much better they are than humans. That's one hurdle that Waymo could have trouble with, whereas Tesla could point to tens of millions of miles where their sensors would have avoided crashes that humans failed to avoid.
        abledon 7y ago
        Do you think they keep all that data on premise or use AWS/Azure/GCP? edit: probably AWS [1] [1] https://www.theinquirer.net/inquirer/news/3027077/hackers-hijack-teslas-unsecured-aws-account-to-mine-cryptocurrency https://www.theinquirer.net/inquirer/news/3027077/hackers-hi...
- MegaButts 7y ago
  Disclaimer: I own no financial stake in Tesla, and while I love the goal of converting the world towards electric vehicles as whole I am not a fan of Tesla or Elon Musk. There are people much more informed than myself that believe Shadow Mode isn't real, and that Tesla is outright lying about its existence. https://twitter.com/greentheonly/status/1096322810694287361 https://twitter.com/greentheonly/status/1096322810694287361
  - ajross 7y ago
    That twitter thread (which is great, btw, everyone interested in this stuff should be following @greentheonly) doesn't remotely substantiate "Shadow Mode isn't real, and that Tesla is outright lying". I don't know the extent of public statements about it, but comparing collected data like the stuff detailed there to driver behavior is certainly feasible and almost certainly being done. It's also likely true that the marketing department had their way with the announcement and made it sound more like an on-vehicle entity, but.. meh. I just don't see how you got from that (again, really great) analysis to such an outraged conclusion.
    - MegaButts 7y ago
      I was stating his opinion, but I'll just quote him directly: > We'll start with the bitter truth. The "shadow driver that just sits there in the computer comparing notes and sending discrepancies and interesting events to Tesla" is a myth. I used to think people just misunderstood Elon, but now I believe Tesla lies about it on purpose I really did not mean for my comment to be inflammatory (hence the disclaimer and including a source), but considering how quickly it was downvoted I guess people do not approve.
      - ricardobeat 7y ago
        That quote has no substance still. In the thread he proves that he can inspect some data logging routines in his Tesla, how does that lead to “shadow mode does not exist” is a mystery. It is an obvious thing to do and the ‘shadowing’ aspect would not be particularly challenging tech, why the disbelief?
dahdum 7y ago
> We are making progress towards stopping at stop signs and traffic lights. This feature is currently operating in “shadow mode” in the fleet, which compares our software algorithm to real-world driver behavior across tens of millions of instances around the world. It's cool that they can collect that data and no doubt will help refine the feature, but otherwise marketing speak. Says nothing about how accurate their algorithm is or how long it may take. There's nothing stopping them from rolling out an algo for anything and running it in "shadow mode".
OliverJones 7y ago
They do this a lot when they're trying out new automation. It's part of the payback for all the investment they've put into field updates and telemetry. I think this data-gathering is part of the reason they continue to absorb the costs of LTE communications. It's an excellent way to prove (or disprove) the effectiveness of new safety-related features. As for non-safety-related features, my 3-yr-old granddaughter calls my Model S the "fartmobile" because of a recent electronic whoopee-cushion easter egg.
rndmize 7y ago
Sounds like they're running their autopilot software on some (all?) of the cars in the wild, where they can compare what the autopilot would do with the actions taken by the human driving the car, and use this to analyze/refine their model/system. In casual talks with friends, this is something I've expected would be the competitive edge Tesla has over Waymo and the others; driving millions of "virtual miles" strikes me as far less useful than having a human baseline to compare to along with data capture from a fleet in all manner of real situations/locations.
- boldslogan 7y ago
  Which makes me imagine a crazy marketing video where they demonstrate their system is driving the right path and the human crashes. In huge black font, then they just show the statistics of how often that happens.
new_realist 7y ago
Doubt the model will coverage as the actions are from human drivers, not generated by the model.
m463 7y ago
They run autopilot in all the cars whether it is turned on or not. If it is not turned on, they still send the inputs to autopilot, without controlling the car. I think they can also compare what the driver does against what the autopilot would have done. They can also ask their "fleet" to match specific conditions, and collect data if they are met. The autonomy day video showed some interesting images, for example "animals" the fleet had seen. I think this is the video https://youtu.be/tbgtGQIygZQ https://youtu.be/tbgtGQIygZQ

leesec 7y ago

This is a known feature unveiled, I believe, on Tesla's autonomy day. It just means they're able to test Autopilot on cars even if they aren't running autopilot.

boldslogan 7y ago
neat. I would say im optomistic on this actual self driving stuff. But coming from almost no machine learning knowledge, doesn't this make tesla way ahead of google/waymo and others? Because they can simply check when they failed and tweak. I'm just curious if it does make them way ahead, and if so how much more ahead? Is this driving a car vs horse wagon levels?
- leesec 7y ago
  Which company is in the lead is a very contentious topic it seems. This one specific feature doesn't seem to push it one way or the other. But IMO Tesla has the best approach and has a huge lead on the data collection pipeline. No other self driving company has even close to as many cars to test on / gather data from.
  - bllguo 7y ago
    contentious? Genuinely curious for a source here. everything ive ever seen says Waymo is far in the lead, though I dont work in this space.
    - leesec 7y ago
      The contentious part is that Waymo doesn't have a product yet. It's easy to imagine something is better when you can't actually use it. In truth there are only 3 self driving products that I know of available today. Tesla's Autopilot, GM's Supercruise (only on CT6), and commma.ai's openpilot. Of those Autopilot is easily the best. I never know why a source would matter in these cases as the writers of these sources usually have no idea and the answer isn't clear anyways. But if you want you can use me as a source because I'm an engineer who worked in automotive and has driven all 3 of aforementiond products. I also am not sure Waymo will ever release a feasible product other than their LIDAR.
      - icelancer 7y ago
        >> It's easy to imagine something is better when you can't actually use it. Yup. Carmack illustrated this very well in his interviews when he would not talk about unreleased games vs. Doom/Quake/whatever. He always said: "It's really easy to be the best graphics engine when it isn't available to the public."
      - bllguo 7y ago
        That is a great point. Would you agree that the _narrative_ is that Waymo has the superior tech right now? Or did I misjudge that? I suppose I used the wrong word. Any expert or well-informed opinion is what I was after.
        leesec 7y ago
        Yes I would agree Waymo owns the narrative and a lot of it is based on their disengagement stats. I wouldn't say those tell the whole story. They are also completely different approaches, i.e. Waymo using LIDAR and Tesla relying on cameras as the primary sensor. But the major question is can LIDAR _ever_ be financially feasible, and if so can it become feasible faster than camera gets good enough at vision. Right now the sensors on the Waymo van probably cost more than the van itself. That's either a very expensive purchase for the consumer or a very large amount of cap-ex Waymo would have to put up for a fleet of self-driving taxis. And if it turns out cameras were the right choice, or visa versa, that means the opposite company would have spent a LOT of their R&D funding on the wrong tech, and would be way behind.
  - mrep 7y ago
    > But IMO Tesla has the best approach and has a huge lead on the data collection pipeline. Not only that, but by using cheap hardware that consumers can reasonably afford, they really only need to get to level 3 (eyes off so you can work/entertain but driver still in the seat) for it to be massively beneficial for everyone who commutes to work. Waymo pretty much requires level 4 as it's too expensive to be sold as a consumer car and nobody will be in a taxi when it is driving between customers.
- Klathmon 7y ago
  I think the real answer is nobody knows. It's obviously got some benefits, and I think most people in ML would agree that real world data from the "average person" (as in "not a Dev or paid to do this") is worlds better than simulations. Still, "ahead" and "behind" isn't really a thing before you reach the finish line here. Many think Tesla will never reach full self driving or won't reach it without additional sensors, others think it's possible but decades or more away, still others think it's right around the corner and can be done with what is on a Tesla right now. Until someone reaches that finish line, we won't have anything to compare it to, so we don't have any idea how close or far anyone really is.
  - jameslevy 7y ago
    Is there any public research or data regarding the value or predicted delta value of long tail "average person" driving data of this kind versus the kind of data captured by driving around the same small portions of SF or MV? I've looked around but haven't found anything. Granted, Tesla and its competitors don't have an incentive to publish this kind of data...but are they even in a position to be able to calculate it currently?
    - Animats 7y ago
      Probably not. Ordinary staying-on-road is pretty much solved. There's years of dashcam imagery available. The last minute of video before each crash would probably be more useful. Training on dashcam data of people doing stupid stuff from Youtube might be useful. You don't need steering, braking, and accel data; you can run a 3D SLAM algorithm to extract the path and get that.
      - xbmcuser 7y ago
        My 2 problems with Musk is that he has dismissed lidar in favor of just cameras because lidar is expensive. So far most of the Tesla auto pilot accidents would have been avoided with lidar. Even 2 very basic lidar sensors on the front and back could make a huge difference. And he calls a driver assistance system autopilot when it hasn't reached the level of autonomy which the word autopilot implies.
- dkonofalski 7y ago
  Probably not way ahead but definitely ahead. Waymo has hundreds, if not thousands of fleet vehicles driving all day, every day around various parts of the country too and they can also check when people took over or when the brakes were hit. I think Tesla's approach is a little neater and nicer but that's just an opinion.
  - leesec 7y ago
    Tesla has hundreds of thousands driving and growing rapidly?
    - dkonofalski 7y ago
      There are a lot of Teslas that do not have the hardware necessary to do the training required. Additionally, Waymo cars are driving all day long while most Teslas are still commuter vehicles.
11thEarlOfMar 7y ago
Or, it means that the feature is running and the neural nets exercising, but not outputting actual control. The results are captured, but not acted on.
- ceejayoz 7y ago
  That's how I took it. They can then compare to what the human driver actually did, and find false positives (driver continued when the AI would've stopped) and false negatives (driver stopped when the AI would've continued).
  - abledon 7y ago
    ugh! the amount of supervised training data that would generate is staggering.
    - deleted 7y ago
      [deleted]
    - kolbe 7y ago
      Hence why many investors don't mind that Tesla isn't generating wonderful cash flow. If there's a reasonable expectation that this data collection will lead to full autonomy, then Tesla should price their cars at whatever level is necessary to move their maximum production into consumers' hands, even if it were a loss (which it's not). And even if they don't use the data to train, they can use it as an argument for how much better they are than humans. That's one hurdle that Waymo could have trouble with, whereas Tesla could point to tens of millions of miles where their sensors would have avoided crashes that humans failed to avoid.
      - abledon 7y ago
        Do you think they keep all that data on premise or use AWS/Azure/GCP? edit: probably AWS [1] [1] https://www.theinquirer.net/inquirer/news/3027077/hackers-hijack-teslas-unsecured-aws-account-to-mine-cryptocurrency https://www.theinquirer.net/inquirer/news/3027077/hackers-hi...
MegaButts 7y ago
Disclaimer: I own no financial stake in Tesla, and while I love the goal of converting the world towards electric vehicles as whole I am not a fan of Tesla or Elon Musk. There are people much more informed than myself that believe Shadow Mode isn't real, and that Tesla is outright lying about its existence. https://twitter.com/greentheonly/status/1096322810694287361 https://twitter.com/greentheonly/status/1096322810694287361
- ajross 7y ago
  That twitter thread (which is great, btw, everyone interested in this stuff should be following @greentheonly) doesn't remotely substantiate "Shadow Mode isn't real, and that Tesla is outright lying". I don't know the extent of public statements about it, but comparing collected data like the stuff detailed there to driver behavior is certainly feasible and almost certainly being done. It's also likely true that the marketing department had their way with the announcement and made it sound more like an on-vehicle entity, but.. meh. I just don't see how you got from that (again, really great) analysis to such an outraged conclusion.
  - MegaButts 7y ago
    I was stating his opinion, but I'll just quote him directly: > We'll start with the bitter truth. The "shadow driver that just sits there in the computer comparing notes and sending discrepancies and interesting events to Tesla" is a myth. I used to think people just misunderstood Elon, but now I believe Tesla lies about it on purpose I really did not mean for my comment to be inflammatory (hence the disclaimer and including a source), but considering how quickly it was downvoted I guess people do not approve.
    - ricardobeat 7y ago
      That quote has no substance still. In the thread he proves that he can inspect some data logging routines in his Tesla, how does that lead to “shadow mode does not exist” is a mystery. It is an obvious thing to do and the ‘shadowing’ aspect would not be particularly challenging tech, why the disbelief?

dahdum 7y ago

> We are making progress towards stopping at stop signs and traffic lights. This feature is currently operating in “shadow mode” in the fleet, which compares our software algorithm to real-world driver behavior across tens of millions of instances around the world. It's cool that they can collect that data and no doubt will help refine the feature, but otherwise marketing speak. Says nothing about how accurate their algorithm is or how long it may take. There's nothing stopping them from rolling out an algo for anything and running it in "shadow mode".

OliverJones 7y ago

They do this a lot when they're trying out new automation. It's part of the payback for all the investment they've put into field updates and telemetry. I think this data-gathering is part of the reason they continue to absorb the costs of LTE communications. It's an excellent way to prove (or disprove) the effectiveness of new safety-related features. As for non-safety-related features, my 3-yr-old granddaughter calls my Model S the "fartmobile" because of a recent electronic whoopee-cushion easter egg.

rndmize 7y ago

Sounds like they're running their autopilot software on some (all?) of the cars in the wild, where they can compare what the autopilot would do with the actions taken by the human driving the car, and use this to analyze/refine their model/system. In casual talks with friends, this is something I've expected would be the competitive edge Tesla has over Waymo and the others; driving millions of "virtual miles" strikes me as far less useful than having a human baseline to compare to along with data capture from a fleet in all manner of real situations/locations.

boldslogan 7y ago
Which makes me imagine a crazy marketing video where they demonstrate their system is driving the right path and the human crashes. In huge black font, then they just show the statistics of how often that happens.

new_realist 7y ago

Doubt the model will coverage as the actions are from human drivers, not generated by the model.

m463 7y ago

They run autopilot in all the cars whether it is turned on or not. If it is not turned on, they still send the inputs to autopilot, without controlling the car. I think they can also compare what the driver does against what the autopilot would have done. They can also ask their "fleet" to match specific conditions, and collect data if they are met. The autonomy day video showed some interesting images, for example "animals" the fleet had seen. I think this is the video https://youtu.be/tbgtGQIygZQ https://youtu.be/tbgtGQIygZQ