5 ms·

The dynamic range is the reason Tesla know counts photons rather than use traditional camera processing. They basically remove the concept of exposure entirely

The dynamic range is the reason Tesla know counts photons rather than use traditional camera processing. They basically remove the concept of exposure entirely and simply pass the sensor photon counts to the neural net. This approach not only simpler as it removes photo processing/encoding but the result is that the NN can operate with a very high dynamic range similar to the human eye and in many cases can be sensitive on the single-photon level.

petilon 4y ago
But it doesn't appear to be helping. Here's an example accident where depth data from Lidar would have helped: "Tesla later said that during the crash, Autopilot’s camera could not distinguish between the white truck and the bright sky." https://www.nytimes.com/2021/12/06/technology/tesla-autopilot-elon-musk.html https://www.nytimes.com/2021/12/06/technology/tesla-autopilo...
- mensetmanusman 4y ago
  Humanity doesn’t know how to solve this yet, so it’s hard to say whether it is helping or not.
  - Gigachad 4y ago
    We already have the solution. LiDAR.
    - ALittleLight 4y ago
      If you could use LiDAR well enough then it would solve the problem. Of course, if you could use vision well enough it would solve the problem too.
      - touch_abs 4y ago
        The big limits of LiDAR are cost, more than anything. There have been dozens of public driving trials where from a functionality level the answer has been positive (apart from traffic lights, the bastards), but nobody wants to buy a solution with a six figure BOM, before integration.
        pokerhobo 4y ago
        Lidar also has problems in rain, fog, snow, etc… FLIR would actually be better
    - mensetmanusman 4y ago
      If LiDAR was a solution, we would have driverless LiDAR based vehicles. No one has solved driverless though.
- aeternum 4y ago
  The crash you referenced occurred in 2016 when they were using radar on the cars and I don't believe they were yet using raw photon counts nor did the NN have any voxel-based memory as it does now.
  - moffkalast 4y ago
    > any voxel-based memory Haha, any WHAT? Seriously though do you have any more info on that, it sounds intriguing. Where and how do voxels come into play in a 2D NN?
    - aeternum 4y ago
      It is pretty cool: https://youtu.be/ODSJsviD_SU?t=4355 https://youtu.be/ODSJsviD_SU?t=4355 They transitioned from 2D to 3D a couple years ago, major transition but it does seem like a critical step. We live in a 3d rather than 2d world.
davidgay 4y ago
> They basically remove the concept of exposure entirely and simply pass the sensor photon counts to the neural net. That sentence does not make sense. There's no such thing as a count without a corresponding interval that count occurred over. That interval is the exposure. You can of course do lots of (very) short exposures to avoid sensor saturation. That's "just" a movie at a very high frame rate. And then you can post-process this in lots of exciting ways, align the frames, average them, etc, etc.
- aeternum 4y ago
  Yeah that's fair. A CCD sensor basically converts individual photons to electrical charges. What Tesla has said they've done is thrown away all the traditional image signal processing & post-processing which often includes a lot of exposure-related averaging. You're right though that we don't typically use real-time neural networks that operate based upon spike rate, so an interval needs to be chosen for photon counting which could be considered a kind of exposure and it is critical that the interval be short enough to avoid saturation.
  - Maxion 4y ago
    Lol this doesn't make any sense. The dynamic range of a fully sunlit California highway during noon in the summer (I.e. the brightness reading of the darket vs the brightest spot) is wayyyy higher than any existing sensor. You cannot ignore exposure, you have to choose which part of the scene you want within the brightness range that your camera sensor can capture. You will have areas of the scene that clip, in other words areas of the scene that are pure black or white with no data. You can do bracketed exposures, but that's literally the opposite of ignoring exposure.
    - aeternum 4y ago
      Just keep the duration low so that you never saturate the sensor even in bright sunlight and let the NN do the summations. At a fundamental level it is somewhat akin to bracketing except all that HDR processing/frame matching is performed within the NN rather than a traditional image processing stack. The NN is better at this anyway since it must already be performing camera/pose motion tracking to correlate what it's seeing from frame to frame.
emkoemko 4y ago
how do you count photons continuously? what... this makes no sense, if you pass "the photon count" you just did a exposure... also how does a photo diode count photons?
moralestapia 4y ago
Does it have electrolytes as well? Nice tech and single photons and whatnot but it still runs into things that a radar with some really simple code wouldn't. ¯\_(ツ)_/¯
lambdasquirrel 4y ago
Counting photons won't keep a camera from being "jammed." Unless you are using a physically perfect polarizing filter, such that each pixel on the sensor only receives photons from the exact angular window, traced back through the lenses, you have a camera that can ultimately be "jammed." The human eye isn't so great on those terms. But humans can raise their hand to block the sun if it's straight at our eyes.
whiddershins 4y ago
The replies to your comment don't seem to understand you at all. in the video link here https://youtu.be/ODSJsviD_SU?t=4424 https://youtu.be/ODSJsviD_SU?t=4424 he clearly states 16x dynamic range as a result of direct photon processing.

petilon 4y ago

But it doesn't appear to be helping. Here's an example accident where depth data from Lidar would have helped: "Tesla later said that during the crash, Autopilot’s camera could not distinguish between the white truck and the bright sky." https://www.nytimes.com/2021/12/06/technology/tesla-autopilot-elon-musk.html https://www.nytimes.com/2021/12/06/technology/tesla-autopilo...

mensetmanusman 4y ago
Humanity doesn’t know how to solve this yet, so it’s hard to say whether it is helping or not.
- Gigachad 4y ago
  We already have the solution. LiDAR.
  - ALittleLight 4y ago
    If you could use LiDAR well enough then it would solve the problem. Of course, if you could use vision well enough it would solve the problem too.
    - touch_abs 4y ago
      The big limits of LiDAR are cost, more than anything. There have been dozens of public driving trials where from a functionality level the answer has been positive (apart from traffic lights, the bastards), but nobody wants to buy a solution with a six figure BOM, before integration.
      - pokerhobo 4y ago
        Lidar also has problems in rain, fog, snow, etc… FLIR would actually be better
  - mensetmanusman 4y ago
    If LiDAR was a solution, we would have driverless LiDAR based vehicles. No one has solved driverless though.
aeternum 4y ago
The crash you referenced occurred in 2016 when they were using radar on the cars and I don't believe they were yet using raw photon counts nor did the NN have any voxel-based memory as it does now.
- moffkalast 4y ago
  > any voxel-based memory Haha, any WHAT? Seriously though do you have any more info on that, it sounds intriguing. Where and how do voxels come into play in a 2D NN?
  - aeternum 4y ago
    It is pretty cool: https://youtu.be/ODSJsviD_SU?t=4355 https://youtu.be/ODSJsviD_SU?t=4355 They transitioned from 2D to 3D a couple years ago, major transition but it does seem like a critical step. We live in a 3d rather than 2d world.

davidgay 4y ago

> They basically remove the concept of exposure entirely and simply pass the sensor photon counts to the neural net. That sentence does not make sense. There's no such thing as a count without a corresponding interval that count occurred over. That interval is the exposure. You can of course do lots of (very) short exposures to avoid sensor saturation. That's "just" a movie at a very high frame rate. And then you can post-process this in lots of exciting ways, align the frames, average them, etc, etc.

aeternum 4y ago
Yeah that's fair. A CCD sensor basically converts individual photons to electrical charges. What Tesla has said they've done is thrown away all the traditional image signal processing & post-processing which often includes a lot of exposure-related averaging. You're right though that we don't typically use real-time neural networks that operate based upon spike rate, so an interval needs to be chosen for photon counting which could be considered a kind of exposure and it is critical that the interval be short enough to avoid saturation.
- Maxion 4y ago
  Lol this doesn't make any sense. The dynamic range of a fully sunlit California highway during noon in the summer (I.e. the brightness reading of the darket vs the brightest spot) is wayyyy higher than any existing sensor. You cannot ignore exposure, you have to choose which part of the scene you want within the brightness range that your camera sensor can capture. You will have areas of the scene that clip, in other words areas of the scene that are pure black or white with no data. You can do bracketed exposures, but that's literally the opposite of ignoring exposure.
  - aeternum 4y ago
    Just keep the duration low so that you never saturate the sensor even in bright sunlight and let the NN do the summations. At a fundamental level it is somewhat akin to bracketing except all that HDR processing/frame matching is performed within the NN rather than a traditional image processing stack. The NN is better at this anyway since it must already be performing camera/pose motion tracking to correlate what it's seeing from frame to frame.

emkoemko 4y ago

how do you count photons continuously? what... this makes no sense, if you pass "the photon count" you just did a exposure... also how does a photo diode count photons?

moralestapia 4y ago

Does it have electrolytes as well? Nice tech and single photons and whatnot but it still runs into things that a radar with some really simple code wouldn't. ¯\_(ツ)_/¯

lambdasquirrel 4y ago

Counting photons won't keep a camera from being "jammed." Unless you are using a physically perfect polarizing filter, such that each pixel on the sensor only receives photons from the exact angular window, traced back through the lenses, you have a camera that can ultimately be "jammed." The human eye isn't so great on those terms. But humans can raise their hand to block the sun if it's straight at our eyes.

whiddershins 4y ago

The replies to your comment don't seem to understand you at all. in the video link here https://youtu.be/ODSJsviD_SU?t=4424 https://youtu.be/ODSJsviD_SU?t=4424 he clearly states 16x dynamic range as a result of direct photon processing.