4 ms·

Orders of magnitude more capable than the best cameras? I wish. I need corrective lenses for my eyes to even work at all. With that fixed they feed my brain an

Orders of magnitude more capable than the best cameras? I wish. I need corrective lenses for my eyes to even work at all. With that fixed they feed my brain an image that's upside down, black and white except in the centre, which is covered in blood vessels and which has a blind spot. They also take a long time to adjust to sudden changes in lighting conditions, don't do any true depth sensing, suffer frequent frame drops and can't run for more than about 20 hours at a time before they basically stop working. My brain tries to hides all this from me, and makes me think that I see the world in glorious 3D technicolor all the time, but that's a lie as revealed by the many amusing optical illusions that have been discovered over the years. Meanwhile, today I used ML that knows more than me, can think and type faster than me, which is a much better artist than me and which can read and react far faster than me to visual stimuli. Oh, it can also easily look in every direction simultaneously without pausing or ever getting distracted or bored. Somehow it doesn't feel like I have a big advantage over computers when it comes to driving.

mlyle 2y ago
The cameras we are talking about have poor angular resolution, worse dynamic range than the human eye, and don’t do any direct depth sensing either.
- mike_hearn 2y ago
  Are we talking about Tesla's cameras or the "best" cameras? There are smartphone cameras that do depth sensing and HDR, and cameras are cheaper than eyeballs so composing them to get more angular resolution seems OK.
  - mlyle 2y ago
    ToF/structured illumination cameras are honestly not that capable. The maximum dynamic range of the eye is ~130dB. It's very difficult to push an imaging system to work well at the dark end of what the eye will do with any decent frame rate. It's not as different as it used to be, but even so: the Mk. I eyeball does pretty damn well compared to quite fancy cameras.
  - kajecounterhack 2y ago
    > There are smartphone cameras that do depth sensing and HDR Depth sensing is again, estimated or using time of flight sensors which is pretty much short-range lidar. HDR is used already in AV perception, but still loses to your eyeballs in dynamic range and processing time.
    - mike_hearn 2y ago
      Yes, but human depth sensing is also estimated. Eyeballs have high dynamic range but with high mode switching times. Walk from a bright area to a dark area and it'll take seconds for your eyes to adjust. Cameras are so cheap you can just have a regular day camera and a dedicated night vision camera together, switching between feeds can be done in milliseconds.
      - kajecounterhack 2y ago
        > Yes, but human depth sensing is also estimated. Robots aren't humans. You need accurate depth perception to maneuver a robot precisely, and you need ground truth depth measurements to train learned depth perceivers as well as to understand their overall performance. Humans learn it by combining their other senses and integrating over very long time using very powerful compute hardware (brain). To date, robots learn it best when you just get the raw supervision signal directly using LiDAR. > Walk from a bright area to a dark area and it'll take seconds for your eyes to adjust You do realize cameras have the same issue, and that HDR isn't free / is very computationally intensive?
kajecounterhack 2y ago
> My brain tries to hides all this from me Your brain is _really really_ good at surmounting challenges including many that you did not mention. We don't know how to get close to this in terms of reliability when using cameras and ML alone. Cameras and ML alone can go very far, but every roboticist understands the problem of compounding errors and catastrophic failure. Every ML person understands how slow our learning loops are. Consider that ML models used in the field have to get by with a fixed amount of power and ram. If you want to process time context of say 5 seconds, and with temporal context 10Hz and with resolution 1080p, how much data bandwidth are you looking at? Comparing what you see with your eyes with a series of 1080p photos, which is better? Up it to 4k: how long does it take to even run object detection and tracking with a limited temporal context? Your brain is working with more temporal context, more world context, and has a much more robust active learning loop than the artificial systems we're composing today. It's really impressive what we can achieve, but to those who've worked on the problem it feels laughable to say you can solve it with just cameras and compute. There are plenty of well respected researchers who think only data and active learning loops are the bottlenecks. In my experience they're focused on treating the self driving task as a canned research problem and not a robotics problem. There are as many if not more respected researchers who've worked on the self-driving problem and see deeper seated issues -- ones that cannot be surmounted without technologies like high fidelity sensors grounded in physics and HD maps. Even if breadth of data is the problem and Tesla's approach is supposedly yielding more data -- there is also the question of the fidelity of said data (e.g. the distances and velocities from camera-only systems are estimated and have noiser gaussians than ones generated with LiDAR). If you make what you measure, and your measurements are noisy, how can you convince yourself or your loss function for that matter that it's doing a good job of learning? It's relatively straightforward to build toy systems where subsystems have something on the order of 95% reliability. But robotics requires you to cut the tail much further. https://wheretheroadmapends.com/game-of-9s.html https://wheretheroadmapends.com/game-of-9s.html

mlyle 2y ago

The cameras we are talking about have poor angular resolution, worse dynamic range than the human eye, and don’t do any direct depth sensing either.

mike_hearn 2y ago
Are we talking about Tesla's cameras or the "best" cameras? There are smartphone cameras that do depth sensing and HDR, and cameras are cheaper than eyeballs so composing them to get more angular resolution seems OK.
- mlyle 2y ago
  ToF/structured illumination cameras are honestly not that capable. The maximum dynamic range of the eye is ~130dB. It's very difficult to push an imaging system to work well at the dark end of what the eye will do with any decent frame rate. It's not as different as it used to be, but even so: the Mk. I eyeball does pretty damn well compared to quite fancy cameras.
- kajecounterhack 2y ago
  > There are smartphone cameras that do depth sensing and HDR Depth sensing is again, estimated or using time of flight sensors which is pretty much short-range lidar. HDR is used already in AV perception, but still loses to your eyeballs in dynamic range and processing time.
  - mike_hearn 2y ago
    Yes, but human depth sensing is also estimated. Eyeballs have high dynamic range but with high mode switching times. Walk from a bright area to a dark area and it'll take seconds for your eyes to adjust. Cameras are so cheap you can just have a regular day camera and a dedicated night vision camera together, switching between feeds can be done in milliseconds.
    - kajecounterhack 2y ago
      > Yes, but human depth sensing is also estimated. Robots aren't humans. You need accurate depth perception to maneuver a robot precisely, and you need ground truth depth measurements to train learned depth perceivers as well as to understand their overall performance. Humans learn it by combining their other senses and integrating over very long time using very powerful compute hardware (brain). To date, robots learn it best when you just get the raw supervision signal directly using LiDAR. > Walk from a bright area to a dark area and it'll take seconds for your eyes to adjust You do realize cameras have the same issue, and that HDR isn't free / is very computationally intensive?

kajecounterhack 2y ago

> My brain tries to hides all this from me Your brain is _really really_ good at surmounting challenges including many that you did not mention. We don't know how to get close to this in terms of reliability when using cameras and ML alone. Cameras and ML alone can go very far, but every roboticist understands the problem of compounding errors and catastrophic failure. Every ML person understands how slow our learning loops are. Consider that ML models used in the field have to get by with a fixed amount of power and ram. If you want to process time context of say 5 seconds, and with temporal context 10Hz and with resolution 1080p, how much data bandwidth are you looking at? Comparing what you see with your eyes with a series of 1080p photos, which is better? Up it to 4k: how long does it take to even run object detection and tracking with a limited temporal context? Your brain is working with more temporal context, more world context, and has a much more robust active learning loop than the artificial systems we're composing today. It's really impressive what we can achieve, but to those who've worked on the problem it feels laughable to say you can solve it with just cameras and compute. There are plenty of well respected researchers who think only data and active learning loops are the bottlenecks. In my experience they're focused on treating the self driving task as a canned research problem and not a robotics problem. There are as many if not more respected researchers who've worked on the self-driving problem and see deeper seated issues -- ones that cannot be surmounted without technologies like high fidelity sensors grounded in physics and HD maps. Even if breadth of data is the problem and Tesla's approach is supposedly yielding more data -- there is also the question of the fidelity of said data (e.g. the distances and velocities from camera-only systems are estimated and have noiser gaussians than ones generated with LiDAR). If you make what you measure, and your measurements are noisy, how can you convince yourself or your loss function for that matter that it's doing a good job of learning? It's relatively straightforward to build toy systems where subsystems have something on the order of 95% reliability. But robotics requires you to cut the tail much further. https://wheretheroadmapends.com/game-of-9s.html https://wheretheroadmapends.com/game-of-9s.html