5 ms·

The parent comment was clearly written by someone with no experience in robotics. In fact, every single problem on this list is "solved" in the sense that there

The parent comment was clearly written by someone with no experience in robotics. In fact, every single problem on this list is "solved" in the sense that there are examples of systems that perform it in some limited context. Generalizing those subsystems to handle the difficult edge cases and integrating those subsystems together is 99% of what makes robotics as a whole difficult, and it would be similarly deceptive to say that AI is "solved" because Microsoft Clippy exists.

sudosysgen 6y ago
I do have experience in robotics. There exists robust solutions to depth estimation and position estimation that are deployed in the real world, in difficult applications where they have been generalized to work with any object you can draw a box around. There is a difference between a problem that is solved in theory and a problem that is solved enough so that you can buy polished products that implement a solution and work essentially without failure. I don't think you can compare optical phase detection in the context of position estimation and depth detection to clippy in the context of AI. Phase detection is quite literally a closed form optical solution to the problem of "how far is this object away from me" as long as the object is within a few thousand times the physical aperture of the lens. It's mature enough that you can use it to drive a motor in response to movements or "that bird", "that teapot", "the closest object in that clump of pixels", "the farthest objects in that clump of pixels", as well as calculate the velocity of the aforementioned object in three dimensions. For all intents and purposes, if you have a problem of the order "what is the distance of that object" as well as "how is that distance changing over time", then you can solve it, and indeed it has been solved to very high reliability, using phase detection. That's what it means for a problem to be solved. In other words, I wouldn't call an implementation where you can click anywhere on an image and receive a distance, all of the time, with almost any lens imaginable in any environment where there are enough photons hitting the sensor "performed in a limited context". The technology simply hasn't been used in mainstream robotics, mainly because it's patented and difficult to implement from scratch, but these are all implementation problems not fundamental problems.
- cpgxiii 6y ago
  Structure from focus is an established technique that can produce quite good results, but it's not quite as easy as the existence of good autofocus in cameras would suggest. SfF techniques often operate the the inverse of an autofocus system - rather than focus on a specific point to figure out depth at that point, you run through a full focus cycle and then for each area of the image, you find the focus setting that made that area in focus. The challenge there is that running through a full focus cycle and capturing frames as you go is quite time-consuming and introduces serious artifacts if anything in view is moving. Phase-detection autofocus is so fast in part because it's only using a sparse set of autofocus points, rather than the full sensor resolution. The other major problems with SfF are the same the photographers encounter all the time - the depth value is only as accurate as the depth of field will allow, and shallow depth of field requires wide apertures (and is generally easier on larger sensors, as well). Large lenses and sensors may work well for stationary scanning applications, but struggle to be useful on robots.
  - sudosysgen 6y ago
    A lot of these problems have been fixed by recent advancements. For example, splitting a few thousand pixels into two and using that as a phase detection sensor provides universal coverage. As far as large lenses and sensors not working well on sensors, I think you'd be surprised just how well large lenses designed for the sensors they belong on can work. Indeed, by far the reason why photography lenses are huge is because of lens IS and zoom. Making small, light, fast normal lenses with larger stabilized sensors works perfectly fine and can be made very light. After all, humans have two huge sensors (slightly bigger than full-frame) with f/3 lenses, and it works perfectly fine. Now, to what I think is missing from the state of the art in robotics: >Phase-detection autofocus is so fast in part because it's only using a sparse set of autofocus points, rather than the full sensor resolution. This was true a decade ago, but now if you look at Canon sensors or some Sony sensors not subject to the patent, every single pixel is a phase detection point. Indeed, the pixels are cut in 2 or 4 photodiodes each only receiving light from one half or one quarter of the aperture. This means that every single pixel can detect phase information, all 45 million of them. Herein lies the major difference between Structure from Focus and phase detection : in a phase detection system, it is not needed at all to run a focus cycle. Instead, two waveforms are generated, one corresponding to one section of the aperture and one corresponding to another section of the aperture. The two waveforms "match" when the incident rays correspond to the same point, that is to say, when focus is achieved. However, it really isn't necessary to actually achieve focus - focus simply offsets the phase of the two waveforms. Therefore, by simply computing the phase difference of the two waveforms, one can instantly know, given knowledge of the lens characteristics, the distance of the subject, without having to achieve a focus cycle (!) Indeed, phase detection actually works very similarly to parallax, in that in actuality you can use it to construct two split, offset images. Of course, you can also use two stereo cameras, but then you have the issue of having to motorize the cameras to achieve convergence, without which stereo overlap is minimimal, wherehas per-pixel phase detection provides complete overlap and is much more precise. If you want to see this in practice, the RAW files of a Canon EOS R5 actually encode distance information for every single pixel. Also, >the depth value is only as accurate as the depth of field Yes. The depth value is precise, in a modern camera, to about one 5000th of the diameter of the aperture, which is, for a 50mm f1.8 normal lens, able to compute depth accurately for any subject within 50-60 meters. Which is better than LiDAR of the the same size, easily. And you can do so for a small fraction of the cost by simply upgrading hardware that is already necessary. It is true that modern cameras actually do move the focus and re-calculate depth, which might give the impression that if you wanted to calculate depth you would actually need to move focus. But in reality, this is done in order to correct for the small misalignment in different lenses, as well as for the fact autofocus motors do not have accurate encoders and will very frequently miss steps. But a modern camera already knows how much the image plane needs to actually shift before even engaging the autofocus motor at all. In essence, SfF in the State of the Art is an inversion of the state of autofocus 20 years ago, using contrast detection as the autofocus method. However, modern autofocus has progressed so much in this time-frame that it has solved almost all of the issues of SfF.

sudosysgen 6y ago

I do have experience in robotics. There exists robust solutions to depth estimation and position estimation that are deployed in the real world, in difficult applications where they have been generalized to work with any object you can draw a box around. There is a difference between a problem that is solved in theory and a problem that is solved enough so that you can buy polished products that implement a solution and work essentially without failure. I don't think you can compare optical phase detection in the context of position estimation and depth detection to clippy in the context of AI. Phase detection is quite literally a closed form optical solution to the problem of "how far is this object away from me" as long as the object is within a few thousand times the physical aperture of the lens. It's mature enough that you can use it to drive a motor in response to movements or "that bird", "that teapot", "the closest object in that clump of pixels", "the farthest objects in that clump of pixels", as well as calculate the velocity of the aforementioned object in three dimensions. For all intents and purposes, if you have a problem of the order "what is the distance of that object" as well as "how is that distance changing over time", then you can solve it, and indeed it has been solved to very high reliability, using phase detection. That's what it means for a problem to be solved. In other words, I wouldn't call an implementation where you can click anywhere on an image and receive a distance, all of the time, with almost any lens imaginable in any environment where there are enough photons hitting the sensor "performed in a limited context". The technology simply hasn't been used in mainstream robotics, mainly because it's patented and difficult to implement from scratch, but these are all implementation problems not fundamental problems.

cpgxiii 6y ago
Structure from focus is an established technique that can produce quite good results, but it's not quite as easy as the existence of good autofocus in cameras would suggest. SfF techniques often operate the the inverse of an autofocus system - rather than focus on a specific point to figure out depth at that point, you run through a full focus cycle and then for each area of the image, you find the focus setting that made that area in focus. The challenge there is that running through a full focus cycle and capturing frames as you go is quite time-consuming and introduces serious artifacts if anything in view is moving. Phase-detection autofocus is so fast in part because it's only using a sparse set of autofocus points, rather than the full sensor resolution. The other major problems with SfF are the same the photographers encounter all the time - the depth value is only as accurate as the depth of field will allow, and shallow depth of field requires wide apertures (and is generally easier on larger sensors, as well). Large lenses and sensors may work well for stationary scanning applications, but struggle to be useful on robots.
- sudosysgen 6y ago
  A lot of these problems have been fixed by recent advancements. For example, splitting a few thousand pixels into two and using that as a phase detection sensor provides universal coverage. As far as large lenses and sensors not working well on sensors, I think you'd be surprised just how well large lenses designed for the sensors they belong on can work. Indeed, by far the reason why photography lenses are huge is because of lens IS and zoom. Making small, light, fast normal lenses with larger stabilized sensors works perfectly fine and can be made very light. After all, humans have two huge sensors (slightly bigger than full-frame) with f/3 lenses, and it works perfectly fine. Now, to what I think is missing from the state of the art in robotics: >Phase-detection autofocus is so fast in part because it's only using a sparse set of autofocus points, rather than the full sensor resolution. This was true a decade ago, but now if you look at Canon sensors or some Sony sensors not subject to the patent, every single pixel is a phase detection point. Indeed, the pixels are cut in 2 or 4 photodiodes each only receiving light from one half or one quarter of the aperture. This means that every single pixel can detect phase information, all 45 million of them. Herein lies the major difference between Structure from Focus and phase detection : in a phase detection system, it is not needed at all to run a focus cycle. Instead, two waveforms are generated, one corresponding to one section of the aperture and one corresponding to another section of the aperture. The two waveforms "match" when the incident rays correspond to the same point, that is to say, when focus is achieved. However, it really isn't necessary to actually achieve focus - focus simply offsets the phase of the two waveforms. Therefore, by simply computing the phase difference of the two waveforms, one can instantly know, given knowledge of the lens characteristics, the distance of the subject, without having to achieve a focus cycle (!) Indeed, phase detection actually works very similarly to parallax, in that in actuality you can use it to construct two split, offset images. Of course, you can also use two stereo cameras, but then you have the issue of having to motorize the cameras to achieve convergence, without which stereo overlap is minimimal, wherehas per-pixel phase detection provides complete overlap and is much more precise. If you want to see this in practice, the RAW files of a Canon EOS R5 actually encode distance information for every single pixel. Also, >the depth value is only as accurate as the depth of field Yes. The depth value is precise, in a modern camera, to about one 5000th of the diameter of the aperture, which is, for a 50mm f1.8 normal lens, able to compute depth accurately for any subject within 50-60 meters. Which is better than LiDAR of the the same size, easily. And you can do so for a small fraction of the cost by simply upgrading hardware that is already necessary. It is true that modern cameras actually do move the focus and re-calculate depth, which might give the impression that if you wanted to calculate depth you would actually need to move focus. But in reality, this is done in order to correct for the small misalignment in different lenses, as well as for the fact autofocus motors do not have accurate encoders and will very frequently miss steps. But a modern camera already knows how much the image plane needs to actually shift before even engaging the autofocus motor at all. In essence, SfF in the State of the Art is an inversion of the state of autofocus 20 years ago, using contrast detection as the autofocus method. However, modern autofocus has progressed so much in this time-frame that it has solved almost all of the issues of SfF.