5 ms·

Converting a fisheye image into a panoramic, spherical or perspective projection

186 points by lnyan 4y ago

JoeAltmaier 4y ago
Transformations like these have wide applicability. Performing optics in software so to speak, is more flexible and cheaper than grinding lenses and building special cameras. I was charged once, with transforming images for projection onto a sphere through special lenses. I didn't know the lens transform, the manufacturer kept that secret. I did know that the intensity across the spherical surface was of uniform brightness - the manufacturer boasted about this. With that bound I could guess the transform, as each pixel of the source image would have to illuminate an equal surface area of sphere once projected. The manufacturer advertised a service to do this transform on your image, but they wanted $10K for each run! Once I presented my insights and a little math, they capitulated and did it at cost. I never got to write my projection software in the end.
- myth2018 4y ago
  Hey, that's a nice case study! Could you give me some pointers on where I could start studying about the math involved? Are those topics usually covered by good books on computer graphics?
  - jeffreygoesto 4y ago
    I loved "Multiple View Geometry" by Hartley & Zisserman, but also "Computer Vision" by Rick Szelisky is pretty comprehensible and covers a lot of (pre-ML) stuff.
  - triggercut 4y ago
    My goto, which has literally saved me weeks of banging my head against a wall, is: Schneider, P.J., Eberly, D.H., 2003. Geometric tools for computer graphics, Morgan Kaufmann Publishers indispensable
- daniel_reetz 4y ago
  Sounds like a classic f-theta lens design. I'm sure you were right about your transform, but it's also possible to equalize illumination by other means, like apodizing filters.
gus_massa 4y ago
> If "straight" lines are not straight that normally means the fisheye center or radius are not specified correctly or the angle is not defined correctly. Is it possible to make the computer guess the correct value with AI? I guess a method is to identify the artificial objects like walls and doors, and assume they have straight lines. The nice curved tops of the windows may be a problem. And probably there must be a special case for circles like the clock.
- hardnose 4y ago
  Calibrating using known objects should be possible. Most road sign dimensions are specified in national laws, motor vehicle headlights must be a specific distance apart, etc.
- bramblerose 4y ago
  For a single image, you'd need some heuristics to distinguish curved objects from camera distortions. If you have multiple frames, you can SLAM your way out of the problem (essentially, match features and solve a big fit which optimized feature positions, camera positions and camera distortions). But by far the easiest option is to just calibrate it, through https://docs.opencv.org/4.x/dc/dbb/tutorial_py_calibration.html https://docs.opencv.org/4.x/dc/dbb/tutorial_py_calibration.h... or one of the many alternative methods.
eldaisfish 4y ago
The applications of this are interesting to say the least. Something i am currently working on is using fisheye images of the sky to predict cloud motion and to then estimate solar PV power output in the short term i.e. one to five minutes out. It sounds simple but it isn’t yet even a child can observe cloud motion and predict to some degree when the disc of the sun will be obscured. Part of the inaccuracy is in the distortion caused by the fisheye lens. I would be interested in seeing how well this works for subjects at infinity i.e. a fisheye lens pointed at the sky.
m463 4y ago
I bought some interesting photos from an artist who has done some things like this, sort of in reverse: https://kovasi.photo/ https://kovasi.photo/ no connection to him other than as a customer. By the way, printing on aluminum seems like a really nice way to mount art on the wall
- rk2 4y ago
  Thanks for sharing ... his photography is surreal.
kazinator 4y ago
What would be good to see on this page: an example of the correction of unwanted fish-eye from a portrait photo of a face. You could fix bad profile photos and such without the need to upgrade to more expensive optics.
maxshm 4y ago
I have seen these pages few years ago. But I could not find source code for these examples
- makeworld 4y ago
  > The source code implementing the projections below is only available on request for a small fee.
naillo 4y ago
Just throw it into im2im stable diffusion (joking but it also would likely literally work).
dcanelhas 4y ago
Paul Bourke's webpage is a true internet gem for 3D reference information. I don't know how many times I must've landed on his page on 3D file format specifications.
- mdswanson 4y ago
  Me too. An amazing resource.
bwigfield 4y ago
I just got stable-diffusion running and thought the title of this would make an amusing prompt. https://imgur.com/a/9TgW7qy https://imgur.com/a/9TgW7qy
atonalfreerider 4y ago
Not sure if this was shared already, but here's an example of the reflection of Buzz Aldrin's visor unwrapped into a 360 image: https://www.reddit.com/r/space/comments/ooexmd/i_unwrapped_buzz_aldrins_visor_to_a_360_sphere_to https://www.reddit.com/r/space/comments/ooexmd/i_unwrapped_b...
abdullahkhalids 4y ago
A few years ago, I was working for a startup in Pakistan which was making small (5-10 person) portable planetariums. The idea was to make astronomy education accessible to people from every socio-economic background. Paul's work on this [1] was amazingly helpful to get our projection done well, and put on successful shows. Also, shout out to the European Space Agency for releasing high quality Planetarium documentaries under the CC license [2]. [1] http://paulbourke.net/dome/ http://paulbourke.net/dome/ [2] https://www.eso.org/public/videos/archive/category/fulldome/?&sort=-release_date https://www.eso.org/public/videos/archive/category/fulldome/...
unterbahn 4y ago
Wow, some of the distortions at the end of the article are really mind bending... For simpler fisheye and defisheye effects, I wrote this web based tool a few years ago which some may find useful! https://jywarren.github.io/fisheyegl/example/ https://jywarren.github.io/fisheyegl/example/
MatthiasWandel 4y ago
Back around 2006 I used "panorama tools" to do the same sort of thing. It had this cool feature of being able to work out lens calibration if it had two photos from the same viewpoint but different angles. It did however require careful tagging of reference points on each image.
- mnw21cam 4y ago
  The logical successor to panorama tools is hugin these days, and yes it is able to perform these transformations very easily. The control point generation can be done automatically now, although I admit it might struggle a little with fisheye images.
politician 4y ago
Has anyone tried asking DALL-E to do this?
triggercut 4y ago
Paul was kind enough to meet with me a few times, probably over a decade ago now, when I was a lost student trying to work out what ideas were worth exploring (I guess I still am). He showed me some photogramatery work he was working on, I was amazed at how smooth everything was running given the precision and having worked with highly detailed tins and pointclouds myself on that days high end hardware. I asked him what kernel/engine he was using. He casually said it was his own. Which (having done toy ray tracing in C and dabling in openGL) blew my mind but also made me realise it would be absolutely necessary in order to do this kind of work with the type of control he needed. Everything from the ground up. One of the few academics/researchers I've met who is 100% legit.
skeletonjelly 4y ago
That shot is so iconically inner Sydney. Thinking Glebe or Paddington
eternityforest 4y ago
This means that open software for handling the raw files from 360 action cams is possible! Would be amazing if kdenlive et al could do it natively.

186 points by lnyan 4y ago

Transformations like these have wide applicability. Performing optics in software so to speak, is more flexible and cheaper than grinding lenses and building special cameras. I was charged once, with transforming images for projection onto a sphere through special lenses. I didn't know the lens transform, the manufacturer kept that secret. I did know that the intensity across the spherical surface was of uniform brightness - the manufacturer boasted about this. With that bound I could guess the transform, as each pixel of the source image would have to illuminate an equal surface area of sphere once projected. The manufacturer advertised a service to do this transform on your image, but they wanted $10K for each run! Once I presented my insights and a little math, they capitulated and did it at cost. I never got to write my projection software in the end.

myth2018 4y ago
Hey, that's a nice case study! Could you give me some pointers on where I could start studying about the math involved? Are those topics usually covered by good books on computer graphics?
- jeffreygoesto 4y ago
  I loved "Multiple View Geometry" by Hartley & Zisserman, but also "Computer Vision" by Rick Szelisky is pretty comprehensible and covers a lot of (pre-ML) stuff.
- triggercut 4y ago
  My goto, which has literally saved me weeks of banging my head against a wall, is: Schneider, P.J., Eberly, D.H., 2003. Geometric tools for computer graphics, Morgan Kaufmann Publishers indispensable
daniel_reetz 4y ago
Sounds like a classic f-theta lens design. I'm sure you were right about your transform, but it's also possible to equalize illumination by other means, like apodizing filters.

gus_massa 4y ago

> If "straight" lines are not straight that normally means the fisheye center or radius are not specified correctly or the angle is not defined correctly. Is it possible to make the computer guess the correct value with AI? I guess a method is to identify the artificial objects like walls and doors, and assume they have straight lines. The nice curved tops of the windows may be a problem. And probably there must be a special case for circles like the clock.

hardnose 4y ago
Calibrating using known objects should be possible. Most road sign dimensions are specified in national laws, motor vehicle headlights must be a specific distance apart, etc.
bramblerose 4y ago
For a single image, you'd need some heuristics to distinguish curved objects from camera distortions. If you have multiple frames, you can SLAM your way out of the problem (essentially, match features and solve a big fit which optimized feature positions, camera positions and camera distortions). But by far the easiest option is to just calibrate it, through https://docs.opencv.org/4.x/dc/dbb/tutorial_py_calibration.html https://docs.opencv.org/4.x/dc/dbb/tutorial_py_calibration.h... or one of the many alternative methods.

eldaisfish 4y ago

The applications of this are interesting to say the least. Something i am currently working on is using fisheye images of the sky to predict cloud motion and to then estimate solar PV power output in the short term i.e. one to five minutes out. It sounds simple but it isn’t yet even a child can observe cloud motion and predict to some degree when the disc of the sun will be obscured. Part of the inaccuracy is in the distortion caused by the fisheye lens. I would be interested in seeing how well this works for subjects at infinity i.e. a fisheye lens pointed at the sky.

m463 4y ago

I bought some interesting photos from an artist who has done some things like this, sort of in reverse: https://kovasi.photo/ https://kovasi.photo/ no connection to him other than as a customer. By the way, printing on aluminum seems like a really nice way to mount art on the wall

rk2 4y ago
Thanks for sharing ... his photography is surreal.

kazinator 4y ago

What would be good to see on this page: an example of the correction of unwanted fish-eye from a portrait photo of a face. You could fix bad profile photos and such without the need to upgrade to more expensive optics.

maxshm 4y ago

I have seen these pages few years ago. But I could not find source code for these examples

makeworld 4y ago
> The source code implementing the projections below is only available on request for a small fee.

naillo 4y ago

Just throw it into im2im stable diffusion (joking but it also would likely literally work).

dcanelhas 4y ago

Paul Bourke's webpage is a true internet gem for 3D reference information. I don't know how many times I must've landed on his page on 3D file format specifications.

mdswanson 4y ago
Me too. An amazing resource.

bwigfield 4y ago

I just got stable-diffusion running and thought the title of this would make an amusing prompt. https://imgur.com/a/9TgW7qy https://imgur.com/a/9TgW7qy

atonalfreerider 4y ago

Not sure if this was shared already, but here's an example of the reflection of Buzz Aldrin's visor unwrapped into a 360 image: https://www.reddit.com/r/space/comments/ooexmd/i_unwrapped_buzz_aldrins_visor_to_a_360_sphere_to https://www.reddit.com/r/space/comments/ooexmd/i_unwrapped_b...

abdullahkhalids 4y ago

A few years ago, I was working for a startup in Pakistan which was making small (5-10 person) portable planetariums. The idea was to make astronomy education accessible to people from every socio-economic background. Paul's work on this [1] was amazingly helpful to get our projection done well, and put on successful shows. Also, shout out to the European Space Agency for releasing high quality Planetarium documentaries under the CC license [2]. [1] http://paulbourke.net/dome/ http://paulbourke.net/dome/ [2] https://www.eso.org/public/videos/archive/category/fulldome/?&sort=-release_date https://www.eso.org/public/videos/archive/category/fulldome/...

unterbahn 4y ago

Wow, some of the distortions at the end of the article are really mind bending... For simpler fisheye and defisheye effects, I wrote this web based tool a few years ago which some may find useful! https://jywarren.github.io/fisheyegl/example/ https://jywarren.github.io/fisheyegl/example/

MatthiasWandel 4y ago

Back around 2006 I used "panorama tools" to do the same sort of thing. It had this cool feature of being able to work out lens calibration if it had two photos from the same viewpoint but different angles. It did however require careful tagging of reference points on each image.

mnw21cam 4y ago
The logical successor to panorama tools is hugin these days, and yes it is able to perform these transformations very easily. The control point generation can be done automatically now, although I admit it might struggle a little with fisheye images.

politician 4y ago

Has anyone tried asking DALL-E to do this?

triggercut 4y ago

Paul was kind enough to meet with me a few times, probably over a decade ago now, when I was a lost student trying to work out what ideas were worth exploring (I guess I still am). He showed me some photogramatery work he was working on, I was amazed at how smooth everything was running given the precision and having worked with highly detailed tins and pointclouds myself on that days high end hardware. I asked him what kernel/engine he was using. He casually said it was his own. Which (having done toy ray tracing in C and dabling in openGL) blew my mind but also made me realise it would be absolutely necessary in order to do this kind of work with the type of control he needed. Everything from the ground up. One of the few academics/researchers I've met who is 100% legit.

skeletonjelly 4y ago

That shot is so iconically inner Sydney. Thinking Glebe or Paddington

eternityforest 4y ago

This means that open software for handling the raw files from 360 action cams is possible! Would be amazing if kdenlive et al could do it natively.