11 ms·
Algorithm Removes Water from Underwater Images
- a_t48 7y agoSome static before/after pictures would _really_ help this article. I get that it's intended to be consumed as a video, but comeon.
- erikig 7y agoFor anyone that’s interested - skip to the last 30s of the video for the best before/after examples.
- deleted 7y ago[deleted]
- lilyball 7y agoThis looks really cool. She described how she takes photos that include her color chart, and I'm wondering if that's actually necessary to calibrate the process, or if that was just done for the purposes of developing it.
- iask 7y agoThe change in the shades of the color chart (due to lighting) is probably used as a tolerance in the algorithm.
- stan_rogers 7y agoIt's necessary for calibration at shooting time. It's a very similar technique to using, say, a ColorChecker Passport or a SpiderCHEKR with their associated profile-creation software for still photography, the main difference being that attenuation underwater is much more variable that lighting conditions on dry land. You need to accommodate attenuation of both the incident light (light hitting the subject has to get there through local water conditions) and light reflected from the subject at different depths at different times of the day and year, and with different "water" compositions (minerals, particulates, etc.).
- zackangelo 7y agoThe researcher that authored the paper answered a few questions on Reddit last year [0]. She explained that the color chart isn’t necessary for every photo. [0] https://www.reddit.com/r/videos/comments/dvts2j/this_researcher_created_an_algorithm_that_removes https://www.reddit.com/r/videos/comments/dvts2j/this_researc...
- ivanech 7y agoactual project site with more detailed description and before+after images: https://www.deryaakkaynak.com/sea-thru https://www.deryaakkaynak.com/sea-thru paper [pdf]: http://openaccess.thecvf.com/content_CVPR_2019/papers/Akkaynak_Sea-Thru_A_Method_for_Removing_Water_From_Underwater_Images_CVPR_2019_paper.pdf http://openaccess.thecvf.com/content_CVPR_2019/papers/Akkayn...
- 2bitencryption 7y agothat's fascinating, and a lot more involved than what I expected (my guess was something more like off-the-shelf color correction)
- loeg 7y agoThe palette cards they carry around remind me of white-balancing.
- mstade 7y agoIf I understand things correctly it's not entirely different to white balancing either, just that the difference between white balancing a monitor or camera is that you don't really have to correct for the scattering of light through air because it's negligible at the distances you're likely to work with. The trick here – again, if I understand correctly – is that this method also finds the light scattering/absorption characteristics and corrects for those as well. It might seem obvious, but I think it's more involving than it seems, because light scatters differently in different environments and I think – again, I might be wrong, this is just my understanding – this method accounts for that by using multiple images at different distances to gather the parameters to find the scattering and absorption characteristics. This is what makes it physically accurate and not just "make these colors mostly look good" which is really quite cool and useful!
- kd5bjo 7y agoIf I remember correctly, the percentage of light that passes through a column of water exponentially decays with distance, like stacking a bunch of fixed-percentage optical filters on top of each other. They’re using parallax shift between multiple images to generate a depth map, so they have a measure of the amount of water obscuring each pixel. I’m not sure how they’re finding the optical properties of the water; there’s something about using the darkest pixels in the image for calibration that I didn’t quite follow.
- Redoubts 7y agohttps://news.ycombinator.com/item?id=21542184 https://news.ycombinator.com/item?id=21542184
- blt 7y agoThis would be a great application for deep learning. Use the authors' method to generate a lot of uncorrected-corrected pairs. Or, use a graphics engine to render realistic underwater scenes with and without water color. Then use a convolutional neural network to learn to mimic the transformation. Then any photographer can apply the learned filter without a color card or depth information. Edit: already been done: https://arxiv.org/abs/1702.07392 https://arxiv.org/abs/1702.07392
- aspaceman 7y agoYou should take a look at the paper itself which is linked above. It does not require a color card in all cases, and the only information required is the depth, which can be obtained from a series of photos instead if required.
- cetra3 7y ago> Sea-thru is protected by a registered patent owned by Carmel Ltd the Economic Co. of the Haifa University and its subsidiary SeaErra Looks like they are going to monetise this technology at some point given the disclaimer at the bottom of the page. This is not wrong. But it feels like a PR exercise dressed up as something academic which is a little creepy.
- sytelus 7y agoAll universities seems to be patenting the hell out of their research these days.
- hijklmno 7y agoWhen you have a colour chart for reference and known distances between the camera and the chart, it's not difficult to colour correct. They might have a better interpretation of the underwater imaging phenomenon, which anyone can reverse engineer and use (from the published work). Patent's are just for reputation.
- lm28469 7y ago> When you have a colour chart for reference and known distances between the camera and the chart, it's not difficult to colour correct. I think the whole point of that project is that yo udon't need a chart.
- smnrchrds 7y agoIt is not unusual in academia to patent all innovations with potential commercial applications. At least in Canada, universities typically have innovation centres whose main job is encouraging and helping professors, graduate students, and other researchers patent their innovations and commercialize them (e.g. by licensing the patent). It is not sinister, it is normal procedure in academia.
- loeg 7y agoAnd the funding for the research that allowed for this invention was paid for by? The inventor? Or the government?
- chairmanwow1 7y agoI hate to say it, but that was actually a really poorly edited and produced video. It spent way too long on b-roll and did a really poor job framing the problem. I would have strongly preferred static images in the article and an interview video buried below.
- ec109685 7y agoThe video let them show a 30 second ad which monetizes much better than static ads. I agree they should have gotten to the punch line and show results rather than the doctor swimming.
- wallflower 7y agoScuba divers already use post-processing in Lightroom or apps like Dive+ (http://dive.plus/ http://dive.plus/). It will be interesting to see if this becomes popular in that community. The results are pretty good already with Dive+.
- peteretep 7y agoLooks a lot stronger than the Dive+ images I've seen (and created)
- oh-4-fucks-sake 7y agoWithout reading the transcript in detail and in total shitpost fashion: Could this algorithm be used to unobscure the bottom half of a bottomless woman submerged in a pool? Asking for a friend.
- stevenjohns 7y agoCoincidentally: the researcher's name, Derya, translates to "sea" or "ocean" in her native Turkish. I wonder if that's something that might have inspired her as part of this.
- teddyh 7y agohttps://en.wikipedia.org/wiki/Nominative_determinism https://en.wikipedia.org/wiki/Nominative_determinism
- nurettin 7y agoOn an unrelated note, the research happens to be sponsored by an Israeli university instead of a turkish one and the intellectual property belongs to their subsidiary. I wonder how that happened.
- netsharc 7y agoWhat a sad state of the world, that it causes bewilderment that a person with a Turkish name went to an Israeli university...
- nurettin 7y agoMore power to them, I'm just wondering why because there are ample subsidies for this kind of research. Surprised I was challenged for this statement, is that sad, too?
- luxuryballs 7y agoI don’t get how it’s not a photoshop though, it’s just a really specific photoshop. See the stuff how it would be on land... but it wouldn’t be like that on land at all. This is no more real or fake than any other filter applied to pixels.
- lilyball 7y agoIt's "not a photoshop" in that it's not someone manipulating the image until it looks good. It's an algorithm that (according to the video) uses precise physical modeling to achieve the correct color, regardless of aesthetic preferences.
- mieses 7y agodoes the method construct some model of the physical environment or does it just modify the color of each pixel based on the color chart (color calibration/correction)?
- DougBTX 7y ago> Sea-thru is a physics-based algorithm using the Akkaynak-Treibitz underwater image formation model (CVPR 2018). It does not use neural networks and was not trained on any dataset. It works without a color chart, or any information about the optical qualities or depth of the water body. https://www.deryaakkaynak.com/sea-thru https://www.deryaakkaynak.com/sea-thru
- JauntyHatAngle 7y agoWhy would it matter if it is a "photoshop" or not? We already use filters all the time on normal camera photos (e.g. Low Light ML on pixels). As long as it's correcting the colour for us to be able to assess it better, and it's accuracy is reasonably high, than it is gravy.
- luxuryballs 7y agoThat’s a question for the maker of the video who thought it was worth differentiating.
- wereHamster 7y agoNot a single image in the article. An article about images. What a shame.
- hinkley 7y agoI was watching a blooper real for an older movie the other day and realized that a very similar color rectangle appeared in some of the shots. Made me wonder how unique this technique is or if they’re just using bog standard movie editing tricks.
- _ph_ 7y agoUsing color-correction charts is a very standard photographic technology for correcting colors in any situation, where there is challenging lighting. So just using one for your underwater photography would give you vastly improved colors in your picture. As I understand their work, it uses a correction model which is distance dependant, so corrects the color shifting with distance. In air, changing the distance doesn't change the color appeareance, but under water, the absoption and scattering effects vary with distance.
- ivanhoe 7y ago> In air, changing the distance doesn't change the color appeareance Actually it does, it works similar to the water, just the effect is weaker in the air. That's why sky and distant mountains look blue, the atmosphere scatters blue light.
- _ph_ 7y agoRight, with lots of air, the same happens :).
- MisterBastahrd 7y agoColor correction is standard in publishing and happened long before movie editing.
- ismepornnahi 7y agoHow is this different from the colour chart idea? If we know how some actual KNOWN RGB pixels look in a particular setup, we can apply the same filter across the image. Right?
- kolinko 7y agoThe difference is that in this case the same colour will look different depending on how far away it is. With regular colour correction things have a slightly different colour regardless of how far away they are from the camera, so the task is way simpler.
- pindab0ter 7y agoTo add to that, as I understand it Sea-Thru calculates what colour should be where based on depth information, even if the original colour information is 'lost'. This makes it more than a simple filter you apply.
- Scaevolus 7y agoThe amount of color shift depends on how much water is between the object and the camera, so you need to have a depth map to recover the true colors. You can see how it compares to naive color transforms in the paper.
- Udik 7y ago> so you need to have a depth map to recover the true colors Isn't the blue shift of a known color already a measure of the amount of water between the object and the camera- and therefore its distance? Knowing the true color of a fish, a seaweed or the sand isn't already enough to infer distances and color-correct?
- Scaevolus 7y agoYes, but how do you know the true color of a fish, seaweed, or sand? That's the unknown part!
- untitled_ 7y agoThats pretty cool. We could do with a GoPro version of this lol.
- lqet 7y ago> It does not use neural networks and was not trained on any dataset.
- DocG 7y agoArticle that is not really an article but video about images. Why? :(
- nighthawk648 7y agoI read this and wonder if the technique can be applied to space. Too bad we can’t take photos of a closing distance of something 50 million light years away. I wonder if the algorithm would become better if not only did author did the swim closer but also took pictures of different path distances and angles simultaneously and map images together. Maybe it will reveal some of the editing work on hazy objects took a little more liberty and will produce more accurate images. Great read!
- iicc 7y agoWhat you want is a few lasers of known wavelengths (RGB?) pointing at known angles such that, from the camera's perspective, they appear as lines (ie they aren't perpendicular to the plane of the photo sensor). A calibration image(s) can be made before each shot. Possibly the resulting image correction can be integrated in to the camera too. The laser wavelengths are a substitute for the color chart. The laser angle means you get a reading at each distance (ie in the image, each point on the laser "line", corresponds to a distance.)
- simonh 7y agolaser beams are only visible if they reflect off something, so in a lot of situations the camera just won't see them except as points where they hit an object.
- deleted 7y ago[deleted]
- ynniv 7y agoHer algorithm doesn't require calibration. The color chart in many of the photos is to demonstrate the effect, or was just habit when she took them.
- londons_explore 7y agoTrue, but the method would totally benefit from calibration, because not all water is made the same - some has different densities of particulates in.
- nicwest 7y agoAlso I imagine that depth would be a significant factor
- rocqua 7y agoThat seems like a bad idea, because lasers are monochromatic. Sure, a red laser looks red. But just because you only see 50% of the brightness of the light of a red laser doesn't mean the absorption of all red colors will be 50%. Seems a lot better to use some very mixed source of light.
- Rainymood 7y agoWhat if I now get a bunch of images before & after and train a neural network on this to "learn" the mapping. Who would the neural network belong to?
- vijay_nair 7y agoPerhaps a real-time version of this can be embedded into the diving goggles(AR) for murk-free dives.
- dmix 7y agoCurrently it’s only photoS and requires a carefully placed colour chart to sync up the colours. I don’t think that last bit is automatable with just glasses.
- bradyd 7y agoThe project website shows a video example and says that a color chart is not required. https://www.deryaakkaynak.com/sea-thru https://www.deryaakkaynak.com/sea-thru
- thrownaway954 7y agoNot trying to be a jerk... but at 2:43 in the video the show video of fish swimming in the reef and those fish are beautiful, their colors aren't affected by the problem she is describing... so my question is, was that footage of the fish run through this algorithm? And if not... then what is the actual point of this ground breaking technology when it can already be done without it?
- tabtab 7y agoNASA and JPL do similar things with Mars surface images to bring out details and color differences. The orange dusty sky normally washes everything in an orange tint and softens shading.
- miguelrochefort 7y agoIs anyone aware of something similar that works when the picture is taken outside the water? Something that could remove reflection and glare.