11 ms·
Learning to See in the Dark (2018)
- ksaxena 7y agoVideo from the paper here: https://www.youtube.com/watch?v=qWKUFK7MWvg https://www.youtube.com/watch?v=qWKUFK7MWvg
- GEBBL 7y agoImpressive of the American news channel, CNN, to convert images in minus one second.
- FreakyT 7y agoI, too, was initially very confused by the headline.
- wiredfool 7y agoWell, Turner used to colorize b/w movies, I guess progress marches on.
- S_A_P 7y agoWould add a new dimension to "fake news" if it were...
- sneakernets 7y agoSpeaking of, This would help a lot of awful dark photos sent in to news sources.
- pbhjpbhj 7y agoWell I got the right CNN by the time I got to the end of the title, but, "Turner"? The British pastoral landscape painter? You lost me.
- wiredfool 7y agoTed Turner founded CNN.
- chungy 7y agoCNN is owned by Turner Broadcasting
- oliveshell 7y agoFor reference: https://en.wikipedia.org/wiki/Ted_Turner https://en.wikipedia.org/wiki/Ted_Turner
- ggregoire 7y agoSeriously, what/who's CNN here?
- wyldfire 7y agoCNN here is a "Convolutional Neural Network"
- whoisjuan 7y agoI think OP was being sarcastic
- wyldfire 7y agoMe too, but lots of people may not know so I answered it as if it were sincere.
- gravitas 7y agoraises hand I interpreted it as the news company, looked at Github and came back going "what does CNN have to do with this, it looks like they're uni students." Thank you for the actual acronym expansion, it's obtuse if you don't know anything about this stuff. Terrible title.
- echelon 7y agoThe title needs to be changed so brains recognize it as such. It either needs a preceding adjective or letter indicating what type of convolutional network it is. The other option is spelling it out. Most people will read CNN as the news channel. Even those familiar with neural networks.
- numbernine 7y agoNot everyone is from the US...
- isoprophlex 7y agoThey should make a CLI tool, it stores your processed images one sec before it is invoked! See also: https://en.m.wikipedia.org/wiki/Thiotimoline https://en.m.wikipedia.org/wiki/Thiotimoline The major peculiarity of the chemical is its "endochronicity": it starts dissolving before it makes contact with water.
- StavrosK 7y agoSee also Ted Chiang's What’s Expected of Us, where there's a device with a button and a LED, and the LED always lights up one second before you press the button.
- ninju 7y agohttps://www.nature.com/articles/436150a https://www.nature.com/articles/436150a The heart of each Predictor is a circuit with a negative time delay — it sends a signal back in time.
- montenegrohugo 7y agoI understand that the title may have brought confusion, and I think your comment calls attention to this whilst also being somewhat funny. But still, could we make an effort not to devolve into what has happened on Reddit, i.e. comment sections which mainly consist of puns and other low effort jokes?
- zoomablemind 7y agoTitle is confusing (?intentionally). Would make sense to add Tensorflow to make it more specific.
- ccktlmazeltov 7y agoI don't understand what you are saying, it is very "HN" to comment on the title and not on the article. And it is even more "HN" to comment on details of the title or the article because you don't really know what to say about the article. Look at my comment.
- hombre_fatal 7y agoSure, and ideally those are downvoted. But low-grade jokes fit in here even less and are usually downvoted. At least that's how it is before HN finishes its transition into Reddit, at which point they won't be downvoted and we'll be back to scrolling through bad jokes like "lol CNN in the title looks like CNN the news network lmao, anyone else notice?" to find serious comments. It's a fight worth fighting. "C'mon it's just a joke lol" or "it's only a comment about the title" only assist in that transformation.
- pbhjpbhj 7y agoMeta: this is when I like the Slashdot method, you allow the jokes, tag them as jokes, and let people use their own settings to show/hide the jokes. Basically be permissive on content, but demand proper tagging, then allow people to filter out what they don't want.
- yingw787 7y agoYeah, my mind jumped to CNN's coverage of the first Gulf War and having some color night vision for combat journalism: https://www.thedrive.com/the-war-zone/25803/this-is-what-color-night-vision-looks-like-and-it-stands-to-be-a-game-changer https://www.thedrive.com/the-war-zone/25803/this-is-what-col...
- dang 7y agoPlease don't do this. What would be a fun crack in person becomes a plastic garbage patch here.
- babuskov 7y agoI was just wondering a couple days ago why the image from my phone is so grainy, while my eyes+brain can see everything clear in the dark (it wasn't completely dark, of course). This seems to replicate the post-processing we do in our brain (which is also a giant neural network). I wonder if the process is similar?
- jacobush 7y agoAlso, our eyes are better. So far.
- xwdv 7y agoI wonder what things would be like when phone cameras are as good as human eyes, the climax of consumer photography?
- koverda 7y agoThat's not really a good analogy. You have a totally different sensor chemistry in your eyes, as well as different processing.
- arpa 7y agoVery small numbers of photons (1) are required to trigger rhodopsin cycle. So primary receptor itself is very very VERY sensitive.
- jrimbault 7y agoWhy is the "page suspended" ? http://cchen156.web.engr.illinois.edu/paper/18CVPR_SID.pdf http://cchen156.web.engr.illinois.edu/paper/18CVPR_SID.pdf
- PStamatiou 7y agoif it's anything like my old college webserver.. too much traffic/bandwidth consumed
- toomuchtodo 7y agoWayback: http://web.archive.org/web/2018*/http://cchen156.web.engr.illinois.edu/paper/18CVPR_SID.pdf http://web.archive.org/web/2018*/http://cchen156.web.engr.il...
- selectodude 7y agoThe State of Illinois is out of money again.
- anigbrowl 7y agoYou shouldn't be downvoted - with a big recession/depression looming, link rot and many sorts of repositories shutting down are a big issue that will slow down the pace of research.
- dyycc 7y agoIt is back now.
- NikhilVerma 7y agoIt's surprising how little code [1] is needed to do this. On the other hand I feel this is quite dependent on the specific camera models and might not work on the RAW data downloaded from my phone. Happy to be corrected. [1] - https://github.com/cchen156/Learning-to-See-in-the-Dark/blob/master/train_Fuji.py https://github.com/cchen156/Learning-to-See-in-the-Dark/blob...
- covidacct 7y agoIt's a huge amount of code, hidden behind the tensorflow import statements. It's common to credit GPUs for the rapid spread of deep learning, but good GPUs were available for quite a few years before deep learning really took off. As someone who wrote * a lot* of OpenCL code, including my own python wrappers, I'm fairly certain this code would be thousands of lines without a computation graph framework library. These frameworks are really amazing pieces of software engineering and deserve some non-trivial fraction of the credit for the rise of deep learning. If you want to know what the next hot thing in software engineering will be, just pay attention to whatever Jeff Dean is doing.
- ganstyles 7y agoI don't know that I agree with this first statement, but even if I do, everything is abstracted by import statements even outside ML. You say this is a huge amount of code abstracted, but it wouldn't be difficult to reimplement this in numpy and pandas directly without using tensorflow at all. The code would expand a bit, and you'd have to deal directly with backprop and calculating derivatives but it wouldn't expand things too much. But then you could make the same claim about numpy abstracting the linear algebra, and I could show you that I could extract that and do it without numpy but then it would be the python math library. It's turtles all the way down. My point is, your comment applies to everything.
- covidacct 7y agoYup, I absolutely agree. Almost all big leaps in software engineering and applied computer science come from building a powerful and simple abstraction. Powerful and simple abstractions are surprisingly difficult to get right.
- css 7y agoWhat camera are they shooting at 409,600 ISO at?
- jeswin 7y agoThe Sony a7s and a7sii have a native iso range of 100 to 102400, which can be boosted to 409600.
- Traster 7y agoIn the video they reference the Sony A7S II, on Sony's website[1] they claim: >Still images: ISO 100-102400 (expandable to ISO 50-409600), [1]:https://www.sony.co.uk/electronics/interchangeable-lens-cameras/ilce-7sm2#product_details_defaul https://www.sony.co.uk/electronics/interchangeable-lens-came...
- dingaling 7y agoWhich is extremely lossy, because any ISO other than the sensor's native level is the result of in-camera processing. Unlike film, adjusting the "ISO" in a digital camera doesn't increase sensitivity; that's physically impossible. Instead, very strong overgain processing is applied. So in this instance they're processing lossily on top of an image already processed lossily in-camera.
- anigbrowl 7y agoThe A7g image is used for comparison rather than input data, as that camera is widely regard as the state of the art for low-light photography on a non-scientific/military budget.
- f- 7y agoAs a photographer, the comparison to "raw" results without color balance or noise removal seems somewhat deceptive. The effects visible in the video seem easy to quickly replicate with existing techniques, such as the "surface blur" filter that averages out pixel values in areas with similar color. This happens at the expense of detail in low-contrast areas, producing a plastic-like appearance of human skin and hair, and making low-contrast text unintelligible, which is why it's generally not done by default.
- sdenton4 7y agoYour example strikes me as the kind of thing neural networks are much better at than a fixed filter. You or I could easily identify regions of an image where it's safe vs unsafe to do the surface averaging, and boundaries where we wouldn't want to mix up the averages. (For example, averaging text should be fine, so long as you don't cross the text boundaries.) A CNN should also be able to learn to do this pretty easily.
- deleted 7y ago[deleted]
- BubRoss 7y agoWhat you are describing is a class of filters known as edge preserving filters. You can look at bilateral filters and guided filters for examples that have been around for decades at this point.
- sdenton4 6y agoSo we can do a decent job with hand designed filters... Why aren't they in use in the problem the parent describes? Are they not good enough to deal with small text boundaries? A lot of hand built filters (I see a lot of these in the audio space) have many hand tuned parameters, which work well in certain circumstances, and less well in other circumstances. One of the big advantages of NN systems is the ability to adapt to context more dynamically. The NN filters can generally emulate the hand designed system, and pick out weightings appropriate to the example.
- Invictus0 7y agoI believe this should have a (2018) tag.
- yokto 7y agoYes, the result image and videos are from 2018 https://github.com/cchen156/Learning-to-See-in-the-Dark/blob/master/images/fig1.png https://github.com/cchen156/Learning-to-See-in-the-Dark/blob...
- coenhyde 7y agoIt's a great result, but it's not perfect. No need for the "perfect" hyperbole in the title.
- mkchoi212 7y agoPretty cool but seems like there’s a big limitation on this for now “ The pretrained model probably not work for data from another camera sensor. We do not have support for other camera data. It also does not work for images after camera ISP, i.e., the JPG or PNG data.” Would be cool to see how they come up with better models that would allow them to overcome the above limitations
- jameshart 7y agoThe problem with techniques like this is that they fundamentally amount to ‘making a plausible guess as to what the image would look like’, since essentially they can’t extract information that is simply not there. There is a Shannon entropy limit here. Machine learning is really machine-enhanced educated-guesswork, which has its place but also has its limits.
- oconnor663 7y agoCounterpoint: The human brain converting a 2D image to a 3D model is educated-guesswork too :)
- war1025 7y agoThis hits on an interesting point. There is an entropy limit to the message, but the message isn't actually the only data. One thing humans are great at is integrating existing knowledge into a messy situation and intuiting more than is available just from the raw message. I.e. The message has an entropy limit, but the message isn't the whole dataset.
- MiroF 7y agoYes and that's what this "lossy" conversion to daytime does as well, incorporate prior knowledge, but that prior knowledge is about how images of real world things function during night versus day.
- gnramires 7y agoThis doesn't quite match Information Theory. First of all, if you were to use the image as a communication channel, how much you could theoretically communicate is exactly the entropy of messages (by definition), and optimal communication means maximum entropy. Information theory already assumes shared existing knowledge in the form of codes; the codes essentially encode all this knowledge, which you could make an analogy in images to digits and shapes, etc. -- what makes them decodable is there is statistical redundancy, shapes do not occur arbitrarily (i.e. not every possible shape occurs, at least not with equal probability) and exhibit dependence between different pixels of a shape and even between shapes elsewhere in the image. Again the dependence is (generally) based on the statistics of the distribution of all possible images -- it essentially encodes all prior knowledge. This redundancy allows reconstruction of losses in on part of the image from data elsewhere. It's the same principle used in error correcting codes, except codes are designed, while shapes are mostly natural (except things like alphabets, which are designed and indeed follow some principles of codes). But because they're not designed there's not guarantee of having a unique/reliable decoding (i.e. you can get a distribution). I think that's a significant issue, because if your estimate doesn't match reality it could have important consequences for the use of the image: perhaps text goes from 'X is good' to 'X sucks'. In this case a few things could be done: 1) Have some kind of watermark indicating the image was enhanced by a neural network, and possibly contains false information; 2) Have some kind of indication of reliability of the image: it should encode the multimodality/confidence of the decoding distribution -- how many different solutions does this have. If it is more or less unique, it would show as high confidence; otherwise it would show a low confidence indicator; 3) Instead of trying to convey uncertainty, the system could simply give up in cases where there is too much uncertainty, i.e. leaving the image dark. This could be done locally or globally, although locally it introduces a lightning consistency problem. --- There's another important observation w.r.t. Information Theory/Statistics: it essentially assumes unbounded computational power (since this distribution could require analyzing arbitrarily large datasets). Of course this isn't true in reality. For example, the entropy of an encrypted of a redundant text is exactly the entropy of the plaintext string plus the entropy of the key (given an encryption ensamble or encryption prior) -- the process of (e.g. through brute force) finding the key doesn't concern statistics. However, with reasonable computational power, the (properly) encrypted stream is indistinguishable from a random string, hence it would have maximum entropy. So there are further computational limits beside statistical limits. In the case of encryption the function is again designed (to be not tractable), while in natural images the correlations are of simpler and hopefully more tractable nature (although I'm sure not always the case).
- robmiller 7y agoI wonder if photographic evidence "enhanced" by such a method would be admissible in court?
- draugadrotten 7y agoPerhaps it depends on "which court". The legal systems in some parts of the world would allow it.
- q3k 7y agoFinally, a way to restore https://en.wikipedia.org/wiki/The_Night_Watch https://en.wikipedia.org/wiki/The_Night_Watch !
- soperj 7y agoJust wondering why for something brand new they'd use python 2.7??
- vehemenz 7y agoWhat would happen if this were paired with license plate recognition? And would it be admissible as evidence?
- penetrarthur 7y agoI always wondered if you can "trust" an image that has been basically recreated. Could that kind of image be used as an evidence in court?
- anigbrowl 7y agoIf you could show a basic level of consistency. Take the correctly rendered title on the book vs the incorrect colors; the odds that it got lucky with the text instead of a different title or a book with random letters on the cover are negligible. But if your evidence revolved around the color of the book the villain stole from the library, not so much. So if you're planning to do crime, make choices where the evidence relies on spectra rather than geometry. Steal Rothkos rather than Mondrians; baggy coveralls are in, form-fitting ninja wear is out.
- mattkrause 7y agoXerox copiers had a bug caused by a failed image (re)construction, which caused it to replace similar (but not identical) parts of an image with other pieces of the image. http://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres_are_switching_written_numbers_when_scanning http://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres_...?
- deleted 7y ago[deleted]
- jordache 7y agoisn't this just some tweaked raw processing algorithm?
- ChrisArchitect 7y ago(2018) and a better title that doesn't say CNN, c'mon
- dang 7y agoWe've added the year above. Submitted title was "CNN converts night images to perfect daylight in – 1s".
- skyde 7y agowhy use iso 8,000 as input and not the camera native ISO ?
- amelius 7y agoSome questions: - Did they create a special network topology for this problem? - Does the network need to see the entire image, or only an NxN subblock at a time? - How did they obtain the training data? Is it possible to take daylight images and automatically turn them into nighttime images somehow?
- todd3834 7y agoCould something like this be done for night vision goggles or is there significant latency?
- riazrizvi 7y agoAn interpolation that looks for movement of a few anchor points? I imagine that would entail much less computation and so deliver apparent real-time night vision. Though sudden big movements in scene would cause blackout regions of about a second?
- throwaway122378 7y agoNow all they have to do is make sure the correct image displayed for the story
- Aardwolf 7y agoI doubt this image is showing the true raw data (a): https://github.com/cchen156/Learning-to-See-in-the-Dark/blob/master/images/fig1.png https://github.com/cchen156/Learning-to-See-in-the-Dark/blob... If you take the dark image (a) from that and balance its color, the information that is present in it simply cannot contain the text from the book covers and so on. In fact, it's full of JPEG artifacts despite the image being a PNG. It would be useful if they presented a histogram equalized image of (a).
- GhostVII 7y agoIt would be good to get a comparison of a brightened version of the sample image compared with the CNN version. Right now the sample image just looks black, but if you scale up the brightness you get an image that looks more like the higher ISO image. That would be a better comparison since it shows what improvements the CNN gives over naive techniques like just bumping up the pixel values.
- baybal2 7y agoI wonder, how much can it improve over this: https://youtu.be/c_0s06ORTkY https://youtu.be/c_0s06ORTkY X27 is also using some kind of neural algorithm to denoise and get maximum out of the CIS
- dang 7y agoDiscussed at the time: https://news.ycombinator.com/item?id=17064079 https://news.ycombinator.com/item?id=17064079
- ZeroCool2u 7y agoInterestingly, this is effectively an extreme version of solving the colorization of black & white photos problem. I wonder what the results would be if you just threw some black and white photos into the model.
- dgellow 7y agoA "Two Minute Papers" on this project from 2018: https://www.youtube.com/watch?v=bcZFQ3f26pA https://www.youtube.com/watch?v=bcZFQ3f26pA
- y7r4m 7y agoHi, I'm a developer at NexOptic[0] and we are a company that was deeply inspired by this paper when it was first published. We had a lot of early success when attempting to replicate the results on our own and ended up running with it, and extending it into our own product line under our ALIIS brand of AI powered solutions. For those curious, our current approach differs in some very significant ways to the author's implementation, such as performing our denoising and enhancement on a raw bayer -> raw bayer basis with a separate pipeline for tone mapping, white-balance, and HDR enhancement. As well, we explored a fair amount of different architectures for the CNN and came to the conclusion that a heavily mixed multi-resolution layering solution produces superior results. As other commentators have pointed out, the most interesting part of it is really coming to terms that, as war1025 pointed out, "The message has an entropy limit, but the message isn't the whole dataset." It is incredibly powerful what can be accomplished with even extraordinarily noisy information as long as one has a extremely "knowledge packed" prior. If anyone has any questions about our research in this space, please feel free to ask. [0] https://nexoptic.com/artificialintelligence/ https://nexoptic.com/artificialintelligence/
- dbeardsl 7y agoSounds like you have taken this pretty far, do you have any example outputs? The only one I found via your website was a PDF with a low res image with no context.
- y7r4m 7y agoSure, we have a short deck[0] that gives an intro to our noise reduction, and also here is a folder[1] that shows off a calibration target we captured with a actual camera (20ms, f22) in low-light conditions: (original, 100x gain, 100x gain + ALIIS) We also have some more raw data[2] where there is the original bayer data available as .npy files with 40db analog gain applied, however I think the calibration targets show off what we are able to accomplish more dramatically. Finally, we have a short youtube video[3] that shows off how it works when applied to video. [0] https://www.dropbox.com/s/0bm4dpxhn35vkhe/ALLIS_Investor_Intro_Q12020.pdf https://www.dropbox.com/s/0bm4dpxhn35vkhe/ALLIS_Investor_Int... [1] https://www.dropbox.com/sh/k861saentyq1cs6/AADmO7X_L49nUkEI_taE7e8Sa https://www.dropbox.com/sh/k861saentyq1cs6/AADmO7X_L49nUkEI_... [2] https://www.dropbox.com/sh/fv8omdf4fbx59m9/AABDnf6sdvv7rtIml-li_Go0a https://www.dropbox.com/sh/fv8omdf4fbx59m9/AABDnf6sdvv7rtIml... [3] https://youtu.be/99Cq1bWCmMM https://youtu.be/99Cq1bWCmMM
- paul7986 7y agoCool and integrating this into AR Glasses would make them almost a must buy! Turn night into Day ..see in the dark, etc!
- deleted 7y ago[deleted]
- pachico 7y agoFunny. I'm walking down the corridor almost in total darkness trying to get my son to sleep. I get bored and with my free hand reach to my phone, open NH and stumble upon this title. Totally unrelated to its content but I had a (quiet) laugh :)
- grumple 7y agoAh, after decades of effort, we have finally replicated the effect of a candle.
- ptrenko 6y agoI think I'll faint seeing AI progress anymore. I didn't even think this was possible. Have people ever done this manually before? Like without AI?