18 ms·
Deep image prior 'learns' on just one image
- giobox 9y agoSome of these image restoration techniques are extremely impressive, but its odd to think how much visual history will be faked in tiny ways if these "fix" techniques become widespread in commonly used photo software. I suppose this is no different really than the extent to which images are edited anyway now, in that many images are never really a fastidiously accurate representation of the original scene, but interesting to consider none the less.
- justonepost 9y agoCool but I think a learned method is more interesting, especially how it can be used for image compression.
- m3kw9 9y agoAm I missing something here? How does this apply to DNNs that tried to recognize images? Does it mean we can train a Net with just image now?
- BatFastard 9y agoHow can it possibly know what was in the white areas of the library? Is there a residual image? Seems impossible that it guesses correctly.
- deleted 9y ago[deleted]
- azeirah 9y agoIt doesn't guess "correctly" at all. Zoom in on the image, and focus on the filled-in areas, they look really blurry. It just doesn't look very bad from a birds-eye view.
- eutectic 9y agoProbably this is a consequence of least-squares training; the 'average' solution is extremely unlikely.
- knolan 9y agoSomewhat similar to content aware fill in Photoshop [0]. The untrained network can latch onto frequent patterns and match them to holes in the data. Why doesn’t it paint everything white? Are these actually transparent images or are they somehow tagged? [0]https://helpx.adobe.com/photoshop/using/content-aware-patch-move.html https://helpx.adobe.com/photoshop/using/content-aware-patch-...
- jmmcd 9y agoYes, for the inpainting, the parts to be painted (big white deleted areas) are supplied as masks, so it doesn't try to match them.
- BatFastard 9y agoBut its generating unique content in those areas...
- jmmcd 9y agoYes, every "run" of the network is generating pixels in those areas, but they're not being compared against the white (deleted) pixels. On the final run of the network, they're still not being compared against anything, except by us, visually, when we look at those pixels.
- theoh 9y agoIt seems like it should be similar in capability to a wavelet-based approach to e.g. image inpainting. In other words the neural network architecture essentially establishes a basis of features onto which the image is projected. Gabor wavelets are known to occur in the human visual system as the 'image elements' in a similar setup -- Gabor wavelets are kind of optimal, but clearly the features that this nn architecture uses are pretty effective too.
- alexlarsson 9y agoIt cannot know, but it can extend what’s around it to something that “looks believable”. I don’t fundamentally think it is different from asking an artist to complete the image. You’ll get an image, but no actual information about what was under the white.
- nametube 9y agoIt doesn't it repeats/generates pixels that minimize E, so the behaviour heavily depends on the choice of E. for inpainting E is the difference between their generated x and the true x but they mask out the error in the white areas. The model gets good results by just copying nearby pixels.
- Zedmor 9y agobecause your brain runs on same CNN software i.e. doing pattern matching, not image matching.
- scentoni 9y agoEnhance. http://knowyourmeme.com/memes/zoom-and-enhance http://knowyourmeme.com/memes/zoom-and-enhance
- trhway 9y agoi find it impressive how it placed a lamp over the library window. After that i was expecting a vase with flowers to be placed on the table in the next, palace shot :)
- nametube 9y agocould you highlight which part this is ?. I think it can only fill in patterns that already exist in the image. I see a big artifact over the window but no lamp.
- trhway 9y agowell, probably it is that deep (i hope) neural network inside my skull that interpreted the artifact as the lamp.
- bitL 9y agoWow, so just the weight sharing architecture does so much already? I am wondering if the same could be done with LSTMs on sequences or CNNs on voice...
- Jokeris 9y agoDo not Echo State Networks already do that?
- cs702 9y agoI'm wondering the same thing too. Note also that this finding strongly suggests that neural net architecture actually is quite important, possibly even more important than having more data -- which contradicts the conventional wisdom!
- deleted 9y ago[deleted]
- jacquesm 9y agoThere is some pretty strong evidence for this: all the toddlers in the world. You only need to show them something once and they'll immediately be able to recognize more examples of the same thing from different angles and even when it is partially hidden. All they have to guide them is the structure of their brains, not the quantity of data they have been exposed.
- rahimnathwani 9y ago"All they have to guide them is the structure of their brains, not the quantity of data they have been exposed." A typical toddler (say 12 months' old) has spent 4000-5000 hours with open eyes. Even if you assume a low frame rate (10fps), resolution (1080p), and a 1000:1 compression ratio, that's still 1TB of training data.
- eanzenberg 9y agoAlso, find me a 12 month old that can recognize an object after seeing it once.
- cdumler 9y agoRed Dwarf - "Uncrop" - http://www.dailymotion.com/video/x2qlmuy http://www.dailymotion.com/video/x2qlmuy
- sarreph 9y agoActually... How cool would it be to have an NN that could extend an image's background with plausible scenery? Not just photoshop 'smart' fill, but for example if it detected a building on the right side of an image, it could draw the rest of it? :)
- deleted 9y ago[deleted]
- nl 9y agoThis exists. This approach takes other photos of the same scene to extend a cropped pic (see also MS PhotoSynth): http://grail.cs.washington.edu/projects/sq_photo_uncrop/ http://grail.cs.washington.edu/projects/sq_photo_uncrop/ This uses a GAN to fill in missing parts of a pic. Those parts could be on the edge of the picture (although the paper doesn't explore that): http://hi.cs.waseda.ac.jp/~iizuka/projects/completion/data/completion_sig2017.pdf http://hi.cs.waseda.ac.jp/~iizuka/projects/completion/data/c...
- dahart 9y agoHere's another earlier example of un-crop, that uses only the source image (doesn't need gps or an internet database of photos), and does both out-painting and in-painting. http://graphics.cs.cmu.edu/people/efros/research/EfrosLeung.html http://graphics.cs.cmu.edu/people/efros/research/EfrosLeung....
- foota 9y agoThis is amazing.
- loser777 9y agoAn interesting question this raises is--what determines when the training has overfitted to the noise/error in the input image? If that's another NN that provides a perceptual model of quality, how was it trained? ;)
- eutectic 9y agoThis is as close to magic as humanly possible.
- looiid 9y agoWhy? A human could do better on most of those examples using paint and brush. Look at that bookstore image. If you actually look at it you will see it is quite fucked up. If a basic human skill done for millennia is magic then that’s a pretty low bar.
- sarreph 9y agoSafe to say that Arthur C Clarke would count this as 'sufficiently advanced technology' then I guess :)
- cs702 9y agoWow: "In this work, we show that, contrary to expectations, a great deal of image statistics are captured by the structure of a convolutional image generator rather than by any learned capability. This is particularly true for the statistics required to solve various image restoration problems, where the image prior is required to integrate information lost in the degradation processes. To show this, we apply untrained ConvNets to the solution of several such problems. Instead of following the common paradigm of training a ConvNet on a large dataset of example images, we fit a generator network to a single degraded image. In this scheme, the network weights serve as a parametrization of the restored image. The weights are randomly initialized and fitted to maximize their likelihood given a specific degraded image and a task-dependent observation model. We show that this very simple formulation is very competitive for standard image processing problems such as denoising, inpainting and super-resolution. This is particularly remarkable because no aspect of the network is learned from data; instead, the weights of the network are always randomly initialized, so that the only prior information is in the structure of the network itself. To the best of our knowledge, this is the first study that directly investigates the prior captured by deep convolutional generative networks independently of learning the network parameters from images." PS. This makes me wonder whether and to what degree the structure of the brain's connectome might be a necessary prior for AGI.
- jmmcd 9y ago> PS. This makes me wonder whether and to what degree the structure of the brain's connectome is a necessary prior for AGI. Well, I wouldn't mix up AGI and AGI by deep learning, and more important I would emphasise that this is a good prior for images. The fundamental insight in CNNs and eventually in this work is that there is a correlation between pairs of nearby pixels. We have something similar for video and audio processing, but nothing remotely similar for other more abstract intelligence tasks.
- cs702 9y agoThanks. I'm not mixing them up! I'm just wondering whether and to what degree architecture, i.e., network structure, will prove important for other, more advanced AI tasks, including up to AGI.
- JumpCrisscross 9y agoCan someone break this down for this layman?
- oh_sigh 9y agoNeural networks are powerful without even training them. By merely designing the structure of them you are creating something.
- theoh 9y agoTruly, an occasion for this koan: Sussman attains enlightenment In the days when Sussman was a novice, Minsky once came to him as he sat hacking at the PDP-6. “What are you doing?”, asked Minsky. “I am training a randomly wired neural net to play Tic-Tac-Toe” Sussman replied. “Why is the net wired randomly?”, asked Minsky. “I do not want it to have any preconceptions of how to play”, Sussman said. Minsky then shut his eyes. “Why do you close your eyes?”, Sussman asked his teacher. “So that the room will be empty.” At that moment, Sussman was enlightened. http://www.catb.org/jargon/html/koans.html http://www.catb.org/jargon/html/koans.html
- jacquesm 9y agoIf there ever was one this is it.
- thefalcon 9y agoOkay, I admit it, I'm not enlightened. Will somebody please Explain Like I'm 5?
- anchpop 9y agoInitializing a neural net with random weights =/= making a neural net with no preconceptions on how to play. It still has those preconceptions, you just don't know what they are. Closing your eyes so you can't see the contents of a room =/= the room is empty. Not knowing what something is, is different from it not existing
- raz32dust 9y agoThis is incredible! Can't help but wonder if the brain does something similar to fill up "gaps" in reality. e.g how we fill up our perception (not only vision, but general mental intuition) based on just context.
- sjg007 9y agoYes it does.
- randcraw 9y agoThat's exactly what happens when the retina is damaged. The brain fills in the void imperceptibly so you aren't distracted by the deficit. But I think biology doesn't fill in the void synthetically. It's more likely that the brain "turns a blind eye" toward the missing 'pixels' and directs your attention elsewhere, perhaps to those remaining regions with high saliency (useful detail).
- rasz 9y agoNot exactly. Brain has prior knowledge (deep learning). Lets take a picture of a car as an example. License plate has letters/numbers, so even if it cuts lower half you will try to guess by fitting the most likely symbol combinations. Approach from the article has no prior knowledge, wouldnt be able to inpaint meaningful letters, nor guess most cars have wheels, bumpers, drive on the road etc.
- chestervonwinch 9y agoThis is unbelievable that it works so well.
- deleted 9y ago[deleted]
- 0x53-61-6C-74 9y agoCSI:Miami...I'm sorry, I take it all back.
- vbuwivbiu 9y agoRemember fractal image compression ? https://youtu.be/AjdogjBxfco?t=260 https://youtu.be/AjdogjBxfco?t=260
- cztomsik 9y agoThx for that!
- eitland 9y agoUpvoted you both. I normally really don't like videos but this was worth the few minutes it took. (The link points to the middle of a 9 min video.)
- ska 9y agoThis can also be looked at as the original source of patch based denoising, etc. In the end it's about capturing the scaling properties and self similarity of natural images. This is also, for example, why wavelets were so effective as a basis. David Mumford particularly did some great work on this sort of thing a couple of decades ago, along with many others. I hope when people are rushing around trying to apply convolutional nets to everything they aren't losing these insights.
- fjsolwmv 9y agoThe benefit of CNNs is like the benefits of SVMs -- they generalize all the great old techniques so you don't have to understand them all, you just throw more CPU at the optimization problem.
- ska 9y agoI don't think that's true particularly in this case. This paper is pointing out that you can encode a structural prior in a CNN - but knowing the "great old techniques" will help you design the right network architecture to do that. SVMs were a surprise when they came out, no so much a generalization as a challenge (at least at first)
- eximius 9y agoHuh. This seems to boil down to 'noise is higher information entropy than realistic content; partial learning will learn realistic content before learning noise' or something like that.
- eref 9y agoI think that can be mainly attributed to the fact that the last few deconvolutional features are overfitted to features in the image and are somewhat robust to noise. The network does not even learn features to produce e.g. white noise as output. This is probably much less magical than the paper makes it seem to be.
- magoghm 9y agoInteresting. Then you might achieve similar results by compressing the image as a JPEG with low quality settings.
- killjoywashere 9y agoThe intel and espionage communities are going to be all over this. This makes the Soviet photo retouching look like child's play. Let's say you want to start a war, and need some evidence of chemical weapons. Now, you can drop in some images of chemical weapons and claim a GAN found them. Sample press releases: "We believe this photo was retouched to hide the chemical weapons. Using a GAN, we recovered clear photographic proof of the chemical weapons." "We believe they are using this tin-roofed building to hide the chemical weapons. Using a GAN on our own satellite images, we recovered clear photographic proof of the chemical weapons."
- ac2u 9y agoCan't see "we used a GAN" as an explanation that would fly. To counter it you can simply say "well, the GAN could be wrong." The technique outlined above is useful because we as humans can eyeball the results and say "the improvement on that blurry zebra photo is great!", but we know that it's an improvement based on a networks hueristics that's good enough for us, not an exact replica of the information lost through noise and compression.
- fenwick67 9y agoI'll grant you that it makes a good scapegoat, but frankly this doesn't really matter... the US went to war with Iraq with bad intel before the machine learning boom.
- albertTJames 9y agook.. this is revolutionary. Using the architecture as a way to capture an image prior hints at how network structure and captured invariance are related. By analogy it leads to thinking of brain areas as both hard coded prior knowledge through their natural arrangement and plastic learning structure. Turning the problem on its head shines a new light to how we could conceive network architectures.
- andbberger 9y agoThis shouldn't really be surprising. Machine learning is specifically not magic. The reason CNNs have seen so much success is precisely because they build in translation-invariance, which massively cuts down on parameters while forcing the final function to have the desired structure regardless of wherever gradient descent takes the weights. Also why most papers in deep learning are network architecture innovation.
- andbberger 9y agoOne more relevant note - (Olshausen and Field, 1997) showed that the filter employed by V1 simple cells could be learned using some simple assumptions about sparse coding and a single image. Translation invariance built in by way of the sampling scheme of the image, small patches. The filters learned by the first layer of CNNs is usually of the same type, Gabor filters. Not a coincidence. That was twenty years ago. What's old is new?
- eref 9y agoWhat do Gabor filters have to do with this?
- ska 9y agoParent is saying Gabor filters are typically effectively recapitulated by the first layer of of the network anyway, as they are a natural representation.
- eref 9y agoBut what does have to do with smoothness and translation invariance which this paper is a demonstration of? You even learn Gabor filters with local connectivity without spatial weight sharing.
- krona 9y agoDeep learning researchers rediscover Compressed Sensing. News at 11.
- foota 9y agoI wonder if you could do something similar with audio
- JoeCoder_ 9y agoAnd turn crappy music into good music.
- zo7 9y agoI feel like this is related to the information bottleneck idea that's been floating around for some time [1]. I only really understand both of these at a superficial level, but from what I think I understand one thing that they observed is that there are two phases when training a deep learning model: a phase which maximizes the mutual information (?) between the input and the output, and a compression phase which compresses the learned representation. In that light this work makes sense, since artifacts are essentially noise that the network would filter out in the fitting process. Very cool work though. [1]: https://youtu.be/bLqJHjXihK8 https://youtu.be/bLqJHjXihK8
- freeflight 9y agoThat Zebra one looks like a "before and after" image with really good MSAA. I realize the techniques are quite different, it's still interesting how the result seems so similar on a superficial level.
- gbrown 9y agoSo what kind of generative stuff does it "learn" from really really sparse data, or from random noise? Trained generative models can produce some crazy things, I wonder if something similar could work here. Recursive geometric patterns perhaps?
- zellyn 9y agoWould a plausible explanation of this be that one part of an image tells you a hell of a lot about another part of it? And that ConvNet structure captures that really well?
- psergeant 9y agoYes
- ac2u 9y agoQuestion to anyone that knows this area in depth. I'm assuming that this impressive feat has a disadvantage (over the traditional example intensive technique of training a CNN with something like ImageNet) in the form of taking a long time to generate the corrected image. With that assumption in mind, could this new technique be reversed? As in feeding in sharp images, getting back out corrupted ones, for the purposes of generating data sets where there isn't much data to begin with? You could then take that data and use it to train a more traditional CNN to sort of amortize the results of the technique in the paper and have the process happening faster.
- wonderous 9y agoDo you mean generative adversarial networks? https://en.m.wikipedia.org/wiki/Generative_adversarial_network https://en.m.wikipedia.org/wiki/Generative_adversarial_netwo...
- ac2u 9y agoNo not really, this isn't generation for the sake of classification. It's generation in order to generate a dataset to train a new network that exhibits the characteristics of the network in the paper that can be used quickly and in a more general fashion.
- wonderous 9y agoMeaning that your assumption is that there is an optimal generalized network structure for generate a neural network as described in the paper and that as such, what is missing from the research is how to create an optimal way to generate a malformed source/target seed. Is that correct?
- ac2u 9y agoI'm not suggesting it's missing from the research as I don't have anywhere near the area of expertise to make that call. But you're correct in your summary: I'm wondering if that's a possible follow up.
- tgb 9y agoI don't see how any choice of a function g(theta) could have the property they desire, ie could eliminate R(g(theta)). Can anyone explain?
- sanxiyn 9y agoIt's expressed somewhat awkwardly, but what's going on is that R(x) is zero if x is in range of g, and infinite otherwise. Choice of g is such that natural images are in range of g, and non-natural images aren't.
- tgb 9y agoSorry I still don't understand. They require g to be surjective. Edit: in their paper they call it f_theta and it's explicitly not surjective. Dunno why their writeup is so confused.
- ReverseCold 9y agoI tried to run their ipynb but it wants something called skip. I don't think it's the "skip" on PyPi (that doesn't work anyway) - so what could they be using?
- sanxiyn 9y agoskip.py is in the repository under models directory.
- frihani 9y agoMaybe it's the presentation of the restoration process, but I'm particularly impressed with the inpainting sample. The idea of not running the simulation `past` the realistic interpretation and using that result makes sense but the results are way beyond what I would have expected! Great work on the write up.
- fpgaminer 9y agoI'm finding it hard to put into words what I find wrong with this paper, but ... here goes nothing. So, the novel thing here is that an encoder-decoder network applied to an image can learn enough from a source image to be useful. In some ways that's obvious, but the effectiveness of it on reconstruction tasks is certainly surprising. I have two problems, though. One is that I would take the reconstruction results with a grain of salt. The examples are clearly lab queens, where the occluded regions are not particularly interesting/challenging. Two is the conclusion the authors reach. Somehow the authors go from the novel discovery I describe above, to saying that somehow the architecture of the network is a prior. Well ... I mean, yeah a network's architecture _is_ a prior. But it's not actually significant. See, in the dark ages of machine learning we had only fully connected networks. They sucked. They'd always overfit and underperform or were impossible to train. Then we finally got convolutional networks, and suddenly a whole slew of machine learning problems became easier and that hurdled us into the current renaissance. But, you see, convolutional networks weren't the _only_ reason for the dawn of this new age. Rather it was three major things: 1) Convolutional layers, 2) more data, 3) more computing power. Some time after the "discovery" of convolutional layers we found out that, hey, our old fully connected networks actually _do_ work. If you give them enough data and enough computational power, you can get them to perform as well as state of the art convolution networks. The great thing about fully connected networks is that they assume nothing. That means A) you can theoretically get better results and B) you don't have to spend time designing an architecture. So we already know that architecture isn't ultimately important. You can have a giant, fully connected network, and it _will_ work, if you feed it enough data and have the computational power necessary to train such a beast. Convolutional layers are just simplifications which make training easier. They are priors in the sense that we know a fully connected layer in image applications would just devolve into a convolutional layer anyway, so we might as well start with a convolution layer. That "design" is the prior. But it's not mandatory; the network would still function without that "prior". So ... I'm not sure how the authors are taking their research and using it to come to the conclusion that their results are because of some magical property imbued into the network by the "priors" of the architecture. They apparently tried other architectures and got poor results, and so they use that to claim that architecture is the only reason their technique works. That's like if you started with ResNet for a classification problem, tried other architectures, saw that they performed worse, and then published a paper saying that Residual Networks somehow embody the fundamental forces of natural images in their architecture, and that's why they work. When the truth is that ResNets aren't special, they are just easier to train. Another example from the annals of machine learning history: time and time again when there is a breakthrough in architectures, it's usually followed a few years later by a simplification of the architecture. For example we started with networks like VGG which are these big, hand crafted architectures. Slowly over time architectures have become _less_ exotic, instead opting to simply define a basic building block repeated N times. The reason for this is because in the intervening years we gather more training data, better training techniques, and more computational power. So we can instead use a more homogeneous architecture which has _less_ assumptions (less priors) and at the end of the day we get _better_ results. I'll repeat that. We put _less_ priors into our networks and we get better results. So on the one hand we have _all_ of machine learning history telling us that priors in architectures are _bad_. On the other hand we have this paper which makes some really weird logical leap from "we tried a few architectures, they were worse, so architecture is _key_ to machine learning and it's important because we need good priors built into the architecture." Anyone remember hand crafted feature vectors? I do. Those were priors. Guess what happened when we got rid of them and used generic networks feeding directly from the raw data? Oh right, all of modern machine learning...
- blauditore 9y agoAs I'm not an expert in the field; what exactly does the term min_x E(x; x0) + R(x) mean? I thought that E(x; x0) would denote the error/difference between original and corrupted images, and R(x) be the (searched-for) correction. But this doesn't seem to make sense with the next parts of their explanation.
- urgoroger 9y agox is actually the generated image they are testing against x0. A lower E(x; x0) means an image which fits well towards the objective based on the original image (depends on the task). The paper gives some examples. For example, for the task of image denoising, E(x; x0) is just the squared distance of the generated (denoised candidate) image x to the pixels to the original image x0. Obviously you would want this to be low in the generated version since it should still look close to x0. R(x) is a regularization term to avoid overfitting. For example, in the denoising example, it could be a measure of the variation in color of x. Clearly, just taking x = x0, the squared error (E(x; x0)) is 0, but it will have high R(x) because of all the noise. That's why they try to minimize both quantities combined, so we get min_x E(x; x0) + R(x), to get close to the objective but also not overfit. https://en.wikipedia.org/wiki/Regularization_(mathematics) https://en.wikipedia.org/wiki/Regularization_(mathematics)
- sigterm 9y agoI think x_0 refers to the corrupted image and x is the (denoised/upscaled/inpainted) image the optimization algorithm tries to come up. Without R(x) the optimal solution is simply x=x0. So we need a good R(x) that can measure how good or natural the solution image is. In this work the R(x) is constraining the valid x to those that can be represented by a CNN.
- cdancette 9y agoYou're right, and here, they're including the prior R(x) in the model architecture. So R(x) = 0 here, and they pose x = f_theta(z), where f_theta is a neural net. So they can avoid overfitting by stopping the training at the right moment. So they're not really trying to minimize E(x, x0), but rather to get a small enough value, so that they obtain good results. A too small value means overfitting.
- CamperBob2 9y agoIs there a straightforward explanation of how to run this anywhere, for those not steeped in the arcane ways of Jupyter?
- zerostar07 9y agoThis is interesting news to neuroscientists as well. Even without plasticity cortical networks might be performing useful functions. Plasticity may then perform some different function.
- artur_makly 9y agocan someone please summarize this for us luddites? also wasnt there an HN summary project out there?
- therealmarv 9y agohow do I get the code running easily? I'm familiar with python but not with ipynb at all. Maybe somebody can give me some easy to follow instructions, thx.
- Quanttek 9y agoCan someone put their results and explanation in laymen's terms please?
- machiaweliczny 9y agoIsn't it simply hardcore overfitting of network? I haven't read the paper yet but it seems to me that it shows you can use overfitting for output interpolation with great results. Same should be possibile for example for interpolation of video images instead od pixels.
- deleted 9y ago[deleted]
- alexasmyths 9y agoTruly congratulations! Amazing.