13 ms·
Wow: "In this work, we show that, contrary to expectations, a great deal of image statistics are captured by the structure of a convolutional image generator r
by cs702 9y ago
Wow:
"In this work, we show that, contrary to expectations, a great deal of image statistics are captured by the structure of a convolutional image generator rather than by any learned capability. This is particularly true for the statistics required to solve various image restoration problems, where the image prior is required to integrate information lost in the degradation processes.
To show this, we apply untrained ConvNets to the solution of several such problems. Instead of following the common paradigm of training a ConvNet on a large dataset of example images, we fit a generator network to a single degraded image. In this scheme, the network weights serve as a parametrization of the restored image. The weights are randomly initialized and fitted to maximize their likelihood given a specific degraded image and a task-dependent observation
model.
We show that this very simple formulation is very competitive for standard image processing problems such as denoising, inpainting and super-resolution. This is particularly remarkable because no aspect of the network is learned from data; instead, the weights of the network are always randomly initialized, so that the only prior information is in the structure of the network itself. To the best of our knowledge, this is the first study that directly investigates the prior captured by deep convolutional generative networks independently of learning the network parameters from images."
PS. This makes me wonder whether and to what degree the structure of the brain's connectome might be a necessary prior for AGI.
- jmmcd 9y ago> PS. This makes me wonder whether and to what degree the structure of the brain's connectome is a necessary prior for AGI. Well, I wouldn't mix up AGI and AGI by deep learning, and more important I would emphasise that this is a good prior for images. The fundamental insight in CNNs and eventually in this work is that there is a correlation between pairs of nearby pixels. We have something similar for video and audio processing, but nothing remotely similar for other more abstract intelligence tasks.
- cs702 9y agoThanks. I'm not mixing them up! I'm just wondering whether and to what degree architecture, i.e., network structure, will prove important for other, more advanced AI tasks, including up to AGI.
- garmaine 9y agoYes, AGI as a discipline distinguishes itself in large part from machine learning by its focus on architecture over training or data.
- EGreg 9y agohttps://en.m.wikipedia.org/wiki/Language_acquisition_device https://en.m.wikipedia.org/wiki/Language_acquisition_device
- mjburgess 9y agoTo throw a dissenting voice into the mix: I do not think intelligence is a consequence of structure. Transparency is a consequence of how light interacts with objects. There is no "transparent gold". And being transparent is not something we can program gold to do. Programming is the application of an electric field across a silicon surface: this can no more transmute silicon into gold as it can into a nervous system. Consciousness is a biological activity. It is not something wood, silicon, sand, metal, glass, water, etc. can do by virtue of moving around in an interesting pattern. To me, this is akin to believing magic, or to thinking that the issue with Bug's Bunny's consciousness is that the ink wasnt laid out in the right way. Rather, there isnt any. As scientists we should begin very clearly by identifying the target phenomenon. Line up everything in the known universe that is conscious and you will find that it engages in a highly specific biochemical activity that requires a whole set of highly specific chemical interactions. To believe these can be achieved by applying an electric field across some silicon seems like the victorian superstition that a person could be raised from the dead by doing likewise. Rather, all we are able to do is imitate in the weakest sense. People fooled by chatbots are no more enlightened than the early victorians who were shocked to watch the movie of a train coming at them. Fooling people is a very easy thing to do. And esp. in the area of consciousness when "alertness to things that might be conscious" seems a deep psychological preoccupation of almost every animal to the point of absurdity (eg. being scared at the wind).
- sanxiyn 9y agoWho said anything about consciousness? Intelligence does not equal consciousness.
- deleted 9y ago[deleted]
- otakucode 9y agoWell, there is correlation between 'nearby' things of all types. An apple is an 'apple' because we encountered the word and the thing in close proximity to one another temporally and perceptively many times during our life. All knowledge is this. Tremendously overlapping and filled with mistakes (spurious correlations induced by coincidence and happenstance), but fundamentally just that same 'insight'. Our brains are association machines. Once sufficient quantity of associations exist in a brain in a body living in a world we recognize, we call the property that results 'consciousness'. Unfortunately there is no reason to think that you can substract the 'in a body living in a world we recognize' part and end up with anything like the same property. All those associations come from perceptive inputs, and result in motive outputs, with the biofeedback that occurs being of profoundly fundamental importance (heck, put a pen in your mouth which causes your mouth to 'smile' and you will 'feel happier'). While of course there is definitely a possibility that a machine-based intelligence could be conscious, it would doubtless have to be very different from anything we could recognize, derived as it would be from its own input, output, and the feedback between the two.
- felippee 9y agoThis is great work. It shows that some of the "amazing" results of deep learning are not as deep and don't even require learning! In my view this also sheds some light on the GAN's and their ability to generate "real looking images". Perhaps there is much less to generating "real looking images" then everyone attributes. E.g. in this work, the network clearly knows exactly nothing about the world and generates good looking in-paint.
- narrator 9y agoThe brain probably has a lot of hard wired networks to recognize faces and to not like being cold for example. I think even feral children have these capabilities.
- deleted 9y ago[deleted]
- eanzenberg 9y agoThe “task dependent observation model” also comes out of thin air, or is it trained with lots of data?
- sanxiyn 9y agoObservation model is trivial and data-free. For example, just mean squared error for denoising, mean squared error of downscaled image for super-resolution, etc.
- eanzenberg 9y agoI'm fairly certain that's not? Where in the paper does it talk about this other network, generally called the discriminator?
- sanxiyn 9y agoThere is no discriminator in this work. Read the paper.
- goialoq 9y agoI don't understand their editorializing > contrary to expectations, ingore the weasel words > a great deal of image statistics are captured by the structure of a convolutional image generator rather than by any learned capability. > To show this, we apply untrained ConvNets to the solution of several such problems. Instead of following the common paradigm of training a ConvNet on a large dataset of example images, we fit a generator network to a single degraded image. How is that _untrained_, if they are training it on an image? Is a generator network different from a ConvNet? > This is particularly remarkable because no aspect of the network is learned from data; but they just said "we fit a generator network to a single degraded image" As others have commented, they appear to be training on all the parts of a single complex image, and using that training to repair local deviations from the global average. This may be better than classical denoising algorithms because the NN can model the image structure better than other approaches, but this doesn't seem like a novel use of NNs.
- lern_too_spel 9y agoThe point is that a fully connected network would not get you the same results. I.e. the convolutional structure allows it to learn interesting things from a single example because by choosing a convolutional architecture, you are implicitly imposing a prior.
- fjsolwmv 9y agoHow is a network structure different from a fully connected network with 0 weights?
- kalal 9y agoAgreed. The network is trained in a sense "online" on the target image and directly applied to it. I don't see much impact of this work outside image denouncing or tasks already presented in the paper.
- jpster 9y agoDoes this mean huge datasets are no longer a prerequisite for this type of computing? Leveling the playing field for smaller teams who may no longer have to rely on Google- or FB-sized datasets?
- cdancette 9y agoI don't think so. They are basically loosing entropy by going from the corrupted image to the original one that's more predictable. I don't think you can do other things like labeling by using the same method
- stanfordkid 9y agoSo are they saying that the topology of a deep-net is intrinsic to "reality" ... somewhat analogous to something like the Fibonacci ratio for organic forms?
- adrianN 9y agoThe Fibonacci thing is mostly a myth perpetuated by confirmation bias tough.
- jon_richards 9y agoMaybe more like a fourier transform.
- vog 9y agoDoes this also explain - on a higher level - why AlphaGo Zero was possible and so successful?
- jeeceebees 9y agoAlpha Go Zero is actually more similar to a GAN: https://arxiv.org/abs/1711.09091 https://arxiv.org/abs/1711.09091
- amelius 9y agoBut I see one problem. Say the image is a photo of a driveway. The gravel on the driveway will look like noise. Will the algorithm now smooth out the driveway, effectively removing the gravel? At least a "learned", context-aware approach can prevent this from happening in theory.
- pasta 9y agoI think you are right. But as a brush in a photo editor program it would be great. Then a used can 'mark' what areas need to be smoothed and the result will be great.
- rsp1984 9y agoIn the examples they show an image of a woman next to a table. The table is covered with a table cloth. In the corrupted image the table cloth looks like all noise to the human eye, however in the reconstructed image you see that the method is able to recover most of the pattern in the cloth.
- amelius 9y agoThat's something different: a demonstration of an "inpainting" problem. The black pixels were specifically marked as corrupt. Also, the pattern on the cloth is much less granular than noise.
- LoSboccacc 9y agostill the problem space is greatly reduced - all we need now is a nn to detect corrupted pixels
- amelius 9y agoWe need more than just the information where corrupted pixels are. For example, consider an image of a woman, severly blurred such that her individual fingers cannot be discerned but you can still see that it's a woman. A "learned" context-aware approach can now deduce that the "blob" that is her hand should be inpainted by fingers. A non-learned approach can't do that.
- rullelito 9y agoCan someone explain this in a way so that an ordinary mortal computer scientist can understand it?
- vbarrielle 9y ago> Can someone explain this in a way so that an ordinary mortal computer scientist can understand it? I'll try. Instead of the common approach that tries to search for image pixels to minimize e.g. a denoising objective, they realize that they can instead search for the weights of an image generator network such that the generated image matches the objective. They argue that the structure of the network then constitutes some prior knowledge over what a natural image should look like. My (probably wrong) interpretation: since a convolutional neural network essentially works by looking for some spatial patterns at different resolutions, their optimization process boils down to finding the high and mid-resolution patterns that best match the input image, and then re-using that information to fill in the missing information (or replacing "noise" that does not match the extracted pattern).
- rdlecler1 9y agoWhat do they mean by ‘the structure of the network’ is this the network topology?
- fny 9y agoThere exist neural network structures that can solve the image processing problems shown when weights are randomly applied. This means that for these problems the solution is encoded entirely in the network structure and not the weights. We could use linear regression as an analogy: y = b f(x). This is akin to saying that for some class of problems, it's the case that you can get as good of a solution from the function you pick alone (x^2, log(x)) as you could from picking a function and then computing the best coefficient. Note this isn't true for linear regression for any meaningful problems, but I feel like it's a decent analogy.
- denzil_correa 9y agoThere's a great explanation of the idea here in the comments : https://news.ycombinator.com/item?id=15820994 https://news.ycombinator.com/item?id=15820994
- amelius 9y agoSince a convolutional network applies parameters (filters) to different parts of an image evenly, this means that this approach still learns, except not from a corpus of images, but from the image itself (i.e. one part of the network learns from different parts of the image). > This makes me wonder whether and to what degree the structure of the brain's connectome might be a necessary prior for AGI. Convnets have been invented to speed up learning (reusing parameters). The human brain (afaik) doesn't contain such speedup. Instead it relies on massive parallellism of neurons, and thus trains all "filters" separately instead of reusing them. Therefore, I suppose this approach would not apply to humans.
- amelius 9y agoAnother way to think of it: Convnets are a method to speed up learning by sharing filter-coefficients. In other words, one could just as well train the network by replacing the convnets by fully-connected layers, except that of course training would take a lot longer. If one views convnets as purely a computational optimization, then the approach of the paper can be considered a "hack" that only works because of this optimization.
- kkylin 9y agoProbably more than structure (which I took to mean connectivity) -- there are a lot of things that affect (biological) neuronal computation in addition to connectivity, such as synaptic weights, distribution & types of receptors for different neurotransmitters, and the (sometimes very different) timescales associated with different neurotransmitters. Not easy things to measure.
- jonmc12 9y ago> PS. This makes me wonder whether and to what degree the structure of the brain's connectome might be a necessary prior for AGI. I've seen an analogy in hardware that is potentially interesting to explore. There is a technique in A/D converters (ADC) where input signal noise can be reduced by RMS averaging the sequence of digital outputs. This increases the effective resolution of the ADC, while lowering the effective sampling frequency of the input signal. In the case that the input noise of a precision ADC is very low, the digital averaging has no effect on effective resolution. Surprisingly, in this case you can actually inject a "hand-crafted" noise source into the input, and still realize the resolution improvements from digital averaging. The key is to find the right amount of noise to randomize the quantization noise. In the case of the ADC, the structure of the quantizing hardware is known, as is the characteristics of quantization noise. This allows for a guassian noise to be injected with the input source (might be analogous to a "prior"). From there, one can balance the input injection noise with the output digital averaging to get an optimal increase in output resolution. This technique is explained well in this paper: http://www.analog.com/en/analog-dialogue/articles/adc-input-noise.html http://www.analog.com/en/analog-dialogue/articles/adc-input-...
- otakucode 9y agoI have been reading about fractal image compression (don't sue me patent trolls, I haven't implemented anything I promise!) lately and so this doesn't surprise me very much. I also wouldn't be surprised if it turned out that what is happening inside the network is essentially similar in many respects to how fractal image compression works. Fractal image compression is inherently pretty simple in concept: Find portions of the image that are similar to transformed other portions of the image. Store the things needed to transform one portion into the other, and patch together enough of those to cover the whole thing. Then you can throw out the image itself. You only need the ways in which the parts can be transformed into the others (affine transformations along with brightness/contrast shifts usually). Once you have those, you can literally start with _any_ source image and iterating the application of the transformations are guaranteed to result in something very close to the original image. A CNN 'rediscovering' this technique feels intuitively like a very natural thing to occur, and the iterative images presented there smack of early iterations of a fractal image 'decoding' from a blank source image. The connection, of course, could be utterly specious and I am just guessing. I am intrigued, however, as I've been wanting to investigate using deep learning to perform VHS video capture cleanup as a side project for awhile now.