4 ms·

I'm finding it hard to put into words what I find wrong with this paper, but ... here goes nothing. So, the novel thing here is that an encoder-decoder network

I'm finding it hard to put into words what I find wrong with this paper, but ... here goes nothing. So, the novel thing here is that an encoder-decoder network applied to an image can learn enough from a source image to be useful. In some ways that's obvious, but the effectiveness of it on reconstruction tasks is certainly surprising. I have two problems, though. One is that I would take the reconstruction results with a grain of salt. The examples are clearly lab queens, where the occluded regions are not particularly interesting/challenging. Two is the conclusion the authors reach. Somehow the authors go from the novel discovery I describe above, to saying that somehow the architecture of the network is a prior. Well ... I mean, yeah a network's architecture _is_ a prior. But it's not actually significant. See, in the dark ages of machine learning we had only fully connected networks. They sucked. They'd always overfit and underperform or were impossible to train. Then we finally got convolutional networks, and suddenly a whole slew of machine learning problems became easier and that hurdled us into the current renaissance. But, you see, convolutional networks weren't the _only_ reason for the dawn of this new age. Rather it was three major things: 1) Convolutional layers, 2) more data, 3) more computing power. Some time after the "discovery" of convolutional layers we found out that, hey, our old fully connected networks actually _do_ work. If you give them enough data and enough computational power, you can get them to perform as well as state of the art convolution networks. The great thing about fully connected networks is that they assume nothing. That means A) you can theoretically get better results and B) you don't have to spend time designing an architecture. So we already know that architecture isn't ultimately important. You can have a giant, fully connected network, and it _will_ work, if you feed it enough data and have the computational power necessary to train such a beast. Convolutional layers are just simplifications which make training easier. They are priors in the sense that we know a fully connected layer in image applications would just devolve into a convolutional layer anyway, so we might as well start with a convolution layer. That "design" is the prior. But it's not mandatory; the network would still function without that "prior". So ... I'm not sure how the authors are taking their research and using it to come to the conclusion that their results are because of some magical property imbued into the network by the "priors" of the architecture. They apparently tried other architectures and got poor results, and so they use that to claim that architecture is the only reason their technique works. That's like if you started with ResNet for a classification problem, tried other architectures, saw that they performed worse, and then published a paper saying that Residual Networks somehow embody the fundamental forces of natural images in their architecture, and that's why they work. When the truth is that ResNets aren't special, they are just easier to train. Another example from the annals of machine learning history: time and time again when there is a breakthrough in architectures, it's usually followed a few years later by a simplification of the architecture. For example we started with networks like VGG which are these big, hand crafted architectures. Slowly over time architectures have become _less_ exotic, instead opting to simply define a basic building block repeated N times. The reason for this is because in the intervening years we gather more training data, better training techniques, and more computational power. So we can instead use a more homogeneous architecture which has _less_ assumptions (less priors) and at the end of the day we get _better_ results. I'll repeat that. We put _less_ priors into our networks and we get better results. So on the one hand we have _all_ of machine learning history telling us that priors in architectures are _bad_. On the other hand we have this paper which makes some really weird logical leap from "we tried a few architectures, they were worse, so architecture is _key_ to machine learning and it's important because we need good priors built into the architecture." Anyone remember hand crafted feature vectors? I do. Those were priors. Guess what happened when we got rid of them and used generic networks feeding directly from the raw data? Oh right, all of modern machine learning...

aqsalose 9y ago
>Another example from the annals of machine learning history: time and time again when there is a breakthrough in architectures, it's usually followed a few years later by a simplification of the architecture. For example we started with networks like VGG which are these big, hand crafted architectures. Slowly over time architectures have become _less_ exotic, instead opting to simply define a basic building block repeated N times. >The reason for this is because in the intervening years we gather more training data, better training techniques, and more computational power. So we can instead use a more homogeneous architecture which has _less_ assumptions (less priors) and at the end of the day we get _better_ results. >I'll repeat that. We put _less_ priors into our networks and we get better results. Is move from VGG-like architectures a question of moving to using less priors or just move to different priors that work better? In addition, I'd claim that a choice of training method also imbues the whole ML system with prior knowledge. If we add a dropout layer, we wish to enforce a particular regularization scheme on the model because we know that we need to regularize to model to avoid overfitting. If we choose to use a particular optimizer instead of another, we use it because we know it has desirable properties on particular objective objection. >So we already know that architecture isn't ultimately important. You can have a giant, fully connected network, and it _will_ work, if you feed it enough data and have the computational power necessary to train such a beast. I'm slightly unclear about your argument here. What is the point of training the fully connected network for a long time and with massive datasets to get equivalent results and structure as convnets if we already do know that convolutional structure with weight sharing is a good fit for natural images? Deep learning has been casted as a magical black box that produces impressive results "yet the scientists don't know why!" by the press. On the contrary, these results suggest there might be less magic in neural net models as the recent hype would suggest. > The examples are clearly lab queens, where the occluded regions are not particularly interesting/challenging. To me they look quite like the standard examples used in inpainting and denoising articles.
warsheep 9y ago
> Convolutional layers are just simplifications which make training easier. They are priors in the sense that we know a fully connected layer in image applications would just devolve into a convolutional layer anyway, so we might as well start with a convolution layer. That "design" is the prior. But it's not mandatory; the network would still function without that "prior". As far as I know this is incorrect. Can you point to a paper that shows this? If by "easier to train" you mean that the models do not overfit training data, then that's the whole point of using correct priors / hypothesis classes. I'm not sure what bugs you in this paper, but the point is that they decouple the prior architecture from the training/optimization mechanism, and that seems interesting.

aqsalose 9y ago

>Another example from the annals of machine learning history: time and time again when there is a breakthrough in architectures, it's usually followed a few years later by a simplification of the architecture. For example we started with networks like VGG which are these big, hand crafted architectures. Slowly over time architectures have become _less_ exotic, instead opting to simply define a basic building block repeated N times. >The reason for this is because in the intervening years we gather more training data, better training techniques, and more computational power. So we can instead use a more homogeneous architecture which has _less_ assumptions (less priors) and at the end of the day we get _better_ results. >I'll repeat that. We put _less_ priors into our networks and we get better results. Is move from VGG-like architectures a question of moving to using less priors or just move to different priors that work better? In addition, I'd claim that a choice of training method also imbues the whole ML system with prior knowledge. If we add a dropout layer, we wish to enforce a particular regularization scheme on the model because we know that we need to regularize to model to avoid overfitting. If we choose to use a particular optimizer instead of another, we use it because we know it has desirable properties on particular objective objection. >So we already know that architecture isn't ultimately important. You can have a giant, fully connected network, and it _will_ work, if you feed it enough data and have the computational power necessary to train such a beast. I'm slightly unclear about your argument here. What is the point of training the fully connected network for a long time and with massive datasets to get equivalent results and structure as convnets if we already do know that convolutional structure with weight sharing is a good fit for natural images? Deep learning has been casted as a magical black box that produces impressive results "yet the scientists don't know why!" by the press. On the contrary, these results suggest there might be less magic in neural net models as the recent hype would suggest. > The examples are clearly lab queens, where the occluded regions are not particularly interesting/challenging. To me they look quite like the standard examples used in inpainting and denoising articles.

warsheep 9y ago

> Convolutional layers are just simplifications which make training easier. They are priors in the sense that we know a fully connected layer in image applications would just devolve into a convolutional layer anyway, so we might as well start with a convolution layer. That "design" is the prior. But it's not mandatory; the network would still function without that "prior". As far as I know this is incorrect. Can you point to a paper that shows this? If by "easier to train" you mean that the models do not overfit training data, then that's the whole point of using correct priors / hypothesis classes. I'm not sure what bugs you in this paper, but the point is that they decouple the prior architecture from the training/optimization mechanism, and that seems interesting.