4 ms·

Cmon, wtf? Some of the criticisms here just aren't even close to valid. He spends a half of the blog post criticizing them spending 100 GPUs on the Imagenet cla

Cmon, wtf? Some of the criticisms here just aren't even close to valid. He spends a half of the blog post criticizing them spending 100 GPUs on the Imagenet classification experiment, > So they trained it using 100 GPUs (100 GPUs dear lord!), and got no difference until fourth decimal digit! 100 GPU's to get a difference on fourth decimal digit! I think somebody at Google of Facebook should reproduce this result using 10000 GPU's, perhaps they will get a difference at a third decimal digit. Or maybe not, but whatever, those GPU's need to do something right? Wow. This is just a blatant mischaracterization of what's going on. First of all, this result is in the appendix. It's not meant to be an important result of the paper. In the appendix, they explicitly write: >Of all vision tasks, we might expect image classification to show the least performance change when using CoordConv instead of convolution, as classification is more about what is in the image than where it is. This tiny amount of improvement validates that. In contrast, they compare against object detection (in which the spatial location matters), and get substantially better results. This is just a standard "negative" result, to validate the fact that what they think is happening is actually happening empirically. The fact that this blog post mocks them for that, and much of HN is laughing along with the blog is seriously disappointing.

317070 8y ago
The first commit of Lasagne in 2014 already had the 'untie bias' option [0], which achieves the same effect as the paper, but in a different way (and is in my opinion more elegant). And while I cannot find a paper, I think it is one of those tricks which have been around since the Schmidhuber days. Moreover, it is one of the tricks which has been actively used since as long as I have been involved with convolutional neural networks (since 2010). So, the Uber paper is kind of silly, other for that I now know where to point to for a confirmation of the effectiveness of the idea. But I agree with you that the mischaracterisation is not appropriate. The main criticisms in the blogpost are missing the point too. The paper is not particularly interesting and might not be appropriate for the big conferences, but in my opinion not for the reasons in the blog post. Also, who cares about 100 GPU's? Nobody is complaining that all algorithms require 1 GPU these days and don't run on a smartphone, but suddenly 100 GPU's is too much? For some researchers (and I think Uber falls in the category), 100 GPU's are pocket change. Science does not require that your algorithm also runs on your lab's DIY Phd GPU cluster. If these guys have the GPU's available and it allowed them to be home earlier to spend time with the family, why would it be a problem for them to use the compute? [0]https://github.com/Lasagne/Lasagne/commit/2f9147497493f71ab6d79f35a041150f0c881d9d https://github.com/Lasagne/Lasagne/commit/2f9147497493f71ab6...
felippee 8y ago
The post mocks them primarily for learning the trivial coordinate transform. That is the core of the paper and ridiculing this piece leaves very little left on the table. The ImageNet test is just an appendix, a cherry on the cake, a curiosity one should say.
albacur 8y ago
The 100 GPUs remark is an aside, but still somewhat significant. It’s a tremendous amount of computing power to show a negative result that the authors don’t even really acknowledge as a negative result. There is a common issue with deep learning papers: deep learning seems to excuse thinking rigorously about the underlying problem at hand. We trust a deep network to do the legwork for us, and this invites an intellectually-lazy approach to research. We’re graduating PhDs who have spent four years twisting knobs. The paper itself is fine, no better or worse than the average paper today. The troubling part is the large number of high-profile ML practitioners praising it within minutes/hours. This suggests to me that notable results in DNN might be drying up and name recognition (Uber AI) is driving attention more than substance.
vicpara 8y ago
I think it's fair criticism. The paper explores effects of known models (convolution functions that exist even outside the NN field) without explaining them and doing the math to understand where the error comes from. You could have taken for example the CNN layer and see exactly what it does. Instead they prefer to pass through the network the full signal they expect to recover at the end and the find it works. Finding sanctuary inside computation and random experimentation against known datasets, without fully explaining what is actually happening on the modelling side should't be acceptable. The OP is flagging in his post the mental model and attitude towards research. The paper being discussed is just an example that confirms that proper rigour is being replaced by the convenience of either computation or parameter fiddling (human or experimental). None of these are going to take the field too far. The OP suggests the scientific field should focus on things that matter and change the world in an irreversible way. Provide a theorem or something that can be proved.

317070 8y ago

The first commit of Lasagne in 2014 already had the 'untie bias' option [0], which achieves the same effect as the paper, but in a different way (and is in my opinion more elegant). And while I cannot find a paper, I think it is one of those tricks which have been around since the Schmidhuber days. Moreover, it is one of the tricks which has been actively used since as long as I have been involved with convolutional neural networks (since 2010). So, the Uber paper is kind of silly, other for that I now know where to point to for a confirmation of the effectiveness of the idea. But I agree with you that the mischaracterisation is not appropriate. The main criticisms in the blogpost are missing the point too. The paper is not particularly interesting and might not be appropriate for the big conferences, but in my opinion not for the reasons in the blog post. Also, who cares about 100 GPU's? Nobody is complaining that all algorithms require 1 GPU these days and don't run on a smartphone, but suddenly 100 GPU's is too much? For some researchers (and I think Uber falls in the category), 100 GPU's are pocket change. Science does not require that your algorithm also runs on your lab's DIY Phd GPU cluster. If these guys have the GPU's available and it allowed them to be home earlier to spend time with the family, why would it be a problem for them to use the compute? [0]https://github.com/Lasagne/Lasagne/commit/2f9147497493f71ab6d79f35a041150f0c881d9d https://github.com/Lasagne/Lasagne/commit/2f9147497493f71ab6...

felippee 8y ago

The post mocks them primarily for learning the trivial coordinate transform. That is the core of the paper and ridiculing this piece leaves very little left on the table. The ImageNet test is just an appendix, a cherry on the cake, a curiosity one should say.

albacur 8y ago

The 100 GPUs remark is an aside, but still somewhat significant. It’s a tremendous amount of computing power to show a negative result that the authors don’t even really acknowledge as a negative result. There is a common issue with deep learning papers: deep learning seems to excuse thinking rigorously about the underlying problem at hand. We trust a deep network to do the legwork for us, and this invites an intellectually-lazy approach to research. We’re graduating PhDs who have spent four years twisting knobs. The paper itself is fine, no better or worse than the average paper today. The troubling part is the large number of high-profile ML practitioners praising it within minutes/hours. This suggests to me that notable results in DNN might be drying up and name recognition (Uber AI) is driving attention more than substance.

vicpara 8y ago

I think it's fair criticism. The paper explores effects of known models (convolution functions that exist even outside the NN field) without explaining them and doing the math to understand where the error comes from. You could have taken for example the CNN layer and see exactly what it does. Instead they prefer to pass through the network the full signal they expect to recover at the end and the find it works. Finding sanctuary inside computation and random experimentation against known datasets, without fully explaining what is actually happening on the modelling side should't be acceptable. The OP is flagging in his post the mental model and attitude towards research. The paper being discussed is just an example that confirms that proper rigour is being replaced by the convenience of either computation or parameter fiddling (human or experimental). None of these are going to take the field too far. The OP suggests the scientific field should focus on things that matter and change the world in an irreversible way. Provide a theorem or something that can be proved.