6 ms·

Towards Principled Design of Deep Convolutional Networks: Introducing SimpNet

3 points by ghosthamlet 9y ago

p1esk 9y ago
I'm sure a lot of effort went to writing this paper, but I have to say I'm underwhelmed with the reported results. First of all, CIFAR results in 2018 are just not going to cut it. The efficiency is only measured in terms of parameter count, and not even compared to CondenseNet. "SAF Pooling" does not deserve a special name, as it's just max pooling followed by dropout. And again, no significant performance benefit is reported. Overall, the paper seems like a solid class project, but the contribution to the field is minimal.
- Mortal_Mind 9y ago
  Have you read the whole paper? cause the whole idea around that paper was not to introduce a new architecture or breaking any new state of the art results but to identify and introduce a series of guidelines by which one can achieve better performance in "any" architecture. Their "demo" architecture which they call simpnet outperforms much deeper and more complex counterparts without the use of any fancy techniques such as skip connections, DenseBlock, etc. Not only their test network which was in fact a demo showcasing the effectiveness of the mentioned guidelines (as they indicate that in section 3 I guess) could outperform much heavier and more complex counterparts, the memory usage and training speed is also different and much better. The "SAF Pooling" part was to show how a design choice concerning something as trivial as a pooling operation can particularly affect the end result, improving network generalization capability, and what the intuition behind such design choice was. In the section they talk about SAF Pooling, they clearly state they deliberately did not choose a complex mechanism such as the ones they had covered, in order to keep everything simple ( a reminder of/reference to what they first claimed in the introduction) and show how much performance can be harnessed out of a simple design with a proper systematic approach. They also show some previously known practices/thoughts should not be followed and why. For example, concerning overlapped pooling they show the idea of overlapped pooling being better than its counter part, advocated by many including professor Hinton does not apply all the time, and interestingly using non overlapped pooling would be a better choice. Thats when they dig into the pooling operation and later show how to get to the SAF pooling idea and why that makes sense. The same thing goes about why architectures face degradation and explain what PLS and PLD issues are and why networks face such problems. They also explain why strided convolution is not a good replacement for maxpooling and many more.
  - p1esk 9y ago
    I've seen at least a dozen of papers providing "guidelines" to good NN architecture design (e.g. "Rethinking the Inception Architecture"). I don't mind another one, and I'm not saying they gave a bad advice, but as a DL practitioner I haven't really learned much from this paper as far as new good tricks to use. Anyone who tried using dropout in convolutional layers would use it after pooling/subsampling. It just does not make sense to do otherwise. Moreover, in my experiments, I didn't see any improvement of doing dropout there. If overfitting is a problem, dropout in conv layers is not going to help much. I've seen a number of various powerful new regularization methods published recently, I'd try them instead. As far as max pooling vs stride two convolutions, I compared them on multiple architectures, and while I agree that max pooling usually provided a slight boost, there were exceptions, and the difference was typically not significant enough to care. In fact I remember max pooling having a significant advantage only in very small networks (such as Lenet5). Same goes for the most of their advices: either you already know it, or it's not always true, or when true it does not result in significant accuracy gains. More importantly, and as I pointed out already, there are two serious shortcomings in the paper: 1. They have not tested their architecture on ImageNet. That alone disqualifies them from providing any advice on how to design a modern image classifier. Cifar is a toy dataset only suitable as proof of concept for new ideas, just to show that whatever crazy thing you came up with works in principle. In 2018 no one cares about results you got on Cifar. ImageNet is much more challenging, and that's why it requires all those "fancy techniques". 2. They have not compared their architecture to the latest state of the art (e.g. CondenseNet, which has been out for several months now). From the numbers I see in the paper, they haven't even bothered to measure the amount of computation (FLOPs). Show me that SimpNet outperforms existing SoA in efficiency models on ImageNet, specifically in regards to latency of inference. Then I will take the advice seriously. EDIT: I just read the section in the appendix which argues for using non-overlapping pooling (vs non-overlapping), and I fail to see any compelling explanation. Yes, higher layer feature maps show the presence of abstract features. Why should non-overlapping pooling be better here?
    - Mortal_Mind 9y ago
      I agree on the ImageNet and I also think they should publish their results on that dataset as well. I don't see why it shouldn't perform well. CIFAR10 is small compared to ImageNet, However, achieving highly competitive accuracy is very hard. Nearly all SOTA architectures have more than 20 ~ 50 Millions of parameters or high memory usage. So I still think their result is impressive enough concerning their "small and simple" architecture with that amount of parameters. If they were to use any new/fancy techniques and achieve the current accuracy, I would say you are right, but concerning everything that's been said, its impressive for its kind. Its as if, a normal car is already on par with top sports car which utilized all they could at hand to achieve their maximum performance. Think what happens if an already well-performing car gets tuned up with those fancy techniques its rivals are utilizing. (The main idea for the paper by the way) Again we need to pay attention here that, models such as CondenseNet cannot be compared with their architecture in that sense because the nature of the work is completely different. It seems you failed at getting their actual point concerning strided/max-pooling. Strided convolution always requires more parameters, it imposes more overhead compared to the pooling operation. Were you to have a limited budget concerning FLOPS/#params, you would clearly see what difference such trivial yet important statement makes. Concerning the ImageNet, while I agree with you that they should also submit their results on ImageNet, I do not think failing to do so would disqualify them completely! Since the nature of the work implies you can improve "any" architecture with the aforementioned principles. (ResNet and SqueezeNet were two simple examples in their paper with a single change!) So you would still be benefiting from their work even if you are not using their demo architecture. Aside from that, they are trying to provide better understanding as to why certain things work better, so that with such gained intuition, a better strategy, a method is devised or developed.(The reason for using dropout and their effect, the use of maxpooling, the PLD, PLS issues, etc) So unlike CondenseNet, they were not focused on a specific technique to improve, rather they tried to focus on the main building blocks that affect the performance the most and why. and in choosing those building blocks, they aimed at the most primitive ones that nearly all architectures can take advantage of. So Simply put, you may start improving the already great CondenseNet or pretty any architecture by following such principles. I could think of a reason as to why they may not have included the CondenseNet results. Looking at their comment on the arXiv page, it seems they had submitted their paper in December 2017 and also looking at the date CondensNet was published,(November 25, 2017) its possible by December they were already in the process of submitting their work and never noticed CondenseNet. They first dismissed the previous claim that advocated: "use overlapped pooling because of the imagelike properties and also because the non-overlapped pooling loses a lot of information in that sense which affects the performance adversely", then they indicated that when considering their intuition and what actually exists, looking at their experiments proved the non-overlapped pooling performing better. Their provided intuition seems plausible, but at the same time it may not apply to all situations.

3 points by ghosthamlet 9y ago

I'm sure a lot of effort went to writing this paper, but I have to say I'm underwhelmed with the reported results. First of all, CIFAR results in 2018 are just not going to cut it. The efficiency is only measured in terms of parameter count, and not even compared to CondenseNet. "SAF Pooling" does not deserve a special name, as it's just max pooling followed by dropout. And again, no significant performance benefit is reported. Overall, the paper seems like a solid class project, but the contribution to the field is minimal.

Mortal_Mind 9y ago
Have you read the whole paper? cause the whole idea around that paper was not to introduce a new architecture or breaking any new state of the art results but to identify and introduce a series of guidelines by which one can achieve better performance in "any" architecture. Their "demo" architecture which they call simpnet outperforms much deeper and more complex counterparts without the use of any fancy techniques such as skip connections, DenseBlock, etc. Not only their test network which was in fact a demo showcasing the effectiveness of the mentioned guidelines (as they indicate that in section 3 I guess) could outperform much heavier and more complex counterparts, the memory usage and training speed is also different and much better. The "SAF Pooling" part was to show how a design choice concerning something as trivial as a pooling operation can particularly affect the end result, improving network generalization capability, and what the intuition behind such design choice was. In the section they talk about SAF Pooling, they clearly state they deliberately did not choose a complex mechanism such as the ones they had covered, in order to keep everything simple ( a reminder of/reference to what they first claimed in the introduction) and show how much performance can be harnessed out of a simple design with a proper systematic approach. They also show some previously known practices/thoughts should not be followed and why. For example, concerning overlapped pooling they show the idea of overlapped pooling being better than its counter part, advocated by many including professor Hinton does not apply all the time, and interestingly using non overlapped pooling would be a better choice. Thats when they dig into the pooling operation and later show how to get to the SAF pooling idea and why that makes sense. The same thing goes about why architectures face degradation and explain what PLS and PLD issues are and why networks face such problems. They also explain why strided convolution is not a good replacement for maxpooling and many more.
- p1esk 9y ago
  I've seen at least a dozen of papers providing "guidelines" to good NN architecture design (e.g. "Rethinking the Inception Architecture"). I don't mind another one, and I'm not saying they gave a bad advice, but as a DL practitioner I haven't really learned much from this paper as far as new good tricks to use. Anyone who tried using dropout in convolutional layers would use it after pooling/subsampling. It just does not make sense to do otherwise. Moreover, in my experiments, I didn't see any improvement of doing dropout there. If overfitting is a problem, dropout in conv layers is not going to help much. I've seen a number of various powerful new regularization methods published recently, I'd try them instead. As far as max pooling vs stride two convolutions, I compared them on multiple architectures, and while I agree that max pooling usually provided a slight boost, there were exceptions, and the difference was typically not significant enough to care. In fact I remember max pooling having a significant advantage only in very small networks (such as Lenet5). Same goes for the most of their advices: either you already know it, or it's not always true, or when true it does not result in significant accuracy gains. More importantly, and as I pointed out already, there are two serious shortcomings in the paper: 1. They have not tested their architecture on ImageNet. That alone disqualifies them from providing any advice on how to design a modern image classifier. Cifar is a toy dataset only suitable as proof of concept for new ideas, just to show that whatever crazy thing you came up with works in principle. In 2018 no one cares about results you got on Cifar. ImageNet is much more challenging, and that's why it requires all those "fancy techniques". 2. They have not compared their architecture to the latest state of the art (e.g. CondenseNet, which has been out for several months now). From the numbers I see in the paper, they haven't even bothered to measure the amount of computation (FLOPs). Show me that SimpNet outperforms existing SoA in efficiency models on ImageNet, specifically in regards to latency of inference. Then I will take the advice seriously. EDIT: I just read the section in the appendix which argues for using non-overlapping pooling (vs non-overlapping), and I fail to see any compelling explanation. Yes, higher layer feature maps show the presence of abstract features. Why should non-overlapping pooling be better here?
  - Mortal_Mind 9y ago
    I agree on the ImageNet and I also think they should publish their results on that dataset as well. I don't see why it shouldn't perform well. CIFAR10 is small compared to ImageNet, However, achieving highly competitive accuracy is very hard. Nearly all SOTA architectures have more than 20 ~ 50 Millions of parameters or high memory usage. So I still think their result is impressive enough concerning their "small and simple" architecture with that amount of parameters. If they were to use any new/fancy techniques and achieve the current accuracy, I would say you are right, but concerning everything that's been said, its impressive for its kind. Its as if, a normal car is already on par with top sports car which utilized all they could at hand to achieve their maximum performance. Think what happens if an already well-performing car gets tuned up with those fancy techniques its rivals are utilizing. (The main idea for the paper by the way) Again we need to pay attention here that, models such as CondenseNet cannot be compared with their architecture in that sense because the nature of the work is completely different. It seems you failed at getting their actual point concerning strided/max-pooling. Strided convolution always requires more parameters, it imposes more overhead compared to the pooling operation. Were you to have a limited budget concerning FLOPS/#params, you would clearly see what difference such trivial yet important statement makes. Concerning the ImageNet, while I agree with you that they should also submit their results on ImageNet, I do not think failing to do so would disqualify them completely! Since the nature of the work implies you can improve "any" architecture with the aforementioned principles. (ResNet and SqueezeNet were two simple examples in their paper with a single change!) So you would still be benefiting from their work even if you are not using their demo architecture. Aside from that, they are trying to provide better understanding as to why certain things work better, so that with such gained intuition, a better strategy, a method is devised or developed.(The reason for using dropout and their effect, the use of maxpooling, the PLD, PLS issues, etc) So unlike CondenseNet, they were not focused on a specific technique to improve, rather they tried to focus on the main building blocks that affect the performance the most and why. and in choosing those building blocks, they aimed at the most primitive ones that nearly all architectures can take advantage of. So Simply put, you may start improving the already great CondenseNet or pretty any architecture by following such principles. I could think of a reason as to why they may not have included the CondenseNet results. Looking at their comment on the arXiv page, it seems they had submitted their paper in December 2017 and also looking at the date CondensNet was published,(November 25, 2017) its possible by December they were already in the process of submitting their work and never noticed CondenseNet. They first dismissed the previous claim that advocated: "use overlapped pooling because of the imagelike properties and also because the non-overlapped pooling loses a lot of information in that sense which affects the performance adversely", then they indicated that when considering their intuition and what actually exists, looking at their experiments proved the non-overlapped pooling performing better. Their provided intuition seems plausible, but at the same time it may not apply to all situations.