3 ms·

I agree on the ImageNet and I also think they should publish their results on that dataset as well. I don't see why it shouldn't perform well. CIFAR10 is small

I agree on the ImageNet and I also think they should publish their results on that dataset as well. I don't see why it shouldn't perform well. CIFAR10 is small compared to ImageNet, However, achieving highly competitive accuracy is very hard. Nearly all SOTA architectures have more than 20 ~ 50 Millions of parameters or high memory usage. So I still think their result is impressive enough concerning their "small and simple" architecture with that amount of parameters. If they were to use any new/fancy techniques and achieve the current accuracy, I would say you are right, but concerning everything that's been said, its impressive for its kind. Its as if, a normal car is already on par with top sports car which utilized all they could at hand to achieve their maximum performance. Think what happens if an already well-performing car gets tuned up with those fancy techniques its rivals are utilizing. (The main idea for the paper by the way) Again we need to pay attention here that, models such as CondenseNet cannot be compared with their architecture in that sense because the nature of the work is completely different. It seems you failed at getting their actual point concerning strided/max-pooling. Strided convolution always requires more parameters, it imposes more overhead compared to the pooling operation. Were you to have a limited budget concerning FLOPS/#params, you would clearly see what difference such trivial yet important statement makes. Concerning the ImageNet, while I agree with you that they should also submit their results on ImageNet, I do not think failing to do so would disqualify them completely! Since the nature of the work implies you can improve "any" architecture with the aforementioned principles. (ResNet and SqueezeNet were two simple examples in their paper with a single change!) So you would still be benefiting from their work even if you are not using their demo architecture. Aside from that, they are trying to provide better understanding as to why certain things work better, so that with such gained intuition, a better strategy, a method is devised or developed.(The reason for using dropout and their effect, the use of maxpooling, the PLD, PLS issues, etc) So unlike CondenseNet, they were not focused on a specific technique to improve, rather they tried to focus on the main building blocks that affect the performance the most and why. and in choosing those building blocks, they aimed at the most primitive ones that nearly all architectures can take advantage of. So Simply put, you may start improving the already great CondenseNet or pretty any architecture by following such principles. I could think of a reason as to why they may not have included the CondenseNet results. Looking at their comment on the arXiv page, it seems they had submitted their paper in December 2017 and also looking at the date CondensNet was published,(November 25, 2017) its possible by December they were already in the process of submitting their work and never noticed CondenseNet. They first dismissed the previous claim that advocated: "use overlapped pooling because of the imagelike properties and also because the non-overlapped pooling loses a lot of information in that sense which affects the performance adversely", then they indicated that when considering their intuition and what actually exists, looking at their experiments proved the non-overlapped pooling performing better. Their provided intuition seems plausible, but at the same time it may not apply to all situations.

p1esk 9y ago
Ok, you have a good point regarding max pooling being more efficient than strided convolutions. I didn't think of that. Regarding overlapping pooling: ok, so they dismissed the previous intuition. But I don't see any new intuition offered! Yes, those higher layer feature maps are more abstract, so what? Why exactly non-overlapping pooling should work better for more abstract "images"? At least in Hinton's paper they tried to explain it. Ultimately, it all comes down to this: 1. I often use a vanilla ResNet-50 with ImageNet as a baseline (e.g. [1]]). Can you tell me what suggestions from the paper I can apply to make it either more accurate or more efficient? 2. If I use CondenseNet [2], what suggestions from the paper would help me improve accuracy/efficiency? 3. Alternatively, if I use the SimpNet as a starting point, what adjustments should I make to make it competitive on ImageNet? I have a 4 GPU workstation sitting idle at the moment, so I can test any of these scenarios right away. [1] https://github.com/tensorflow/models/blob/master/official/resnet/imagenet_main.py https://github.com/tensorflow/models/blob/master/official/re... [2] https://github.com/ShichenLiu/CondenseNet https://github.com/ShichenLiu/CondenseNet
- Mortal_Mind 9y ago
  Forget about more abstract "images", you are not dealing with "images" from a certain point forward in a network. The concept and nature gets very different. You are dealing with "feature"-maps. when this is the case, you can no more make assumptions based on dealing with images. If we are dealing with some features, it may make sense not to include duplicated features which will ultimately result in not very distinctive or accurate patterns. I guess, when you avoid the duplicated features or let me rephrase it better, prevent the overlapping, this way, you are essentially paying attention to the unique combination of features in the input and as the network is further trained, such patterns get more developed and the network can have much easier time to discriminate between classes, because of better feature entanglement. that's what I think is happening. 1.There are more than 10+ points in the paper, simple things that I guess you can readily do are : 1.Increase the interlayer connectivity 2.Use larger feature-maps, 3.Use max-pooling (or its dropout variant) (change the upper layers that do downsampling, to have the stride of 1, instead of 2 and then use a max-pooling after them for downsampling) 4.Use Global Max-pooling instead of Global Avg-Pooling 5.ResNet seems to be using padding and stride of 2 for early layers: 1.Do padding = 1, set stride = 2, remove the immediate pooling layer and instead use the stride 2 for the outgoing volumes(side branch and the main one, use stride of for the others). (the early local information are important, so the operation max-pooling carries out would not be interesting. 2.Experiment with the removal of 1x1 kernels in early layers, (as a starting point, change the first/two resblocks and later on in the next round of changes, add more blocks). 3.remember to use GP at the end as well. 4.Since we are removing 1x1 kernels in early layers, we will be facing an increase in #params, we first carry out the test, run the counterpart resnet with increased #params, or balance out the neurons in the next round (as they show in their paper, this balancing can be tricky and may need a well-planned distribution) 2.You can go about this like ResNet, they have similar basic building blocks, multiple poolings, use of stride 2 and average pooling. 3.SimpNet was a demo architecture, I don't think that would make sense :-/, but anyway if we are to enhance it based on what we have available, I would say, SimpNet would benefit much more if it was deeper and had interlayer connectivity. So since they point out to papers such as DenseNet, stating one of the main reasons they perform very well is because of the better and richer information pools made available by using interlayer connectivity, we can use that as well. Aside from these, they have not included their ImageNet tuned network. I couldn't find one in their repo as of today. The ImageNet model may differ, so we can either run our tests on the CIFAR10 model that we have or wait until they update their repo with the ImageNet model. One important thing to note here is that, after all the changes, the result should then be finetuned (hyperparameter tuning, multiple runs to check the variance and account for random initialization effect, etc ) so we can come to the right conclusion. (By the way whatever result you get, I think it would be a good idea to send the authors an email and let them know about your experiments) Good luck and keep me posted

p1esk 9y ago

Ok, you have a good point regarding max pooling being more efficient than strided convolutions. I didn't think of that. Regarding overlapping pooling: ok, so they dismissed the previous intuition. But I don't see any new intuition offered! Yes, those higher layer feature maps are more abstract, so what? Why exactly non-overlapping pooling should work better for more abstract "images"? At least in Hinton's paper they tried to explain it. Ultimately, it all comes down to this: 1. I often use a vanilla ResNet-50 with ImageNet as a baseline (e.g. [1]]). Can you tell me what suggestions from the paper I can apply to make it either more accurate or more efficient? 2. If I use CondenseNet [2], what suggestions from the paper would help me improve accuracy/efficiency? 3. Alternatively, if I use the SimpNet as a starting point, what adjustments should I make to make it competitive on ImageNet? I have a 4 GPU workstation sitting idle at the moment, so I can test any of these scenarios right away. [1] https://github.com/tensorflow/models/blob/master/official/resnet/imagenet_main.py https://github.com/tensorflow/models/blob/master/official/re... [2] https://github.com/ShichenLiu/CondenseNet https://github.com/ShichenLiu/CondenseNet

Mortal_Mind 9y ago
Forget about more abstract "images", you are not dealing with "images" from a certain point forward in a network. The concept and nature gets very different. You are dealing with "feature"-maps. when this is the case, you can no more make assumptions based on dealing with images. If we are dealing with some features, it may make sense not to include duplicated features which will ultimately result in not very distinctive or accurate patterns. I guess, when you avoid the duplicated features or let me rephrase it better, prevent the overlapping, this way, you are essentially paying attention to the unique combination of features in the input and as the network is further trained, such patterns get more developed and the network can have much easier time to discriminate between classes, because of better feature entanglement. that's what I think is happening. 1.There are more than 10+ points in the paper, simple things that I guess you can readily do are : 1.Increase the interlayer connectivity 2.Use larger feature-maps, 3.Use max-pooling (or its dropout variant) (change the upper layers that do downsampling, to have the stride of 1, instead of 2 and then use a max-pooling after them for downsampling) 4.Use Global Max-pooling instead of Global Avg-Pooling 5.ResNet seems to be using padding and stride of 2 for early layers: 1.Do padding = 1, set stride = 2, remove the immediate pooling layer and instead use the stride 2 for the outgoing volumes(side branch and the main one, use stride of for the others). (the early local information are important, so the operation max-pooling carries out would not be interesting. 2.Experiment with the removal of 1x1 kernels in early layers, (as a starting point, change the first/two resblocks and later on in the next round of changes, add more blocks). 3.remember to use GP at the end as well. 4.Since we are removing 1x1 kernels in early layers, we will be facing an increase in #params, we first carry out the test, run the counterpart resnet with increased #params, or balance out the neurons in the next round (as they show in their paper, this balancing can be tricky and may need a well-planned distribution) 2.You can go about this like ResNet, they have similar basic building blocks, multiple poolings, use of stride 2 and average pooling. 3.SimpNet was a demo architecture, I don't think that would make sense :-/, but anyway if we are to enhance it based on what we have available, I would say, SimpNet would benefit much more if it was deeper and had interlayer connectivity. So since they point out to papers such as DenseNet, stating one of the main reasons they perform very well is because of the better and richer information pools made available by using interlayer connectivity, we can use that as well. Aside from these, they have not included their ImageNet tuned network. I couldn't find one in their repo as of today. The ImageNet model may differ, so we can either run our tests on the CIFAR10 model that we have or wait until they update their repo with the ImageNet model. One important thing to note here is that, after all the changes, the result should then be finetuned (hyperparameter tuning, multiple runs to check the variance and account for random initialization effect, etc ) so we can come to the right conclusion. (By the way whatever result you get, I think it would be a good idea to send the authors an email and let them know about your experiments) Good luck and keep me posted