Forget about more abstract "images", you are not dealing with "images" from a certain point forward in a network. The concept and nature gets very different. Yo

Forget about more abstract "images", you are not dealing with "images" from a certain point forward in a network. The concept and nature gets very different. You are dealing with "feature"-maps. when this is the case, you can no more make assumptions based on dealing with images. If we are dealing with some features, it may make sense not to include duplicated features which will ultimately result in not very distinctive or accurate patterns. I guess, when you avoid the duplicated features or let me rephrase it better, prevent the overlapping, this way, you are essentially paying attention to the unique combination of features in the input and as the network is further trained, such patterns get more developed and the network can have much easier time to discriminate between classes, because of better feature entanglement. that's what I think is happening. 1.There are more than 10+ points in the paper, simple things that I guess you can readily do are : 1.Increase the interlayer connectivity 2.Use larger feature-maps, 3.Use max-pooling (or its dropout variant) (change the upper layers that do downsampling, to have the stride of 1, instead of 2 and then use a max-pooling after them for downsampling) 4.Use Global Max-pooling instead of Global Avg-Pooling 5.ResNet seems to be using padding and stride of 2 for early layers: 1.Do padding = 1, set stride = 2, remove the immediate pooling layer and instead use the stride 2 for the outgoing volumes(side branch and the main one, use stride of for the others). (the early local information are important, so the operation max-pooling carries out would not be interesting. 2.Experiment with the removal of 1x1 kernels in early layers, (as a starting point, change the first/two resblocks and later on in the next round of changes, add more blocks). 3.remember to use GP at the end as well. 4.Since we are removing 1x1 kernels in early layers, we will be facing an increase in #params, we first carry out the test, run the counterpart resnet with increased #params, or balance out the neurons in the next round (as they show in their paper, this balancing can be tricky and may need a well-planned distribution) 2.You can go about this like ResNet, they have similar basic building blocks, multiple poolings, use of stride 2 and average pooling. 3.SimpNet was a demo architecture, I don't think that would make sense :-/, but anyway if we are to enhance it based on what we have available, I would say, SimpNet would benefit much more if it was deeper and had interlayer connectivity. So since they point out to papers such as DenseNet, stating one of the main reasons they perform very well is because of the better and richer information pools made available by using interlayer connectivity, we can use that as well. Aside from these, they have not included their ImageNet tuned network. I couldn't find one in their repo as of today. The ImageNet model may differ, so we can either run our tests on the CIFAR10 model that we have or wait until they update their repo with the ImageNet model. One important thing to note here is that, after all the changes, the result should then be finetuned (hyperparameter tuning, multiple runs to check the variance and account for random initialization effect, etc ) so we can come to the right conclusion. (By the way whatever result you get, I think it would be a good idea to send the authors an email and let them know about your experiments) Good luck and keep me posted