Paper is stretching by picking favorable benchmarks. It’s not ResNet on ImageNet, folks. It’s AlexNet on Fashion-MNIST. AlexNet (2012) is out of date. The perf

Paper is stretching by picking favorable benchmarks. It’s not ResNet on ImageNet, folks. It’s AlexNet on Fashion-MNIST. AlexNet (2012) is out of date. The performance each parameter brings is less than newer models. Newer models, apart from just being better, use convolutions more extensively, which have less parameters than linear/affine layers. The authors describe this choice in Section 4, even showing how they have to retrofit a 224x224 model to the small fashion MNIST 28x28 dataset they use. Therefore, when you embed a big payload into AlexNet, you should not be surprised that you lose little accuracy. Your model is a low accuracy model to begin with and the model’s parameter counts are fluffed up by its choice of layers.