This seems bad, though? They introduce a complicated training regime to make up for tearing out a lot of known-good ideas like residual connections. (ie, not s

This seems bad, though? They introduce a complicated training regime to make up for tearing out a lot of known-good ideas like residual connections. (ie, not so vanilla after all.) They end up needing a lot more parameters and flops to match resnet quality... (Edit to add) Like why not use some clever distillation to get things to work well? Take the large well performing model and try to match intermediate activations to avoid gradient collapse. Training should then be very straightforward. Really any time I see multi stage training in a paper I'm afraid... Tends to indicate that the whole system is very fragile, and that we'll be searching for just the right hyperparameters for months...