Architecture search always works for improving efficiency, but it never beats a real algorithmic improvement (and there were lots of interesting ones being prop

Architecture search always works for improving efficiency, but it never beats a real algorithmic improvement (and there were lots of interesting ones being proposed for increasing context window / improving inference time in the last few months).