not turned on by default but they are sort of a default choice for vision tasks. it may also be that they were useful on an earlier iteration of the model, and

not turned on by default but they are sort of a default choice for vision tasks. it may also be that they were useful on an earlier iteration of the model, and only became harmful later. nobody’s quite sure why anything works, though. hence comparisons to alchemy — we have lots of cool empirical results, and a highly elaborated set of theories and ideas trying to explain what works and what doesn’t, but in fact these theories often fail to make correct predictions. So the only way to make progress is constant blind experimentation.