When I was looking at AL query strategies (including random sampling, which I agree should always be included as a baseline), hyperparameter tuning (and model s

When I was looking at AL query strategies (including random sampling, which I agree should always be included as a baseline), hyperparameter tuning (and model selection more generally) was definitely performed and optimized on a per-strategy basis. (This is a lot of combinations but it's reasonable to prune things that are clearly dead-ends, and that's how we ended up with a non-linear SVC.) We just didn't do it between iterations due to the practical concerns. I understand the view that from a scientific perspective it is difficult to quantify the utility of a technique if it is providing multiple benefits at the same time. Not necessarily saying that's what's happening here, but it's worth pointing out that you can also look at this from a practical angle.For any given dataset your chosen model and set of hyperparameters probably isn't optimal, so if indeed AL is compensating for suboptimal choices in model selection, that may still be valuable. Another potentially interesting angle to consider is label imbalance; because we were constructing binary classification problems out of multiclass problems (as discussed earlier), some evidence shows AL strategies outperform random sampling to a significantly greater degree in imbalanced scenarios [1]. I agree with you that we're moving towards a world in which transformer models are so good that zero-shot or few-shot approaches to classification are going to be the way to go. Stuff I did 3 years ago I probably would not do today. [1] https://aclanthology.org/2020.emnlp-main.638.pdf https://aclanthology.org/2020.emnlp-main.638.pdf