Thanks for the good discussion! * Reg. model selection/calibration: I acknowledge that in interactive setups cross-validation (CV) is not practical, but that i

Thanks for the good discussion! * Reg. model selection/calibration: I acknowledge that in interactive setups cross-validation (CV) is not practical, but that is a particular kind of model selection. And faster alternatives exist for specific model families, such as approx. gradients [1,2], out-of-bag error (Random Forests), Infinitesimal Jacknife [3]. The bigger problem to me (for no model selection) is that there is a philosophical gap here: why would we fit the model with a default parameter blessed by a specific library? For ex., for LinearSVC in scikit, what is so great about C=1? And then, papers should mention that their query strategies are evaluated against C=1, but that of course doesn't sound as cool :-) Problem #2 is what I had mentioned earlier: the AL query strategy seems to be doing two things now - makes up for the lack of good hyperparameters in the current model, and picking up "good" points - how do I know which it does more of? Or what is it really good at? If its performance is influenced by a lot of the former, it is hardly surprising that when you actually perform model selection, the gap between a query strategy and random decreases. I guess my point is it doesn't have to be CV - but there needs to be model selection, because there is no well defined notion of a "default model". Calibration is less of an issue, because you can do this just once on the best model, and its not as expensive. * In that paper [4], the effect of AL is shown for upto 5k points, but the unlabelled pool size is 20k points ("Other Settings" in Page 5). So we do require good instance selection. On a different note, my hunch is that as language models get better at holding a broad variety of concept information, and it keeps getting easier to adapt them to specific tasks with just few examples/shots, we'll see whatever benefit AL has over random diminish. First (the bigger reason), the number of examples required to get good performance would reduce to the point that it might not make practical sense to set up an AL pipeline. And second, it would get easier to adapt the model per se, so that the difference (in terms of final model accuracy) between k good points and k randomly selected points might not be large. I guess we already see this in some form with SetFit [5] where examples are randomly picked to reach a good accuracy. I wonder if you have a view on that. [1] Hyperparameter optimization with approximate gradient, https://proceedings.mlr.press/v48/pedregosa16.html https://proceedings.mlr.press/v48/pedregosa16.html [2] Optimizing Millions of Hyperparameters by Implicit Differentiation, https://proceedings.mlr.press/v108/lorraine20a.html https://proceedings.mlr.press/v108/lorraine20a.html [3] A Swiss Army Infinitesimal Jackknife, https://proceedings.mlr.press/v89/giordano19a.html https://proceedings.mlr.press/v89/giordano19a.html [4] On the Fragility of Active Learners, https://arxiv.org/pdf/2403.15744.pdf https://arxiv.org/pdf/2403.15744.pdf [5] Efficient Few-Shot Learning Without Prompts, https://neurips2022-enlsp.github.io/papers/paper_17.pdf https://neurips2022-enlsp.github.io/papers/paper_17.pdf