4 ms·

Unfortunately I don't have anything to share, as much of the data now rests with my former employer. That being said, a few comments: * Understood that dataset

by whakim 3y ago

Unfortunately I don't have anything to share, as much of the data now rests with my former employer. That being said, a few comments: * Understood that dataset selection exists within the context of what others are doing and trying to provide comparable results. That being said, binary sentiment classifiers are (and should be) a pretty low bar for any reasonable technique in 2024, so it necessarily makes the reader wonder whether the conclusions are actually applicable to problems they might be facing. The point is taken that you're also looking at some older methods like TF-IDF which are prevalent in the (historical) AL literature. * Performing model selection and calibration (cross-validation) at each iteration makes intuitive sense, but doesn't really make practical sense. If the goal of AL in the real world is getting frequent updates from a human oracle, new models need to be trained in seconds (at absolute maximum). Even if performance improves by spending time doing model selection/CV, humans simply won't tolerate waiting around for new responses to label. * In your first link, the sample sizes seem extremely...small? If I was consistently faced with datasets only containing 1000 or 2000 data points, I'd probably just do supervised learning because the effort to set up a good AL pipeline (not to mention dealing with stopping criteria etc.) probably wouldn't be worth it. My experience has mostly been with datasets containing tens of thousands of data points (if not more). * 100% agree that sampling bias is a huge issue with a number of AL strategies and one we worked on extensively. A component of that is trying to design the interaction such that the human labeler has some insight into parts of the data distribution that the model is making assumptions about; your second link is very nice and we tried a number of clustering-type techniques alongside our active learner to try and offset sample bias. More generally, I think this points to the importance of considering any human-in-the-loop ML problem in a way that takes into account more than just algorithmic efficiency.

abhgh 3y ago
Thanks for the good discussion! * Reg. model selection/calibration: I acknowledge that in interactive setups cross-validation (CV) is not practical, but that is a particular kind of model selection. And faster alternatives exist for specific model families, such as approx. gradients [1,2], out-of-bag error (Random Forests), Infinitesimal Jacknife [3]. The bigger problem to me (for no model selection) is that there is a philosophical gap here: why would we fit the model with a default parameter blessed by a specific library? For ex., for LinearSVC in scikit, what is so great about C=1? And then, papers should mention that their query strategies are evaluated against C=1, but that of course doesn't sound as cool :-) Problem #2 is what I had mentioned earlier: the AL query strategy seems to be doing two things now - makes up for the lack of good hyperparameters in the current model, and picking up "good" points - how do I know which it does more of? Or what is it really good at? If its performance is influenced by a lot of the former, it is hardly surprising that when you actually perform model selection, the gap between a query strategy and random decreases. I guess my point is it doesn't have to be CV - but there needs to be model selection, because there is no well defined notion of a "default model". Calibration is less of an issue, because you can do this just once on the best model, and its not as expensive. * In that paper [4], the effect of AL is shown for upto 5k points, but the unlabelled pool size is 20k points ("Other Settings" in Page 5). So we do require good instance selection. On a different note, my hunch is that as language models get better at holding a broad variety of concept information, and it keeps getting easier to adapt them to specific tasks with just few examples/shots, we'll see whatever benefit AL has over random diminish. First (the bigger reason), the number of examples required to get good performance would reduce to the point that it might not make practical sense to set up an AL pipeline. And second, it would get easier to adapt the model per se, so that the difference (in terms of final model accuracy) between k good points and k randomly selected points might not be large. I guess we already see this in some form with SetFit [5] where examples are randomly picked to reach a good accuracy. I wonder if you have a view on that. [1] Hyperparameter optimization with approximate gradient, https://proceedings.mlr.press/v48/pedregosa16.html https://proceedings.mlr.press/v48/pedregosa16.html [2] Optimizing Millions of Hyperparameters by Implicit Differentiation, https://proceedings.mlr.press/v108/lorraine20a.html https://proceedings.mlr.press/v108/lorraine20a.html [3] A Swiss Army Infinitesimal Jackknife, https://proceedings.mlr.press/v89/giordano19a.html https://proceedings.mlr.press/v89/giordano19a.html [4] On the Fragility of Active Learners, https://arxiv.org/pdf/2403.15744.pdf https://arxiv.org/pdf/2403.15744.pdf [5] Efficient Few-Shot Learning Without Prompts, https://neurips2022-enlsp.github.io/papers/paper_17.pdf https://neurips2022-enlsp.github.io/papers/paper_17.pdf
- whakim 3y ago
  When I was looking at AL query strategies (including random sampling, which I agree should always be included as a baseline), hyperparameter tuning (and model selection more generally) was definitely performed and optimized on a per-strategy basis. (This is a lot of combinations but it's reasonable to prune things that are clearly dead-ends, and that's how we ended up with a non-linear SVC.) We just didn't do it between iterations due to the practical concerns. I understand the view that from a scientific perspective it is difficult to quantify the utility of a technique if it is providing multiple benefits at the same time. Not necessarily saying that's what's happening here, but it's worth pointing out that you can also look at this from a practical angle.For any given dataset your chosen model and set of hyperparameters probably isn't optimal, so if indeed AL is compensating for suboptimal choices in model selection, that may still be valuable. Another potentially interesting angle to consider is label imbalance; because we were constructing binary classification problems out of multiclass problems (as discussed earlier), some evidence shows AL strategies outperform random sampling to a significantly greater degree in imbalanced scenarios [1]. I agree with you that we're moving towards a world in which transformer models are so good that zero-shot or few-shot approaches to classification are going to be the way to go. Stuff I did 3 years ago I probably would not do today. [1] https://aclanthology.org/2020.emnlp-main.638.pdf https://aclanthology.org/2020.emnlp-main.638.pdf
  - abhgh 3y ago
    It becomes valuable if it either compensates for the lack of hyperparam tuning in all or a broad class of models, or the result is precisely presented to say something like "better than random if the model is a linear svm with C=1". Right now we are somewhere in between where the results are presented to seem somewhat generally useful, but when you tweak the hyperparams you realize that the utility is potentially quite narrow. We either need broad utility or increased rigor in reporting - we cannot abandon both. Yeah that's an interesting paper [1], and label imbalance is a case we have not investigated (we were able to reproduce some results from it - the DAL experiments in Section A.3 "Reproducibility Experiments" in [2] come from there). [1] Active Learning for BERT: An Empirical Study, https://aclanthology.org/2020.emnlp-main.638.pdf https://aclanthology.org/2020.emnlp-main.638.pdf [2] One the Fragility of Active Learners, https://arxiv.org/abs/2403.15744 https://arxiv.org/abs/2403.15744

by whakim 3y ago

abhgh 3y ago

Thanks for the good discussion! * Reg. model selection/calibration: I acknowledge that in interactive setups cross-validation (CV) is not practical, but that is a particular kind of model selection. And faster alternatives exist for specific model families, such as approx. gradients [1,2], out-of-bag error (Random Forests), Infinitesimal Jacknife [3]. The bigger problem to me (for no model selection) is that there is a philosophical gap here: why would we fit the model with a default parameter blessed by a specific library? For ex., for LinearSVC in scikit, what is so great about C=1? And then, papers should mention that their query strategies are evaluated against C=1, but that of course doesn't sound as cool :-) Problem #2 is what I had mentioned earlier: the AL query strategy seems to be doing two things now - makes up for the lack of good hyperparameters in the current model, and picking up "good" points - how do I know which it does more of? Or what is it really good at? If its performance is influenced by a lot of the former, it is hardly surprising that when you actually perform model selection, the gap between a query strategy and random decreases. I guess my point is it doesn't have to be CV - but there needs to be model selection, because there is no well defined notion of a "default model". Calibration is less of an issue, because you can do this just once on the best model, and its not as expensive. * In that paper [4], the effect of AL is shown for upto 5k points, but the unlabelled pool size is 20k points ("Other Settings" in Page 5). So we do require good instance selection. On a different note, my hunch is that as language models get better at holding a broad variety of concept information, and it keeps getting easier to adapt them to specific tasks with just few examples/shots, we'll see whatever benefit AL has over random diminish. First (the bigger reason), the number of examples required to get good performance would reduce to the point that it might not make practical sense to set up an AL pipeline. And second, it would get easier to adapt the model per se, so that the difference (in terms of final model accuracy) between k good points and k randomly selected points might not be large. I guess we already see this in some form with SetFit [5] where examples are randomly picked to reach a good accuracy. I wonder if you have a view on that. [1] Hyperparameter optimization with approximate gradient, https://proceedings.mlr.press/v48/pedregosa16.html https://proceedings.mlr.press/v48/pedregosa16.html [2] Optimizing Millions of Hyperparameters by Implicit Differentiation, https://proceedings.mlr.press/v108/lorraine20a.html https://proceedings.mlr.press/v108/lorraine20a.html [3] A Swiss Army Infinitesimal Jackknife, https://proceedings.mlr.press/v89/giordano19a.html https://proceedings.mlr.press/v89/giordano19a.html [4] On the Fragility of Active Learners, https://arxiv.org/pdf/2403.15744.pdf https://arxiv.org/pdf/2403.15744.pdf [5] Efficient Few-Shot Learning Without Prompts, https://neurips2022-enlsp.github.io/papers/paper_17.pdf https://neurips2022-enlsp.github.io/papers/paper_17.pdf

whakim 3y ago
When I was looking at AL query strategies (including random sampling, which I agree should always be included as a baseline), hyperparameter tuning (and model selection more generally) was definitely performed and optimized on a per-strategy basis. (This is a lot of combinations but it's reasonable to prune things that are clearly dead-ends, and that's how we ended up with a non-linear SVC.) We just didn't do it between iterations due to the practical concerns. I understand the view that from a scientific perspective it is difficult to quantify the utility of a technique if it is providing multiple benefits at the same time. Not necessarily saying that's what's happening here, but it's worth pointing out that you can also look at this from a practical angle.For any given dataset your chosen model and set of hyperparameters probably isn't optimal, so if indeed AL is compensating for suboptimal choices in model selection, that may still be valuable. Another potentially interesting angle to consider is label imbalance; because we were constructing binary classification problems out of multiclass problems (as discussed earlier), some evidence shows AL strategies outperform random sampling to a significantly greater degree in imbalanced scenarios [1]. I agree with you that we're moving towards a world in which transformer models are so good that zero-shot or few-shot approaches to classification are going to be the way to go. Stuff I did 3 years ago I probably would not do today. [1] https://aclanthology.org/2020.emnlp-main.638.pdf https://aclanthology.org/2020.emnlp-main.638.pdf
- abhgh 3y ago
  It becomes valuable if it either compensates for the lack of hyperparam tuning in all or a broad class of models, or the result is precisely presented to say something like "better than random if the model is a linear svm with C=1". Right now we are somewhere in between where the results are presented to seem somewhat generally useful, but when you tweak the hyperparams you realize that the utility is potentially quite narrow. We either need broad utility or increased rigor in reporting - we cannot abandon both. Yeah that's an interesting paper [1], and label imbalance is a case we have not investigated (we were able to reproduce some results from it - the DAL experiments in Section A.3 "Reproducibility Experiments" in [2] come from there). [1] Active Learning for BERT: An Empirical Study, https://aclanthology.org/2020.emnlp-main.638.pdf https://aclanthology.org/2020.emnlp-main.638.pdf [2] One the Fragility of Active Learners, https://arxiv.org/abs/2403.15744 https://arxiv.org/abs/2403.15744