6 ms·
Presented with no comment :) https://twitter.com/chhillee/status/1635790330854526981?s=46&t=IVF1sX_TGndxvax1l-hJ0Q https://twitter.com/chhillee/status/163579033
by FanaHOVA 3y ago
Presented with no comment :) https://twitter.com/chhillee/status/1635790330854526981?s=46&t=IVF1sX_TGndxvax1l-hJ0Q https://twitter.com/chhillee/status/1635790330854526981?s=46...
- deleted 3y ago[deleted]
- lumost 3y agoHaving worked on ML products, there is sometimes debate on whether you should train on the test partition prior to prod deployment - after all, why would you ship a worse model to prod? Obviously you can't tell whether the model is better at generalization compared to an alternate technique, and you also incur some overfit risk. But many industrial problems are solvable through memorization.
- sangnoir 3y ago> after all, why would you ship a worse model to prod? ...because you need a control to evaluate how well your product is doing? I know it's a young field, but boy, do some folk love removing the "science" from "data science"
- baobabKoodaa 3y agoYou can evaluate a version of the model that has been trained on one set of data, and ship to production a different model that has been trained on the complete set of data. In many cases one can reasonably infer that the model which has seen all of the data will be better than the model which has seen only some of the data. I'm not claiming that's what happened here, nor am I interested in nitpicking "what counts as 'science'". I'm just saying this is a reasonable thing to do.
- mafuy 3y agoThis is possible if you use e.g. train 1000 models on different subsets of data and verify that each and every one of them is performing well. In that case, you can reasonably infer that another model trained on all data would work well, too. But this is, of course, 1000 times more expensive to do. And if you only train 100, or 10, or 1 model, then the deduction becomes increasingly unstable. So from a practical point of view, it's probably not feasible, because you would put those resources into something else instead that has more ROI.
- baobabKoodaa 3y agoI have personally never seen a situation where more training data (of similar quality) causes the model to perform worse. Have you seen such a situation? Please provide example. Your suggestion of running 1000 training runs with different subsets of data sounds excessive and unnecessary to me.
- nightski 3y agoYou have to know when to stop training. How are you going to do that without a test set? How do you know when you have achieved generalization without over-fitting?
- wedesoft 3y agoEarly stopping is just one way of regularization. You can use L2 or dropout and then you can train until your model converges.
- deleted 3y ago[deleted]
- baobabKoodaa 3y agoUsually I develop models with a train/validation/test split, where I'm measuring results on the validation set to decide the appropriate number of epochs to use. Then I burn the test set to evaluate performance. Then I train from scratch on the entire dataset (no split) and I use the same number of epochs to train here. Is this number of epochs optimal when the dataset is different? Of course not. But when you use regularization and other methods to combat overfitting appropriately, your training is not going to be overly sensitive to changes in epoch number anyway.
- deleted 3y ago[deleted]
- Naracion 3y ago>infer that the model which has seen all of the data will be better than the model which has seen only some of the data. It really depends upon the data. A smaller set of data that mostly consists of "truth" might be better than a larger dataset that also has many "lies". Perhaps what you mean is that the model might be more representative, rather than _better_.
- janalsncm 3y agoThere are offline metrics and online metrics. Offline metrics might be something like AUROC on a test set. Once you’ve pushed the model online, you can check the online metrics. Ultimately the online metrics are more important, that’s the whole reason the model exists in the first place. Your control in an online environment is the current baseline. You don’t need to save the test set anymore, you can push it online and test it directly.
- snowstormsun 3y agoWhy would you want to ship an untested model? That's insane.
- baobabKoodaa 3y agoThis is a common approach, for example, in data science competitions. Why? Well, if you want to maximize the model's abilities, this is what you have to do. (Not saying Llama 2 is released like this; it probably isn't)
- snowstormsun 3y agoYeah but in competitions there's a secret test set used to evaluate the model.
- baobabKoodaa 3y agoI have personally shipped "untested" models in production in situations where a "secret test set" does not exist. (Train on subset of data -> evaluate on different subset of data -> train again on entire dataset). I do not consider myself to be insane.
- snowstormsun 3y agoI didn't mean to insult anyone. The idea of not knowing the actual performance of the model just intuitively seems to me like it's a bit of a gamble. I have only trained models in a scientific context before, where this was never an option.
- DougBTX 3y agoHere's another way to look at it. The test set is an approximation for how the model will perform against production data, but the actual performance of the model is how it performs for actual end-users. So real _actual_ results are always unknown util after the fact. Given that, if the metrics from training clearly show that more data == better model, and there's no reason to expect that trend to reverse, then the logical thing to do is maximise the data used for training to get the best results for actual production data. Doing this does complicate decisions for releasing subsequent model updates, as the production model can't be directly compared against new iterations any more. Instead a pre-production model would need to be used, that has not seen the test set. However, if data drift is likely, then re-using the old test set wouldn't be useful anyway.
- sundarurfriend 3y agoNitter link: https://nitter.net/chhillee/status/1635790330854526981/ https://nitter.net/chhillee/status/1635790330854526981/