4 ms·

I haven't got a clue about meteorology, but I know a thing or two about machine learning and the majority of papers in the field remain interesting prototypes b

I haven't got a clue about meteorology, but I know a thing or two about machine learning and the majority of papers in the field remain interesting prototypes because they are tested in the same way: namely, on a test dataset that is already known and readily accessible during training -by the experimenters, and thereofore, indirectly, by the training model itself. This is the case with this paper, from which I quote (see section "Dataset creation and splits"): >> The available data spans a period from July 2017 to August 2020. In other words, they have shown that their system works better than an earlier system on old data. It can predict rainfall in a three-ish year interval in the past. Show that your approach works on real-world data, that you didn't have access to when training your model. Show that your models can predict the future, not the past. Otherwise, your experiments don't mean, what you think they mean. Edit: and to make it perfectly clear for the uninitiated: when you, as an experimenter, know the "ground truth" of your test data, there is nothing easier than to tune the training of your model to maximise its performance on the test data. That's doubly so, and doubly as dangerous, with neural nets that are very, very good at reproducing their training set, but very, very bad at generalising beyond it. Unfortunately, this is what the vast majority of experiments in machine learning do: they test on known data. The result is that nobody really knows how good the tested systems are until they deploy them in a real environment (if they ever do, which they usually don't, because the whole point was writing a paper to report improved performance, and then move on to the next).

thrwawy74 4y ago
As an anecdote, I just started a masters in data analytics and I've been having a lot of fun asking ChatGPT for help detecting overfitting and useful terms like the mean squared error (MSE).
- m463 4y ago
  lol, reminds me of "The brain is the most important part of the body, until you realize who's telling you"
  - thrwawy74 4y ago
    Absolutely; i just mean I love it for guiding my learning :) i don't use it for final projects, but it does help me navigate the material if I'm not comprehending it well. I hope ChatGPT is embraced for that reason, because it's really like finding a key to a lock with professors. They explain things in the most effective way they know how, and ChatGPT wouldn't get exhausted or impatient with you if you ask it the same question in different ways. That said, it does produce erroneous information. I hope less than my professors. :)
candiodari 4y ago
> Show that your approach works on real-world data, that you didn't have access to when training your model. Show that your models can predict the future, not the past. Otherwise, your experiments don't mean, what you think they mean. Great, but ... that requires you have collection infrastructure and almost by definition means you can't make it work on a dataset. The way ML research works is like the Kaggle competitions. You get a dataset. This can be anything, as long as it can be expressed as a tensor (which are vectors, but can have more dimensions). Then split into seen and unseen data. You split into 3 portions/partitions. One for training. One for testing, and one for blind validation (80% training and 10% each for test and validation sets). In order to simulate what you're suggesting researchers can, and often do, take the most recent datapoints as validation and testing sets. But extending the data set is an expensive process that generally can't be done by the machine learning researchers themselves. So it is not done.
zamfi 4y ago
Best practice here is to keep a chunk of your dataset held out from your entire development process (a "held out test set"), and then only evaluate it when you write your final report on your work. The idea is that you don't even look at this data until you're confident you've solved the problem, and then you have one shot to confirm that you have, indeed, solved it. Obviously you have to be careful in selecting this set -- if you're working with time-series, for example, you want it to be temporally disjoint -- but done well it's a reasonable approximation for "future results". About as good as you can get without collecting new data. I can't tell if they did that here, but if they did, they didn't use the common jargon for it.
- dr_dshiv 4y ago
  > About as good as you can get without collecting new data. So let’s call holdout datasets “second-best practice”?
  - zamfi 4y ago
    LOL, love it! Sadly too common that collecting new data is practically speaking impossible, because the actual collection process is underspecified.
- martincmartin 4y ago
  Suppose you do this, try some different models, get the error down on the training sample. Then you apply to the test sample and the results are ... about the same as the well known results. The best practice says you should now quit and move on to another project, and either publish a negative result or don't bother publishing at all. Human nature, and publish-or-perish rewards, mean the researcher will say "well, what if I try this other model? Or tweak this parameter, or ..." In other words, it's unrealistic to say you have one shot to confirm that have, indeed, solved it.
  - zamfi 4y ago
    Not unrealistic -- it just causes people to be very, very careful about not fooling themselves. Do you really believe that this practice is unrealistic because anyone with a model that doesn't do well on the held-out set will simply engage in fraud? That is a very, very cynical take. FWIW, I have published this way [0]. There were definitely some aspects of the model's performance on the held-out set that did not meet test set performance; it's not fatal to the paper, it's an invitation to discuss. [0] https://arxiv.org/pdf/2205.07333.pdf https://arxiv.org/pdf/2205.07333.pdf
muraiki 4y ago
It's worth noting that the paper was submitted in August of 2021. So perhaps it took a year to develop the methodology and get the paper ready for publishing. They did use separate training, validation, and test sets for data, although we do need to trust that the test set was only used a single time. To whatever extent the model doesn't do as well in 2022 as it did at the time it was created, that comes down to concept drift and data drift.
m463 4y ago
Reminds me of the visual cancer recognition model that was actually keying off of the ruler in the picture.
j7ake 4y ago
In competition formats, the test data is not seen by anybody and after submission, the best algorithm is ranked based on how well it works on this unseen test data.

thrwawy74 4y ago

As an anecdote, I just started a masters in data analytics and I've been having a lot of fun asking ChatGPT for help detecting overfitting and useful terms like the mean squared error (MSE).

m463 4y ago
lol, reminds me of "The brain is the most important part of the body, until you realize who's telling you"
- thrwawy74 4y ago
  Absolutely; i just mean I love it for guiding my learning :) i don't use it for final projects, but it does help me navigate the material if I'm not comprehending it well. I hope ChatGPT is embraced for that reason, because it's really like finding a key to a lock with professors. They explain things in the most effective way they know how, and ChatGPT wouldn't get exhausted or impatient with you if you ask it the same question in different ways. That said, it does produce erroneous information. I hope less than my professors. :)

candiodari 4y ago

> Show that your approach works on real-world data, that you didn't have access to when training your model. Show that your models can predict the future, not the past. Otherwise, your experiments don't mean, what you think they mean. Great, but ... that requires you have collection infrastructure and almost by definition means you can't make it work on a dataset. The way ML research works is like the Kaggle competitions. You get a dataset. This can be anything, as long as it can be expressed as a tensor (which are vectors, but can have more dimensions). Then split into seen and unseen data. You split into 3 portions/partitions. One for training. One for testing, and one for blind validation (80% training and 10% each for test and validation sets). In order to simulate what you're suggesting researchers can, and often do, take the most recent datapoints as validation and testing sets. But extending the data set is an expensive process that generally can't be done by the machine learning researchers themselves. So it is not done.

zamfi 4y ago

Best practice here is to keep a chunk of your dataset held out from your entire development process (a "held out test set"), and then only evaluate it when you write your final report on your work. The idea is that you don't even look at this data until you're confident you've solved the problem, and then you have one shot to confirm that you have, indeed, solved it. Obviously you have to be careful in selecting this set -- if you're working with time-series, for example, you want it to be temporally disjoint -- but done well it's a reasonable approximation for "future results". About as good as you can get without collecting new data. I can't tell if they did that here, but if they did, they didn't use the common jargon for it.

dr_dshiv 4y ago
> About as good as you can get without collecting new data. So let’s call holdout datasets “second-best practice”?
- zamfi 4y ago
  LOL, love it! Sadly too common that collecting new data is practically speaking impossible, because the actual collection process is underspecified.
martincmartin 4y ago
Suppose you do this, try some different models, get the error down on the training sample. Then you apply to the test sample and the results are ... about the same as the well known results. The best practice says you should now quit and move on to another project, and either publish a negative result or don't bother publishing at all. Human nature, and publish-or-perish rewards, mean the researcher will say "well, what if I try this other model? Or tweak this parameter, or ..." In other words, it's unrealistic to say you have one shot to confirm that have, indeed, solved it.
- zamfi 4y ago
  Not unrealistic -- it just causes people to be very, very careful about not fooling themselves. Do you really believe that this practice is unrealistic because anyone with a model that doesn't do well on the held-out set will simply engage in fraud? That is a very, very cynical take. FWIW, I have published this way [0]. There were definitely some aspects of the model's performance on the held-out set that did not meet test set performance; it's not fatal to the paper, it's an invitation to discuss. [0] https://arxiv.org/pdf/2205.07333.pdf https://arxiv.org/pdf/2205.07333.pdf

muraiki 4y ago

It's worth noting that the paper was submitted in August of 2021. So perhaps it took a year to develop the methodology and get the paper ready for publishing. They did use separate training, validation, and test sets for data, although we do need to trust that the test set was only used a single time. To whatever extent the model doesn't do as well in 2022 as it did at the time it was created, that comes down to concept drift and data drift.

m463 4y ago

Reminds me of the visual cancer recognition model that was actually keying off of the ruler in the picture.

j7ake 4y ago

In competition formats, the test data is not seen by anybody and after submission, the best algorithm is ranked based on how well it works on this unseen test data.