5 ms·

What is "data leakage"? Do the authors define it? They reference Kaufman et al, but that makes it sound just like "errors". But what are the errors?

Flashtoo 4y ago
When you evaluate an ML approach, you should use one part of the data to train your model and a completely separate part to evaluate it. Otherwise, your model can just memorize parts of the data (or overfit in some other way), resulting in artificially high performance. Data leakage is when there is a problem in this separation and you somehow use information about the evaluation dataset in the model training process. The table in the article lists various examples. The simplest would be to just not have a separate evaluation set. A more subtle one is if you normalize your input data based on both the training and evaluation sets; this way the normalization will be better suited to the evaluation set than it should be if you had no knowledge of it, resulting in artificially high performance.
- YeGoblynQueenne 4y ago
  Great, now could you please email the authors and explain to them how to explain "lekage" in their draft paper, so the rest of us can read it also?
YeGoblynQueenne 4y ago
I'm reading the draft paper linked from the article and "data leakage" is used multiple times without any attempt at defining it. Oh well, I guess this is only meant for insiders who understand the in-group jargon. The rest of us need not be interested at all. Edit: for context, here is the paragraph titled "Leakage" from the draft paper: Leakage. Data leakage has long been recognized as a lead- ing cause of errors in ML applications (Nisbet et al., 2009). In formative work on leakage, Kaufman et al. (2012) provide an overview of different types of errors and give several rec- ommendations for mitigating these errors. Since this paper was published, the ML community has investigated leak- age in several engineering applications and modeling com- petitions (Fraser, 2016; Ghani et al., 2020; Becker, 2018; Brownlee, 2016; Collins-Thompson). However, leakage oc- curring in ML-based science has not been comprehensively investigated. As a result, mitigations for data leakage in scientific applications of ML remain understudied https://arxiv.org/pdf/2207.07048.pdf https://arxiv.org/pdf/2207.07048.pdf Yes, alright. But what the flying fuck is "leakage"? Am I supposed to go read the "formative work on leakage" cited? What if that work also leaves it undefined and points to an earlier source? What the hell is this paper about? Hhow hard it is to explain what your main subject is, so I can read your paper knowing what you're talking about? How frustrating.
- esalman 4y ago
  Looks like you only read the first page of the draft. There is a whole one and a half page section (2.4. Towards a solution: A taxonomy of data leakage) that describes every kind of leakage the authors considered for this work.
  - YeGoblynQueenne 4y ago
    Yes of cours I only read the beginning of the draft paper because it was very frustrating to read and I had no particularly strong reason to keep on reading it. This is what will happen to most people who read the draft paper. They will start with the abstract, feel confused, get annoyed and leave before they get to the "texonomy". In fact I did glance at the "taxonomy" as I was scanning through the paper dejectedly in the last few seconds I bothered with it and it still didn't seem to explain what the different kinds of data leakage are different kinds of. That's basically when I stopped reading. And this is why you don't write papers like that, because it makes them less likely to be read. And because, as an author in a field where people put tens of thousands of preprints on the internet, you want to make absolutely sure that your paper is as likely to be read as possible, by as many people as possible. Basically, if you want people to read your papers, the absolute worst thing you can do is to expect your readers to "keep reading to see what we mean". Most people will give up and reviewers will be so annoyed you're not respecting their time that they'll skim your paper looking for reasons to skewer it. The next worst thing is to expect your readers to try and guess what you mean. I could very easily make an educated guess at what "data leakage" means, but then I would have to read the paper while never being sure I know what exactly I'm reading and whether I have it subtly wrong. Again, you don't want to cause that kind of stress to your readers. You want everyone who reads your paper to feel happy and calm and admire your bright, squeaky clean ideas. So make things as clear as possible, as hassle-free as possible, as friction-free as possible. Otherwise, you're kicking yourself in the butt. That's my free advice.

Flashtoo 4y ago

When you evaluate an ML approach, you should use one part of the data to train your model and a completely separate part to evaluate it. Otherwise, your model can just memorize parts of the data (or overfit in some other way), resulting in artificially high performance. Data leakage is when there is a problem in this separation and you somehow use information about the evaluation dataset in the model training process. The table in the article lists various examples. The simplest would be to just not have a separate evaluation set. A more subtle one is if you normalize your input data based on both the training and evaluation sets; this way the normalization will be better suited to the evaluation set than it should be if you had no knowledge of it, resulting in artificially high performance.

YeGoblynQueenne 4y ago
Great, now could you please email the authors and explain to them how to explain "lekage" in their draft paper, so the rest of us can read it also?

YeGoblynQueenne 4y ago

I'm reading the draft paper linked from the article and "data leakage" is used multiple times without any attempt at defining it. Oh well, I guess this is only meant for insiders who understand the in-group jargon. The rest of us need not be interested at all. Edit: for context, here is the paragraph titled "Leakage" from the draft paper: Leakage. Data leakage has long been recognized as a lead- ing cause of errors in ML applications (Nisbet et al., 2009). In formative work on leakage, Kaufman et al. (2012) provide an overview of different types of errors and give several rec- ommendations for mitigating these errors. Since this paper was published, the ML community has investigated leak- age in several engineering applications and modeling com- petitions (Fraser, 2016; Ghani et al., 2020; Becker, 2018; Brownlee, 2016; Collins-Thompson). However, leakage oc- curring in ML-based science has not been comprehensively investigated. As a result, mitigations for data leakage in scientific applications of ML remain understudied https://arxiv.org/pdf/2207.07048.pdf https://arxiv.org/pdf/2207.07048.pdf Yes, alright. But what the flying fuck is "leakage"? Am I supposed to go read the "formative work on leakage" cited? What if that work also leaves it undefined and points to an earlier source? What the hell is this paper about? Hhow hard it is to explain what your main subject is, so I can read your paper knowing what you're talking about? How frustrating.

esalman 4y ago
Looks like you only read the first page of the draft. There is a whole one and a half page section (2.4. Towards a solution: A taxonomy of data leakage) that describes every kind of leakage the authors considered for this work.
- YeGoblynQueenne 4y ago
  Yes of cours I only read the beginning of the draft paper because it was very frustrating to read and I had no particularly strong reason to keep on reading it. This is what will happen to most people who read the draft paper. They will start with the abstract, feel confused, get annoyed and leave before they get to the "texonomy". In fact I did glance at the "taxonomy" as I was scanning through the paper dejectedly in the last few seconds I bothered with it and it still didn't seem to explain what the different kinds of data leakage are different kinds of. That's basically when I stopped reading. And this is why you don't write papers like that, because it makes them less likely to be read. And because, as an author in a field where people put tens of thousands of preprints on the internet, you want to make absolutely sure that your paper is as likely to be read as possible, by as many people as possible. Basically, if you want people to read your papers, the absolute worst thing you can do is to expect your readers to "keep reading to see what we mean". Most people will give up and reviewers will be so annoyed you're not respecting their time that they'll skim your paper looking for reasons to skewer it. The next worst thing is to expect your readers to try and guess what you mean. I could very easily make an educated guess at what "data leakage" means, but then I would have to read the paper while never being sure I know what exactly I'm reading and whether I have it subtly wrong. Again, you don't want to cause that kind of stress to your readers. You want everyone who reads your paper to feel happy and calm and admire your bright, squeaky clean ideas. So make things as clear as possible, as hassle-free as possible, as friction-free as possible. Otherwise, you're kicking yourself in the butt. That's my free advice.