5 ms·
How is this overfitting, rather than a data quality / classification issue?
by stingraycharles 1y ago
How is this overfitting, rather than a data quality / classification issue?
- hsn915 1y agoُThe Arabic text is the translator's self credit "Translated by Nancy Qanfar"
- wongarsu 1y agoAnd the German is “subtitles of [public broadcaster] for [content network], 2017 I'm not sure this is really overfitting, the network does exactly what the training data demands. According to the training data silence art the end transcribes to a copyright notice or subtitle credits
- baobabKoodaa 1y ago> I'm not sure this is really overfitting, the network does exactly what the training data demands. What do you think overfitting is, if not that?
- wongarsu 1y agoOverfitting would be replicating overly specific details. Like if a specific pattern of silence (or quiet noise) matched to specific copyright notices. But in this case the behavior seems to generalize over multiple languages, with the model choosing representative "outro silence" captions depending on the language. Which is consistent with the training data showing that outro silence is captioned. If the model was generalizing perfectly it would show something like "[subtitle credits here]" but that'd be demanding a bit much. Transcribing outro silence as silence despite the training data consistently transcribing outro silence differently from regular silence would be underfitting
- maxbond 1y agoThe optimizer is functioning correctly, and the pattern really exists in the training data. But consider: - This behavior damages the model's performance on out of sample data; every word you predict during silence increases the transcript's Word Error Rate. - These translation credits are an artifact of our training data, and not a reflection of the process we are modeling (spoken language). So, while you are correct about the mechanism at work here, it is still correct to call learning a spurious pattern which damages our performance "overfitting".
- bmacho 1y agoOverfitting is achieving better and better scores on the training material and worse and worse scores on unseen tasks. More at: https://en.wikipedia.org/wiki/Overfitting#Machine_learning https://en.wikipedia.org/wiki/Overfitting#Machine_learning This is just wrong training data.
- samrus 1y agofitting on noise in the training data is exactly what overfitting is. underfitting is smoothing out signal
- wongarsu 1y agoExactly. Underfitting would be if the model doesn't pick up on the fact that outro silence is labeled differently from regular silence and transcribes them the same
- xigoi 1y agoOverfitting implies a failure to properly generalize the training data. Here it generalized them correctly. Garbage in, garbage out.
- samrus 1y agoNo. Because there would have been indtances in the data where silence was labelled correctly. But the model couldnt handle the null case, so it over fit on the outros. But generally it fit on the random error in the label of the null feature. Which is what overfitting is
- andrepd 1y agoThat's literally what overfitting means. Side-note: it's also yet more evidence that AI companies hoover all data with no regard for legality or copyright status, the very same offences that got other people in jail or with heavy fines.
- efitz 1y agoI know it’s off topic, but it reminded me that translators like to put in Easter eggs, or at least they used to: https://learn.microsoft.com/en-us/archive/blogs/ericfitz/i-always-wondered-who-bjrn-was https://learn.microsoft.com/en-us/archive/blogs/ericfitz/i-a...
- maxbond 1y agoIt is a data quality issue which caused the model to overfit.
- mort96 1y agoIsn't overfitting just when the model picks up on an unintended pattern in the training data? Isn't that precisely what this is?
- deleted 1y ago[deleted]
- RamblingCTO 1y agonot necessarily, no. if you have 60% of examples for silence being the hallucination, it just learns the (what you detect as) wrong connection.
- mort96 1y agoWhich ... would be overfitting. It picks up on a pattern in the training data that we don't want it to pick up on and which causes it to generalize poorly.
- RamblingCTO 1y agoHow is it overfitting if the data is garbage in the first place? Saying it's overfitting in this context has no meaning as there is no alternative that maximizes the utility function we're training for?
- bGl2YW5j 1y agoIf the model was able to generalise, you’d expect it to output something like “[silence]” or “…”, in response to silence. Instead, it reverted to what it has seen before (in the training data), hence the overfit.
- deleted 1y ago[deleted]
- stingraycharles 1y agoRight, maybe my definition of overfitting was wrong, I always understood it more as trying to optimize for a specific benchmark / use case, and then it starts failing in other areas. But the way you phrase it, it’s just “the model is not properly able to generalize”, ie it doesn’t understand the concept of silence also makes sense. But couldn’t you then argue that any type of mistake / unknown could be explained as “overfitting” ? Where do you draw the line ?
- heavyset_go 1y agoYour definition is one, but the one the OP is using is overfitting to training data.
- stingraycharles 1y agoThat’s exactly my point: by that definition any incorrect answer can be explained by “overfitting to training data”. Where do you draw the line between “overfitting to training data” and “incorrect data” ?
- tempaccount420 1y ago> That’s exactly my point: by that definition any incorrect answer can be explained by “overfitting to training data”. Not really, getting 94381294*123=... wrong, but close within the actual answer, cannot be overfitting since it wasn't in the training data.
- 1y ago