4 ms·

A large number of commenters fundamentally misunderstand what happened. They are saying "why did he upload his scraped training data? Why didn't he just train o

by dlkf 7y ago

A large number of commenters fundamentally misunderstand what happened. They are saying "why did he upload his scraped training data? Why didn't he just train on it and upload the resulting model?" If you are making this argument, it means that you don't understand the contest. User rahimnathwani explains: > In this competition, the training code was run on Kaggle's system, so you'd still need to smuggle in the extra data. The question then becomes, how do you smuggle in the data? This is a much more interesting discussion than pontificating about the ethics of Pleskov's actions. In particular, a better understanding of this problem could have ramifications for how Kaggle could combat hacks of this variety. (By contrast, "shame on him" and "aww but he's a nice guy" are both useless, except perhaps as a form of virtue signalling). It's essentially a cryptography problem. Does anyone know if this has been widely studied?

KaoruAoiShiho 7y ago
AFAIK it's impossible to smuggle it in a way that you can't be caught. Maybe the use of MD5 made it easier to see but if examined I don't think there's a totally hidden solution.
- AstralStorm 7y ago
  Yes, he could have trained a network to recognize them and return memorized values. It would not be nearly as obvious. Solving this would require changing the contest so that it comes with algorithm and instructions only (no data files, entropy checks) and is trained by contest operators.
  - dlkf 7y ago
    This would also be very fishy to anyone who inspects the source code. Immediately they would ask "uh where did this giant file of floats come from?" One approach would be to write some handcrafted rules/features that look likecthey were plausibly a priori, but have the effect of memorizing the scraped data. (I don't know if this is actually possible.) User KaoruAoiShiho is probably right that any approach along would look out of place. Coupled with the fact that its removal would massively boost accuracy, it's hard to imagine how this would get past a curious reviewer. Perhaps peer review should be a component of the Kaggle process.

by dlkf 7y ago

KaoruAoiShiho 7y ago

AFAIK it's impossible to smuggle it in a way that you can't be caught. Maybe the use of MD5 made it easier to see but if examined I don't think there's a totally hidden solution.

AstralStorm 7y ago
Yes, he could have trained a network to recognize them and return memorized values. It would not be nearly as obvious. Solving this would require changing the contest so that it comes with algorithm and instructions only (no data files, entropy checks) and is trained by contest operators.
- dlkf 7y ago
  This would also be very fishy to anyone who inspects the source code. Immediately they would ask "uh where did this giant file of floats come from?" One approach would be to write some handcrafted rules/features that look likecthey were plausibly a priori, but have the effect of memorizing the scraped data. (I don't know if this is actually possible.) User KaoruAoiShiho is probably right that any approach along would look out of place. Coupled with the fact that its removal would massively boost accuracy, it's hard to imagine how this would get past a curious reviewer. Perhaps peer review should be a component of the Kaggle process.