5 ms·

They train winning code from scratch, I think, so you wouldn't have been able to just train the model with the "unseen" data.

by MiroF 7y ago

harry8 7y ago
They scraped data from the website of the org that wanted the results and funded the competition. That data was supposed to be unseen for the competitors and used to grade the models. This cheat was to use that scraped data in its training set and, beyond that, hard code some predictions. It's looking up the answers in the grading sheet while taking an exam.
- iudqnolq 7y ago
  Your parent is saying kaggle takes the training code and runs it themselves. This makss secretly training on the known right answers impossible.
  - harry8 7y ago
    There is no such thing as "training code". You have a model that you have trained on some provided data - the training set. You give kaggle this model. Kaggle grades your model on some different data your model has never seen. The better your model classifies this data it hasn't seen the higher it scores and the more money you win. So again if you trained your model (code) on a training set that you have illegally obtained. That is broken the rules of the competition to get the additional data. Data that is also going to be used for competition verification and grading, then you have cheated. Doubly so if you just hardcode outputs to boost your score, which they did here. They took the official training set. Said, we need more. And scraped websites to get a bigger, illegal training set. This is against the rules and is cheating. They got caught. It really is equivalent to looking up the answer sheet while taking an exam.
    - Hercuros 7y ago
      People are saying that you do not just submit your trained model to Kaggle. You also submit the code that was used to train the model from the training set, which is used in the winning models to train them from scratch on the training set. Of course, that wouldn't have prevented this type of cheating of course, but it does mean that you can't submit a model that was trained on your own private data set.
      - harry8 7y ago
        You can overfit a model to your hold out set quite easily with repeated trials. It's a trap you have to avoid in normal circumstances! (Feynman: "You are the easiest person to fool"). Even if you have to submit code to generate your model parameters from "the training set" (which hasn't been explained at all well by "People" if that is indeed the case) you could do that overfitting deliberately here with the illegal unseen data as your hold out set. Aside from the advantage of a bigger training set. Aside from the advantage in model selection, which is not done with code from a training set. Aside from the advantage to your feature engineering also not done in code. Aside from the advantage to your regularization choices, bias parameters etc etc. So yes you absolutely /can/ submit a model trained on your own private data set even if what you submit is a model code that will be re-trained. Even if "the training set" is different to the provided - you still have that scraped data so you can slice it up with the provided training set so that any selected training set does well against the rest. Now the overfit you've just carefully engineered should win against the honest models unless you suck, right? It's kind of risible that they had to go further and hard code certain results, don't you think? Perhaps if they still couldn't win with scraped, illegal additional data then everyone else had illegal data too? Perhaps Kaggle is not a good indicator of how good ML techniques are in practise? Perhaps Kaggle systematically overstates ML effectiveness due to this kind of uncaught cheating in many of their competitions? I bet kaggle won't look too hard at that.
    - iudqnolq 7y ago
      > There is no such thing as "training code". I'm by no means an expert in ML, but my understanding is there's some code that is run to train the modal. I meant that by "training code". My regrets if my terminology was unclear. > They took the official training set. Said, we need more. And scraped websites to get a bigger, illegal training set. This is against the rules and is cheating. They got caught. No, this is wrong I suggest you look at one of the other comment where people have explained why this is wrong. They did a better job than me. https://news.ycombinator.com/item?id=22124760 https://news.ycombinator.com/item?id=22124760 https://news.ycombinator.com/item?id=22126489 https://news.ycombinator.com/item?id=22126489 https://news.ycombinator.com/item?id=22124193 https://news.ycombinator.com/item?id=22124193
      - harry8 7y ago
        No it isn't. See above. Overfittiing to a holdout set is so easy and common you usually need to take steps to avoid it. See my comment directly above. Best.
        iudqnolq 7y ago
        Thank you. I found your comment extremely helpful.
  - aratauto 7y ago
    This competition allowed submissions to include extra data files that can be used by the model. The cheaters added a file with data from another website that seemed innocent, but secretly encoded extra information (perfect answers) in IDs. For 10% of predictions, the code via a set of obfuscated operation retrieved this information and presented it as the answer.

They train winning code from scratch, I think, so you wouldn't have been able to just train the model with the "unseen" data.

by MiroF 7y ago

They train winning code from scratch, I think, so you wouldn't have been able to just train the model with the "unseen" data.

harry8 7y ago

They scraped data from the website of the org that wanted the results and funded the competition. That data was supposed to be unseen for the competitors and used to grade the models. This cheat was to use that scraped data in its training set and, beyond that, hard code some predictions. It's looking up the answers in the grading sheet while taking an exam.

iudqnolq 7y ago
Your parent is saying kaggle takes the training code and runs it themselves. This makss secretly training on the known right answers impossible.
- harry8 7y ago
  There is no such thing as "training code". You have a model that you have trained on some provided data - the training set. You give kaggle this model. Kaggle grades your model on some different data your model has never seen. The better your model classifies this data it hasn't seen the higher it scores and the more money you win. So again if you trained your model (code) on a training set that you have illegally obtained. That is broken the rules of the competition to get the additional data. Data that is also going to be used for competition verification and grading, then you have cheated. Doubly so if you just hardcode outputs to boost your score, which they did here. They took the official training set. Said, we need more. And scraped websites to get a bigger, illegal training set. This is against the rules and is cheating. They got caught. It really is equivalent to looking up the answer sheet while taking an exam.
  - Hercuros 7y ago
    People are saying that you do not just submit your trained model to Kaggle. You also submit the code that was used to train the model from the training set, which is used in the winning models to train them from scratch on the training set. Of course, that wouldn't have prevented this type of cheating of course, but it does mean that you can't submit a model that was trained on your own private data set.
    - harry8 7y ago
      You can overfit a model to your hold out set quite easily with repeated trials. It's a trap you have to avoid in normal circumstances! (Feynman: "You are the easiest person to fool"). Even if you have to submit code to generate your model parameters from "the training set" (which hasn't been explained at all well by "People" if that is indeed the case) you could do that overfitting deliberately here with the illegal unseen data as your hold out set. Aside from the advantage of a bigger training set. Aside from the advantage in model selection, which is not done with code from a training set. Aside from the advantage to your feature engineering also not done in code. Aside from the advantage to your regularization choices, bias parameters etc etc. So yes you absolutely /can/ submit a model trained on your own private data set even if what you submit is a model code that will be re-trained. Even if "the training set" is different to the provided - you still have that scraped data so you can slice it up with the provided training set so that any selected training set does well against the rest. Now the overfit you've just carefully engineered should win against the honest models unless you suck, right? It's kind of risible that they had to go further and hard code certain results, don't you think? Perhaps if they still couldn't win with scraped, illegal additional data then everyone else had illegal data too? Perhaps Kaggle is not a good indicator of how good ML techniques are in practise? Perhaps Kaggle systematically overstates ML effectiveness due to this kind of uncaught cheating in many of their competitions? I bet kaggle won't look too hard at that.
  - iudqnolq 7y ago
    > There is no such thing as "training code". I'm by no means an expert in ML, but my understanding is there's some code that is run to train the modal. I meant that by "training code". My regrets if my terminology was unclear. > They took the official training set. Said, we need more. And scraped websites to get a bigger, illegal training set. This is against the rules and is cheating. They got caught. No, this is wrong I suggest you look at one of the other comment where people have explained why this is wrong. They did a better job than me. https://news.ycombinator.com/item?id=22124760 https://news.ycombinator.com/item?id=22124760 https://news.ycombinator.com/item?id=22126489 https://news.ycombinator.com/item?id=22126489 https://news.ycombinator.com/item?id=22124193 https://news.ycombinator.com/item?id=22124193
    - harry8 7y ago
      No it isn't. See above. Overfittiing to a holdout set is so easy and common you usually need to take steps to avoid it. See my comment directly above. Best.
      - iudqnolq 7y ago
        Thank you. I found your comment extremely helpful.
- aratauto 7y ago
  This competition allowed submissions to include extra data files that can be used by the model. The cheaters added a file with data from another website that seemed innocent, but secretly encoded extra information (perfect answers) in IDs. For 10% of predictions, the code via a set of obfuscated operation retrieved this information and presented it as the answer.