4 ms·
I'm not going to defend Pleskov but organizers shouldn't have put out the competition with money attached that can simply be solved by scraping data. Good ML co
by sytelus 7y ago
I'm not going to defend Pleskov but organizers shouldn't have put out the competition with money attached that can simply be solved by scraping data. Good ML competition in fact should even invite cheats because the end goal is not ML for the sake of ML but rather cracking the prediction problem by whatever shortest path possible.
- rasz 7y agoNo doubt he is brilliant. Many investment banks will be happy to have him. He will fit right in something like Merrill Lynch or Goldman Sachs.
- zkid18 7y agoHe made a good career in quant finance before diving into ML and Kaggle
- Traster 7y agoI wouldn't necessarily describe him as having a 'good' career in quant finance. He spent 18 months as a quant and then left to go and cheat at ML competitions. It's not a traditional sign of a successful quant - giving up after 18 months.
- zkid18 7y agoAfair, he worked for about 3 years in two HFT funds. Initially he was hired as a quant researcher but lately moved to business role.
- trhway 7y ago>No doubt he is brilliant How? According to the article his model without the cheat rated at ~100th place, and the article mentions him cheating the same way before (by scraping Quora for some Quora related competition).
- mannykannot 7y ago"Winning" by anything that can reasonably called cheating, as in this case, does not advance the general state of the art. Innovation is best served through appropriate rules and competition structure.
- sytelus 7y agoYes, and that's the right thing to do in the academic research setting ("advance the state of the art"). But the public competitions with monetary rewards are not the same setting. I can imagine scenarios where the guy stole the test set from Kaggle servers (i.e. unlawful access) should disqualify him permanently. But the essence of the competition should be the focus on cracking a given problem, not about a specific technique. One test a good of ML competition: Can it be solved by simply hiring lots of humans to make predictions without incurring significantly more costs than the prize money?
- deleted 7y ago[deleted]
- joe_the_user 7y agoOne test a good of ML competition: Can it be solved by simply hiring lots of humans to make predictions without incurring significantly more costs than the prize money? What value to the organizers, to society or to whatever are you imagining coming out of a free-for-all style competition? I think the organizers now imagine that the result would identifying good, generic prediction algorithms along with identifying good AI programmers capable of producing general prediction algorithms. It seems like the contest framework already has become a bit problematic through context winners just being good at contests and not otherwise achieving anything. But what are you thinking of? There are already hacking competitions btw.
- sytelus 7y agoAgain, I'm not defending Pleskov. If he had come forth with the hack, things would have been different. Instead, he pretended that he had ML solution, pocket the money and put an extraordinary effort into making sure that people can't actually figure out his true doings. He was disingenuous, fully self-aware that he was in the wrong and did his best to cover up his tracks. It wasn't fair to other competitor and it was most certainly not fair to the organization trying to do something good. So yes, Pleskov, remains indefensible.
- Gatsky 7y agoI would have a different interpretation. If he cheated once, he has most likely done it before and since the competition in question. Usually when 'good people' cheat there is a series of escalating transgressions before they get caught.
- halflings 7y agoHe scraped the test set's labels. How is that useful? It's not about "ML for the sake of ML", it's the equivalent of stealing the answers to a math test then writing them down. Why should that be rewarded? The goal of this competition is to build a system (using ML or not) which is useful for predicting how quickly pets will be adopted. Any information used during the competition should be realistically available at inference time for future predictions... clearly, the expected answer cannot be available at the time you're trying to predict when a pet will be adopted.
- nl 7y agoPeople do scrape test answers and use them to optimise hyperparameters in ML models and blends of them in Kaggle. That's cheating, but at least they built a learning system here.
- nl 7y agoHe scraped the test set, then encoded the correct answer in the id. If one can excuse scraping the data to build a better model, I can't see how one can excuse this. He wasn't really making predictions at all, just looking up the answer. And the goal really is a prediction system, not something that looks up previously known answers.