3 ms·

I'm confused by this reaction here - this was a _brilliant_ hack of the system and I think his work should be celebrated. Was it in keeping with the intention

by luma 7y ago

I'm confused by this reaction here - this was a _brilliant_ hack of the system and I think his work should be celebrated. Was it in keeping with the intention of the competition? Of course not. Were lives threatened by their creative solution to the competition? Also no. At the end of the day it was a fun, inventive approach to a made-up problem. So what's the problem?

bleuarff 7y ago
That's not a hack, that's blatant cheating, their solution litterally looked at the anwsers. It bypassed the ML model prediction, so that's not a ML solution, which to my understanding was the constraint of the competition. And in the end, that solution is useless for the adoption site, since the objective is to get adoption predictions (the animal has not been adopted yet). I'm confused too, by how can anyone think it's an acceptable behavior.
- luma 7y ago
  Their solution made use of the data available to them in a "prize" that was nothing more than a made-up competition for fun. Did the rules expressly say somewhere that one could not make use of available datasets?
  - gpm 7y ago
    Kaggle rules routinely restrict use of datasets to approved ones, almost certainly yes but you would have to check to be certain. The prize was $10,000, not "fun".
  - zentiggr 7y ago
    I think you misunderstood that the team in question scraped prospective test profiles from the website in question outside of the competition data flow. They blatantly cheated the competition process using data that was not supposed to be available to them.
Townley 7y ago
For this particular problem, they were working on pet adoption timing prediction algorithms. The proposed solution wouldn’t work as efficiently in production as one of the other competitors. That leads to inaccurate predictions on pet adoption times, which increases costs for adoption centers, and maybe euthanasia rates among kill shelters. So animal lives would be impacted by this cheat. More importantly, Kagggle does competitions across dozens of industries. If a culture of getting better at hiding your cheat pervades the platform, that could impact finance, transportation, and medical research. In those scenarios, lives would either be threatened or at least subject to sub-optimal systems.
nomadluap 7y ago
The problem was that there was a prize at stake, and everyone else was playing by the rules.
- luma 7y ago
  Not trying to be argumentative here, but is what they did actually against the rules? I'm not at all familiar with this competition which might be why I'm not quite so worked up about this, so maybe I have misunderstood something. Did the competition really require that they only train on the provided data? I compare this to my favorite sport, Formula 1 racing. In F1, teams of engineers with nearly unlimited budgets spend an absurd amount of effort doing everything in their power to bend the regulations (the "formula") to squeeze out some extra advantage. For example, in this past season Ferrari was suddenly outperforming the pack (and their own recent performance) and it was clear something had changed on the car, they had power in places they didn't before. What finally came down is a clarification of the rules around fuel-rate metering, without directly calling out Ferrari. After the clarification, Ferrari power was back where it used to be. Nothing more was said of the matter by the FIA. What we all _think_ happened is that Ferrari, knowing the fuel rate meters ran at 10kHz, discovered they could pulse their fuel pump so that the low-end of the flow rate cycle happened during that sampling interval. This means they could increase their overall fuel rate beyond what was technically allowed, due to how that technical requirement was being measured on the car (and reported back to the FIA). Is it in keeping with the spirit of the rules? Of course not! Does it make for an interesting engineering puzzle on top of an already-exciting sport? Sure does! Clearly I'm in the minority here, but I think this sort of problem-solving approach can be useful. If you're looking to compete against a field of entrants who are all looking for obvious and well-understood approaches to solving the problem at hand, I think sometimes the best solution to stand out is to look where the other teams aren't looking.
  - bart_spoon 7y ago
    The issue is that the entire reason the competition exists is because the company is sponsoring it and putting forward the prize money so that the top performing models can then be put into production, thereby solving some problem the company has. This type of cheating is dishonest and against the spirit of the competition, but it also defeats the entire purpose of the exercise. Simply keeping a lookup table of answers for the data isn't machine learning, and will not generalize into a production system. As stated in the article, without these hacks, he wouldn't have even placed in the top 100. To use your F1 analogy, this isn't the equivalent of tweaking the cars in whatever way possible is within the rules. This is the equivalent of completely cutting across the grass and bypassing 90% of the track, which is indeed illegal and would get you penalized.
Can_Not 7y ago
That reminds me of another _brilliant_ hack I had my kid do at a youth checkers tournament. I got him to swat the other players pieces off of the board so the opponent would eventually run out of pieces before my child. I think the opposing kids parents were biased against me just because I'm an inventive genius.
throwawayjava 7y ago
> this was a _brilliant_ hack of the system Jesus fucking christ. He fucking ran MD5 on some shit he pulled down from a web crawler. Over-training a model on the validation set would be a lot more "brilliant", and even that is a dumb script kiddie level hack. Maybe finding an algorithm that computes weights s.t. the preimage of the training algorithm on the training set matches the result of training with truly random weights using the validation set. That could be a "clever hack". And even then _brilliant_ would be.... a real fucking stretch.

by luma 7y ago

bleuarff 7y ago

That's not a hack, that's blatant cheating, their solution litterally looked at the anwsers. It bypassed the ML model prediction, so that's not a ML solution, which to my understanding was the constraint of the competition. And in the end, that solution is useless for the adoption site, since the objective is to get adoption predictions (the animal has not been adopted yet). I'm confused too, by how can anyone think it's an acceptable behavior.

luma 7y ago
Their solution made use of the data available to them in a "prize" that was nothing more than a made-up competition for fun. Did the rules expressly say somewhere that one could not make use of available datasets?
- gpm 7y ago
  Kaggle rules routinely restrict use of datasets to approved ones, almost certainly yes but you would have to check to be certain. The prize was $10,000, not "fun".
- zentiggr 7y ago
  I think you misunderstood that the team in question scraped prospective test profiles from the website in question outside of the competition data flow. They blatantly cheated the competition process using data that was not supposed to be available to them.

Townley 7y ago

For this particular problem, they were working on pet adoption timing prediction algorithms. The proposed solution wouldn’t work as efficiently in production as one of the other competitors. That leads to inaccurate predictions on pet adoption times, which increases costs for adoption centers, and maybe euthanasia rates among kill shelters. So animal lives would be impacted by this cheat. More importantly, Kagggle does competitions across dozens of industries. If a culture of getting better at hiding your cheat pervades the platform, that could impact finance, transportation, and medical research. In those scenarios, lives would either be threatened or at least subject to sub-optimal systems.

nomadluap 7y ago

The problem was that there was a prize at stake, and everyone else was playing by the rules.

luma 7y ago
Not trying to be argumentative here, but is what they did actually against the rules? I'm not at all familiar with this competition which might be why I'm not quite so worked up about this, so maybe I have misunderstood something. Did the competition really require that they only train on the provided data? I compare this to my favorite sport, Formula 1 racing. In F1, teams of engineers with nearly unlimited budgets spend an absurd amount of effort doing everything in their power to bend the regulations (the "formula") to squeeze out some extra advantage. For example, in this past season Ferrari was suddenly outperforming the pack (and their own recent performance) and it was clear something had changed on the car, they had power in places they didn't before. What finally came down is a clarification of the rules around fuel-rate metering, without directly calling out Ferrari. After the clarification, Ferrari power was back where it used to be. Nothing more was said of the matter by the FIA. What we all _think_ happened is that Ferrari, knowing the fuel rate meters ran at 10kHz, discovered they could pulse their fuel pump so that the low-end of the flow rate cycle happened during that sampling interval. This means they could increase their overall fuel rate beyond what was technically allowed, due to how that technical requirement was being measured on the car (and reported back to the FIA). Is it in keeping with the spirit of the rules? Of course not! Does it make for an interesting engineering puzzle on top of an already-exciting sport? Sure does! Clearly I'm in the minority here, but I think this sort of problem-solving approach can be useful. If you're looking to compete against a field of entrants who are all looking for obvious and well-understood approaches to solving the problem at hand, I think sometimes the best solution to stand out is to look where the other teams aren't looking.
- bart_spoon 7y ago
  The issue is that the entire reason the competition exists is because the company is sponsoring it and putting forward the prize money so that the top performing models can then be put into production, thereby solving some problem the company has. This type of cheating is dishonest and against the spirit of the competition, but it also defeats the entire purpose of the exercise. Simply keeping a lookup table of answers for the data isn't machine learning, and will not generalize into a production system. As stated in the article, without these hacks, he wouldn't have even placed in the top 100. To use your F1 analogy, this isn't the equivalent of tweaking the cars in whatever way possible is within the rules. This is the equivalent of completely cutting across the grass and bypassing 90% of the track, which is indeed illegal and would get you penalized.

Can_Not 7y ago

That reminds me of another _brilliant_ hack I had my kid do at a youth checkers tournament. I got him to swat the other players pieces off of the board so the opponent would eventually run out of pieces before my child. I think the opposing kids parents were biased against me just because I'm an inventive genius.

throwawayjava 7y ago

> this was a _brilliant_ hack of the system Jesus fucking christ. He fucking ran MD5 on some shit he pulled down from a web crawler. Over-training a model on the validation set would be a lot more "brilliant", and even that is a dumb script kiddie level hack. Maybe finding an algorithm that computes weights s.t. the preimage of the training algorithm on the training set matches the result of training with truly random weights using the validation set. That could be a "clever hack". And even then _brilliant_ would be.... a real fucking stretch.