15 ms·
A Kaggle Grandmaster cheated in $25k AI contest with hidden code
- mmhsieh 7y agoIs there a public list of all known Kaggle cheats?
- quickthrower2 7y agoCan you write a ML program to predict if someone is cheating based on their submission?
- nourse 7y agoYes. Well I call it ML, but in reality it just checks HN for articles about their username cheating.
- mmhsieh 7y agoIf only some organization has a large dataset of ML algorithms and identified cheaters...
- bitL 7y agoBetter, make it based on your e-mail: "Sorry, you are likely a cheater, we logged you out and banned forever."
- bitxbit 7y agoNever understood Kaggle. The Netflix Prize was great. Now it’s just people gaming Kaggle to get a job.
- omarhaneef 7y agoFeels like you just answered your own question. (Which happens a lot on HN)
- raverbashing 7y agoI don't see a problem with that. It seems to me most companies sponsoring the contests have legitimate problems they wanted solved. Now it really illustrated to me how data science is still an incipient field. Even the basic Titanic example has a lot of fake solution and a bit of "cheating" (overfitting). And don't get me started on the multitude of tutorials and examples that only show the loss decreasing graph but the actual results are not evaluated (and thus might have some glaring defects).
- blurps 7y agoThe Netflix Prize winners regularly compete on Kaggle. One of the top 10 Netflix Prize winners joined Kaggle early on (he got a job out of it). Kaggle competitions are a game. Most notorious players already have a good job.
- bitxbit 7y agoI think the original intent was great. And you can learn from it still. However, as the job market for data scientists heated up, Kaggle became something else entirely.
- deleted 7y ago[deleted]
- endorphone 7y agoThe Netflix Prize was interesting, and drew a lot of attention to it, but ultimately wasn't that simply stuffed into the trash bin? Not long after that, Netflix basically abandoned both user ratings of significance, and realistic recommendations. Now it's just a nonsense engine with some sort of meaningless overlap or whatever they call it.
- thrwaway69 7y agoI guess they realised giving user swhat they want isn't necessarily profitable for business.
- blurps 7y ago> into the trash bin? They did not implement the winning solution as is, but a lot of good stuff came from the winning teams, including techniques (such as SVD) that are in use as of today (or maybe a few years back). > just a nonsense engine No, it is safe to assume the recommendation engine of Netflix is close to the state-of-the-art. A lot of money and talent went into it.
- endorphone 7y ago"They did not implement the winning solution as is" Shortly after that contest finished Netflix removed the five star rating system, and dramatically subdued the recommendation engine. Now the vast majority of the content surfacing is universal beyond some small category filtering (e.g. You like crime dramas and horrors so here's a bunch of the most popular stuff from those categories). Now they have a "match" rating that I have not met a single person who finds useful (it is almost at the point of farce and seems more like a randomization engine). A lot of money and talent went into something that Netflix clearly decided just wasn't important or useful enough for them. Now they just push Don't F*ck With Cats on everyone.
- Lev1a 7y agoOne of the Kaggle competitions was used as an assignment for the Statistics course at my university. IIRC it was really fiddly.
- bitL 7y agoIt's pretty common these days, many courses at top universities feature a Kaggle competition/datasets.
- bitL 7y ago> Now it’s just people gaming Kaggle to get a job. How is that different from "LeetCode PhD" doing "DP Hard" for 6 months to get into FAANG?
- floatingatoll 7y ago> "For me, it was never about the money but rather about the Kaggle points" Accumulated integer counts that represent social value and/or standing encourage destructive behavior.
- nefitty 7y agoIt’s so fascinating to see how easily a human brain can be hijacked by status. I wonder if there’s a way to harness that power for improving individual performance...
- deleted 7y ago[deleted]
- gdsdfe 7y agoThere is and it's very effective ... A company I worked in briefly used to have this badge system to honor people that go the extra mile at work, and people were very keen on getting those even though it didn't have any financial compensation behind it
- marta_morena 7y agoUm that is kindof missing the point. A badge is nothing more than a number expressed in symbols, so you demonstrated just another variation of how to hijack the brain, even if you just read an article showing it to you with numbers... If "improving individual performance" means to you that you perform better at your job, you are already lost.
- gdsdfe 7y agoI'm sorry but if performing better at your job doesn't improve your individual performance, you need another job!
- mmhsieh 7y agoNapoleon Bonaparte: “Give me enough medals and I'll win you any war”
- sandGorgon 7y agoI remember reading about this on Twitter because of a reply from h2o.ai account. https://twitter.com/ppleskov/status/1215983188876709888?s=19 https://twitter.com/ppleskov/status/1215983188876709888?s=19 The person was originally employed at h2o.ai and as a consequence of this was fired. Not sure if that was completely appropriate. Wasn't this a personal participation? Or are there "company teams" on Kaggle ?
- daenz 7y ago>Not sure if that was completely appropriate. In my opinion, keeping him onboard compromises h2o.ai's image as a trustworthy AI company. To many people, AI is magic, so the last thing you want is the impression that the magic is really a con.
- throwawayjava 7y ago> Not sure if that was completely appropriate. He has only been at h20 for a few months. He was almost certainly hired at least in part because of his participation in the contest.
- ramraj07 7y agoThe article says people mention kaggle grandmaster in their CVs and H2o uses this to hire them as well. This is essentially lying on the resume then.
- deleted 7y ago[deleted]
- starpilot 7y agoAt will employment.
- C1sc0cat 7y ago"bringing the company into disrepute" is gross misconduct and normally a firing offence, even in countries with more liberal employment laws than the USA
- 7y ago
- eanzenberg 7y agoI never understood Kaggle. Most competitions don't require code to be submitted, just predictions to be made on a test set with missing labels. So, you don't even need to apply machine learning and I'd bet money that lots of winners don't and label by hand or outsource. I don't understand the fascination and appeal of these so-called "grandmasters".
- throwawayjava 7y agoWithout very careful contest design, the best performers are obviously going to be over-fitting. Especially if the entire distribution is public. That's exactly what this team did. This is true of academic contests in general, btw, even without cheating. They stop being interesting/fun/good signals as soon as people start treating them as an independent skill set. Comparing performance on chess games for the first N games between two new players might be a good signal for some general intellectual capabilities. Comparing experienced players against one another is mostly just testing who's spent more time learning about chess.
- blurps 7y agoWinners have to submit code most of the times. There are sporadic computer vision competitions where it is possible to hand-label (such as geo-int competitions where people reverse engineered the flight paths), but vast majority it is impossible to hand-label. Nobody becomes a grandmaster without applying world-class machine learning skills. The "cheater" in this competition is both a world-class data scientist and a reverse engineering hacker. Heck, people used to write papers about how they crawled the ground truth. Now they see their name in the newspaper.
- MasterScrat 7y ago> I never understood Kaggle. Most competitions don't require code to be submitted Your point of view is outdated ;-) In recent ML competitions, participants do submit code that is run on a held-out dataset - as was the case in the PetFinder.my challenge in question here. Most competition platforms are migrating to this format, as otherwise you can just label by hand as you said. Note that this competition went even further: not only was the evaluation code run on Kaggle, the training code was also run there. This means that you couldn't even train a gigantic model then submit it: your model had to be trainable within well defined time and resource constraints, which is a great way to level the playing field. Of course, there's still some unfairness as people with more resources can try out more solutions before submitting a model to be trained on the platform. No platform has a solution for this yet!
- paulgb 7y agoFor an HN audience, the "How Bestspotting cheated" post on Kaggle may be a better place to start: https://www.kaggle.com/bminixhofer/how-bestpetting-cheated https://www.kaggle.com/bminixhofer/how-bestpetting-cheated
- 7777fps 7y agoIt's an odd way to cheat too, if they realised they had data from the validation set, couldn't they have over-trained a model with the validation set in the training data?
- RandomWorker 7y agoAgreed, this actually happened in the buildings ASREA comp. the data was found online.
- paulgb 7y agoYes, that would have been harder to detect if it is allowed. My understanding is that for a kernel competition (like this one), you can't use model weights that you've trained outside the kernel. Oddly, I can't find a rule explicitly prohibiting it.
- koheripbal 7y agoIs it me, or has the HN user-sphere changes significantly this year. I feel as though comments are less informative, more judgemental, and chalk full of emotions. ...as if a younger crowd of users has started using it. Have the older-wiser HNers started using another site?
- deleted 7y ago[deleted]
- quickthrower2 7y agoWouldn’t it be smarter to use the hidden data as training data rather than hard code it in?
- blurps 7y agoThis only works when you don't win. You have to upload your code, including model training code, they won't accept a trained model binary.
- MiroF 7y agoIf they were smart, they would have used the whole set for hyperparameter tuning. That would be essentially undetectable.
- jeffshek 7y agoNot if the underlying model was bad, no tweaking of hyperparameters can change that. It's safe to say, he probably did consider this (and it probably didn't work well enough).
- shkkmo 7y agoAm I missing something? It seems like "adoption speed" really isn't something you want to over optimize. The whole point is to match abandoned animals with suitable homes. Adoption speed seems secondary to post adoption measures of adopter satisfaction and adoptee welfare.
- singron 7y agoThey probably don't have a way to measure that reliably, and adoption speed might make more sense to their bottom line.
- xiphias2 7y agoIt looks like the shelter has limited capacity, and too many dogs to be taken care of, therefore the faster dogs are adopted, the less need to be euthanized. Also what you are writing are extremely sparse and weak signals compared to adoption speed.
- TrackerFF 7y agoIt's a classic optimization problem. With high speed, but poor suitability, animals are bound to return. With low speed, but good suitability, there's low return rate - but slow turnover, which leads to overcrowding and rejection of new animals. Like in business and tech, you want a high as posible throughput / inventory turnover rate - but obviously with constraints.
- jeffshek 7y agoAdoption speed is probably the only proxy that could be used. Measuring quality of pet life in suitable homes is rarely collected. I'm not even certain of how that could be objectively measured for children, let alone pets.
- Thorentis 7y ago> The goal was to create an algorithm that could predict how quickly a pet would be adopted based on its profile details, from its photo to its breed, sex, size, age, and whether it had been vaccinated or not. > These predictions would be used to optimize and tweak future critters' profiles so that they are adopted as soon as possible. Sorry but, how is this useful? You can't just change the age of an animal to make it more likely to be adopted. The profile is meant to be an accurate representation of the animal so people know what they're getting. What exactly was the algorithm meant to achieve aside from being a predictor?
- cjauvin 7y agoExactly my thought when I read this also, and I think it generalizes to a lot of cases where the value of trying to predict something is dubious at best, when there's nothing obvious you can do with the prediction itself.
- Thorentis 7y agoSadly, many ML comps involve exactly this. Making a predictor for something, where the prediction has no real world value, and even dubious value at a thought experiment level. I think one of the biggest skills to have in the ML space, is knowing what is worth training a model to know, and what isn't. Just like in engineering, the most successful products are those that solve a real world problem, no matter how elegantly the others might have been made.
- mxcrossb 7y agoWell optimizing the photo seems a sensible task for AI.
- singron 7y agoYou can use this to select which pets to put on a platform. For instance, no-kill shelters have to decide which animals they intake since they have finite room. They can save more animals if they pick animals that are likely to be adopted quickly. Obviously, kill shelters have a similar calculus when deciding which animals to cull (and indeed, animals that don't fit in the no-kill shelter go to the kill shelter). I'm not sure how this website manages "inventory", but they might have similar problems.
- harry8 7y agotl;dr Competive ML model grading with a common training set using unseen data. Cheat was to scrape the data that would then be used as unseen by the organisers. Unseen is now seen for this model. Then, instead of training the model with the "unseen" data, which would have been cheating and an advantage it apparently wasn't enough of an advantage so they hard code 10% of the cases to boost metrics and win. Having more data to train your model is google & facebricks competitive advantage. Their attempts to use that advantage for something actually useful to society rather than just as a method of selling ads seems to have been a complete bust so far. If that is wrong you know better, please link us up. I'm suspicious that their predictive power to sell ads actually works for the people who buy those ads but I guess we aren't likely to know for sure. I do wonder "who dominated their industry segment in sales by being an early adopter of google ads" I don't know anyone. It's not a great metric but what else do we have?
- MiroF 7y agoThey train winning code from scratch, I think, so you wouldn't have been able to just train the model with the "unseen" data.
- harry8 7y agoThey scraped data from the website of the org that wanted the results and funded the competition. That data was supposed to be unseen for the competitors and used to grade the models. This cheat was to use that scraped data in its training set and, beyond that, hard code some predictions. It's looking up the answers in the grading sheet while taking an exam.
- jungletime 7y agoThis contest was probably BS anyway. Google wanted to solve an analogous problem, not pet adoption. As such, both parties cheated.
- sytelus 7y agoI'm not going to defend Pleskov but organizers shouldn't have put out the competition with money attached that can simply be solved by scraping data. Good ML competition in fact should even invite cheats because the end goal is not ML for the sake of ML but rather cracking the prediction problem by whatever shortest path possible.
- rasz 7y agoNo doubt he is brilliant. Many investment banks will be happy to have him. He will fit right in something like Merrill Lynch or Goldman Sachs.
- zkid18 7y agoHe made a good career in quant finance before diving into ML and Kaggle
- Traster 7y agoI wouldn't necessarily describe him as having a 'good' career in quant finance. He spent 18 months as a quant and then left to go and cheat at ML competitions. It's not a traditional sign of a successful quant - giving up after 18 months.
- zkid18 7y agoAfair, he worked for about 3 years in two HFT funds. Initially he was hired as a quant researcher but lately moved to business role.
- trhway 7y ago>No doubt he is brilliant How? According to the article his model without the cheat rated at ~100th place, and the article mentions him cheating the same way before (by scraping Quora for some Quora related competition).
- mannykannot 7y ago"Winning" by anything that can reasonably called cheating, as in this case, does not advance the general state of the art. Innovation is best served through appropriate rules and competition structure.
- gojomo 7y agoBefore reading they'd scraped public data that was likely to be the "hidden" evaluation set, I thought they might have cheated using Python introspection: inspect the caller's frame, find some variable already loaded with the expected answer, return that. Has anyone cheated at Kaggle/similar using that approach?
- anoncareer0212 7y agoNo, and this pattern of thought is bizarre, lacks technical grounding, and more importantly, scruples
- vladislav 7y agoThey would have able to win and get away with it if they incorporated the knowledge of the external dataset directly into the ML model, provided they had a reasonable estimate on the fraction of overlap between the external data and the test set. A weak version of this would be to just train on the external data in addition to the provided data. A stronger version would train regularly on the provided training data and in addition overfit on a random subset of some percentage of the external data (with some small random prediction error thrown in to obfuscate), which would get equivalent results to what they did with logic.
- m3kw9 7y agoPart of the reason h2o.ai fired him? He was a cheat, ok, but you also cheated so stupidly
- rahimnathwani 7y ago"A weak version of this would be to just train on the external data in addition to the provided data." In this competition, the training code was run on Kaggle's system, so you'd still need to smuggle in the extra data.
- alanfranz 7y agoThis. You've got the testing set. Create random HPs and tune them to fit. The way they cheated is stupid. And the way the testing set can be obtained is silly.
- t0mk 7y agoThis is a really good point! Considering the guy was smart (he is kaggle grandmaster), I would really like to know what prevented him from training on the scraped data, and what motivated him to obfuscate the known sample lookup. Maybe there's some technicality they made it impossible to tune the model on the additional scraped training data.
- mellosouls 7y agoI'm surprised by some of the overly sympathetic comments here, the guy cheated, not "cheated" etc. Of course, we're all human, and he's come clean, but his actions potentially had a negative effect on the non-profit and the animals it places; and competing talents were denied their rightful places. This comment isn't about condemning him or anything, just let's be honest about what happened here; it wasn't ok, or just system-gaming caught out.
- archi42 7y ago> competing talents were denied their rightful places Absolutely. What are the odds he (and/or others in his team?) did this just for this one competition? It's possible, but unlikely he/they invented this (or other) code hiding technique(s) just for this single occasion. Also, the employer did the right thing and kick him out. Now they only have to scrutinize the last few months of his work instead of looking over his shoulders the next few years.
- honestoHeminway 7y agoWhich is why he is now in the blackbook.db. Professional social death. And silence. All is forgiven.
- m3kw9 7y agoAlso there may not be much correlation with profiles and speed of adoption. Why don’t they prove that first then have the competition instead of assuming it would be the solution?
- fizx 7y agoQuick, give him a growth-hacking gig!
- C1sc0cat 7y agoBlack hat seo will get your site tanked no sane seo agency / client would take some one like that on.
- ribs 7y agoIt really worries me how many people are so quick to forgive him and tell him so. In my family if someone cheated they got called a cheater and suffered consequences. At least, they would have, if someone did something like that. But my parents didn’t raise mendacious villains. Look at this crap on Twitter: “Everyone makes mistakes. Thank you for the apology”. “Kagglers will still love to have you back” “It's great that you realize your mistakes. Looking forward to see your comeback with more cool DS solutions and ethics than before.” “Thanks for doing this. It's okay to make errors in judgement, we've all been there to varying degrees. Y'all be gonna be fine.1!” Those are the worst. I’m not so crazy about these below, either, although there’s just a hint of steel in them, at least: “I’m glad to see that you had a change of heart after sleeping on it and that you will be returning the prize money. I hope you will consider donating to or volunteering at a local animal shelter as well. Atonement here is more than returning the money and apologizing.” “I hope this can be used as a teaching moment as well. Many people clearly look up to you because of your work. What can we learn from this? Something to ponder in the days to come.”
- ribs 7y agoI’m not saying that Pleskov is a sociopath, but I’m beginning to see how sociopaths manage to make their way in the world without getting punished.
- raverbashing 7y agoYou're not too far off, it usually starts with the family attitude.
- mosselman 7y agoAny sources for this? You can't just claim things like this. At least now without getting some reaction like mine here.
- bitL 7y agoImagine you have 100 sociopaths, all of them aiming for the top, all able to recognize each other in a crowd. Now you both compete and cooperate together to get to the top; the ones caught are written off, the uncaught ones sooner or later achieve the goal, even if only 10 out of 100 make it there. Some headhunters specifically try to identify high-performing sociopaths for top management positions.
- puneetchawla 7y agoHilarious! I have picked my favorite text from the article. https://youtu.be/r6H14MIJRk4 https://youtu.be/r6H14MIJRk4
- niceworkbuddy 7y agoMy two nuggets. This reminded me of Ijon Tichy's saying after one of his voyages: " Thus concluded one of the most unusual of my adventures and voyages. Notwithstanding all the hardship and pain it had occasioned me, I was glad of the outcome, since it restored my faith, shaken by corrupt cosmic officeholders, in the natural decency of electronic brains. Yes, it’s comforting to know, when you think about it, that only man can be a bastard. " (source: The Star Diaries, Stanisław Lem)
- reedwolf 7y agoAlways happy to see a Stanislaw Lem reference.
- juskrey 7y agoKaggle has nothing to do with real world. It is a system which invites everyone to cheat it and to be cheated. Similar with school: how many times you, excellent grades holder, was cheating because no one scrutinizes high grade holders too much?
- nootropicat 7y agoHow is this not criminal fraud for $10k? He deserves to go to prison. "Boo cheater" would be an appropriate response if he did it purely for ranking, not for money (or something easily sold for money).
- elteto 7y agoHow _is it_ criminal though? Please do tell which law his team broke here. Put down the pitchfork and calm down. The guy is a raging a* and a cheater, but a criminal he is not. He lost his job, was publicly shamed and his reputation is tarnished basically forever. I think he got enough coming his way.
- laydn 7y agoSlightly off topic: The nature of the competition is a bit worrying to me. They're essentially letting an algorithm decide which dogs have the best chance of adoption and which to euthanize, aren't they?
- alanfranz 7y agoAre you sure? Aren't they just trying to show the most appropriate pets to the most appropriate people in order to make adoptions quicker? Where do you read the euthanization part?
- dh5 7y agoUnfortunately, most pet shelters get more pets than their facilities are able to handle. Unless they're a no-kill shelter (in order to be considered one they need to kill <10%), they will need to euthanize in order to make room for incoming animals. This is as opposed to promoting adoption, fostering, etc. An algorithm that can prematurely decide what animals will be adopted, or have the best chance to be, can create a perverse incentive to euthanize early the less optimal pets.
- nnq 7y agoTHIS! Drop the "slightly"... I can't understand how people focus so much on the competition cheating, and so less on "wait, wtf are they doing here"... I mean, even if they are not deciding whether to euthanize or not based on this, you're still building a system that introduces "good looks" as a factor in a life-and-death decision regarding a living being. It's not hard to jump from this a system that would use your facebook file to grant or deny medical health coverage or a similar life-and-death thing. Shift the Overton-window a little, push is a few notches further, and you're re-inventing phrenology with deep learning... This is bone-chilling! I mean the fact that so many people overlook this...
- RomanBob 7y agoIf some animals are going to be euthanised, would you rather you euthanise at random, or euthanise those who are likely not going to be adopted anyway. The problem here isn't AI, it's the problem of pet ownership. This problem has existed. And you are criticising an imperfect attempt at solving a part of this problem, because the solution isn't perfect. shake my head
- tastyminerals 7y agoKaggle doesn't make a lot of sense to me. It's a good self education platform but that's it. It is kind of sad to see job interview candidates for ML positions whose only ML experience is Kaggle.
- prodent 7y agoWhere/how else would you get that experience if your current job does not involve ML?
- 75dvtwin 7y agoIt was briefly discussed about 7 days ago. https://news.ycombinator.com/item?id=22045696 https://news.ycombinator.com/item?id=22045696 I posted there, same self-addressed question, that I cannot figure out the answer to... It seems that intensives to cheat, and environment where 'means justify the ways' -- are overpowering. For people who are naturally gifted, successful at young age -- why cheat? Was this historically, always like this? These insensitive to cheat, to gain unfair advantage, to treat life opportunities without any 'honor code' just seem to be so pervasive now, it seems. There is a cheating scandal every other week involving most prestigious institutions, competitions, and so on. These incentives to cheat, basically destroy from inside our commercial model, academics, judicial system, political system and probably military too. This also creates a new type of powerful currency, and therefore the 'billionaires' in that currency have infinite power -- and that currency is 'dirt on somebody'. Dirt on somebody who cheated before -- forever makes the cheaters into tools of injustice. --- Public shaming is reactive, there we need something more proactive at various points. There are needs to be incentives for work verification, as an example. I also think it is unfortunate but at least civil/commercial law in many countries is pretty much riddled with 'more expensive lawyers produce better results'. And it skews society into basically thinking 'anything goes, really. means justify the ways, and cheating something one can get away with'
- Tenoke 7y ago>Was this historically, always like this? Of course. 20$ bills left on the floor are bound to be picked by someone even if most won't. ..At least as long as the expexted punishment comes down to less than the value of said bills.
- jl2718 7y agoI have a 5€ note I found blowing in the wind while running at an abandoned construction site. I asked everybody in the vicinity if they dropped it, but I still feel awful about picking it up, because I didn’t and don’t feel I can do justice by dropping it. It’s made me realize I may have moral failures from earlier in my life that I never bothered to commit to memory because I wouldn’t have thought of them that way. A child would have felt glee, and I don’t know when that should have or did change in me.
- jpdus 7y agoOriginal submission: https://news.ycombinator.com/item?id=22012763 https://news.ycombinator.com/item?id=22012763
- deleted 7y ago[deleted]
- dlkf 7y agoA large number of commenters fundamentally misunderstand what happened. They are saying "why did he upload his scraped training data? Why didn't he just train on it and upload the resulting model?" If you are making this argument, it means that you don't understand the contest. User rahimnathwani explains: > In this competition, the training code was run on Kaggle's system, so you'd still need to smuggle in the extra data. The question then becomes, how do you smuggle in the data? This is a much more interesting discussion than pontificating about the ethics of Pleskov's actions. In particular, a better understanding of this problem could have ramifications for how Kaggle could combat hacks of this variety. (By contrast, "shame on him" and "aww but he's a nice guy" are both useless, except perhaps as a form of virtue signalling). It's essentially a cryptography problem. Does anyone know if this has been widely studied?
- KaoruAoiShiho 7y agoAFAIK it's impossible to smuggle it in a way that you can't be caught. Maybe the use of MD5 made it easier to see but if examined I don't think there's a totally hidden solution.
- AstralStorm 7y agoYes, he could have trained a network to recognize them and return memorized values. It would not be nearly as obvious. Solving this would require changing the contest so that it comes with algorithm and instructions only (no data files, entropy checks) and is trained by contest operators.
- dlkf 7y agoThis would also be very fishy to anyone who inspects the source code. Immediately they would ask "uh where did this giant file of floats come from?" One approach would be to write some handcrafted rules/features that look likecthey were plausibly a priori, but have the effect of memorizing the scraped data. (I don't know if this is actually possible.) User KaoruAoiShiho is probably right that any approach along would look out of place. Coupled with the fact that its removal would massively boost accuracy, it's hard to imagine how this would get past a curious reviewer. Perhaps peer review should be a component of the Kaggle process.
- bart_spoon 7y agoI was pretty surprised by how common this type of behavior is on Kaggle. I work in machine learning and data science, but I don't use Kaggle much because it quickly became clear competitions boiled down to who could eek out the last hundredths of a percent in accuracy from models trained for weeks on multithousand dollar machines, and because the behavior described in the article was surprisingly common. That said, the site is a fantastic resource for datasets. Lots of fantastic data uploaded both from old competitions and by the community.
- QuercusMax 7y agoIs a "multithousand dollar machine" supposed to be expensive? Any company with even half-decent resources should be able to put hundreds of thousands of dollars of hardware toward training a model.
- bart_spoon 7y agoIt is when the competitions are aimed at the machine learning practitioning public, and not companies themselves. If having a machine with a bare minimum of $3000 of GPU power is the entry point for having a competitive model, then most of these competitions aren't actually a matter of coming up with a unique and clever model. It's simply throwing the biggest neural network you can at it. Which is fine, but it definitely flies in the face of the perception of Kaggle.
- jl2718 7y agoJust a reminder how unique the conditions were around the discovery process, which suggests something a lot more common.