4 ms·
Hell, I did an analysis on "Causality in Crypto Markets": https://blog.projectpiglet.com/2018/01/causality-in-cryptomarkets/ https://blog.projectpiglet.com/201
by lettergram 9y ago
Hell, I did an analysis on "Causality in Crypto Markets":
https://blog.projectpiglet.com/2018/01/causality-in-cryptomarkets/ https://blog.projectpiglet.com/2018/01/causality-in-cryptoma...
I provided code and the data (since I knew no one would believe me, without evidence). Why do we believe researchers who have more incentive than I do to publish? Their jobs require publication, or at least are greatly improved by it, thus they have every incentive to be less than honest about their "experiments". Most of the "A.I. revolution" isn't about algorithms, it's about having more data and the right data. That's why Google and Facebook can out compete in research, as much as they do. They have all the data.
Personally, I've almost made it a life mission to try and replicate every study I can. After watching people in labs (private and public biology labs), just straight up make up data - I never believe anything.
I also wrote some stuff on backtesting (which in industry is viewed as the gold standard, but IMO is deeply prone to flaws): https://blog.projectpiglet.com/2018/01/perils-of-backtesting/ https://blog.projectpiglet.com/2018/01/perils-of-backtesting...
Having worked in industry and worked in academia... I don't believe 90% of the research I hear.
- nonbel 9y agoYes, I have come to much prefer a blog post and/or github to any "published" paper. It is almost like these journals are actively impeding progress at this point in every way possible.
- IntronExon 9y agoYou might enjoy this site if you don’t already: http://retractionwatch.com http://retractionwatch.com (No affiliation)
- d0lph 9y agoThere should be a major hit to the reputation of anyone publishing fake data. I am a bit surprised studies aren't independently reproduced immediately.
- lettergram 9y agoI considered starting a non-profit company around that, basically - you could donate to reproduce research. Then we could provide grants to reproduce the research. Kind of like a bounty system. Even registered the domain: http://researchdonor.com/ http://researchdonor.com/ I know a few people with terminal illness who would donate large sums of money for reproducible research.
- madenine 9y agoIts gotten to the point where the first thing I do when reading a paper is skim for their training data. If its not from a public/standard/replicable data source my bullshit meter hops up a couple steps. "We improved on the current SotA by 15% but you can't see the data or the code - but trust us, its for real"
- MaxBarraclough 9y agoA similar line of thinking to the always provide the source code behind your paper position.
- dmreedy 9y agoIn fact, I'd go so far to say that data is source code in the case of Machine Learning; it just goes through a kind of compilation step.
- MaxBarraclough 9y agoCareful now, don't tempt the LISPers.
- TuringNYC 9y agoI'm honestly amazed at how some papers (esp in medicine) magically train deep nets using 1000 or 2000 cases and get absurdly high 90s AUC.
- madenine 9y agoI imagine getting samples/training sets that don't have some hefty source of bias is really tough in medicine, and probably leads to over-fitting.
- ashelmire 9y agoDitto. Data can be manipulated, so can results. Small changes demanded by a PM can have a huge impact on results. The input data and the way it's handled are going to have a huge impact on any ML result.