3 ms·

Sigh this beautifully captures the tragedy of biostats and epidemiology which is held hostage by lack of systems thinking. Unlike say a clinical trials, ML comp

Sigh this beautifully captures the tragedy of biostats and epidemiology which is held hostage by lack of systems thinking. Unlike say a clinical trials, ML competitions are not limited to data available at start of the competition in fact by having a fair measurement of performance there is strong incentive to label and share more data and run checkpointed models. Further since the goal is not “publishing” with arbitrary requirements of P values / Power an empirical strategy is likely to provide much better long term accountability and better models. Sadly the cult of bio stats is so deeply vested in publishing rather than thinking end to end (designing a system as whole) any other models is quickly rejected by the community.

AstralStorm 7y ago
You're making a critical mistake. Why make a network for a competition that won't produce great results without major modifications on real data? That is the main problem, and the lack of systems thinking is on your side. There is a strong pressure to cheat and overfit. Sharing in fact makes this even stronger. We've had some fun with that when trying to use ML for something as complex as music rhythm envelope extraction. (Which is easy in comparison to CT test.) Best results were approaching 90% accuracy on the big suite, but real results were closer to 40%. A slightly worse solution did reliably 70%. (And was not a neural network even, plus possible to improve.) General best approaches sometimes indeed work, but sometimes (often?) they are overfitted in architecture, not even dataset.