4 ms·

> Also, in DL especially, it's quite difficult to perform ablation studies or get statistical significance numbers due experiments taking a long time to run. A

by gumby 3y ago

> Also, in DL especially, it's quite difficult to perform ablation studies or get statistical significance numbers due experiments taking a long time to run. Aww, the poor babies. This the the nature of working with complex systems (life sciences, physics, social sciences, etc): teasing out signal from very noisy data that is quite expensive (in time at least) to obtain. On the engineering side things can (but don't always) move faster with a different kind of rigour. And that applies to observational reports in the sciences as well. All of which is to say that there's nothing wrong with the flood of papers that have a bunch of colored boxes and some running code. But consider those to be the equivalent of most of the reports in the 18th century Royal Society: "I have just returned from being a passenger on the HMS Foobar; while on the Malabar coast I saw an odd insect that looked like this drawing" And conversely spend referees' precious time on actual science.

godelski 3y ago
I mean I'm with you on the side of "ML needs more math" because we're about as sciencey as psychology, but I'm not quite on the "aww poor babies" side (yet). It's just the nature of how sciences evolve. The truth is that with deep rigorous theory, things like physics are easier. I'd say the same for psych too. It's a crazy amount of stats you need to do to account for all the unknowns and uncertainties but I'm not sure why these sciences push the theory people away when they can't even be bothered to add an epsilon to their models (which are always linear), but I digress. The thing that concerns me more though is the push against math. We see it a lot in ML but weirdly enough reviews will reject you if you don't have equations so you get tons of papers just copy pasting math that adds nothing to the paper. It's really weird and a clear indication of the laziness of reviews and area chairs. What I tell my students is "you don't need math to make good models but you need math to know why your models are wrong." This seems to be becoming more important than ever. Hell, how many people know about the Whitney embedding theorem? That itself helps answer half my undergrad questions and I think it's relatively unknown. But you need both. The thinkers and tinkerers. Referring to my longer rant, industry has captured the research field. I'm really happy they're involved but we're responding in insane ways. Let academia be the thinkers and let them make small models and don't expect a $100M budget for every paper. It's the railroading that annoys me and I think it's anti science. But no one wants to even acknowledge it's existence. I'll go ahead and say there's not much use to GPT papers unless you work for OpenAI (not to be confused with LLM papers). Let people explore new architectures and don't make them compete with sota models that have billions of dollars and man hours put into them. Hype is killing our field and it's hard to see because we're a high functioning addict.

by gumby 3y ago

godelski 3y ago

I mean I'm with you on the side of "ML needs more math" because we're about as sciencey as psychology, but I'm not quite on the "aww poor babies" side (yet). It's just the nature of how sciences evolve. The truth is that with deep rigorous theory, things like physics are easier. I'd say the same for psych too. It's a crazy amount of stats you need to do to account for all the unknowns and uncertainties but I'm not sure why these sciences push the theory people away when they can't even be bothered to add an epsilon to their models (which are always linear), but I digress. The thing that concerns me more though is the push against math. We see it a lot in ML but weirdly enough reviews will reject you if you don't have equations so you get tons of papers just copy pasting math that adds nothing to the paper. It's really weird and a clear indication of the laziness of reviews and area chairs. What I tell my students is "you don't need math to make good models but you need math to know why your models are wrong." This seems to be becoming more important than ever. Hell, how many people know about the Whitney embedding theorem? That itself helps answer half my undergrad questions and I think it's relatively unknown. But you need both. The thinkers and tinkerers. Referring to my longer rant, industry has captured the research field. I'm really happy they're involved but we're responding in insane ways. Let academia be the thinkers and let them make small models and don't expect a $100M budget for every paper. It's the railroading that annoys me and I think it's anti science. But no one wants to even acknowledge it's existence. I'll go ahead and say there's not much use to GPT papers unless you work for OpenAI (not to be confused with LLM papers). Let people explore new architectures and don't make them compete with sota models that have billions of dollars and man hours put into them. Hype is killing our field and it's hard to see because we're a high functioning addict.