3 ms·
As a ML researcher, I don't think you're far off the point. w.r.t HN, there's almost all hype and no "science". People have strong convictions but not strong e
by godelski 3y ago
As a ML researcher, I don't think you're far off the point.
w.r.t HN, there's almost all hype and no "science". People have strong convictions but not strong evidence. They happily cite papers, but only read the abstracts and miss the essential nuance. Especially in a field where suggesting limitations puts you at high risk of rejection (reviewers just copy paste that and thank you for the work).
w.r.t academia, it is a bit better, but I find that in general there are a lot of researchers missing math fundamentals. I know or have met people at top universities or top labs that don't know the difference between likelihood and probability. Similarly ones that don't understand probability density. Even ones working on diffusion. But I will say, that in general the most prominent researchers do have these skills. But you'll notice that they aren't publishing as fast and their works might not even be as popular. A lot of research right now goes into parameter tuning and throwing compute at the problem. I've been a bit vocal about this though. Mostly due to it being a barrier to other types of research (because I'll admit that the tuning is needed, but we need to be honest that it isn't high innovation either and that it is hard to prove these are better given that we haven't tuned other models/architectures to the same degree).
tldr: You're pretty spot on. There's a shit ton of noise in ML/AI. Especially on HN
Edit:
I thought I should also suggest Richard McElreath's Statistical Rethinking (https://xcelab.net/rm/statistical-rethinking/ https://xcelab.net/rm/statistical-rethinking/), which is a more enjoyable read than ISLR and will also introduce you to Bayesian stats (Lectures are also on youtube). I'd also suggest Gelman's Regression and Other Stories (https://avehtari.github.io/ROS-Examples/ https://avehtari.github.io/ROS-Examples/).
- mathisfun123 3y ago>don't know the difference between likelihood and probability. Similarly ones that don't understand probability density. I'm a phd student in a "top university", in a research group primarily focused on data science (NLP, LLMs, blah blah blah). I'm 100% sure I am the only person in the group of ~25 (including profs/postdocs) that knows the difference between f(θ|x) and f(x|θ). In fact I'm pretty sure I'm the only person that has ever even seen f(θ|x) (because I took a stats sequence out of casella+berger). This group puts out dozens of papers a year. My research focus is not data science (compilers).
- godelski 3y agoWell your name says something about who you are (and might mean you can guess at the roots of mine :). I often find PL people are more likely to have good math chops because it is taken seriously in their field. Fwiw, at CVPR last year I asked every author of a diffusion paper about likelihood or score and only 2 gave me meaningful answers (1 compared their model's density against the data's density which was estimated through an explicit density method. Yeah, parametric vs parametric, but diffusion is not a tractable density method). It is really impressive that people who are working with probability and likelihood every day do not understand the difference (I see many assume they are the same, not just not know the difference).
- mathisfun123 3y ago>and might mean you can guess at the roots of mine :) your services are required on the busy beaver thread! >It is really impressive that people who are working with probability and likelihood every day do not understand the difference i think i come away from the whole experience (the phd, even though i'm not done yet) with a deep skepticism/cynicism of very many things. but it's probably not what you expect. i just don't think the math is at all relevant/important epistemically as long as you can run the experiments efficiently. which is exactly what you see happening - people with access to gobs of compute develop good intuition that leads them towards breakthroughs, and people that don't have access to compute struggle and make do with the formalisms. it's not much different in physics, where the good experimentalists aren't born that way, they're made in the well-funded labs. i firmly believe that in ML, the math does not matter at all, beyond the tiny bit of calculus and linear algebra you need to kind of understand forwards and backwards. of course everytime i say this on here i'm skewered/debated to death on it, as if i don't know what i'm talking about :shrug:
- sjkoelle 3y agoyou havent seen skewered until you make a comment saying that bayesians and frequentists should bury the hatchet