3 ms·
This really brings me back to my days in college, when this was the exact sort of stuff that ML classes focused on. You could have an entire course on various i
by eachro 3y ago
This really brings me back to my days in college, when this was the exact sort of stuff that ML classes focused on. You could have an entire course on various interpretations and extensions of kernel based methods. I wonder how much these insights are worth anymore in the age of LLMs, deep neural networks. I havent kept up too much with the NTK literature but it seems like the theoretical understanding of kernel based methods, gaussian processes does not confer you any advantage in being a better modern ML (specifically working on LLMs) engineer, where the skill set is more heavily geared towards systems engineering and/or devops for babysitting all your experiments.
- lars 3y agoI basically share your sentiment. However, Greg Yangs work seems to have produced something of direct practical benefit for training large neural nets, based on the NTK literature. µ-Parametrization is apparently very useful in real world practice.
- touisteur 3y agoI guess the systems engineering / babysitting part is an intermediate step in the process (fits and starts of a new engineering discipline) and I hope that CS/ML research goes sideways or backs up a bit there, to (maybe less brute force) alternatives or better/deeper understanding.
- snovv_crash 3y agoWhile the DevOps and data management is critical, if you are able to understand what the different components and layers do you can go a lot further than someone who can only retrain someone else's models on different data. Academia hasn't provided reference architectures for a lot of difficult industry problems that still need solving.