3 ms·
While most people think this "knowledge" should be organized and even shared, I strongly disagree. For context, I have worked in large research labs, ML enginee
by machinelearning 6y ago
While most people think this "knowledge" should be organized and even shared, I strongly disagree. For context, I have worked in large research labs, ML engineering organizations and startups and have encountered many people across the engineer and research spectrum.
These intuitions are often wrong and arise due to the lack of vocabulary in correctly describing the mechanisms that occur.
From a researcher's standpoint, learning these is counterproductive if the goal is to study and understand the underlying mechanisms from first principles.
From a beginner engineer perspective, these intuitions may be effective "functional truths" but there's the danger of perceiving these handwavy intuitions as truths. This leads to inflexibility in light of empirical evidence that contradicts these intuitions and even worse - not debugging enough since the pattern seems to match roughly the intuition. The latter results in flawed institutional knowledge being accrued over time. An engineer might say: "Engineer X tried Y and it didn't work because of ML intuition Z, since this is a related problem, we should not prioritize Y due to the precedent."
I think its much better for a beginner engineer to learn the methods from first principles and develop an appreciation for them. They can then learn the distinction between what's true and the intuitive language people use to describe a phenomenon they don't completely understand but can pattern match to. This will help them avoid making the mistakes that people who rely on this intuitive language too much, mistaking it for ML theory.
- bigyikes 6y ago> I think its much better for a beginner engineer to learn the methods from first principles and develop an appreciation for them. They can then learn the distinction between what's true and the intuitive language people use to describe a phenomenon they don't completely understand but can pattern match to. This seems super meta, because I am a beginner engineer and your comment is totally pattern matching to distillation, but for humans instead of models. Maybe beginners should start with those “functional“ (read: distilled) truths and train from there :)
- machinelearning 6y agoIf you want to go more meta, I can point out that your comment is pattern matching to training models and applying them to humans vs. thinking from first principles and realizing that humans and models learn differently. That's exactly the issue with applying functional truths as truths.
- p1esk 6y agoThis is a very good point. Especially in ML, there have been many cases where very smart people were wrong with their intuitions (the original explanation of batchnorm comes to mind), or maybe their intuitions were correct, but attempts to explain the intuition (especially when there's a race to publish) led to wrong conclusions. I still think intuitions should be discussed and shared, but with a clearly stated caveat like: "that's just my guess, we don't know what's really happening there". This is how I usually explain to others (and to myself) my experimental results.