5 ms·

Setting aside the primary subject, this is an excellent observation: > What I find however is that there are a base of unspoken intuitions that underlie expert

Setting aside the primary subject, this is an excellent observation: > What I find however is that there are a base of unspoken intuitions that underlie expert understanding of a field, that are never directly stated in the literature, because they can’t be easily proved with the rigor that the literature demands. And as a result, the insights exist only in conversation and subtext, which make them inaccessible to the casual reader.

wenc 6y ago
This is very true. This is why conferences are important. A lot of knowledge is sociological in nature. Failures and tricks of the trade are discussed at the bar after 5. I've experienced this first hand. Academic publications and lectures, in their (rightful) pursuit of rigor, aren't usually the right space for conversations around hunches and experiences. Reputations are at stake, and most people are doing impression management. When experienced readers read a journal publication, they read it with the implicit understanding that it is a highly curated view of the messiness behind the scenes.
- brundolf 6y ago
  It seems toxic to me that there's no accepted public venue for that stuff, though. The fact that people are too afraid to relay certain useful information until they're tipsy; the emphasis on "doing impression management". Maybe it should remain separate from the rigorous stuff, but where's the "Op-Ed section" of academic publishing?
  - wenc 6y ago
    I feel the word "toxic" is maybe a bit too pejorative for what it is? Another framing is that it is "guild knowledge". The incentives in academia are complex and it affects how open some folks are or can afford to be with such knowledge. That said, there are certain open avenues for making such knowledge public. MathOverflow is one. Some academics document their guild knowledge in "technical papers" which they put up on their website. The ML community (which I'm not part of, but that I'm able to observe as an outsider) seems to be particularly open when it comes to publishing blog posts -- sometimes to gain reputational points? In some journals, arguments over publications are carried out in the Letters to the Editor section. Sometimes this leads to public feuds however, and some academic communities are small enough that if you make too many enemies your publications may be visited upon with disfavor when it comes time for peer review. It's not worth getting into public tiffs unless there's a principle at stake.
    - brundolf 6y ago
      > The incentives in academia are complex and it affects how open some folks are or can afford to be with such knowledge. > Sometimes this leads to public feuds however, and some academic communities are small enough that if you make too many enemies your publications may be visited upon with disfavor when it comes time for peer review. It's not worth getting into public tiffs unless there's a principle at stake. This is more the kind of thing I was using the word "toxic" to describe. Of course I know this is a widespread and deep-seeded problem and not one that could be fixed overnight, I was just commenting on it
    - kerng 6y ago
      That's why publishing and presenting outside the formal academic realm is important. Especially presentations that have one level of indirection from original creators often provide much better intuition and also the presenter isn't afraid sharing that he/she doesn't know certain things or they present content in a more creative funny way.
  - p1esk 6y ago
    Good tricks or intuitions don't stay hidden for long. Everything good is quickly published, can easily be found in high quality implementations, and is discussed on GH issues, Pytorch forums, r/MachineLearning, or Twitter.
    - cwyers 6y ago
      That just kicks the can down the road a step, though: that just makes the folklore knowing where to find the good GitHub issues.
      - p1esk 6y ago
        The point is, the experts do freely share this knowledge. Making it accessible to "casual reader" would be low on my priority list.
  - bArray 6y ago
    Personally I've wanted a 1-page IEEE publication for a while that accepts smaller contributions, where people can share these kinds of insights. Just a "we tried this, this is what happened" or "we were not able to repeat this" or "we found this interesting, but we need more data". You could argue it's kind of like a long abstract, but a long abstract really indicates you intend to probe it further, but in actual fact you might just want to indicate that there might be something there for somebody else.
    - Fishysoup 6y ago
      People have been floating the idea of doing this for the life sciences too, where the cost of performing an experiment can be very high (animals, enzymes, local permitting and legal considerations etc. on top of the man hours). It would be very useful to see the results of an experiment you want to do(or something resembling it) but didn't follow up on it for whatever reason and would definitely accelerate research. And would probably help with the weird incentive structure and secrecy issues academia is pointlessly built around.
    - tastroder 6y ago
      I'm not sure that would be a net benefit over research blogs or something like a technical report directly published by a lab/working group to be quite honest. In a few domains I've recently looked at these notes to the editor / commentary sections seem to only be pseudo-reviewed and I'd say the likelihood of an IEEE rubberstamped one pager on perpetual motion would be non-zero.
      - bArray 6y ago
        > I'm not sure that would be a net benefit over research > blogs or something like a technical report directly > published by a lab/working group to be quite honest. I think this is kind of the problem, half of this stuff sits on a webpage somewhere completely unread and not really reference-able. > I'd say the likelihood of an IEEE rubberstamped one pager > on perpetual motion would be non-zero. I would hope that each single page would be reviewed with the same integrity as a six pager. Of course, it's not impossible crap leaks through to any conference/journal.
  - moron4hire 6y ago
    Other fields have their popular message boards. It's not just programmers.
    - philipov 6y ago
      What are the popular message boards for practicing physicists and mathematicians?
      - Zee2 6y ago
        The Mathematics Stack Exchange has consistently surprised me (in a good way) in their helpfulness and rigor.
  - hobofan 6y ago
    There is a venue for that stuff and it's in blog posts. The main problem however is that for a hype topic like this there are so many blog posts that it's hard to find the gems with valueable insights between all of the badly regurgitated common knowledge.
    - moultano 6y ago
      I'm curious which category you put this one in. :D
      - hobofan 6y ago
        Definitely not the latter ;) I think it's too new of an article to already qualify as a gem, but it's certainly made me think about a few things with a new perspective, so there is that :)
  - sjg007 6y ago
    Require people to publish their code.
  - munchbunny 6y ago
    It’s the stuff you 60-80% believe but you don’t want to put your name on it because once you do people assume you believe it 99% or more. That’s unfortunately how publishing it works.
  - renewiltord 6y ago
    All of this stuff already exists. By design it excludes people who would likely disparage the lack of rigor. And sometimes other people get caught in that net. That's a good way. It's called knowing your audience. No one wants to be savaged on HN or Reddit by a bunch of people.
  - TrackerFF 6y ago
    There are blogs But, then again, lots of academics will outright ignore blogs - if they're ever brought up in "serious" discussion.
- dehrmann 6y ago
  > Reputations are at stake, and most people are doing impression management What's the saying about academia? The politics are so vicious because stakes are so small? It's pretty ironic because the stakes are larger for a lot of decisions made by engineers in industry, but we seem content with swapping best practices.
m0zg 6y ago
This is true in a lot of other fields. This is also the reason why experienced people make a lot more money than inexperienced ones. They know where the rakes are buried under the leaves.
machinelearning 6y ago
While most people think this "knowledge" should be organized and even shared, I strongly disagree. For context, I have worked in large research labs, ML engineering organizations and startups and have encountered many people across the engineer and research spectrum. These intuitions are often wrong and arise due to the lack of vocabulary in correctly describing the mechanisms that occur. From a researcher's standpoint, learning these is counterproductive if the goal is to study and understand the underlying mechanisms from first principles. From a beginner engineer perspective, these intuitions may be effective "functional truths" but there's the danger of perceiving these handwavy intuitions as truths. This leads to inflexibility in light of empirical evidence that contradicts these intuitions and even worse - not debugging enough since the pattern seems to match roughly the intuition. The latter results in flawed institutional knowledge being accrued over time. An engineer might say: "Engineer X tried Y and it didn't work because of ML intuition Z, since this is a related problem, we should not prioritize Y due to the precedent." I think its much better for a beginner engineer to learn the methods from first principles and develop an appreciation for them. They can then learn the distinction between what's true and the intuitive language people use to describe a phenomenon they don't completely understand but can pattern match to. This will help them avoid making the mistakes that people who rely on this intuitive language too much, mistaking it for ML theory.
- bigyikes 6y ago
  > I think its much better for a beginner engineer to learn the methods from first principles and develop an appreciation for them. They can then learn the distinction between what's true and the intuitive language people use to describe a phenomenon they don't completely understand but can pattern match to. This seems super meta, because I am a beginner engineer and your comment is totally pattern matching to distillation, but for humans instead of models. Maybe beginners should start with those “functional“ (read: distilled) truths and train from there :)
  - machinelearning 6y ago
    If you want to go more meta, I can point out that your comment is pattern matching to training models and applying them to humans vs. thinking from first principles and realizing that humans and models learn differently. That's exactly the issue with applying functional truths as truths.
mam2 6y ago
This is NOT TRUE that it's NOT SHARED, especially in learning. Just browse quora on any deep learning realted fields, it's a bunch of guys shareing their (more or less relevant and exact) hunch. Also most ML vulgarisation articles focus on that too. Not sure it's true for the other fields tho.

wenc 6y ago

This is very true. This is why conferences are important. A lot of knowledge is sociological in nature. Failures and tricks of the trade are discussed at the bar after 5. I've experienced this first hand. Academic publications and lectures, in their (rightful) pursuit of rigor, aren't usually the right space for conversations around hunches and experiences. Reputations are at stake, and most people are doing impression management. When experienced readers read a journal publication, they read it with the implicit understanding that it is a highly curated view of the messiness behind the scenes.

brundolf 6y ago
It seems toxic to me that there's no accepted public venue for that stuff, though. The fact that people are too afraid to relay certain useful information until they're tipsy; the emphasis on "doing impression management". Maybe it should remain separate from the rigorous stuff, but where's the "Op-Ed section" of academic publishing?
- wenc 6y ago
  I feel the word "toxic" is maybe a bit too pejorative for what it is? Another framing is that it is "guild knowledge". The incentives in academia are complex and it affects how open some folks are or can afford to be with such knowledge. That said, there are certain open avenues for making such knowledge public. MathOverflow is one. Some academics document their guild knowledge in "technical papers" which they put up on their website. The ML community (which I'm not part of, but that I'm able to observe as an outsider) seems to be particularly open when it comes to publishing blog posts -- sometimes to gain reputational points? In some journals, arguments over publications are carried out in the Letters to the Editor section. Sometimes this leads to public feuds however, and some academic communities are small enough that if you make too many enemies your publications may be visited upon with disfavor when it comes time for peer review. It's not worth getting into public tiffs unless there's a principle at stake.
  - brundolf 6y ago
    > The incentives in academia are complex and it affects how open some folks are or can afford to be with such knowledge. > Sometimes this leads to public feuds however, and some academic communities are small enough that if you make too many enemies your publications may be visited upon with disfavor when it comes time for peer review. It's not worth getting into public tiffs unless there's a principle at stake. This is more the kind of thing I was using the word "toxic" to describe. Of course I know this is a widespread and deep-seeded problem and not one that could be fixed overnight, I was just commenting on it
  - kerng 6y ago
    That's why publishing and presenting outside the formal academic realm is important. Especially presentations that have one level of indirection from original creators often provide much better intuition and also the presenter isn't afraid sharing that he/she doesn't know certain things or they present content in a more creative funny way.
- p1esk 6y ago
  Good tricks or intuitions don't stay hidden for long. Everything good is quickly published, can easily be found in high quality implementations, and is discussed on GH issues, Pytorch forums, r/MachineLearning, or Twitter.
  - cwyers 6y ago
    That just kicks the can down the road a step, though: that just makes the folklore knowing where to find the good GitHub issues.
    - p1esk 6y ago
      The point is, the experts do freely share this knowledge. Making it accessible to "casual reader" would be low on my priority list.
- bArray 6y ago
  Personally I've wanted a 1-page IEEE publication for a while that accepts smaller contributions, where people can share these kinds of insights. Just a "we tried this, this is what happened" or "we were not able to repeat this" or "we found this interesting, but we need more data". You could argue it's kind of like a long abstract, but a long abstract really indicates you intend to probe it further, but in actual fact you might just want to indicate that there might be something there for somebody else.
  - Fishysoup 6y ago
    People have been floating the idea of doing this for the life sciences too, where the cost of performing an experiment can be very high (animals, enzymes, local permitting and legal considerations etc. on top of the man hours). It would be very useful to see the results of an experiment you want to do(or something resembling it) but didn't follow up on it for whatever reason and would definitely accelerate research. And would probably help with the weird incentive structure and secrecy issues academia is pointlessly built around.
  - tastroder 6y ago
    I'm not sure that would be a net benefit over research blogs or something like a technical report directly published by a lab/working group to be quite honest. In a few domains I've recently looked at these notes to the editor / commentary sections seem to only be pseudo-reviewed and I'd say the likelihood of an IEEE rubberstamped one pager on perpetual motion would be non-zero.
    - bArray 6y ago
      > I'm not sure that would be a net benefit over research > blogs or something like a technical report directly > published by a lab/working group to be quite honest. I think this is kind of the problem, half of this stuff sits on a webpage somewhere completely unread and not really reference-able. > I'd say the likelihood of an IEEE rubberstamped one pager > on perpetual motion would be non-zero. I would hope that each single page would be reviewed with the same integrity as a six pager. Of course, it's not impossible crap leaks through to any conference/journal.
- moron4hire 6y ago
  Other fields have their popular message boards. It's not just programmers.
  - philipov 6y ago
    What are the popular message boards for practicing physicists and mathematicians?
    - Zee2 6y ago
      The Mathematics Stack Exchange has consistently surprised me (in a good way) in their helpfulness and rigor.
- hobofan 6y ago
  There is a venue for that stuff and it's in blog posts. The main problem however is that for a hype topic like this there are so many blog posts that it's hard to find the gems with valueable insights between all of the badly regurgitated common knowledge.
  - moultano 6y ago
    I'm curious which category you put this one in. :D
    - hobofan 6y ago
      Definitely not the latter ;) I think it's too new of an article to already qualify as a gem, but it's certainly made me think about a few things with a new perspective, so there is that :)
- sjg007 6y ago
  Require people to publish their code.
- munchbunny 6y ago
  It’s the stuff you 60-80% believe but you don’t want to put your name on it because once you do people assume you believe it 99% or more. That’s unfortunately how publishing it works.
- renewiltord 6y ago
  All of this stuff already exists. By design it excludes people who would likely disparage the lack of rigor. And sometimes other people get caught in that net. That's a good way. It's called knowing your audience. No one wants to be savaged on HN or Reddit by a bunch of people.
- TrackerFF 6y ago
  There are blogs But, then again, lots of academics will outright ignore blogs - if they're ever brought up in "serious" discussion.
dehrmann 6y ago
> Reputations are at stake, and most people are doing impression management What's the saying about academia? The politics are so vicious because stakes are so small? It's pretty ironic because the stakes are larger for a lot of decisions made by engineers in industry, but we seem content with swapping best practices.

m0zg 6y ago

This is true in a lot of other fields. This is also the reason why experienced people make a lot more money than inexperienced ones. They know where the rakes are buried under the leaves.

machinelearning 6y ago

While most people think this "knowledge" should be organized and even shared, I strongly disagree. For context, I have worked in large research labs, ML engineering organizations and startups and have encountered many people across the engineer and research spectrum. These intuitions are often wrong and arise due to the lack of vocabulary in correctly describing the mechanisms that occur. From a researcher's standpoint, learning these is counterproductive if the goal is to study and understand the underlying mechanisms from first principles. From a beginner engineer perspective, these intuitions may be effective "functional truths" but there's the danger of perceiving these handwavy intuitions as truths. This leads to inflexibility in light of empirical evidence that contradicts these intuitions and even worse - not debugging enough since the pattern seems to match roughly the intuition. The latter results in flawed institutional knowledge being accrued over time. An engineer might say: "Engineer X tried Y and it didn't work because of ML intuition Z, since this is a related problem, we should not prioritize Y due to the precedent." I think its much better for a beginner engineer to learn the methods from first principles and develop an appreciation for them. They can then learn the distinction between what's true and the intuitive language people use to describe a phenomenon they don't completely understand but can pattern match to. This will help them avoid making the mistakes that people who rely on this intuitive language too much, mistaking it for ML theory.

bigyikes 6y ago
> I think its much better for a beginner engineer to learn the methods from first principles and develop an appreciation for them. They can then learn the distinction between what's true and the intuitive language people use to describe a phenomenon they don't completely understand but can pattern match to. This seems super meta, because I am a beginner engineer and your comment is totally pattern matching to distillation, but for humans instead of models. Maybe beginners should start with those “functional“ (read: distilled) truths and train from there :)
- machinelearning 6y ago
  If you want to go more meta, I can point out that your comment is pattern matching to training models and applying them to humans vs. thinking from first principles and realizing that humans and models learn differently. That's exactly the issue with applying functional truths as truths.

mam2 6y ago

This is NOT TRUE that it's NOT SHARED, especially in learning. Just browse quora on any deep learning realted fields, it's a bunch of guys shareing their (more or less relevant and exact) hunch. Also most ML vulgarisation articles focus on that too. Not sure it's true for the other fields tho.