5 ms·

A high bias low-variance introduction to Machine Learning for physicists

200 points by yoquan 8y ago

pure-awesome 8y ago
I get the idea that the title "High Bias, Low Variance" is some sort of reference or pun, but I'm not quite getting it... I mean, I know that there's a bias-variance tradeoff in stats and ML, but what does it mean in the context of introduction to ML for physicists? My guess is they mean they aren't going into as heavy detail in ML, which means the reader may lack some knowledge (high bias) but won't miss the forest/wood for the trees (low variance). Anyone else care to speculate?
- gabrielgoh 8y ago
  I guess the author is not claiming a well rounded (low bias) introduction to ML to statistics, but a highly biased and specialized (low variance) course that is tailored to the author's own interests and tastes.
  - pure-awesome 8y ago
    Quite possibly.
- tanilama 8y ago
  High-bias, meaning it is opinionated from author's own perspective. Low-variance, I think the author means to have a sharp and quality focus on the subjects he/she is going to talk about.
  - yoquan 8y ago
    Almost agree to what you said. Just a piece of more context: the 1st author wrote about connection between physics's Renormalization Group and Deep learning which was featured in [1] 4 years ago. [1]: https://www.quantamagazine.org/deep-learning-relies-on-renormalization-physicists-find-20141204 https://www.quantamagazine.org/deep-learning-relies-on-renor...
  - pure-awesome 8y ago
    Seems the most plausible explanation to me, so far.
- ivan_ah 8y ago
  Yeah after looking through the PDF, I'm still wondering what they meant. Here are some conjectures: - The book has a high bias (since written from physics perspective), but low variance since assuming physics background allows authors to write a very focussed narrative that works for physicists (low variance on subset of physisicts readers). - If we define the problem of choosing which topics t1, t2, ..., tn to cover in a ML book, the authors are saying the using the physicists approach will lead to consistency good results (i.e. not all over the place). It something like an argument from physicists-are-sensible people, unlike CS theoreticians, industry practicians, and mathematicians.
- aisofteng 8y ago
  The reference is to the bias-variance tradeoff: https://en.wikipedia.org/wiki/Bias–variance_tradeoff https://en.wikipedia.org/wiki/Bias–variance_tradeoff
  - pure-awesome 8y ago
    Yes, I know. I mention that in my original comment.
- salty_biscuits 8y ago
  The "model" presented for ML is low complexity, hence lives on the high bias, low variance side of the space of possible models.
- curious1212 8y ago
  my take on this: The high bias, low variance means that the models used in the book are the standard used in physic. The low variance means that the author doesn't use any artificial model or parameter except those that arise from physical considerations. Care is taken to motivate physically ML models and those far from physic can't be used with many parameters (hyperparameters).
- currymj 8y ago
  I interpreted “high-bias” as meaning it has an idiosyncratic and specific audience, but “low variance” as meaning with high probability it will be an effective teaching tool for that audience.
  - pure-awesome 8y ago
    Yeah, seems like a sane interpretation.
    - currymj 8y ago
      all these different interpretations suggest that lots of people, myself included, have a fuzzier understanding of the bias-variance tradeoff than we'd care to admit.
ivan_ah 8y ago
This is very, very, very good. Book is here: https://arxiv.org/abs/1803.08823 https://arxiv.org/abs/1803.08823 ____ My review: The authors provide a condensed summary of all central topics in machine learning. Topics include ML basics, ML theory, optimization algorithms, but also a detailed introduction to modern methods deep learning methods. Code examples and tutorials are provided as jupyter notebooks for each chapter [2]. The book uses three datasets (MNIST digit recognition, SUSY physics data, and simulated Nearest Neighbor Ising Model) as running examples throughout the book to help learners understand what different ML techniques can bring when analyzing the same problems from different perspectives. The book has a high bias (since written from physics perspective), but low variance since assuming physics background allows authors to write a very focussed narrative that gets to the point, and communicates three-books-worth of information in 100 pages. This is somewhat of a repeat of the general physics-ML-explanations-for-the-win pattern established in Bishop's `Pattern Recognition and Machine Learning`. The authors are wrong to label this book as useful only to people with a physics background, and in fact it will be useful for everyone who wants to learn modern ML. An estimator with high-bias but high efficiency is always useful. ____ For all my hacker news peeps that wants to learn ML and/or DL, you need to drop everything right now, go print this on the office printer, and sit outside with coffee for the next two weeks and read through this entire thing. Turn off the computer and phone. Stop checking HN for two weeks. Trust me, nothing better than this will come around on HN anytime soon. [1] book pdf => https://arxiv.org/pdf/1803.08823 https://arxiv.org/pdf/1803.08823 [2] jupyter notebooks zip => http://physics.bu.edu/~pankajm/ML-Notebooks/NotebooksforMLReview.zip http://physics.bu.edu/~pankajm/ML-Notebooks/NotebooksforMLRe...
- rerx 8y ago
  Wow, that must be one of the most enticing endorsements I have ever read. I am definitely checking this out soon. (Physicist by training, now working with deep learning)
  - Kroniker 8y ago
    There are literally dozens of us!
shamino 8y ago
I love it when physicists teach. Concise and pedagogical. Gives us physicists a good name :) Michael Hartl (Caltech Physicist) also does a killer job with Rails Tutorial. There's definitely a trend.

200 points by yoquan 8y ago

I get the idea that the title "High Bias, Low Variance" is some sort of reference or pun, but I'm not quite getting it... I mean, I know that there's a bias-variance tradeoff in stats and ML, but what does it mean in the context of introduction to ML for physicists? My guess is they mean they aren't going into as heavy detail in ML, which means the reader may lack some knowledge (high bias) but won't miss the forest/wood for the trees (low variance). Anyone else care to speculate?

gabrielgoh 8y ago
I guess the author is not claiming a well rounded (low bias) introduction to ML to statistics, but a highly biased and specialized (low variance) course that is tailored to the author's own interests and tastes.
- pure-awesome 8y ago
  Quite possibly.
tanilama 8y ago
High-bias, meaning it is opinionated from author's own perspective. Low-variance, I think the author means to have a sharp and quality focus on the subjects he/she is going to talk about.
- yoquan 8y ago
  Almost agree to what you said. Just a piece of more context: the 1st author wrote about connection between physics's Renormalization Group and Deep learning which was featured in [1] 4 years ago. [1]: https://www.quantamagazine.org/deep-learning-relies-on-renormalization-physicists-find-20141204 https://www.quantamagazine.org/deep-learning-relies-on-renor...
- pure-awesome 8y ago
  Seems the most plausible explanation to me, so far.
ivan_ah 8y ago
Yeah after looking through the PDF, I'm still wondering what they meant. Here are some conjectures: - The book has a high bias (since written from physics perspective), but low variance since assuming physics background allows authors to write a very focussed narrative that works for physicists (low variance on subset of physisicts readers). - If we define the problem of choosing which topics t1, t2, ..., tn to cover in a ML book, the authors are saying the using the physicists approach will lead to consistency good results (i.e. not all over the place). It something like an argument from physicists-are-sensible people, unlike CS theoreticians, industry practicians, and mathematicians.
aisofteng 8y ago
The reference is to the bias-variance tradeoff: https://en.wikipedia.org/wiki/Bias–variance_tradeoff https://en.wikipedia.org/wiki/Bias–variance_tradeoff
- pure-awesome 8y ago
  Yes, I know. I mention that in my original comment.
salty_biscuits 8y ago
The "model" presented for ML is low complexity, hence lives on the high bias, low variance side of the space of possible models.
curious1212 8y ago
my take on this: The high bias, low variance means that the models used in the book are the standard used in physic. The low variance means that the author doesn't use any artificial model or parameter except those that arise from physical considerations. Care is taken to motivate physically ML models and those far from physic can't be used with many parameters (hyperparameters).
currymj 8y ago
I interpreted “high-bias” as meaning it has an idiosyncratic and specific audience, but “low variance” as meaning with high probability it will be an effective teaching tool for that audience.
- pure-awesome 8y ago
  Yeah, seems like a sane interpretation.
  - currymj 8y ago
    all these different interpretations suggest that lots of people, myself included, have a fuzzier understanding of the bias-variance tradeoff than we'd care to admit.

ivan_ah 8y ago

This is very, very, very good. Book is here: https://arxiv.org/abs/1803.08823 https://arxiv.org/abs/1803.08823 ____ My review: The authors provide a condensed summary of all central topics in machine learning. Topics include ML basics, ML theory, optimization algorithms, but also a detailed introduction to modern methods deep learning methods. Code examples and tutorials are provided as jupyter notebooks for each chapter [2]. The book uses three datasets (MNIST digit recognition, SUSY physics data, and simulated Nearest Neighbor Ising Model) as running examples throughout the book to help learners understand what different ML techniques can bring when analyzing the same problems from different perspectives. The book has a high bias (since written from physics perspective), but low variance since assuming physics background allows authors to write a very focussed narrative that gets to the point, and communicates three-books-worth of information in 100 pages. This is somewhat of a repeat of the general physics-ML-explanations-for-the-win pattern established in Bishop's `Pattern Recognition and Machine Learning`. The authors are wrong to label this book as useful only to people with a physics background, and in fact it will be useful for everyone who wants to learn modern ML. An estimator with high-bias but high efficiency is always useful. ____ For all my hacker news peeps that wants to learn ML and/or DL, you need to drop everything right now, go print this on the office printer, and sit outside with coffee for the next two weeks and read through this entire thing. Turn off the computer and phone. Stop checking HN for two weeks. Trust me, nothing better than this will come around on HN anytime soon. [1] book pdf => https://arxiv.org/pdf/1803.08823 https://arxiv.org/pdf/1803.08823 [2] jupyter notebooks zip => http://physics.bu.edu/~pankajm/ML-Notebooks/NotebooksforMLReview.zip http://physics.bu.edu/~pankajm/ML-Notebooks/NotebooksforMLRe...

rerx 8y ago
Wow, that must be one of the most enticing endorsements I have ever read. I am definitely checking this out soon. (Physicist by training, now working with deep learning)
- Kroniker 8y ago
  There are literally dozens of us!

shamino 8y ago

I love it when physicists teach. Concise and pedagogical. Gives us physicists a good name :) Michael Hartl (Caltech Physicist) also does a killer job with Rails Tutorial. There's definitely a trend.