4 ms·

> Personally I'd advise against both SVM's and Bayesian methods for a beginner. Bayesian statistics is very much the deep end of the pool. I don’t know, I thin

> Personally I'd advise against both SVM's and Bayesian methods for a beginner. Bayesian statistics is very much the deep end of the pool. I don’t know, I think it depends on what you mean by Bayesian. I would say understanding loss functions and regularization requires some understanding of Bayesian stats (just knowing that it comes from log p(x|q) + log p(q) and what both of those terms mean). > Graphical models and Bayesian methods generally may make a comeback but such approaches have been superseded by other methods for good reasons, i.e. scaling Can you be more specific here? It sounds like you’re talking about a particular problem or class of methods. PGMs/Bayesian methods can mean basically anything from logistic regression to running HMC on some hierarchical model using 10,000 CPU hours. I just mean aspiring to learn PGMs will force you to quickly learn and gain a deeper understanding of and appreciation for Bayesian stats, and then you can build on that for years and years. But it depends on what you’re interested in doing —- there’s a difference between model building and inference; you can spend your whole life using the same loss function and just focus on making your NN architecture better, you don’t need much Bayesian stats to do that. > i.e. "MAP" estimation which is more of a hack to ML than an actual Bayesian method Huh? Maybe we mean different things by Bayesian — the mode of your posterior seems pretty Bayesian to me! > Meanwhile a strong basis in "deterministic methods", as an alternative way to spend that learning effort, has its own rewards. The training algorithms for deep learning are also the hottest algorithm research area in machine learning, and are certainly applicable beyond deep learning. For that matter a thorough understanding of SVM delves into convex optimization, an extremely powerful framework as well. Would agree that optimization is an important part of ML/DS, but since nowadays virtually all of the most popular optimization algorithms are available at our fingertips in e.g. pytorch, I would still think its better to start with trying to build a fundamental understanding of how to frame problems. But that’s colored by my own experience and background, people’s priorities should be different depending on what they want to do.

KKKKkkkk1 6y ago
How does understanding loss functions and regularization require understanding Bayesian statistics? Those notions are literally part of linear regression theory.
- ssivark 6y ago
  Only if you use them as a black box. What is considered l2 norm in regression theory corresponds to a Gaussian likelihood, whose log-likelihood becomes quadratic. And one typically might not have any priors (which is subtly misleading, because the "flat" prior is highly dependent on the chosen representation for the underlying degree of freedom). Note that regularizations are just log-priors. Without an understanding of the underlying Bayesian formulation, linear regression theory might look like a vast and somewhat ad-hoc collection of separate ideas (eg: robust loss functions, and the many different kinds of regularizations), but seen in the correct light, it is easy to start with a general formulation and specialize it nicely to your problem. Working that way, you can often design a good solution for your problem without searching through handbooks for possible pre-defined methods. You can also combine multiple ideas very easily. eg: A couple of weeks ago, working from first principles I rediscovered what is called "Lavrentyev_regularization": https://en.wikipedia.org/wiki/Tikhonov_regularization#Lavrentyev_regularization https://en.wikipedia.org/wiki/Tikhonov_regularization#Lavren...
  - KKKKkkkk1 6y ago
    I get it, understanding the Bayesian perspective on linear regression requires understanding Bayesian statistics. But linear regression is not solely a Bayesian technique.
    - astrophysician 6y ago
      You’re right, but in ML/DS (and I would argue, even in other contexts you might not consider ML/DS) you usually are doing things in a Bayesian context (“what can I say about my model parameters given my observations and assumptions”). If you’re only doing OLS with maybe a L1 or L2 regularization, it’s not critical you understand the Bayesian interpretation, but it helps, and as soon as you start venturing out into other ideas and messier/biased data and want to modify something, you’ll need a Bayesian understanding of what you’re doing.
  - unishark 6y ago
    Regularization is used by mathematicians and engineers with a different theoretical perspective. A regular solution is one that is smooth or well-behaved in some sense. The motivation to choose the penalty is based on desired properties of the solution. L2 regularization leads to the least-length solution, L1 is chosen if one wants the sparsest solution, and so on. It is a bit less of a deep motivation, but then the decision of which model to choose is always somewhat ad hoc.
ssivark 6y ago
To elaborate on the point: When doing probabilistic modeling, whether one realizes or not, there typically is an underlying Bayesian formulation which explains what one is doing. Now, that might be well-aligned with the problem of interest (or not), and being clear on the fundamentals helps understand that, and also to compose distinct ideas which make sense together in the context of the problem. eg: see my comments below, in the context of linear regression from a Bayesian perspective. Also, while "scaling" with data is a very hip thing these days, for most problems of interest it is very difficult/expensive to get lots of data (or afford compute). Further, humans often have very useful domain-models which are worth encoding into the structure of the model. This also helps nicely mix together a conventional "software" modeling with probabilistic aspects (for those who weren't aware, this flows towards what is called "probabilistic programming", and recent developments have made significant progress towards methods which work for an "intermediate" dimensionality, if not "large" dimensionality). -- @astrophysician: Feels nice to see expressed so clearly, a perspective that I share! Feel free to get in touch if you'd like to discuss ML.
- prox 6y ago
  Is there a good book for a beginner to get acquainted with ML? I have experience in Python, JS if that helps narrow it down.
  - disgruntledphd2 6y ago
    You probably want this book: https://faculty.marshall.usc.edu/gareth-james/ISL/ISLR%20Seventh%20Printing.pdf https://faculty.marshall.usc.edu/gareth-james/ISL/ISLR%20Sev... Additionally, you should probably learn a (little) R (which you can get from this book). This is not because R is a wonderful language (though I'm pretty fond of it myself) but because it's a great tool for the communication and expression of statistical methods. Most good stats (which will help you be actually good at ML) books tend to either be written in mathematics, or R (or both). Given that you're already a programmer, R will probably make it easier for you to learn a bunch of this stuff (and the docs for R functions tend to point towards useful literature). I actually travelled the other way (i.e. from stats to code) and I found R very very helpful. Of course, your mileage may vary, but the link above is probably the best single book that you could read to start learning ML.
unishark 6y ago
By Bayesian methods I mean methods which also describe everything with statistical distributions including hyperparameters, and which solve for a distribution (or some hard-to-get feature of it like its mean or confidence intervals), instead of just a maximum probability estimate where you can make shortcuts. For example Ridge regression where you put a gaussian prior on the predictor then guess at its variance as a hyperparameter for a quadratic penalty (with cross validation or something) is MAP but not actually "Bayesian". Bayesians also put a prior distribution on the hyperparameter itself making life extra difficult. I agree it's an awkward definition but it does differentiate between the easy tricks and the tough approaches Bayesian researchers actually work on. Even in deep learning it's simple to put a penalty term on the loss or weights for that matter, so you could call it a MAP estimate. That's implemented everywhere. For that matter you can claim any neural net classifier with a sigmoid or softmax at the final layer is doing logistic regression. With multilayer networks just adding a representation learning stage to it. As I said cheers to statistics, just that stopping at MAP estimates will cover most everything. If one goes looking for a Bayesian methods text or review article it will be well beyond that. By scaling problems I basically the fact that monte carlo doesn't work in large numbers of dimensions, where the parameters of the joint distribution of everything are vast.
- astrophysician 6y ago
  > By Bayesian methods I mean methods which also describe everything with statistical distributions including hyperparameters, and which solve for a distribution (or some hard-to-get feature of it like its mean or confidence intervals), instead of just a maximum probability estimate where you can make shortcuts. Ah I see yea, if you’re talking about hyperpriors and marginalization and MCMC I totally agree; these are really valuable in science, and this is the “full Bayesian solution” but with many caveats, one big one being that it’s very easy to be overly confident in the results of some unwieldy hierarchical model and ignore the fact that the priors (or hyperpriors) are often times fudge factors because it’s so damn hard to articulate your prior belief as a bona fide distribution. A lot of times you run into a “garbage in garbage out” problem and it might not be obvious right away since we are probably busy patting ourselves on the back and popping champagne bottles because we’ve gotten our MCMC chains to converge. > For example Ridge regression where you put a gaussian prior on the predictor then guess at its variance as a hyperparameter for a quadratic penalty (with cross validation or something) is MAP but not actually "Bayesian". Bayesians also put a prior distribution on the hyperparameter itself making life extra difficult. Yea right — nothing wrong with cross validation in a lot of cases. Since yea we could go with the “full Bayesian solution” and marginalize over some distribution of our hyperprior, but that very well might not give you anything more than cross validation (or maybe will give you something worse if you have a bad hyperprior). > Even in deep learning it's simple to put a penalty term on the loss or weights for that matter, so you could call it a MAP estimate. That's implemented everywhere. Agreed, my only argument is that it’s helpful to know where all of these terms come from in a general sense rather than treat each one as some ad hoc solution. > For that matter you can claim any neural net classifier with a sigmoid or softmax at the final layer is doing logistic regression. With multilayer networks just adding a representation learning stage to it. Agree and that’s exactly the sort of valuable insight that’s helpful to have (though maybe this one isn’t really an insight from Bayesian stats). > As I said cheers to statistics, just that stopping at MAP estimates will cover most everything. If one goes looking for a Bayesian methods text or review article it will be well beyond that. Sure, if you know MAP and what it means, then you know what a posterior means and you know that every model you fit and loss function you use comes from that, and you know if you care about more than a mode or mean of the posterior you can use lots of tools to deal with that. I’ve never really encountered MCMC in the work I do and i don’t think it would really bring any additional insight into what we’re doing, but sometimes we care about uncertainty and Laplace approximation or variational inference can do the job just fine. > By scaling problems I basically the fact that monte carlo doesn't work in large numbers of dimensions, where the parameters of the joint distribution of everything are vast. While I would agree MCMC and it’s variants are certainly not a good idea a lot of the time because they require a lot of time and computation, very rarely they are (you want insights from your data to use internally or something and you really want to explore implications of many different assumptions), and HMC etcetera can deal with very high dimensional settings (> millions of parameters). But generally I agree, if you’re talking about “scalable solutions” it usually means MCMC is not really what you want to be doing.

KKKKkkkk1 6y ago

How does understanding loss functions and regularization require understanding Bayesian statistics? Those notions are literally part of linear regression theory.

ssivark 6y ago
Only if you use them as a black box. What is considered l2 norm in regression theory corresponds to a Gaussian likelihood, whose log-likelihood becomes quadratic. And one typically might not have any priors (which is subtly misleading, because the "flat" prior is highly dependent on the chosen representation for the underlying degree of freedom). Note that regularizations are just log-priors. Without an understanding of the underlying Bayesian formulation, linear regression theory might look like a vast and somewhat ad-hoc collection of separate ideas (eg: robust loss functions, and the many different kinds of regularizations), but seen in the correct light, it is easy to start with a general formulation and specialize it nicely to your problem. Working that way, you can often design a good solution for your problem without searching through handbooks for possible pre-defined methods. You can also combine multiple ideas very easily. eg: A couple of weeks ago, working from first principles I rediscovered what is called "Lavrentyev_regularization": https://en.wikipedia.org/wiki/Tikhonov_regularization#Lavrentyev_regularization https://en.wikipedia.org/wiki/Tikhonov_regularization#Lavren...
- KKKKkkkk1 6y ago
  I get it, understanding the Bayesian perspective on linear regression requires understanding Bayesian statistics. But linear regression is not solely a Bayesian technique.
  - astrophysician 6y ago
    You’re right, but in ML/DS (and I would argue, even in other contexts you might not consider ML/DS) you usually are doing things in a Bayesian context (“what can I say about my model parameters given my observations and assumptions”). If you’re only doing OLS with maybe a L1 or L2 regularization, it’s not critical you understand the Bayesian interpretation, but it helps, and as soon as you start venturing out into other ideas and messier/biased data and want to modify something, you’ll need a Bayesian understanding of what you’re doing.
- unishark 6y ago
  Regularization is used by mathematicians and engineers with a different theoretical perspective. A regular solution is one that is smooth or well-behaved in some sense. The motivation to choose the penalty is based on desired properties of the solution. L2 regularization leads to the least-length solution, L1 is chosen if one wants the sparsest solution, and so on. It is a bit less of a deep motivation, but then the decision of which model to choose is always somewhat ad hoc.

ssivark 6y ago

To elaborate on the point: When doing probabilistic modeling, whether one realizes or not, there typically is an underlying Bayesian formulation which explains what one is doing. Now, that might be well-aligned with the problem of interest (or not), and being clear on the fundamentals helps understand that, and also to compose distinct ideas which make sense together in the context of the problem. eg: see my comments below, in the context of linear regression from a Bayesian perspective. Also, while "scaling" with data is a very hip thing these days, for most problems of interest it is very difficult/expensive to get lots of data (or afford compute). Further, humans often have very useful domain-models which are worth encoding into the structure of the model. This also helps nicely mix together a conventional "software" modeling with probabilistic aspects (for those who weren't aware, this flows towards what is called "probabilistic programming", and recent developments have made significant progress towards methods which work for an "intermediate" dimensionality, if not "large" dimensionality). -- @astrophysician: Feels nice to see expressed so clearly, a perspective that I share! Feel free to get in touch if you'd like to discuss ML.

prox 6y ago
Is there a good book for a beginner to get acquainted with ML? I have experience in Python, JS if that helps narrow it down.
- disgruntledphd2 6y ago
  You probably want this book: https://faculty.marshall.usc.edu/gareth-james/ISL/ISLR%20Seventh%20Printing.pdf https://faculty.marshall.usc.edu/gareth-james/ISL/ISLR%20Sev... Additionally, you should probably learn a (little) R (which you can get from this book). This is not because R is a wonderful language (though I'm pretty fond of it myself) but because it's a great tool for the communication and expression of statistical methods. Most good stats (which will help you be actually good at ML) books tend to either be written in mathematics, or R (or both). Given that you're already a programmer, R will probably make it easier for you to learn a bunch of this stuff (and the docs for R functions tend to point towards useful literature). I actually travelled the other way (i.e. from stats to code) and I found R very very helpful. Of course, your mileage may vary, but the link above is probably the best single book that you could read to start learning ML.

unishark 6y ago

By Bayesian methods I mean methods which also describe everything with statistical distributions including hyperparameters, and which solve for a distribution (or some hard-to-get feature of it like its mean or confidence intervals), instead of just a maximum probability estimate where you can make shortcuts. For example Ridge regression where you put a gaussian prior on the predictor then guess at its variance as a hyperparameter for a quadratic penalty (with cross validation or something) is MAP but not actually "Bayesian". Bayesians also put a prior distribution on the hyperparameter itself making life extra difficult. I agree it's an awkward definition but it does differentiate between the easy tricks and the tough approaches Bayesian researchers actually work on. Even in deep learning it's simple to put a penalty term on the loss or weights for that matter, so you could call it a MAP estimate. That's implemented everywhere. For that matter you can claim any neural net classifier with a sigmoid or softmax at the final layer is doing logistic regression. With multilayer networks just adding a representation learning stage to it. As I said cheers to statistics, just that stopping at MAP estimates will cover most everything. If one goes looking for a Bayesian methods text or review article it will be well beyond that. By scaling problems I basically the fact that monte carlo doesn't work in large numbers of dimensions, where the parameters of the joint distribution of everything are vast.

astrophysician 6y ago
> By Bayesian methods I mean methods which also describe everything with statistical distributions including hyperparameters, and which solve for a distribution (or some hard-to-get feature of it like its mean or confidence intervals), instead of just a maximum probability estimate where you can make shortcuts. Ah I see yea, if you’re talking about hyperpriors and marginalization and MCMC I totally agree; these are really valuable in science, and this is the “full Bayesian solution” but with many caveats, one big one being that it’s very easy to be overly confident in the results of some unwieldy hierarchical model and ignore the fact that the priors (or hyperpriors) are often times fudge factors because it’s so damn hard to articulate your prior belief as a bona fide distribution. A lot of times you run into a “garbage in garbage out” problem and it might not be obvious right away since we are probably busy patting ourselves on the back and popping champagne bottles because we’ve gotten our MCMC chains to converge. > For example Ridge regression where you put a gaussian prior on the predictor then guess at its variance as a hyperparameter for a quadratic penalty (with cross validation or something) is MAP but not actually "Bayesian". Bayesians also put a prior distribution on the hyperparameter itself making life extra difficult. Yea right — nothing wrong with cross validation in a lot of cases. Since yea we could go with the “full Bayesian solution” and marginalize over some distribution of our hyperprior, but that very well might not give you anything more than cross validation (or maybe will give you something worse if you have a bad hyperprior). > Even in deep learning it's simple to put a penalty term on the loss or weights for that matter, so you could call it a MAP estimate. That's implemented everywhere. Agreed, my only argument is that it’s helpful to know where all of these terms come from in a general sense rather than treat each one as some ad hoc solution. > For that matter you can claim any neural net classifier with a sigmoid or softmax at the final layer is doing logistic regression. With multilayer networks just adding a representation learning stage to it. Agree and that’s exactly the sort of valuable insight that’s helpful to have (though maybe this one isn’t really an insight from Bayesian stats). > As I said cheers to statistics, just that stopping at MAP estimates will cover most everything. If one goes looking for a Bayesian methods text or review article it will be well beyond that. Sure, if you know MAP and what it means, then you know what a posterior means and you know that every model you fit and loss function you use comes from that, and you know if you care about more than a mode or mean of the posterior you can use lots of tools to deal with that. I’ve never really encountered MCMC in the work I do and i don’t think it would really bring any additional insight into what we’re doing, but sometimes we care about uncertainty and Laplace approximation or variational inference can do the job just fine. > By scaling problems I basically the fact that monte carlo doesn't work in large numbers of dimensions, where the parameters of the joint distribution of everything are vast. While I would agree MCMC and it’s variants are certainly not a good idea a lot of the time because they require a lot of time and computation, very rarely they are (you want insights from your data to use internally or something and you really want to explore implications of many different assumptions), and HMC etcetera can deal with very high dimensional settings (> millions of parameters). But generally I agree, if you’re talking about “scalable solutions” it usually means MCMC is not really what you want to be doing.