4 ms·

One often neglected idea is transduction. The idea is that you can learn to label data better by augmenting your labeled dataset with unlabeled data. The unlab

One often neglected idea is transduction. The idea is that you can learn to label data better by augmenting your labeled dataset with unlabeled data. The unlabeled data helps discover the structure of the data which in turns helps the supervised learning. There is a lot of 2-phase approach to this problem, by first discovering feature with unsupervised learning and then using those features in supervised learning, but in many cases it's possible to do both at the same time. If you have a generative model for your data, you can treat the labels as potentially missing values and learn the joint distribution.

benanne 12y ago
Afaik what you're talking about is more commonly referred to as "semi-supervised learning". Transduction is a more specific case of semi-supervised learning where you know the test set, i.e. you know which data points the model will have to make predictions for. That means you can exploit this data for training the model, for example by unsupervised pre-training or with a pseudo-labeling approach.
- murbard2 12y ago
  Bayesians don't need a test set 8)
  - turnersr 12y ago
    Why is that?
    - murbard2 12y ago
      Because the prior on your parameters smooths out the prediction. Most cookbook techniques such as ridge regressions, cross-validation, etc have a Bayesian interpretation as a prior on the parameter. Bayesian techniques allow you to use all the data available. That said, sometimes they are computationally expensive, and it's better to approximate them by using a test set.
      - moultano 12y ago
        Unless you have an infinite regress on priors for your priors, and uncomputable Komolgorov penalties on the structure of your model, I think you need a test set. (This means you need a test set.)
        ced 12y ago
        There's an interesting chapter in MacKay's book on Occam's razor. I'm not sure how I feel about it, but it's very thought-provoking.
      - GFK_of_xmaspast 12y ago
        If your priori are that strong, why bother with the data?
      - Totient 12y ago
        You don't need a validation set. I'm pretty sure you still want a test set.
    - deleted 12y ago
      [deleted]
      - murbard2 12y ago
        Nuh uh, anyone doing LOCV is basically using AIC. There are also other principles, such as MDL which do not rely on a test set.
hankcutter 12y ago
I think the last point is important. Since it connects the ML and Stats ideas. It comes down to verbiage really - some general form of label propagation vs censored data.
- murbard2 12y ago
  All ML is density estimation using a parametric probability distribution. Some algorithm make the estimation easy and the distribution quite implicit, others make the distribution explicit but training is harder.
  - srean 12y ago
    > All ML is density estimation using a parametric probability distribution. Umm is there a typo there ? Otherwise it is so wrong that I cannot even begin to pick the flaws. In fact the key revolutionary idea in classification theory, courtesy Vapnik, was that one needn't estimate the density at all. I cannot over-emphasize the revolutionary part, prior to Vapnik, that over-arching consensus was that one should try to learn the the density and then threshold it to obtain the classifier. The motivation for this is the result that the thresholded conditional probability is the optimal classifier for a {0,1} loss. What Vapnik showed was that it is better to learn the classifier directly (he called it structural risk minimization, SRM, for short), without any parametric assumptions whatsoever. The main reasoning is that density estimation is freaking hard, and depending on a freaking hard task as a pre-requisite is backwards. Learning to classify is actually easier than learning densities. It was one of those paradigm shifts that come to a field only once in a while. Even if we let SRM be, even in traditional statistics there have been methods as old as time that do not assume any parametric model. Those into non-parametric statistics (I am not one) would be very upset with your claim :) The problem with parametric models is that you almost never know the parametric family. You can of course test, but how many would you test, there are infinite number of such families. Non-parametric methods dont make parametric assumptions, (they do make some and considerably weaker assumptions. Assumptions are necessary for learning). The drawback is that they are more expensive, similarly if you happen to know the correct form of the family parametric model would be hard to beat.

benanne 12y ago

Afaik what you're talking about is more commonly referred to as "semi-supervised learning". Transduction is a more specific case of semi-supervised learning where you know the test set, i.e. you know which data points the model will have to make predictions for. That means you can exploit this data for training the model, for example by unsupervised pre-training or with a pseudo-labeling approach.

murbard2 12y ago
Bayesians don't need a test set 8)
- turnersr 12y ago
  Why is that?
  - murbard2 12y ago
    Because the prior on your parameters smooths out the prediction. Most cookbook techniques such as ridge regressions, cross-validation, etc have a Bayesian interpretation as a prior on the parameter. Bayesian techniques allow you to use all the data available. That said, sometimes they are computationally expensive, and it's better to approximate them by using a test set.
    - moultano 12y ago
      Unless you have an infinite regress on priors for your priors, and uncomputable Komolgorov penalties on the structure of your model, I think you need a test set. (This means you need a test set.)
      - ced 12y ago
        There's an interesting chapter in MacKay's book on Occam's razor. I'm not sure how I feel about it, but it's very thought-provoking.
    - GFK_of_xmaspast 12y ago
      If your priori are that strong, why bother with the data?
    - Totient 12y ago
      You don't need a validation set. I'm pretty sure you still want a test set.
  - deleted 12y ago
    [deleted]
    - murbard2 12y ago
      Nuh uh, anyone doing LOCV is basically using AIC. There are also other principles, such as MDL which do not rely on a test set.

hankcutter 12y ago

I think the last point is important. Since it connects the ML and Stats ideas. It comes down to verbiage really - some general form of label propagation vs censored data.

murbard2 12y ago
All ML is density estimation using a parametric probability distribution. Some algorithm make the estimation easy and the distribution quite implicit, others make the distribution explicit but training is harder.
- srean 12y ago
  > All ML is density estimation using a parametric probability distribution. Umm is there a typo there ? Otherwise it is so wrong that I cannot even begin to pick the flaws. In fact the key revolutionary idea in classification theory, courtesy Vapnik, was that one needn't estimate the density at all. I cannot over-emphasize the revolutionary part, prior to Vapnik, that over-arching consensus was that one should try to learn the the density and then threshold it to obtain the classifier. The motivation for this is the result that the thresholded conditional probability is the optimal classifier for a {0,1} loss. What Vapnik showed was that it is better to learn the classifier directly (he called it structural risk minimization, SRM, for short), without any parametric assumptions whatsoever. The main reasoning is that density estimation is freaking hard, and depending on a freaking hard task as a pre-requisite is backwards. Learning to classify is actually easier than learning densities. It was one of those paradigm shifts that come to a field only once in a while. Even if we let SRM be, even in traditional statistics there have been methods as old as time that do not assume any parametric model. Those into non-parametric statistics (I am not one) would be very upset with your claim :) The problem with parametric models is that you almost never know the parametric family. You can of course test, but how many would you test, there are infinite number of such families. Non-parametric methods dont make parametric assumptions, (they do make some and considerably weaker assumptions. Assumptions are necessary for learning). The drawback is that they are more expensive, similarly if you happen to know the correct form of the family parametric model would be hard to beat.