6 ms·

At the end, the most important step is good feature engineering. (including dimensionality reduction, regularisation and stuff like that)

by dejv 10y ago

RangerScience 10y ago
Can you tell me more about what you mean by "features"? I taught myself part of how computer vision works, and now I'm wondering if "feature points" in CV map to... something, elsewhere, like NP problems map to NP problems. Or do you mean features like a website?
- mabcat 10y ago
  Features is another term for inputs, variables, etc. It has a connotation of being recognisable as a property of the thing being modeled, that you might have cleaned up from the raw data.
- nl 10y ago
  Features are variables which are inputs into your model. Imagine you are predicting a basketball game. Features would be things like: Season Win/Loss ration of each team Number of wins in a row Proportion of points by best player on team etc etc. These are pretty easy, but then sometimes you need to do work to generate the features - this is called feature engineering. Some features which might need more "feature engineering": Average sentiment of social network posting by team members 1-on-1 effectiveness of defensive players vs opposition offense (something like this might require video analysis and sometimes manual annotation) In the computer vision world, things like edges and points are often used as features. Deep learning is nice because it automatically learns these features given enough time. Otherwise people spend a lot of time building algorithms like SIFT etc.
  - dejv 10y ago
    Good examples. Lets do another one, more text oriented. Say you want to classify if text is spam or not. Features could be each word. Then you can build additional features on top of those words, like: - is header all caps? - is text contain some particular keyword (viagra, enlargement...) - are there typos? - link to weird domains? and much more. You can create tons of those features. Some would be helpful, some neutral, some will decrease quality of your model. Core of the machine learning is creating those features, sending them to algo and measuring impact. Choosing algo is actually smallest portion of your job (on most projects)
    - nl 10y ago
      NGrams. Text processing almost always benefits from NGrams features.
- chestervonwinch 10y ago
  Functionally speaking, the "features" are a transformation of the image pixel values values, F: R^{n x n} => R^d So `F(image)` is a vector of length `d`, containing `d` derived features from the image. Note that `F` could be a an identity transformation of sorts, which simply flattens the image. In this case, the "features" are simply the image pixel values. However, in general, the feature vector attempts to use a priori information about the problem domain to make things easier down the line. Now, for binary classification, the classifier is a map, C: R^d => {0,1} To obtain a "good" classifier, one optimizes the parameters of the classifier so that C( F(image) ) = label fairly well, by using database of `( F(image), label )` pairs. You could a imagine a database of 100px by 100px images of circles and squares. In this case, you might have a hard time obtaining an accurate classifier, if the "features" are simply the image pixel values. On the other hand, a small feature indicating the presence of corners and curvature through some mathematical transformations would likely perform well.
  - nl 10y ago
    That's a very image-centric explanation, and I'm not at all sure it makes things any easier. Conceptually, in the image case, features are "things" in images that ML tools use to perform tasks. A image with lots of blue it it has a chance that it is of sky. An image with lots of hard edges might be something human made - a house, a book etc. Pixels are really a distraction - there are alternative representations of images which don't use pixels at all. Think wavelet based compressed sensing techniques.
- nabla9 10y ago
  Feature is an individual measurable property. Features are the input variables you give to your model. Feature engineering is the work process where you massage, select, transform, normalize, regularize, standardize, adjust, whiten your input into format that is easy to digest into next phase. To confuse things even more, another name for feature learning is representation learning. Deep does just this. One layer learns new features that are inputs for next layer. Real world example I'm working with: I have raw EEG data from multiple channels and I want to use machine learning to classify it. The raw input is rows of integer values. That represent EEG reading 1 per channel. There are 300 readings per second. Consider all these issues: 1) every person and every electrode has different level of impedance and noise every time they are attached, 2) during a measurement the impedance changes and noise level changes. 3) there are several periods where person moves messing with the measures. These measurements should be removed automatically. The pipeline from raw data to the beginning of deep learning pipeline may be as follows: linear and constant detrending, normalize to standard deviation, remove 50 Hz noise, whiten, STFT into 0.5 second frequency segments, (remove epochs where patient moves) run trough carefully adjusted logit function.
  - RangerScience 10y ago
    So, if my understanding from this (and the other comments) is correct, then yeah; kinda. In CV, the first-ish level of processing determines the points on the image that will be used as "individual measurable properties" (or, maybe more accurately, the sources for those). These are the features, and then they go into the rest of the system for recognition, etc. The basic theme is stripping something analogous to "noise" and "bias" - noise being the stuff you don't want to pay attention to, and "bias" being the predictable/regular offset per thing - because you want to recognize that two things are actually the same thing, despite differences in noise and bias. (An example of "bias" might be the color of the lights where the picture was taken). Then the idea is pretty much that, if the same thing is seen in two different places, after stripping the noise and the bias from it, they "look" identical (diff is within an epsilon). That sound about right?
    - nl 10y ago
      Yes. Manually correcting the bias by engineering a feature with it stripped out is a good example.

At the end, the most important step is good feature engineering. (including dimensionality reduction, regularisation and stuff like that)

by dejv 10y ago

At the end, the most important step is good feature engineering. (including dimensionality reduction, regularisation and stuff like that)

RangerScience 10y ago

Can you tell me more about what you mean by "features"? I taught myself part of how computer vision works, and now I'm wondering if "feature points" in CV map to... something, elsewhere, like NP problems map to NP problems. Or do you mean features like a website?

mabcat 10y ago
Features is another term for inputs, variables, etc. It has a connotation of being recognisable as a property of the thing being modeled, that you might have cleaned up from the raw data.
nl 10y ago
Features are variables which are inputs into your model. Imagine you are predicting a basketball game. Features would be things like: Season Win/Loss ration of each team Number of wins in a row Proportion of points by best player on team etc etc. These are pretty easy, but then sometimes you need to do work to generate the features - this is called feature engineering. Some features which might need more "feature engineering": Average sentiment of social network posting by team members 1-on-1 effectiveness of defensive players vs opposition offense (something like this might require video analysis and sometimes manual annotation) In the computer vision world, things like edges and points are often used as features. Deep learning is nice because it automatically learns these features given enough time. Otherwise people spend a lot of time building algorithms like SIFT etc.
- dejv 10y ago
  Good examples. Lets do another one, more text oriented. Say you want to classify if text is spam or not. Features could be each word. Then you can build additional features on top of those words, like: - is header all caps? - is text contain some particular keyword (viagra, enlargement...) - are there typos? - link to weird domains? and much more. You can create tons of those features. Some would be helpful, some neutral, some will decrease quality of your model. Core of the machine learning is creating those features, sending them to algo and measuring impact. Choosing algo is actually smallest portion of your job (on most projects)
  - nl 10y ago
    NGrams. Text processing almost always benefits from NGrams features.
chestervonwinch 10y ago
Functionally speaking, the "features" are a transformation of the image pixel values values, F: R^{n x n} => R^d So `F(image)` is a vector of length `d`, containing `d` derived features from the image. Note that `F` could be a an identity transformation of sorts, which simply flattens the image. In this case, the "features" are simply the image pixel values. However, in general, the feature vector attempts to use a priori information about the problem domain to make things easier down the line. Now, for binary classification, the classifier is a map, C: R^d => {0,1} To obtain a "good" classifier, one optimizes the parameters of the classifier so that C( F(image) ) = label fairly well, by using database of `( F(image), label )` pairs. You could a imagine a database of 100px by 100px images of circles and squares. In this case, you might have a hard time obtaining an accurate classifier, if the "features" are simply the image pixel values. On the other hand, a small feature indicating the presence of corners and curvature through some mathematical transformations would likely perform well.
- nl 10y ago
  That's a very image-centric explanation, and I'm not at all sure it makes things any easier. Conceptually, in the image case, features are "things" in images that ML tools use to perform tasks. A image with lots of blue it it has a chance that it is of sky. An image with lots of hard edges might be something human made - a house, a book etc. Pixels are really a distraction - there are alternative representations of images which don't use pixels at all. Think wavelet based compressed sensing techniques.
nabla9 10y ago
Feature is an individual measurable property. Features are the input variables you give to your model. Feature engineering is the work process where you massage, select, transform, normalize, regularize, standardize, adjust, whiten your input into format that is easy to digest into next phase. To confuse things even more, another name for feature learning is representation learning. Deep does just this. One layer learns new features that are inputs for next layer. Real world example I'm working with: I have raw EEG data from multiple channels and I want to use machine learning to classify it. The raw input is rows of integer values. That represent EEG reading 1 per channel. There are 300 readings per second. Consider all these issues: 1) every person and every electrode has different level of impedance and noise every time they are attached, 2) during a measurement the impedance changes and noise level changes. 3) there are several periods where person moves messing with the measures. These measurements should be removed automatically. The pipeline from raw data to the beginning of deep learning pipeline may be as follows: linear and constant detrending, normalize to standard deviation, remove 50 Hz noise, whiten, STFT into 0.5 second frequency segments, (remove epochs where patient moves) run trough carefully adjusted logit function.
- RangerScience 10y ago
  So, if my understanding from this (and the other comments) is correct, then yeah; kinda. In CV, the first-ish level of processing determines the points on the image that will be used as "individual measurable properties" (or, maybe more accurately, the sources for those). These are the features, and then they go into the rest of the system for recognition, etc. The basic theme is stripping something analogous to "noise" and "bias" - noise being the stuff you don't want to pay attention to, and "bias" being the predictable/regular offset per thing - because you want to recognize that two things are actually the same thing, despite differences in noise and bias. (An example of "bias" might be the color of the lights where the picture was taken). Then the idea is pretty much that, if the same thing is seen in two different places, after stripping the noise and the bias from it, they "look" identical (diff is within an epsilon). That sound about right?
  - nl 10y ago
    Yes. Manually correcting the bias by engineering a feature with it stripped out is a good example.