6 ms·

The death of feature engineering is greatly exaggerated

1cvmask 5y ago
Here is a post on why feature engineering is easier in fact: https://gab41.lab41.org/feature-engineering-is-just-easier-1928d935ed17 https://gab41.lab41.org/feature-engineering-is-just-easier-1...
anonymousDan 5y ago
As a non ML expert, I never quite understand where you draw the line between feature engineering and the architecture engineering required for Deep Learning to work. Is it just a spectrum or is there some fundamental difference?
- ford 5y ago
  I'm assuming by architecture engineering you mean the process of choosing the model + hyperparameters that perform best. I think of feature vs architecture engineering as two not-quite perpendicular axes that you can do work on. Feature engineering involves selecting & transforming inputs such that the model has access to the most meaningful/easiest to interpret data. Architecture engineering involves choosing the appropriate model architecture. Suppose we were building a model to predict a business's sales for the next 12 months based on the prior 12 months. An important input is the sales for the prior year. A feature engineering task would be choosing to take the sum, median, or average sales per day based on what you know about the distribution. Architecture engineering would involve trying a few different models & tuning their hyperparameters to be most accurate. The two areas are related - because you may know that the business's sales on one day are highly dependent on the previous few days. Knowing this, you may instead choose features + an architecture that instead of predicting sales for an entire 12 months period, predicts sales on a day-by-day basis (which can then be summed).
criticaltinker 5y ago
What’s dying is classical feature engineering: complex ad-hoc pipelines created to manipulate and transform data to improve a specific model. Contemporary machine learning instead prefers to create complex model architectures and loss functions which imbue certain inductive biases that guide a model toward learning useful transformations. In this way, modern machine learning is just “meta” feature engineering. It’s one level abstracted, but still feature engineering under the hood. Of course feature engineering will always be around, especially in the abstract sense of “transforming data”, which the OP alludes to with examples like transforming CCD sensor readings to RGB pixel space.
dgreensp 5y ago
In the examples given, there is an existing pipeline from the “raw” input (CCD or unfused sensor data) to a simpler or less idiosyncratic thing (RGB pixel grid, position in space). The point seems to be, you still have a design choice to make about whether to use the raw or processed data. The processed data is conceptually cleaner to work with, but the raw data may be cleaner in the sense of more accurate. The processed data doesn’t depend so much on the details of the sensor (such as the layout of the pixels on the CCD), but it does depend on the details of the processing (in the sense that it relies on them). That’s not the same as having to choose or design a set of features, though, I don’t think.

Here is a post on why feature engineering is easier in fact: https://gab41.lab41.org/feature-engineering-is-just-easier-1928d935ed17 https://gab41.lab41.org/feature-engineering-is-just-easier-1...

anonymousDan 5y ago

As a non ML expert, I never quite understand where you draw the line between feature engineering and the architecture engineering required for Deep Learning to work. Is it just a spectrum or is there some fundamental difference?

ford 5y ago
I'm assuming by architecture engineering you mean the process of choosing the model + hyperparameters that perform best. I think of feature vs architecture engineering as two not-quite perpendicular axes that you can do work on. Feature engineering involves selecting & transforming inputs such that the model has access to the most meaningful/easiest to interpret data. Architecture engineering involves choosing the appropriate model architecture. Suppose we were building a model to predict a business's sales for the next 12 months based on the prior 12 months. An important input is the sales for the prior year. A feature engineering task would be choosing to take the sum, median, or average sales per day based on what you know about the distribution. Architecture engineering would involve trying a few different models & tuning their hyperparameters to be most accurate. The two areas are related - because you may know that the business's sales on one day are highly dependent on the previous few days. Knowing this, you may instead choose features + an architecture that instead of predicting sales for an entire 12 months period, predicts sales on a day-by-day basis (which can then be summed).

criticaltinker 5y ago

What’s dying is classical feature engineering: complex ad-hoc pipelines created to manipulate and transform data to improve a specific model. Contemporary machine learning instead prefers to create complex model architectures and loss functions which imbue certain inductive biases that guide a model toward learning useful transformations. In this way, modern machine learning is just “meta” feature engineering. It’s one level abstracted, but still feature engineering under the hood. Of course feature engineering will always be around, especially in the abstract sense of “transforming data”, which the OP alludes to with examples like transforming CCD sensor readings to RGB pixel space.

dgreensp 5y ago

In the examples given, there is an existing pipeline from the “raw” input (CCD or unfused sensor data) to a simpler or less idiosyncratic thing (RGB pixel grid, position in space). The point seems to be, you still have a design choice to make about whether to use the raw or processed data. The processed data is conceptually cleaner to work with, but the raw data may be cleaner in the sense of more accurate. The processed data doesn’t depend so much on the details of the sensor (such as the layout of the pixels on the CCD), but it does depend on the details of the processing (in the sense that it relies on them). That’s not the same as having to choose or design a set of features, though, I don’t think.