8 ms·

Paraphrased from the article: AutoML tables takes a generic table and predicts can predict a columns value (for an unseen partial row, I assume). Their product

Paraphrased from the article: AutoML tables takes a generic table and predicts can predict a columns value (for an unseen partial row, I assume). Their product page emphasises how easy it is: for developers with limited machine learning experience. What are the consequences of using such structured data predictions devoid of interpretation or quantified uncertainty? Presumably such predictions could be from a set of discrete values, how is that smoothed? What is the result of providing definite results in the absence of intention?

perturbation 8y ago
AutoML is essentially training a ML model using some heuristics or optimization algorithm to select model architecture and train a model. Feature engineering / feature synthesis as well as interpretability remain open challenges. If I'm understanding your questions correctly, the main problems I see with this are: - Using raw data instead of feature engineering (less of a problem given feature synthesis libraries like https://www.featuretools.com/ https://www.featuretools.com/ and other heuristic methods). I'd expect Google to do a good job of basic things like normalization of raw input features before training. - Using features that it really shouldn't (if you just throw ML at your database for say, loan applications, then sensitive / personally identifying information can/will be used as features) - Lack of insight / understanding as to what is driving the model. This can be partially overcome with post-training methods like LIME, Shapley values, etc. I wouldn't expect predictions to be from a set of discrete values - if (say) predicting housing values and training a NN, the output should be continuous and based on the input features.
- mritchie712 8y ago
  Another common error I see is timing (e.g. using data from the "future" to predict an event). To build on your loan example, if you inadvertently included the current FICO score of an applicant that applied 12 months ago, it will be unfairly correlated with the loans current performance.
  - kmax12 8y ago
    This is very important! If you use Featuretools, we provide a mechanism to avoid this very problem. See how we handle time in our documentation here: https://docs.featuretools.com/automated_feature_engineering/handling_time.html https://docs.featuretools.com/automated_feature_engineering/...

perturbation 8y ago

AutoML is essentially training a ML model using some heuristics or optimization algorithm to select model architecture and train a model. Feature engineering / feature synthesis as well as interpretability remain open challenges. If I'm understanding your questions correctly, the main problems I see with this are: - Using raw data instead of feature engineering (less of a problem given feature synthesis libraries like https://www.featuretools.com/ https://www.featuretools.com/ and other heuristic methods). I'd expect Google to do a good job of basic things like normalization of raw input features before training. - Using features that it really shouldn't (if you just throw ML at your database for say, loan applications, then sensitive / personally identifying information can/will be used as features) - Lack of insight / understanding as to what is driving the model. This can be partially overcome with post-training methods like LIME, Shapley values, etc. I wouldn't expect predictions to be from a set of discrete values - if (say) predicting housing values and training a NN, the output should be continuous and based on the input features.

mritchie712 8y ago
Another common error I see is timing (e.g. using data from the "future" to predict an event). To build on your loan example, if you inadvertently included the current FICO score of an applicant that applied 12 months ago, it will be unfairly correlated with the loans current performance.
- kmax12 8y ago
  This is very important! If you use Featuretools, we provide a mechanism to avoid this very problem. See how we handle time in our documentation here: https://docs.featuretools.com/automated_feature_engineering/handling_time.html https://docs.featuretools.com/automated_feature_engineering/...