4 ms·
Data scientists have the right approach here. Crop the outliers, use the mode for sparse data, make an embedding of categorical variables. In short, just preten
by ipnon 4y ago
Data scientists have the right approach here. Crop the outliers, use the mode for sparse data, make an embedding of categorical variables. In short, just pretend like the data is normalized because that makes for a more accurate model of reality in practice.
- brundolf 4y agoI'm not sure that translates to situations where you aren't aggregating data
- defrost 4y agoGood data science only "psuedo crops" for scale. Eg: Sat imagery is scan lines full of instrument return values. The first normalisation is to use 99.9% of the returned value range to setup a colour lookup table. .1% of the return values could be : * lens flare * instrument error * actual valid but extreme data values. Depending on the problem domain, after removing (actual) error and bogus values (where possible) you might actually be using the 99.9% of the data to "train" for normal expected background stuff ... and you're really looking for the edge case that is Gold | Uranium | a Hidden tank, etc.
- astrobe_ 4y agoTranslation please! ;-) Does this means that e.g. you just ignore the recipes with optional ingredients, or "normalize" them by making them mandatory (or deleting them), or you create two recipes (one with and one without the optional part)?
- yccs27 4y agoCrop the outliers -> ignore weird recipes/inedibles Embed categorical variables -> naively convert cups, pinches, "to taste" etc I guess data scientists would also just ignore the "optional" tag before running their analysis.
- vajrabum 4y agoCropping the outliers is often a hidden abstraction. That might be good for recognizing things but for making things it's likely to be a problem--take the cupcake example. What's the robot (program) going to do with the cupcake? Probably don't want the cups to go in the mixer.