3 ms·

In the article, the report from Data Wrangler actually tells about class imbalance and report a whole other bunch of ML specific issues like target leakage. The

In the article, the report from Data Wrangler actually tells about class imbalance and report a whole other bunch of ML specific issues like target leakage. There’s a blog post on it (granted there are other pre canned tool you could use to do this) https://aws.amazon.com/blogs/machine-learning/accelerate-data-preparation-with-data-quality-and-insights-in-amazon-sagemaker-data-wrangler/ https://aws.amazon.com/blogs/machine-learning/accelerate-dat.... Canvas also seem to have nice histograms in their UI

CarbonCycles 4y ago
There is a difference between reporting a stat and understanding the stat. Another thing that many ppl (including ppl who work in this field) fail to understand is the theoretical (or statistical) underpinnings of the algorithms they are trying to deploy. Many assume IID or NID to make the problem tractable, but that's not how the world typically works even on the very very large data scale. More things on why many data science teams/groups/fail because too many ppl treat it as a BI/BA organization...I really should step away from the keyboard now. LOL.
time_to_smile 4y ago
> class imbalance Worrying about class imbalance is a classic example of how ignorant many data scientists are about how the statistical properties of the models they're using work. For example if you are to try to "solve" class imbalance issues in a logistic model by over/under sampling you're actually throwing out important information about the prior probability of an event. Class imbalance in the data, if it reflects the real distribution of observed events, is often valuable information that will give you a better model. Yet so many data scientists I've known see class imbalance as a major issue and when you ask "why?" it's clear they don't know even the basic principles underlying the models they're using. There are times we you are forced to deal with a class imbalance issue due to the way this can impact training data size, but these cases are relatively rare in modern data environments.
- CarbonCycles 4y ago
  And this is where DS with a deeper more rigorous understanding begin to differentiate themselves by being able to step back and reformulate/re-model the problem into something more like an anomaly detector (in keeping with this example). I can see the argument where ML Engineers and DS w/out a more advanced statistics/STEM background would fail since they would continue down their list of libraries w/in their prescribed toolbox. Granted many problems can be approximated to be good enough, and let's face it, the FAANG/MAANG/whatever companies aren't running things so super critical where a user getting one extra email or ad presentation will cause them serious injury or death. Btw, appreciate your comments.
- matmatmatmat 4y ago
  Could you expand a bit about what kind of information class imbalance can reveal?
  - timy2shoes 4y ago
    It's not about revealing information, it's about how you model the problem. Say you're using logistic regression. If you upsample the minority class (never downsample the majority class because that's statistically inadmissible), then your classifier might get better at modeling the decision boundary. However, that comes at the cost of losing the best part of logistic regression: probability prediction and calibration. Sometimes the trade-off is worth it, sometimes not. Depends on the type of problem. For more details I suggest reading Frank Harrell: https://www.fharrell.com/post/classification/ https://www.fharrell.com/post/classification/
  - time_to_smile 4y ago
    The class imbalance quite literally represents the prior probability in the model. As an example, train a logistic model on imbalanced data with both the real distribution of imbalanced classes and over/down sampled classes and you'll notice that the distribution of the predictions from each model is different. This is hugely important because if you are using the output of your model as an expectation to feed into another model (for example predict expected value of a new user account given P(purchase) * E[value of purchase]), rebalancing will give you the wrong answer. This can be corrected in a logistic model by tweaking the intercept but doing this requires you to make some assumptions, and it's more artful of a process than one would hope. You can also see this if you compare the log likelihood of the true data with the given model and the new data. The model trained on the real data will have a notably higher log likelihood. Then there are separate issues with over/down sampling. If you are over sampling the standard error on your coefficients will be artificially reduced by the over representation of examples from the one class, leading you to be more sure than you should be about your parameters. The point estimates should be more or less the same, but if you care at all about inference this change in standard error is a big deal. I believe, but would have to double check, that this would likewise impact the effects of regularization on your model. If you are down sampling you are quite literally throwing out observations, and using less information than you have when modeling. Additionally you'll get the inverse problem which is artificially increased variance in the parameter estimates. Again, even if you don't care about inference you still have the problem that your regularization is very likely not working how you expect. In my experience nearly all data scientists think they are experts in logistic regression when very few really understand this "simple" model. All of these issues here will manifest themselves in more complex models but understanding how (and how to correct them) is a much trickier question.

CarbonCycles 4y ago

There is a difference between reporting a stat and understanding the stat. Another thing that many ppl (including ppl who work in this field) fail to understand is the theoretical (or statistical) underpinnings of the algorithms they are trying to deploy. Many assume IID or NID to make the problem tractable, but that's not how the world typically works even on the very very large data scale. More things on why many data science teams/groups/fail because too many ppl treat it as a BI/BA organization...I really should step away from the keyboard now. LOL.

time_to_smile 4y ago

> class imbalance Worrying about class imbalance is a classic example of how ignorant many data scientists are about how the statistical properties of the models they're using work. For example if you are to try to "solve" class imbalance issues in a logistic model by over/under sampling you're actually throwing out important information about the prior probability of an event. Class imbalance in the data, if it reflects the real distribution of observed events, is often valuable information that will give you a better model. Yet so many data scientists I've known see class imbalance as a major issue and when you ask "why?" it's clear they don't know even the basic principles underlying the models they're using. There are times we you are forced to deal with a class imbalance issue due to the way this can impact training data size, but these cases are relatively rare in modern data environments.

CarbonCycles 4y ago
And this is where DS with a deeper more rigorous understanding begin to differentiate themselves by being able to step back and reformulate/re-model the problem into something more like an anomaly detector (in keeping with this example). I can see the argument where ML Engineers and DS w/out a more advanced statistics/STEM background would fail since they would continue down their list of libraries w/in their prescribed toolbox. Granted many problems can be approximated to be good enough, and let's face it, the FAANG/MAANG/whatever companies aren't running things so super critical where a user getting one extra email or ad presentation will cause them serious injury or death. Btw, appreciate your comments.
matmatmatmat 4y ago
Could you expand a bit about what kind of information class imbalance can reveal?
- timy2shoes 4y ago
  It's not about revealing information, it's about how you model the problem. Say you're using logistic regression. If you upsample the minority class (never downsample the majority class because that's statistically inadmissible), then your classifier might get better at modeling the decision boundary. However, that comes at the cost of losing the best part of logistic regression: probability prediction and calibration. Sometimes the trade-off is worth it, sometimes not. Depends on the type of problem. For more details I suggest reading Frank Harrell: https://www.fharrell.com/post/classification/ https://www.fharrell.com/post/classification/
- time_to_smile 4y ago
  The class imbalance quite literally represents the prior probability in the model. As an example, train a logistic model on imbalanced data with both the real distribution of imbalanced classes and over/down sampled classes and you'll notice that the distribution of the predictions from each model is different. This is hugely important because if you are using the output of your model as an expectation to feed into another model (for example predict expected value of a new user account given P(purchase) * E[value of purchase]), rebalancing will give you the wrong answer. This can be corrected in a logistic model by tweaking the intercept but doing this requires you to make some assumptions, and it's more artful of a process than one would hope. You can also see this if you compare the log likelihood of the true data with the given model and the new data. The model trained on the real data will have a notably higher log likelihood. Then there are separate issues with over/down sampling. If you are over sampling the standard error on your coefficients will be artificially reduced by the over representation of examples from the one class, leading you to be more sure than you should be about your parameters. The point estimates should be more or less the same, but if you care at all about inference this change in standard error is a big deal. I believe, but would have to double check, that this would likewise impact the effects of regularization on your model. If you are down sampling you are quite literally throwing out observations, and using less information than you have when modeling. Additionally you'll get the inverse problem which is artificially increased variance in the parameter estimates. Again, even if you don't care about inference you still have the problem that your regularization is very likely not working how you expect. In my experience nearly all data scientists think they are experts in logistic regression when very few really understand this "simple" model. All of these issues here will manifest themselves in more complex models but understanding how (and how to correct them) is a much trickier question.