4 ms·

This post essentially described Pincipal Component Analysis in its entirety. Trying to find eigencomponents in a mixing of incompatible units means that your r

by xeonmc 2y ago

This post essentially described Pincipal Component Analysis in its entirety. Trying to find eigencomponents in a mixing of incompatible units means that your result is completely arbitrary to the scale factor of your units. And "fixing" the dimensionality by choosing an arbitrary normalization factor only further increases its mathiness factor.

dr_kiszonka 2y ago
1. If I wanted to use PCA, is there a better approach than normalization? I am asking because you seem better at math than I (based on your comment history). 2. Not that PCA is my favorite technique, but you can apply PCA to data that are expressed in the same units. 3. Do you oppose most clustering techniques since one can apply them to mixed units data?
- xeonmc 2y ago
  1. Yes, take the logarithm, especially if your data domain is a constitutive quantity (i.e. there exists an unequovical "absolute zero"). It converts multiplicative scale factors into translational offsets. If your data domain have negatives, consider whether if it's just an artifact of your relative measurement and try to re-offset your data based on reasoning. If your data is a positional quantity, consider if there exists a corresponding metric for which your data is an exponential measurement of (e.g. acidity <-> concentration), and re-shape your log base appropriately. 2. Almost yes, but you also need to make sure that there can exist a rotational symmetry in the measurement metrics, unless it is precisely the asymmetry which you are measuring (but then that would mean you have determined a priori that it's axis-aligned?) 3. Depends, I cannot say I am encyclopedically well-versed to give a judgement on the proportion of how many techniques are well-founded and how many are not. However, I can give you an example of one which is rigorous and spectacularly elegant -- see Michael F. Ashby's work.
bbor 2y ago
Woah that’s an unexpected and very hot take! What application of PCA are you familiar with that’s flawed? I learned it in the context of data science, where a) parameter normalization is a very important setup step, and b) it has directly observable empirical success. Like, I believe you that it’s used for Mathiness bs by some people, but disregarding it in its entirety seems like disregarding linear regression, or, idk, long division. I didn’t realize that was an option!
- mrbungie 2y ago
  I guess parent is talking about using PCA for doing something akin to Factor Analysis (and making interpretations over the components and the loadings / contribution from each variable towards each component). I have at least seen that in psychology, for example the Big 5 personality traits were extracted and then analyzed/discussed in that way, using FA though. I also don't think this is about disregarding the methods usefulness, but rather being very careful with assumptions and use critical thinking when interpretating results in studies that use a lot of math. In the case of DS/ML for prediction, it doesn't matter at all. As long as it empirically works, you're good. Unless you want very explainable/understandable models that is. PS: I don't like using wikipedia as a source, but you can take a look here as an example of what parent is talking about: https://en.m.wikipedia.org/wiki/Big_Five_personality_traits#Methodological_issues https://en.m.wikipedia.org/wiki/Big_Five_personality_traits#... . As I said, in this case is for FA, but a lot of people use PCA for making the same kind of studies, specially in less rigorous data-science for business settings.

by xeonmc 2y ago

dr_kiszonka 2y ago

1. If I wanted to use PCA, is there a better approach than normalization? I am asking because you seem better at math than I (based on your comment history). 2. Not that PCA is my favorite technique, but you can apply PCA to data that are expressed in the same units. 3. Do you oppose most clustering techniques since one can apply them to mixed units data?

xeonmc 2y ago
1. Yes, take the logarithm, especially if your data domain is a constitutive quantity (i.e. there exists an unequovical "absolute zero"). It converts multiplicative scale factors into translational offsets. If your data domain have negatives, consider whether if it's just an artifact of your relative measurement and try to re-offset your data based on reasoning. If your data is a positional quantity, consider if there exists a corresponding metric for which your data is an exponential measurement of (e.g. acidity <-> concentration), and re-shape your log base appropriately. 2. Almost yes, but you also need to make sure that there can exist a rotational symmetry in the measurement metrics, unless it is precisely the asymmetry which you are measuring (but then that would mean you have determined a priori that it's axis-aligned?) 3. Depends, I cannot say I am encyclopedically well-versed to give a judgement on the proportion of how many techniques are well-founded and how many are not. However, I can give you an example of one which is rigorous and spectacularly elegant -- see Michael F. Ashby's work.

bbor 2y ago

Woah that’s an unexpected and very hot take! What application of PCA are you familiar with that’s flawed? I learned it in the context of data science, where a) parameter normalization is a very important setup step, and b) it has directly observable empirical success. Like, I believe you that it’s used for Mathiness bs by some people, but disregarding it in its entirety seems like disregarding linear regression, or, idk, long division. I didn’t realize that was an option!

mrbungie 2y ago
I guess parent is talking about using PCA for doing something akin to Factor Analysis (and making interpretations over the components and the loadings / contribution from each variable towards each component). I have at least seen that in psychology, for example the Big 5 personality traits were extracted and then analyzed/discussed in that way, using FA though. I also don't think this is about disregarding the methods usefulness, but rather being very careful with assumptions and use critical thinking when interpretating results in studies that use a lot of math. In the case of DS/ML for prediction, it doesn't matter at all. As long as it empirically works, you're good. Unless you want very explainable/understandable models that is. PS: I don't like using wikipedia as a source, but you can take a look here as an example of what parent is talking about: https://en.m.wikipedia.org/wiki/Big_Five_personality_traits#Methodological_issues https://en.m.wikipedia.org/wiki/Big_Five_personality_traits#... . As I said, in this case is for FA, but a lot of people use PCA for making the same kind of studies, specially in less rigorous data-science for business settings.