4 ms·

> Yet, the company does not justify which version of the program is the correct one to use in order to get as close as possible to the underlying true relations

> Yet, the company does not justify which version of the program is the correct one to use in order to get as close as possible to the underlying true relationship. Even if they did that's still not good enough. I guess it's my own ignorance of how science is done, but I'd expect a higher standard of certainty for which algorithm you're using than "some vendor said so".

RobertRoberts 8y ago
One project I have worked with was analyzing data in the education industry. Mainly public schools, think student grades and educational performance in general. You'd think a simple percentage grade would have meaning, but it doesn't. Often times federal and state rules alter how data is to be "perceived" and therefore, analayzed and reported on. There are a few major players in the data collection and analysis game for public school system, and they all not only have their own way of determining skill level (eg, reading skill level) with their own testing systems, they also analyze their custom data with their own algorithms. Thus leading to quite a variation in their ultimate recommendations/conclusions between vendors. Then, take into account some schools have many students with English as a second language, and based on how the state views that school, it may be analyzed as "all equal" or take into account the language barriers. Then mix in that you have to analyze students over multiple years, with students entering and leaving districts, sometimes with data vendors changing, etc... Then there is state and federal laws mixing with multiple vendors data collection _and_ analysis. So the next time you hear a particular school district is under/over performing based on these vendor's analysis (especially related to budgets), keep in mind how variant the actual reality is to what is being reported. Maybe every complex system has the same core problems?

RobertRoberts 8y ago

One project I have worked with was analyzing data in the education industry. Mainly public schools, think student grades and educational performance in general. You'd think a simple percentage grade would have meaning, but it doesn't. Often times federal and state rules alter how data is to be "perceived" and therefore, analayzed and reported on. There are a few major players in the data collection and analysis game for public school system, and they all not only have their own way of determining skill level (eg, reading skill level) with their own testing systems, they also analyze their custom data with their own algorithms. Thus leading to quite a variation in their ultimate recommendations/conclusions between vendors. Then, take into account some schools have many students with English as a second language, and based on how the state views that school, it may be analyzed as "all equal" or take into account the language barriers. Then mix in that you have to analyze students over multiple years, with students entering and leaving districts, sometimes with data vendors changing, etc... Then there is state and federal laws mixing with multiple vendors data collection _and_ analysis. So the next time you hear a particular school district is under/over performing based on these vendor's analysis (especially related to budgets), keep in mind how variant the actual reality is to what is being reported. Maybe every complex system has the same core problems?