7 ms·

This is a surprising blog post in that I draw the complete opposite conclusion the author does. The author seems to think that the averaging hides volatility (

This is a surprising blog post in that I draw the complete opposite conclusion the author does. The author seems to think that the averaging hides volatility (which it does), which leads to incorrect conclusions drawn. Whereas to me it looks like it removes visual noise to show an actual trend. In his original scatter plot, because of the big ball in the middle it's easy to handwave and say, "look a random blob" -- but even a slightly longer look seems to indicate there is a positive correlation in the data. His averaging that he does at the end, IMO, makes it clear.

cryptoz 14y ago
I agree 100%. A scatter plot, with a low but very obvious correlation, shown for two years only seems to me to be absolutely fantastic evidence that progress is being made.
- pfortuny 14y ago
  Yes, that is what the people at Lehman Brothers were saying about the Mortgage market for some years. But anyway, each one is free to reach his own conclussions.
pdonis 14y ago
You're missing the point. The author agreed that there is a slight positive correlation in the data. But the averaging makes it look like there is a strong positive correlation in the data, which there isn't. In other words: without the averaging, you are seeing the full implications of the data: slight positive correlation, but a lot of scatter--i.e., not much meat there. With the averaging, you are seeing only the positive correlation, with all other information filtered out--i.e., you are seeing only part of the picture, and it's the part that, surprise, surprise, makes the person who ordered the data collected look good.
- kenjackson 14y ago
  Huh? The averaging doesn't give you only the positive correlation. It simply averages it. If the correlation was negative, you'd see that just as plainly. Averaging doesn't change the trend -- it just removes the volatility from the graph and makes the graph more readable (what's the density at y=x? I suspect it's a lot denser than he'd have you believe, but you can't tell from the chart at all). Sure, you could add a volatility measure as well, but the trend is clear -- you'd rather have your child by a teacher with a high teacher score than a low one. Or are you saying that over 12 years of schooling you don't think you'd have any preference having your child taught by teachers with a score in the lowest 5%ile versus a teacher with a predicted score in the top 5%ile? Of course there's some volatility, but over time, it's seems pretty obvious to me.
  - streptomycin 14y ago
    The problem is that averaging can make a very small correlation look like a very strong correlation. You might be interested in this: http://en.wikipedia.org/wiki/Ecological_fallacy http://en.wikipedia.org/wiki/Ecological_fallacy Nobody is arguing that a small correlation doesn't exist. They are arguing that a small correlation isn't very meaningful. Or are you saying that over 12 years of schooling you don't think you'd have any preference having your child taught by teachers with a score in the lowest 5%ile versus a teacher with a predicted score in the top 5%ile? If you base those percentiles on a metric that has a very low correlation with performance, then there won't be much of a difference in outcome. Or are you saying that you want to pick who to fire and who to promote based on a metric that we objectively know has only a very weak correlation with reality?
    - yummyfajitas 14y ago
      Or are you saying that you want to pick who to fire and who to promote based on a metric that we objectively know has only a very weak correlation with reality? What alternatives do we have? Do you know of a more accurate metric?
      - streptomycin 14y ago
        One alternative is to not use these flawed/complicated/expensive metrics. Imagine if I came up with a metric that had an 0.1 correlation coefficient with your performance at your job. If you're a programmer, maybe that'd be something like measuring the lines of code written per day. Would you put much faith in that? Of course not, there would be myriad problems with such a system. Same thing here. The status quo in teaching is to evaluate teachers based on qualitative observations from management, same as it is in most other industries. As the same blogger points out in another post http://garyrubinstein.teachforus.org/2013/01/13/50-million-3-years-no-clue/ http://garyrubinstein.teachforus.org/2013/01/13/50-million-3... "Conspicuously missing from the various weighting schemes they compare is one with 100% classroom observations. As this is what many districts currently do and since this report is supposed to guide those who are designing new systems, wouldn’t it be scientifically necessary to include the existing system as the ‘control’ group? As implementing a change is a costly and difficult process, shouldn’t we know what we could expect to gain over the already existing system?" The status quo also has the advantage of not being ridiculously expensive and complicated. Ask Bill Gates to release the data if you're curious about its performance, but given what I've seen, I'd be shocked if it was terribly different than these complicated new metrics.
FourthProtocol 14y ago
The other thing the author does is to derive absolute findings when the author himself admits to making an assumption about how the numbers were obtained. To me, using the word "lie" in the title is therefore nothing more than click-baiting. And on a separate note - there's a lot of anti-Gates sentiment flying around lately. Whether his methods are the best or not, he's using his own money. I find it really difficult to give any criticism directed at Bill Gates' foundation any credence. And that not knowing much at all about what he does. Maybe people would like Bill more if he did a Larry Ellison and bought a yacht instead.
- zasz 14y ago
  Bill Gates's position as a famous wealthy guy means he has more of a responsibility than the rest of us to not put out misleading graphs. Even if he is spreading misinformation with his own money.
  - btilly 14y ago
    But Bill Gates is not putting out the graphs. This looks to me like someone trying to make the report look good so that Bill Gates will get positive press, think this was worthwhile, and fund them again.
- Evbn 14y ago
  Many would argue that Bill came by that money via illegal and unethical behavior, paid for by millions of consumers, so it is more like tax money than his own money.
- Evbn 14y ago
  Luxury taxes crimped the yacht industry and many workers. Fiscal conservatives would appreciate some support for the yacht industry.
- jacalata 14y ago
  People are arguing that he is using his own money to actively damage society. I'm not sure if I agree with that, but to say you can't criticize someone for what they do with their money is ridiculous - as an extreme, he could use his money to poison people, or to lobby for life sentences for marijuana use.
sadfaceunread 14y ago
+1. The blog author saying there is no clear trend in his random blob of points is misleading. The trend is there, it is just difficult to tell the density of points in his scatter plot because of the marker size and plot size.
Alex3917 14y ago
"Whereas to me it looks like it removes visual noise to show an actual trend." Except for that in this case the noise is more important than the trend. Think about it, if you're firing or sanctioning perhaps 30%+ of teachers each year for no reason, then only complete morons would go into teaching. It's the same as airport security, where a .1% false positive rate is unacceptable, whereas a 75% false negative rate is just fine.
- jellicle 14y ago
  Currently the number of public school teachers fired or sanctioned per year is much less than 0.1%. Which is a bit different than 30%. So I'm thinking they're not too worried about this.
  - deleted 14y ago
    [deleted]
  - Alex3917 14y ago
    "Currently the number of public school teachers fired or sanctioned per year is much less than 0.1%." This is comparing apples and oranges.
- ryanmolden 14y ago
  Did you transpose false negative and false positive in your statement about airport security? I think airport security would much rather have false positive (i.e. This guy has something suspicious, let's do further checks, oh, turns out it was nothing) vs. False negatives (i.e. This guy is clean, oh, turns out he wasn't and blew up a plane).
  - Alex3917 14y ago
    No, that's correct. The issue is that the overwhelming majority of people aren't terrorists. So even though it's worse to let a terrorist on a plane than it is to ban someone who isn't a terrorist from flying, in aggregate the harms of banning non-terrorists from flying become greater than the harms letting a few terrorists fly even when the false positive rate is very small. Bruce Schneier has a good explanation of this somewhere on his blog. (Actually, this is one of his pet issues, so there are probably dozens of blog posts about this.)

cryptoz 14y ago

I agree 100%. A scatter plot, with a low but very obvious correlation, shown for two years only seems to me to be absolutely fantastic evidence that progress is being made.

pfortuny 14y ago
Yes, that is what the people at Lehman Brothers were saying about the Mortgage market for some years. But anyway, each one is free to reach his own conclussions.

pdonis 14y ago

You're missing the point. The author agreed that there is a slight positive correlation in the data. But the averaging makes it look like there is a strong positive correlation in the data, which there isn't. In other words: without the averaging, you are seeing the full implications of the data: slight positive correlation, but a lot of scatter--i.e., not much meat there. With the averaging, you are seeing only the positive correlation, with all other information filtered out--i.e., you are seeing only part of the picture, and it's the part that, surprise, surprise, makes the person who ordered the data collected look good.

kenjackson 14y ago
Huh? The averaging doesn't give you only the positive correlation. It simply averages it. If the correlation was negative, you'd see that just as plainly. Averaging doesn't change the trend -- it just removes the volatility from the graph and makes the graph more readable (what's the density at y=x? I suspect it's a lot denser than he'd have you believe, but you can't tell from the chart at all). Sure, you could add a volatility measure as well, but the trend is clear -- you'd rather have your child by a teacher with a high teacher score than a low one. Or are you saying that over 12 years of schooling you don't think you'd have any preference having your child taught by teachers with a score in the lowest 5%ile versus a teacher with a predicted score in the top 5%ile? Of course there's some volatility, but over time, it's seems pretty obvious to me.
- streptomycin 14y ago
  The problem is that averaging can make a very small correlation look like a very strong correlation. You might be interested in this: http://en.wikipedia.org/wiki/Ecological_fallacy http://en.wikipedia.org/wiki/Ecological_fallacy Nobody is arguing that a small correlation doesn't exist. They are arguing that a small correlation isn't very meaningful. Or are you saying that over 12 years of schooling you don't think you'd have any preference having your child taught by teachers with a score in the lowest 5%ile versus a teacher with a predicted score in the top 5%ile? If you base those percentiles on a metric that has a very low correlation with performance, then there won't be much of a difference in outcome. Or are you saying that you want to pick who to fire and who to promote based on a metric that we objectively know has only a very weak correlation with reality?
  - yummyfajitas 14y ago
    Or are you saying that you want to pick who to fire and who to promote based on a metric that we objectively know has only a very weak correlation with reality? What alternatives do we have? Do you know of a more accurate metric?
    - streptomycin 14y ago
      One alternative is to not use these flawed/complicated/expensive metrics. Imagine if I came up with a metric that had an 0.1 correlation coefficient with your performance at your job. If you're a programmer, maybe that'd be something like measuring the lines of code written per day. Would you put much faith in that? Of course not, there would be myriad problems with such a system. Same thing here. The status quo in teaching is to evaluate teachers based on qualitative observations from management, same as it is in most other industries. As the same blogger points out in another post http://garyrubinstein.teachforus.org/2013/01/13/50-million-3-years-no-clue/ http://garyrubinstein.teachforus.org/2013/01/13/50-million-3... "Conspicuously missing from the various weighting schemes they compare is one with 100% classroom observations. As this is what many districts currently do and since this report is supposed to guide those who are designing new systems, wouldn’t it be scientifically necessary to include the existing system as the ‘control’ group? As implementing a change is a costly and difficult process, shouldn’t we know what we could expect to gain over the already existing system?" The status quo also has the advantage of not being ridiculously expensive and complicated. Ask Bill Gates to release the data if you're curious about its performance, but given what I've seen, I'd be shocked if it was terribly different than these complicated new metrics.

FourthProtocol 14y ago

The other thing the author does is to derive absolute findings when the author himself admits to making an assumption about how the numbers were obtained. To me, using the word "lie" in the title is therefore nothing more than click-baiting. And on a separate note - there's a lot of anti-Gates sentiment flying around lately. Whether his methods are the best or not, he's using his own money. I find it really difficult to give any criticism directed at Bill Gates' foundation any credence. And that not knowing much at all about what he does. Maybe people would like Bill more if he did a Larry Ellison and bought a yacht instead.

zasz 14y ago
Bill Gates's position as a famous wealthy guy means he has more of a responsibility than the rest of us to not put out misleading graphs. Even if he is spreading misinformation with his own money.
- btilly 14y ago
  But Bill Gates is not putting out the graphs. This looks to me like someone trying to make the report look good so that Bill Gates will get positive press, think this was worthwhile, and fund them again.
Evbn 14y ago
Many would argue that Bill came by that money via illegal and unethical behavior, paid for by millions of consumers, so it is more like tax money than his own money.
Evbn 14y ago
Luxury taxes crimped the yacht industry and many workers. Fiscal conservatives would appreciate some support for the yacht industry.
jacalata 14y ago
People are arguing that he is using his own money to actively damage society. I'm not sure if I agree with that, but to say you can't criticize someone for what they do with their money is ridiculous - as an extreme, he could use his money to poison people, or to lobby for life sentences for marijuana use.

sadfaceunread 14y ago

+1. The blog author saying there is no clear trend in his random blob of points is misleading. The trend is there, it is just difficult to tell the density of points in his scatter plot because of the marker size and plot size.

Alex3917 14y ago

"Whereas to me it looks like it removes visual noise to show an actual trend." Except for that in this case the noise is more important than the trend. Think about it, if you're firing or sanctioning perhaps 30%+ of teachers each year for no reason, then only complete morons would go into teaching. It's the same as airport security, where a .1% false positive rate is unacceptable, whereas a 75% false negative rate is just fine.

jellicle 14y ago
Currently the number of public school teachers fired or sanctioned per year is much less than 0.1%. Which is a bit different than 30%. So I'm thinking they're not too worried about this.
- deleted 14y ago
  [deleted]
- Alex3917 14y ago
  "Currently the number of public school teachers fired or sanctioned per year is much less than 0.1%." This is comparing apples and oranges.
ryanmolden 14y ago
Did you transpose false negative and false positive in your statement about airport security? I think airport security would much rather have false positive (i.e. This guy has something suspicious, let's do further checks, oh, turns out it was nothing) vs. False negatives (i.e. This guy is clean, oh, turns out he wasn't and blew up a plane).
- Alex3917 14y ago
  No, that's correct. The issue is that the overwhelming majority of people aren't terrorists. So even though it's worse to let a terrorist on a plane than it is to ban someone who isn't a terrorist from flying, in aggregate the harms of banning non-terrorists from flying become greater than the harms letting a few terrorists fly even when the false positive rate is very small. Bruce Schneier has a good explanation of this somewhere on his blog. (Actually, this is one of his pet issues, so there are probably dozens of blog posts about this.)