5 ms·

Serious question: who here who has at least a BS in CS didn't have a mandatory stats class where you learned all about picking sample sizes to give you an accur

Serious question: who here who has at least a BS in CS didn't have a mandatory stats class where you learned all about picking sample sizes to give you an accuracy you're happy with or that an average without a standard deviation is nigh-meaningless? It's just a fact of our profession, there is a significant percentage of people who just slap together APIs and have zero understanding of the maths behind it. I don't see why it takes Zed 1000 words to say it or why he has to get sanctimonious about it.

stcredzero 17y ago
There's a big percentage of programmers who don't understand O(n^2). On multiple occasions, I've seemed as a guru because of this one little tidbit. What if the medical profession has as much egregious and widespread ignorance of the basics as programming? Would you be in favor of certification?
- dschobel 17y ago
  We have de facto certification. It's called moving to silicon valley :)
  - stcredzero 17y ago
    And everywhere else, people have to put up with crappy programming? I think this system leaves a bit to be desired.
- duncanj 17y ago
  In medical professions, you go through a period of training (clinicals, internship, residency) where people with tons of experience point out that you never learned anything. This humbling experience is not really available in the CS world.
  - stcredzero 17y ago
    Particularly disturbing, since the only good way I know of to learn programming and system architecture is through mentoring. Basically, our mentoring system is haphazard. There's even a lot of anti-mentoring happening out there.
- SapphireSun 17y ago
  I find this incredibly hard to believe. Anyone who has gone through a single intro course in CS should understand at least the difference between linear and quadratic time. Perhaps I've been in the university too long. Are these people non technical majors who picked up programming in PHP?
megaduck 17y ago
The problem isn't that people know nothing about statistics. The problem is that they think they know more than they actually do. It's fairly easy to get past your "intro to stats" class with only a passing familiarity with the subject and no deep understanding. Only once you truly grasp stats do you realize that you don't actually know anything. Any statistical formula will predict its own failure. That's kind of mind-blowing, and the implications aren't always grasped after just one semester (or quarter) studying the topic.
- m_ 17y ago
  I studied CS and I got past my "intro to stats" class with only a passing familiarity with the subject and no deep understanding (aside from the probability aspects of the course). The professor was very poor overall. I was able to cram enough to do well, but I retained very little after the final. I would love a deep understanding built from the foundations-- the coursework we had was a lot more obtuse and "take my word for it". In a sense, I felt I got a better foundation for statistics from a freshman sociology class than from the Math professor.
  - stcredzero 17y ago
    Why aren't there statistics for CS classes that put stats primarily in the context of system profiling and metrics? Are there any out there?
    - cschep 17y ago
      That would have been cool. Stats wasn't required for us, but if it would have been Statistical Analysis of Computer Systems(or programs, or whatever), I would have signed up for it anyway. I heard the stats class offered where I went to school was a joke so I didn't pursue it optionally. I'm sure it's different depending on your university. Also, my school was not renowned for any sciences.
      - stcredzero 17y ago
        A really good textbook along these lines might really be something.
baguasquirrel 17y ago
At CMU we had a mandatory stats class but it didn't have anything concerning sample size picking. Ironically, the stats class I had to take when I was in the humanities dept. was more comprehensive. When I walked into industry, I realized after two or three years that practically everyone I'd met knew jack shit about statistics. I myself learned most of what I know from humanities-major friends and books like NLP by Manning and Schutze. Having been through what Zed's been through (on both the giving and receiving end to wit), I can empathize with why he'd get all sanctimonious about it, but I feel that it just makes the problem worse (at least in person). Programmers are already an egotistical lot, and I've learned that directly attacking their ego tends to make things worse. What can be done then? I don't know. It feels like this is at the core a personality problem, in particular one in which people associate their ability to know with their sense of self, and personality problems are terribly hard to correct.
- Periodic 17y ago
  I'm currently doing a CS masters and I did my undergrad in Physics. I'm really glad that I have the background in mathematics and stats that I have. In our physics classes they really drilled into us that although our theory and homework often used exact numbers, no measurement meant anything without a measurement of the error. "A number without error is meaningless", they would say. One thing I never fully understood was how to calculate the propagation of error. There are many useful tricks for reducing the impact of error on your final calculation, and a few things you need to watch to make sure it doesn't increase.
randome 17y ago
it seems pretty mind boggling that people are that bad at stats ... we have a mandatory stats class for CS ... it doesn't teach some of the stuff mentioned in the article but the stuff mentioned is completely intuitive -- you shouldn't need to take a stats class to understand that. like if the variance of your result is really high and you're estimating the mean its clear that the number of samples you take is important !?!
- megaduck 17y ago
  You're proving his point, which is that people's intuition is often dangerously wrong. The sample size is actually often nowhere near as important as your sampling method. For an extreme example, I can conduct a billion coin flips with a weighted coin, but it's going to tell me jack about the general behavior of coin flips. Avoiding sample bias is a hard problem, far harder than most people appreciate. To complicate matters further, "accuracy" is an extremely tricky concept in statistical analysis because error rates work very differently than they do in, say, physics. In physics, when you measure something you can be sure that your results are accurate, so long as you stay outside your instrument's range of error. In stats, your confidence interval just tells you how likely it is that your results are completely wrong, or even worse, wrong by a completely unknown amount. Every statistical inference you make has a chance of completely blowing up on you. That chance can be defined and reduced, but it can never be eliminated. There's also things like frequentist vs Bayesian statistics, where the interpretation of the same data can be completely different. Zed may be a jerk sometimes, but on this topic he's dead right. Most programmers are far more confident about this stuff than they should be.
  - megaduck 17y ago
    After doing some research, I think I've completely abused the notion of the confidence interval. Which I think just helps to prove Zed's point. This stuff be hard.
menloparkbum 17y ago
Data point: I am about Zed's age and majored in math and was one class away from a CS double major and I didn't have to take any stats courses. When I was in school stats was a totally different track than what most math people took. For some reason biology and business majors had to take stats but physics and math majors did not. I feel like the physics/math/CS people assumed stats was easy and you'd pick it up as needed.
voberoi 17y ago
I didn't, and I'm graduating with a BS in CS from Stanford in a few weeks. We have an introductory stats requirement, but I found that the stats class I took my sophomore year for my (dropped) economics major was far more enlightening than my CS stats requirement. I think the fundamental issue is that there's very little focus on statistical practice in most of these courses. Social scientists have it good: they're always taught how to deal with and interpret statistics in the same way they'll have to in their line of work. It's totally useless to throw a bunch of theory at students. I think that teaching statistics in the context of real problems is the only way (most) students will actually learn and come to appreciate how useful it is. The department revamped the major this year, and they've introduced a mandatory stats class tailored to CS students: cs109.stanford.edu. I haven't looked through it at all, but I think it's a step in the right direction. Finally, I have to give a shout-out to The Little Handbook of Statistical Practice (http://www.tufts.edu/~gdallal/LHSP.HTM http://www.tufts.edu/~gdallal/LHSP.HTM) in this thread. It's an amazing resource for anyone who works with statistics. I've referenced it while doing performance testing, building an A/B testing system, and working on problem sets. From the website: "My aim is to describe, for better or worse, what I do rather than simply present theory and methods as they appear in standard textbooks. This is about statistical practice--what happens when a statistician (me) deals with data on a daily basis." Read it now!
- lutorm 17y ago
  For a different view of how useful statistical practice (when applied mindlessly) is, read "The Black Swan". It's all about how people use statistical models with Normal distributions in places where it's patently unjustified, and the price we pay for it. I played this game myself with a friend. I sent him ten samples of a (for him) unknown distribution and asked him to estimate the mean. Then 100, then 1000. His estimate of the mean kept changing to higher and higher values, because the samples were drawn from a Pareto (power-law) distribution with a mean of 1000. Such a distribution is almost indistinguishable from one with a mean of infinity, because all the signal is in the very rare, large outliers. If you try to analyze samples from such a process assuming it's Gaussian, nothing will make sense, and the standard deviation will give you an estimated uncertainty of the mean that is far, far below the actual uncertainty.

stcredzero 17y ago

There's a big percentage of programmers who don't understand O(n^2). On multiple occasions, I've seemed as a guru because of this one little tidbit. What if the medical profession has as much egregious and widespread ignorance of the basics as programming? Would you be in favor of certification?

dschobel 17y ago
We have de facto certification. It's called moving to silicon valley :)
- stcredzero 17y ago
  And everywhere else, people have to put up with crappy programming? I think this system leaves a bit to be desired.
duncanj 17y ago
In medical professions, you go through a period of training (clinicals, internship, residency) where people with tons of experience point out that you never learned anything. This humbling experience is not really available in the CS world.
- stcredzero 17y ago
  Particularly disturbing, since the only good way I know of to learn programming and system architecture is through mentoring. Basically, our mentoring system is haphazard. There's even a lot of anti-mentoring happening out there.
SapphireSun 17y ago
I find this incredibly hard to believe. Anyone who has gone through a single intro course in CS should understand at least the difference between linear and quadratic time. Perhaps I've been in the university too long. Are these people non technical majors who picked up programming in PHP?

megaduck 17y ago

The problem isn't that people know nothing about statistics. The problem is that they think they know more than they actually do. It's fairly easy to get past your "intro to stats" class with only a passing familiarity with the subject and no deep understanding. Only once you truly grasp stats do you realize that you don't actually know anything. Any statistical formula will predict its own failure. That's kind of mind-blowing, and the implications aren't always grasped after just one semester (or quarter) studying the topic.

m_ 17y ago
I studied CS and I got past my "intro to stats" class with only a passing familiarity with the subject and no deep understanding (aside from the probability aspects of the course). The professor was very poor overall. I was able to cram enough to do well, but I retained very little after the final. I would love a deep understanding built from the foundations-- the coursework we had was a lot more obtuse and "take my word for it". In a sense, I felt I got a better foundation for statistics from a freshman sociology class than from the Math professor.
- stcredzero 17y ago
  Why aren't there statistics for CS classes that put stats primarily in the context of system profiling and metrics? Are there any out there?
  - cschep 17y ago
    That would have been cool. Stats wasn't required for us, but if it would have been Statistical Analysis of Computer Systems(or programs, or whatever), I would have signed up for it anyway. I heard the stats class offered where I went to school was a joke so I didn't pursue it optionally. I'm sure it's different depending on your university. Also, my school was not renowned for any sciences.
    - stcredzero 17y ago
      A really good textbook along these lines might really be something.

baguasquirrel 17y ago

At CMU we had a mandatory stats class but it didn't have anything concerning sample size picking. Ironically, the stats class I had to take when I was in the humanities dept. was more comprehensive. When I walked into industry, I realized after two or three years that practically everyone I'd met knew jack shit about statistics. I myself learned most of what I know from humanities-major friends and books like NLP by Manning and Schutze. Having been through what Zed's been through (on both the giving and receiving end to wit), I can empathize with why he'd get all sanctimonious about it, but I feel that it just makes the problem worse (at least in person). Programmers are already an egotistical lot, and I've learned that directly attacking their ego tends to make things worse. What can be done then? I don't know. It feels like this is at the core a personality problem, in particular one in which people associate their ability to know with their sense of self, and personality problems are terribly hard to correct.

Periodic 17y ago
I'm currently doing a CS masters and I did my undergrad in Physics. I'm really glad that I have the background in mathematics and stats that I have. In our physics classes they really drilled into us that although our theory and homework often used exact numbers, no measurement meant anything without a measurement of the error. "A number without error is meaningless", they would say. One thing I never fully understood was how to calculate the propagation of error. There are many useful tricks for reducing the impact of error on your final calculation, and a few things you need to watch to make sure it doesn't increase.

randome 17y ago

it seems pretty mind boggling that people are that bad at stats ... we have a mandatory stats class for CS ... it doesn't teach some of the stuff mentioned in the article but the stuff mentioned is completely intuitive -- you shouldn't need to take a stats class to understand that. like if the variance of your result is really high and you're estimating the mean its clear that the number of samples you take is important !?!

megaduck 17y ago
You're proving his point, which is that people's intuition is often dangerously wrong. The sample size is actually often nowhere near as important as your sampling method. For an extreme example, I can conduct a billion coin flips with a weighted coin, but it's going to tell me jack about the general behavior of coin flips. Avoiding sample bias is a hard problem, far harder than most people appreciate. To complicate matters further, "accuracy" is an extremely tricky concept in statistical analysis because error rates work very differently than they do in, say, physics. In physics, when you measure something you can be sure that your results are accurate, so long as you stay outside your instrument's range of error. In stats, your confidence interval just tells you how likely it is that your results are completely wrong, or even worse, wrong by a completely unknown amount. Every statistical inference you make has a chance of completely blowing up on you. That chance can be defined and reduced, but it can never be eliminated. There's also things like frequentist vs Bayesian statistics, where the interpretation of the same data can be completely different. Zed may be a jerk sometimes, but on this topic he's dead right. Most programmers are far more confident about this stuff than they should be.
- megaduck 17y ago
  After doing some research, I think I've completely abused the notion of the confidence interval. Which I think just helps to prove Zed's point. This stuff be hard.

menloparkbum 17y ago

Data point: I am about Zed's age and majored in math and was one class away from a CS double major and I didn't have to take any stats courses. When I was in school stats was a totally different track than what most math people took. For some reason biology and business majors had to take stats but physics and math majors did not. I feel like the physics/math/CS people assumed stats was easy and you'd pick it up as needed.

voberoi 17y ago

I didn't, and I'm graduating with a BS in CS from Stanford in a few weeks. We have an introductory stats requirement, but I found that the stats class I took my sophomore year for my (dropped) economics major was far more enlightening than my CS stats requirement. I think the fundamental issue is that there's very little focus on statistical practice in most of these courses. Social scientists have it good: they're always taught how to deal with and interpret statistics in the same way they'll have to in their line of work. It's totally useless to throw a bunch of theory at students. I think that teaching statistics in the context of real problems is the only way (most) students will actually learn and come to appreciate how useful it is. The department revamped the major this year, and they've introduced a mandatory stats class tailored to CS students: cs109.stanford.edu. I haven't looked through it at all, but I think it's a step in the right direction. Finally, I have to give a shout-out to The Little Handbook of Statistical Practice (http://www.tufts.edu/~gdallal/LHSP.HTM http://www.tufts.edu/~gdallal/LHSP.HTM) in this thread. It's an amazing resource for anyone who works with statistics. I've referenced it while doing performance testing, building an A/B testing system, and working on problem sets. From the website: "My aim is to describe, for better or worse, what I do rather than simply present theory and methods as they appear in standard textbooks. This is about statistical practice--what happens when a statistician (me) deals with data on a daily basis." Read it now!

lutorm 17y ago
For a different view of how useful statistical practice (when applied mindlessly) is, read "The Black Swan". It's all about how people use statistical models with Normal distributions in places where it's patently unjustified, and the price we pay for it. I played this game myself with a friend. I sent him ten samples of a (for him) unknown distribution and asked him to estimate the mean. Then 100, then 1000. His estimate of the mean kept changing to higher and higher values, because the samples were drawn from a Pareto (power-law) distribution with a mean of 1000. Such a distribution is almost indistinguishable from one with a mean of infinity, because all the signal is in the very rare, large outliers. If you try to analyze samples from such a process assuming it's Gaussian, nothing will make sense, and the standard deviation will give you an estimated uncertainty of the mean that is far, far below the actual uncertainty.