6 ms·

Programmers Need To Learn Statistics Or I Will Kill Them All

130 points by jmonegro 17y ago

dylanz 17y ago
It's a great post, albeit, probably about 3 years old.
- deleted 17y ago
  [deleted]
manvsmachine 17y ago
In case you're interested in the (lengthy) original discussion: http://news.ycombinator.com/item?id=626771 http://news.ycombinator.com/item?id=626771 EDIT: that wasn't the original, but it was the longest. This is the original (I think): http://news.ycombinator.com/item?id=48006 http://news.ycombinator.com/item?id=48006
- jmonegro 17y ago
  It's a really good article - I came across it today on my twitter feed, and it surprised me that it still has the same impact it had in 2005.
amoeba 17y ago
Statistics will save us all. And for the HN folks that like to Lisp it up: http://incanter.org/ http://incanter.org/
smanek 17y ago
I strongly disagree with Zed on this one (at least part of his diatribe - he makes some good points about sample size/ramp-up/etc). I don't care what your mean or standard deviation are. If your performance isn't symmetrically distributed about the mean (and it probably isn't), standard deviation probably doesn't mean what you expect. I only look at performance in 99th percentile (or sometimes, 99.9th percentile). If 99% of responses are being served in 100ms on my webapp, I'm happy. I don't care if the mean is 5ms with a 20ms standard deviation. As long as the performance at the 99th (or 99.9th) percentile meets a level of 'acceptability' that I have predetermined, I'm satisfied. And you should be too. This is much easier to deal with than trying to do real math. Just make sure that no more than 1% (or 0.1%, as the case may be) of requests take longer than X. Pre-emptive answer: I usually get someone asking me why I don't demand acceptable performance at the 99.99999th percentile or at the slowest request. It's simply an issue of resources. To get that kind of performance, you can't use garbage collection, context switches become an issue, disk buffering even comes into play. You have to basically write a hard real-time (or pseudo real-time) app, and that level of effort isn't worth it for a consumer facing webapp. If I was writing code for a pacemaker or something (and please, dear god, no one ever let me do that) it damn sure better be hard real-time though.
- Locke1689 17y ago
  I'm pretty sure your system is useless without a statistically significant sample size.
  - jerf 17y ago
    Sampling is not relevant to smanek's point, since he can look at all events. You don't need to sample when you have all the data. On the other hand, I guess this argues against smanek's points being relevant to Zed's point, since statistics aren't really involved for the same reason. You don't need stats to analyze your performance if you have all the performance data; just read it off. (Edit: I suppose his point that you shouldn't be using averages but just thresholds is still a statistical argument, against using the wrong tool like "average" on the grounds that you don't actually want the "average".)
    - lmkg 17y ago
      Even if you have all the data of your current run, that's still a sample (broadly speaking) of all the possible runs in the future. This is one of the core aspects of statistics, but it's so fundamental that it gets glossed over rather than getting drilled into people's head like they need. As a result, I have to tell people that when running a survey, total responses are more important than response rate.
- viraptor 17y ago
  I can't really agree with you. If you care only about 99.9% of your requests being on time, it means that one in a 1000 can time-out and die. Not that bad? Ok - let's have an example: Facebook - one random user-page view == 63 requests (reported by firebug). That means 16 people generate >1000 requests. Even with 99.9% requests within your "acceptable range", you get every 16'th person either missing an element on a page due to some DB timeout (for example), or seeing an incomplete page for a not "acceptable" time (with 1/63'th probability of it being the main document, if we don't know which element fails). Is that really an acceptable behaviour? I don't believe that a requirement saying "the complete website loads in less than 1s" is affected by GC or context switches at all, tbh. If you actually tested the website with load of 10x the average expected one (over a couple of hours of random scripted walk with a natural write/read balance - testing you'd do on a service you really care about) and you still get random glitches with the expected load, then there's most likely something wrong on a higher level (disk seeks failing? shared hosting overloaded? network? etc. etc.)
thisisnotmyname 17y ago
This is a great resource if you're studying / reviewing on your own: http://spartan.ac.brocku.ca/~jvrbik/MATH2P82/Statistics.PDF http://spartan.ac.brocku.ca/~jvrbik/MATH2P82/Statistics.PDF
Perceval 17y ago
I'm trying to teach myself statistics (for social sciences) right now. I got two books (Agresti & Finlay along with Knoke, Bohrnstedt & Mee) but both seem to be written in jargon instead of English. Does anyone have recommendations for statistics books that are actually written in English?
- pibefision 17y ago
  Mario Triola's book is excellent. Covers all the topics with very clear explanations. Elementary Statics 10th Edition.
- jbl 17y ago
  The Cartoon Guide to Statistics by Gonick and Smith is actually quite good for dipping your toes in. It should give you an okay conceptual foundation from which you can tackle the textbooks you've already got.
  - silentbicycle 17y ago
    I was just going to recommend this, too. It's hokey as hell, but it's actually a great intro -- After reading it, you'll know the vocabulary and fundamentals well enough to ask meaningful questions. It's also cheap, and a quick read. I just loaned my copy to my little brother for his stats class, and I'm betting he'll find it more useful than the dreadful (and expensive) intro textbook his professor chose.
  - hadley 17y ago
    I'd second the recommendation. But avoid the manga guide to statistics - it's nowhere near as good.
- kqr2 17y ago
  Zed actually recommends quite a few books in his post: * Statistics; by Freedman, Pisani, Purves, and Adhikari. Norton publishers. * Introductory Statistics with R; by Dalgaard. Springer publishers. * Statistical Computing: An Introduction to Data Analysis using S-Plus; by Crawley. Wiley publishers. * Statistical Process Control; by Grant, Leavenworth. McGraw-Hill publishers. * Statistical Methods for the Social Sciences; by Agresti, Finlay. Prentice-Hall publishers. * Methods of Social Research; by Baily. Free Press publishers. * Modern Applied Statistics with S-PLUS; by Venables, Ripley. Springer publishers.
AlphaMonkey 17y ago
Zed Shaw should stick to his area of expertise. His angry, opinionated rants on topics he clearly lacks deep knowledge of are perfect examples of the pseudo-intellectual junk that litters HN from time to time.
- Raphael_Amiard 17y ago
  thanks for that argumentated non opinionated answer !
  - AlphaMonkey 17y ago
    At least I don't write like a slightly retarded teenager, and I don't pretend to be an expert on Statistics. Thanks for the "kudos" anyways.
    - Raphael_Amiard 17y ago
      Sorry for the sarcastic answer, never a good thing. My point was, despite Zed's tone , that i found irritating, he does actually provide content in this article, that a newbie like me can use and put at profit (wich i did when the article first came out). On the other hand, your post sounds angry, gives no insight as to why / how he lacks knowledge (which would have been interresting) and use condescending expressions like "pseudo intellectual garbage"
      - mariorz 17y ago
        > he does actually provide content in this article, that a newbie like me can use and put at profit (wich i did when the article first came out). what could you possibly use from this article? if you're really interested in learning stats concepts and tools you can be better served by blogs with actual content. here's a good one: http://incanter-blog.org/ http://incanter-blog.org/
        Pistos2 17y ago
        Vb > Va does not imply that Va == 0. We cannot conclude that A has no value just because B has more value than A.
asb 17y ago
Just thought I'd share an example where the graphs really help you to understand the performance profile. http://s3.amazonaws.com/four.livejournal/20090911/benchmark.html http://s3.amazonaws.com/four.livejournal/20090911/benchmark....
jrockway 17y ago
Hacker News readers need to stop submitting linkbait titles, or I will kill them all.
- hyperbovine 17y ago
  Quoi? That is the verbatim title of the post.
  - jrockway 17y ago
    That means you took the linkbait. Why reward articles like this with free views?
    - raganwald 17y ago
      If the article delivers on the hype, why not reward the author for having the walk to match the talk?
      - jrockway 17y ago
        Does this article deliver on the hype? What I got out of it was: "Some people are dumber than me." This is a well-known problem both inside and outside the programming world, and unfortunately killing the people dumber than you is not the solution to the problem.
        raganwald 17y ago
        I took it as "some people are ignorant of basic statistics," which isn't quite as daunting as "some people are dumber than me." The cure is motivation to learn, and one can take the view that the hyperbole of the title broadens the post's reach and simultaneously provides a little negative motivation. I didn't take the title quite that literally. I personally take the view that you catch more flies with honey than vinegar, but I suggest the article as a whole including the title provides a general good.
- zedshaw 17y ago
ez77 17y ago
P = "Programmers Need To Learn Statistics" Q = "Zed Will Kill All Programmers" If P || Q must hold, go with P!
- deleted 17y ago
  [deleted]
stcredzero 17y ago
This article is my call for all programmers to finally learn enough about statistics to at least know they don’t know shit. His call can be generalized: "This article is my call for all programmers to finally learn enough about [insert here] to at least know they don’t know shit." The power that we programmers have as little gods inside systems of our own making, tends to have us over-estimate how much we know. Thus our penchant for posting stuff that makes experts in various technical and scientific fields roll their eyes.
MikeTLive 17y ago
once again slashdot posts hackernews as "new" news. would be nice if they linked HERE to gain from the new comments not the 2.5yr old comments. oh well. http://developers.slashdot.org/story/10/01/09/2154224/Why-Programmers-Need-To-Learn-Statistics http://developers.slashdot.org/story/10/01/09/2154224/Why-Pr...
sireat 17y ago
Two great books for those who need to re/freshen up their statistics: http://www.amazon.com/How-Lie-Statistics-Darrell-Huff/dp/0393310728 http://www.amazon.com/How-Lie-Statistics-Darrell-Huff/dp/039... http://www.amazon.com/Cartoon-Guide-Statistics-Larry-Gonick/dp/0062731025 http://www.amazon.com/Cartoon-Guide-Statistics-Larry-Gonick/... I haven't had a chance to check out the manga guide to statistics but that might be a decent introduction, as well.

130 points by jmonegro 17y ago

dylanz 17y ago

It's a great post, albeit, probably about 3 years old.

deleted 17y ago
[deleted]

manvsmachine 17y ago

In case you're interested in the (lengthy) original discussion: http://news.ycombinator.com/item?id=626771 http://news.ycombinator.com/item?id=626771 EDIT: that wasn't the original, but it was the longest. This is the original (I think): http://news.ycombinator.com/item?id=48006 http://news.ycombinator.com/item?id=48006

jmonegro 17y ago
It's a really good article - I came across it today on my twitter feed, and it surprised me that it still has the same impact it had in 2005.

amoeba 17y ago

Statistics will save us all. And for the HN folks that like to Lisp it up: http://incanter.org/ http://incanter.org/

smanek 17y ago

I strongly disagree with Zed on this one (at least part of his diatribe - he makes some good points about sample size/ramp-up/etc). I don't care what your mean or standard deviation are. If your performance isn't symmetrically distributed about the mean (and it probably isn't), standard deviation probably doesn't mean what you expect. I only look at performance in 99th percentile (or sometimes, 99.9th percentile). If 99% of responses are being served in 100ms on my webapp, I'm happy. I don't care if the mean is 5ms with a 20ms standard deviation. As long as the performance at the 99th (or 99.9th) percentile meets a level of 'acceptability' that I have predetermined, I'm satisfied. And you should be too. This is much easier to deal with than trying to do real math. Just make sure that no more than 1% (or 0.1%, as the case may be) of requests take longer than X. Pre-emptive answer: I usually get someone asking me why I don't demand acceptable performance at the 99.99999th percentile or at the slowest request. It's simply an issue of resources. To get that kind of performance, you can't use garbage collection, context switches become an issue, disk buffering even comes into play. You have to basically write a hard real-time (or pseudo real-time) app, and that level of effort isn't worth it for a consumer facing webapp. If I was writing code for a pacemaker or something (and please, dear god, no one ever let me do that) it damn sure better be hard real-time though.

Locke1689 17y ago
I'm pretty sure your system is useless without a statistically significant sample size.
- jerf 17y ago
  Sampling is not relevant to smanek's point, since he can look at all events. You don't need to sample when you have all the data. On the other hand, I guess this argues against smanek's points being relevant to Zed's point, since statistics aren't really involved for the same reason. You don't need stats to analyze your performance if you have all the performance data; just read it off. (Edit: I suppose his point that you shouldn't be using averages but just thresholds is still a statistical argument, against using the wrong tool like "average" on the grounds that you don't actually want the "average".)
  - lmkg 17y ago
    Even if you have all the data of your current run, that's still a sample (broadly speaking) of all the possible runs in the future. This is one of the core aspects of statistics, but it's so fundamental that it gets glossed over rather than getting drilled into people's head like they need. As a result, I have to tell people that when running a survey, total responses are more important than response rate.
viraptor 17y ago
I can't really agree with you. If you care only about 99.9% of your requests being on time, it means that one in a 1000 can time-out and die. Not that bad? Ok - let's have an example: Facebook - one random user-page view == 63 requests (reported by firebug). That means 16 people generate >1000 requests. Even with 99.9% requests within your "acceptable range", you get every 16'th person either missing an element on a page due to some DB timeout (for example), or seeing an incomplete page for a not "acceptable" time (with 1/63'th probability of it being the main document, if we don't know which element fails). Is that really an acceptable behaviour? I don't believe that a requirement saying "the complete website loads in less than 1s" is affected by GC or context switches at all, tbh. If you actually tested the website with load of 10x the average expected one (over a couple of hours of random scripted walk with a natural write/read balance - testing you'd do on a service you really care about) and you still get random glitches with the expected load, then there's most likely something wrong on a higher level (disk seeks failing? shared hosting overloaded? network? etc. etc.)

thisisnotmyname 17y ago

This is a great resource if you're studying / reviewing on your own: http://spartan.ac.brocku.ca/~jvrbik/MATH2P82/Statistics.PDF http://spartan.ac.brocku.ca/~jvrbik/MATH2P82/Statistics.PDF

Perceval 17y ago

I'm trying to teach myself statistics (for social sciences) right now. I got two books (Agresti & Finlay along with Knoke, Bohrnstedt & Mee) but both seem to be written in jargon instead of English. Does anyone have recommendations for statistics books that are actually written in English?

pibefision 17y ago
Mario Triola's book is excellent. Covers all the topics with very clear explanations. Elementary Statics 10th Edition.
jbl 17y ago
The Cartoon Guide to Statistics by Gonick and Smith is actually quite good for dipping your toes in. It should give you an okay conceptual foundation from which you can tackle the textbooks you've already got.
- silentbicycle 17y ago
  I was just going to recommend this, too. It's hokey as hell, but it's actually a great intro -- After reading it, you'll know the vocabulary and fundamentals well enough to ask meaningful questions. It's also cheap, and a quick read. I just loaned my copy to my little brother for his stats class, and I'm betting he'll find it more useful than the dreadful (and expensive) intro textbook his professor chose.
- hadley 17y ago
  I'd second the recommendation. But avoid the manga guide to statistics - it's nowhere near as good.
kqr2 17y ago
Zed actually recommends quite a few books in his post: * Statistics; by Freedman, Pisani, Purves, and Adhikari. Norton publishers. * Introductory Statistics with R; by Dalgaard. Springer publishers. * Statistical Computing: An Introduction to Data Analysis using S-Plus; by Crawley. Wiley publishers. * Statistical Process Control; by Grant, Leavenworth. McGraw-Hill publishers. * Statistical Methods for the Social Sciences; by Agresti, Finlay. Prentice-Hall publishers. * Methods of Social Research; by Baily. Free Press publishers. * Modern Applied Statistics with S-PLUS; by Venables, Ripley. Springer publishers.

AlphaMonkey 17y ago

Zed Shaw should stick to his area of expertise. His angry, opinionated rants on topics he clearly lacks deep knowledge of are perfect examples of the pseudo-intellectual junk that litters HN from time to time.

Raphael_Amiard 17y ago
thanks for that argumentated non opinionated answer !
- AlphaMonkey 17y ago
  At least I don't write like a slightly retarded teenager, and I don't pretend to be an expert on Statistics. Thanks for the "kudos" anyways.
  - Raphael_Amiard 17y ago
    Sorry for the sarcastic answer, never a good thing. My point was, despite Zed's tone , that i found irritating, he does actually provide content in this article, that a newbie like me can use and put at profit (wich i did when the article first came out). On the other hand, your post sounds angry, gives no insight as to why / how he lacks knowledge (which would have been interresting) and use condescending expressions like "pseudo intellectual garbage"
    - mariorz 17y ago
      > he does actually provide content in this article, that a newbie like me can use and put at profit (wich i did when the article first came out). what could you possibly use from this article? if you're really interested in learning stats concepts and tools you can be better served by blogs with actual content. here's a good one: http://incanter-blog.org/ http://incanter-blog.org/
      - Pistos2 17y ago
        Vb > Va does not imply that Va == 0. We cannot conclude that A has no value just because B has more value than A.

asb 17y ago

Just thought I'd share an example where the graphs really help you to understand the performance profile. http://s3.amazonaws.com/four.livejournal/20090911/benchmark.html http://s3.amazonaws.com/four.livejournal/20090911/benchmark....

jrockway 17y ago

Hacker News readers need to stop submitting linkbait titles, or I will kill them all.

hyperbovine 17y ago
Quoi? That is the verbatim title of the post.
- jrockway 17y ago
  That means you took the linkbait. Why reward articles like this with free views?
  - raganwald 17y ago
    If the article delivers on the hype, why not reward the author for having the walk to match the talk?
    - jrockway 17y ago
      Does this article deliver on the hype? What I got out of it was: "Some people are dumber than me." This is a well-known problem both inside and outside the programming world, and unfortunately killing the people dumber than you is not the solution to the problem.
      - raganwald 17y ago
        I took it as "some people are ignorant of basic statistics," which isn't quite as daunting as "some people are dumber than me." The cure is motivation to learn, and one can take the view that the hyperbole of the title broadens the post's reach and simultaneously provides a little negative motivation. I didn't take the title quite that literally. I personally take the view that you catch more flies with honey than vinegar, but I suggest the article as a whole including the title provides a general good.
zedshaw 17y ago

ez77 17y ago

P = "Programmers Need To Learn Statistics" Q = "Zed Will Kill All Programmers" If P || Q must hold, go with P!

deleted 17y ago
[deleted]

stcredzero 17y ago

This article is my call for all programmers to finally learn enough about statistics to at least know they don’t know shit. His call can be generalized: "This article is my call for all programmers to finally learn enough about [insert here] to at least know they don’t know shit." The power that we programmers have as little gods inside systems of our own making, tends to have us over-estimate how much we know. Thus our penchant for posting stuff that makes experts in various technical and scientific fields roll their eyes.

MikeTLive 17y ago

once again slashdot posts hackernews as "new" news. would be nice if they linked HERE to gain from the new comments not the 2.5yr old comments. oh well. http://developers.slashdot.org/story/10/01/09/2154224/Why-Programmers-Need-To-Learn-Statistics http://developers.slashdot.org/story/10/01/09/2154224/Why-Pr...

sireat 17y ago

Two great books for those who need to re/freshen up their statistics: http://www.amazon.com/How-Lie-Statistics-Darrell-Huff/dp/0393310728 http://www.amazon.com/How-Lie-Statistics-Darrell-Huff/dp/039... http://www.amazon.com/Cartoon-Guide-Statistics-Larry-Gonick/dp/0062731025 http://www.amazon.com/Cartoon-Guide-Statistics-Larry-Gonick/... I haven't had a chance to check out the manga guide to statistics but that might be a decent introduction, as well.