8 ms·
Programmers Need To Learn Statistics Or I Will Kill Them All (2005)
- bena 17y agoI think I've read this before when Zed was "so fucking awesome". Is his whole "dropping the persona" gig just a way to get more mileage out of old articles?
- edmccaffrey 17y agoHis favorite blog posts are being reposted as essays.
- petercooper 17y agoHe's the new PG, with most of the wisdom replaced by machismo.
- e4m 17y agoMa Kiz e Moe
- deleted 17y ago[deleted]
- deleted 17y ago[deleted]
- tokenadult 17y agoThe article does seem familiar, and I think I've even seen it on HN before. http://news.ycombinator.com/item?id=48006 http://news.ycombinator.com/item?id=48006 Yep. (There seems to be a slight change in the base URL for the submitted article, but this has been discussed here before.) Still this is well worth discussing again. It's amazing how much most college-educated people think they know about statistics that they really don't. Two of my favorite quick overview articles about statistics, both by Ph.D. professors of statistics, are "Advice to Mathematics Teachers on Evaluating Introductory Statistics Textbooks" http://statland.org/MAAFIXED.PDF http://statland.org/MAAFIXED.PDF (the link is dead because of site maintenance now, but should be fixed soon) and "The Introductory Statistics Course: A Ptolemaic Curriculum?" http://repositories.cdlib.org/cgi/viewcontent.cgi?article=1002&context=uclastat/cts/tise http://repositories.cdlib.org/cgi/viewcontent.cgi?article=10... Both are thought-provoking articles about what usually isn't taught to undergraduates about statistics.
- pasbesoin 17y agoWayback has a copy of MAAFIXED.PDF (I'm not sure whether it's appropriate / a good idea to include the link).
- deleted 17y ago[deleted]
- stcredzero 17y agoReminds me of the film clip Alan Kay showed at one of his talks. It was taken at a Harvard graduation, and it shows many, many Harvard graduates, students, and faculty saying that the Earth was warmer in Summer because the orbit takes it closer to the sun. http://www.sigchi.org/chi98/chikids/vol2/headlines/head2.html http://www.sigchi.org/chi98/chikids/vol2/headlines/head2.htm...
- travisjeffery 17y agoIt's caused by tilt/angle of Earth, correct?
- deleted 17y ago[deleted]
- edu 17y agoSo after Ruby and Ruby on Rails, Zed has learnt some R and statistics. Because after reading almost all the article, he basically talks about: average, median, standard deviation. He should learn a little bit of complexity and calculability by the way, best case, worst case, Big O notation, etc.
- chollida1 17y ago> So after Ruby and Ruby on Rails, Zed has learnt some R and statistics. From the article it sounds like he learned stats first. > studied statistics in grad school,
- jerf 17y agoBecause when edu writes a blog post, he includes his entire life history, and, for that matter, the entire history of the universe in it. I know you've already been smacked down but this is a big pet peeve of mine and worth pointing out. You can always complain that something should have had more information in it, and since that is always true no matter what, the complaint is information free. (A value that comes from a universe of one value contributes zero bits of information.)
- SapphireSun 17y agoI'm not so sure about that. A request for more information can be incredibly important (for instance: Tell me who committed the murder! also see: stubs on Wikipedia). Essentially, there is a point of diminishing returns, and after a while, possibly negative returns as the work becomes too hard to assimilate. However, in this case, I tend to think that Zed included just enough information to get someone who's clueless started - he even included references at the end so that if you DO want more, you can easily get it.
- trezor 17y agoI can see some people here claiming this is old (it is) and that it's a repost (it is), but I'll still vote this up, because especially now with people yelling "scaling! scaling!" all over the place, I can't imagine a more fitting time for developers to read this.
- Tamerlin 17y agoScaling? People keep using that word. I don't think it means what they think it means.
- dschobel 17y agoSerious question: who here who has at least a BS in CS didn't have a mandatory stats class where you learned all about picking sample sizes to give you an accuracy you're happy with or that an average without a standard deviation is nigh-meaningless? It's just a fact of our profession, there is a significant percentage of people who just slap together APIs and have zero understanding of the maths behind it. I don't see why it takes Zed 1000 words to say it or why he has to get sanctimonious about it.
- stcredzero 17y agoThere's a big percentage of programmers who don't understand O(n^2). On multiple occasions, I've seemed as a guru because of this one little tidbit. What if the medical profession has as much egregious and widespread ignorance of the basics as programming? Would you be in favor of certification?
- dschobel 17y agoWe have de facto certification. It's called moving to silicon valley :)
- stcredzero 17y agoAnd everywhere else, people have to put up with crappy programming? I think this system leaves a bit to be desired.
- duncanj 17y agoIn medical professions, you go through a period of training (clinicals, internship, residency) where people with tons of experience point out that you never learned anything. This humbling experience is not really available in the CS world.
- stcredzero 17y agoParticularly disturbing, since the only good way I know of to learn programming and system architecture is through mentoring. Basically, our mentoring system is haphazard. There's even a lot of anti-mentoring happening out there.
- msluyter 17y agoI hadn't seen this, so I was glad it was resubmitted. I think he has a fair point. Here on HN I see a lot of armchair sociologists critique the various articles in the social sciences that get posted, but it's rather unclear to me whether these are well grounded in an actual understanding of the issues involved, or simply habitual incantations of rules of thumb such as "correlation doesn't imply causation."
- nazgulnarsil 17y agoum...rules of thumb are a bit different from identifying logical fallacies.
- Freaky 17y agoSpeaking of statistics, FreeBSD comes with a tool called "ministat", which accepts a bunch of input files filled with numbers and tells you how statistically significant the difference is between them. It's frequently used to demonstrate performance improvements in kernel code: http://www.freebsd.org/cgi/man.cgi?query=ministat&apropos=0&sektion=0&manpath=FreeBSD+8-current&format=html http://www.freebsd.org/cgi/man.cgi?query=ministat&apropo...
- stcredzero 17y agoOh, and you wonder why I say, “he”? I never have this problem with female programmers...I think women are better programmers because they have less ego and are typically more interested in the gear rather than the pissing contest. When I was hanging out in Homer Alaska, squatting in the tent city on the beach, I heard that on the halibut fishing tours, the women would catch more and bigger fish. (And halibut are big fish. A 25 pound halibut is puny. You can catch 300 pound halibut, and it's not a super-rare event.) This is because the husbands would bring their fishing egos with them from the lower 48 states and not listen to the deckhands. But the wives, with no preconceived ideas, would listen carefully and do things the right way to catch halibut. (As opposed to trout.)
- biohacker42 17y agoI wonder if this is a "feature" of women or all minorities? For example, do male nurses show the same egolessness as female programmers?
- stcredzero 17y agoAs a minority Irish musician, I find that this seems to have a dampening effect on my ego. (Not 100% sufficient to keep me from making a fool of myself occasionally, though.) It's also caused me to be very, very paranoid about whether or not I "get it" and as a result I've had bouts of beneficial studiousness. And yes, I have to do better to win respect, and sometimes I am judged by my skin color, not my actual playing.
- nailer 17y agoI thought Irish people were renowned for their musicality. Eg, singing is a common social activity in Ireland (to the point where it's socially unacceptable not to sing), and Ireland has produced far more commercially successful musicians than their population would suggest. What country are you playing in?
- menloparkbum 17y ago
- chanux 17y agoI always hated stats lecture :( . But the book "Statistics for utterly confused" helped me in exams. But still I'm not in love with stats. Would he kill me?
- Tichy 17y agoI don't like your chances.
- jhancock 17y agoI've read this before. Zed does not provide any details on learning or understanding statistics in this rant.
- billswift 17y agoStatistics and probability is a major course of study - far more than could be usefully taught in a blog. What he tries to do is to motivate people to learn it - which is the first major step to actually learning it.
- jhancock 17y agoahh..the old "motivation by calling people stupid" trick ;)
- ironkeith 17y agoYou know, I've read a bunch of Zed's articles, and I always end up thinking that he's the source of most of his problems. There was a talk I saw him give (can't find the link) to a bunch of college kids where he basically said "phone in your job, do the stuff you love on your own time, everyone else is retarded and they'll never understand you." Great attitude you have there. I know guys like that; guys who are extremely egotistical, are always right, and know everything about everything. They're team killing, energy sucking, wastes. He may be able to hack like a dream, and "thanks for Mongrel" and all, but I wouldn't even want to be in the same room as him. I think from now on I'll do my best to ignore Zed's perspectives on life; they're more than a little skewed.
- batasrki 17y agoThe talk is from CUSEC, probably this one: http://vimeo.com/2723800 http://vimeo.com/2723800 As for his "attitude", I've actually met him and talked to him face-to-face. His internet persona is a put-on and he only talks like that to piss people off. In actuality, he's a really nice guy who is super-smart and who does not take shit from people. It's actually too bad that there aren't more of him in our industry. Scope creeps, memory leaks, etc. would be a thing of the past.
- ironkeith 17y agoYou are who your project yourself as, and to the billions of people who haven't met him who may stumble across his stuff online, he comes across as a selfish dick. IRL he could be happiness and teddy bears stuffed with sunshine, but so far as I'm concerned if the industry was full of people like him it would be in ruins. Also, that was the exact video I was talking about, so thanks.
- PonyGumbo 17y agoThanks for posting this. I went from despising Shaw (based on his writings) to kind of liking him.
- mahmud 17y ago
- grandalf 17y agoI think a better way to help would have been to spend the time writing a blog post introducing some core stats concepts and showing how to use R to do useful things...
- callmeed 17y agoCan anyone recommend a decent (read "not boring") book on statistics? Something along the lines of "Naked Economics" for the stats realm ...
- sp332 17y agoThere's the Cartoon Guide to Statistics ( http://www.amazon.com/Cartoon-Guide-Statistics-Larry-Gonick/dp/0062731025 http://www.amazon.com/Cartoon-Guide-Statistics-Larry-Gonick/... ), which, despite its name, is pretty solid and comprehensive book on basics. And definitely not boring!
- billswift 17y agoHow to Lie with Statistics (http://www.amazon.com/How-Lie-Statistics-Darrell-Huff/dp/0393310728/ref=sr_1_1?ie=UTF8&s=books&qid=1243384018&sr=8-1 http://www.amazon.com/How-Lie-Statistics-Darrell-Huff/dp/039...) is a short, enjoyable read. It doesn't tell you how to do statistics, but it gives some warning about common problems.
- srn 17y agoAlso the manga guide to statistics http://nostarch.com/mg_statistics.htm http://nostarch.com/mg_statistics.htm
- nkurz 17y agoI don't like this article. Yes, he's right: an intuitive knowledge of the relationship between averages and standard deviations is essential. But to presume that the standard tools of statistics should be applied to every problem is to miss the point. On a superficial level, if you are doing overnight processing of log files, then you probably care more about throughput than latency. In this case, averages are probably a fine metric. On a slightly deeper level, standard deviation is only a useful measure if the distribution is known, and in a lot of real world cases it is not. The right question isn't whether 100 or 1000 tests on the same data provides sufficient statistical power, but whether range of inputs is sufficient to trigger worst case perfomance. Now, I presume that Zed knows these things and applies them appropriately, but the article strikes me as more snide than helpful. Perhaps as others say he's a great guy in person, but I prefer my stats with less attitude and more insight. Here, for example: http://yudkowsky.net/rational/bayes http://yudkowsky.net/rational/bayes [edit: changed my sloppy language from 'has no meaning unless to the distribution is normal' to 'is only a useful measure if the distribution is known']
- tokenadult 17y agoOn a slightly deeper level, standard deviation only has meaning if the distribution is presumed to be normal Are you completely sure about that? I suppose many readers of this thread are more knowledgeable about statistics than I am. I would appreciate hearing from the knowledgeable readers whether or not variance in the observed values makes a difference in the cases discussed in the submitted article.
- nkurz 17y agoAre you completely sure about that? You could quibble about my exact phrasing, but yes, I'm completely sure about that. This is the 'black swan' problem writ small. I don't mean that a high standard deviation should be ignored for real-world distributions, but I do mean that a low standard deviation carries very little weight unless a normal distribution is presumed. I'm hard pressed to relate this to the cases discussed in the article, as those cases are shy on detail, but the DB2 example seems most applicable. Although he points to standard deviation as the tell-tale flag here, this is sort of misleading. The exact numerical value for the standard deviation across all queries is meaningless here, as not every query has an equal likelihood of being slow. As he states, the real problem was the terrible performance of an single query. How many similar queries exist? Will a new query added to the system trigger a similar bug? We don't know, and standard statistics isn't going to help us unless we have an understanding of the underlying mechanism. The key here is not to test a statistically significant subset of all possible queries, but to check the performance of the actual queries executed (as he did).
- WilliamLP 17y ago> I think women are better programmers because they have less ego and are typically more interested in the gear rather than the pissing contest. People need to stop writing shit like this. If it's not okay to say that men are better programmers simply because of their gender (it's not), then it's not okay to say it the other way around either.
- tsally 17y agoExactly. Why not just say that ego tends to prohibit good programmers? Bringing gender into it is hardly necessary, and it actually detracts from the important point about ego.
- cellis 17y agoThen again, Zed Shaw isn't exactly known for being PC.
- tjic 17y ago> If it's not okay to say that men are better programmers simply because of their gender (it's not), then it's not okay to say it the other way around either. God forbid that someone make an argument and try to support it with data, when one person has already decided for all of us what the correct conclusion is. WT* ever happened to debate, intellectual discourse, and the marketplace of ideas? This is one of the things that I find most disgusting about political correctness: that it tries to just wall off huge swaths of POTENTIAL CONCLUSIONS based on an argument that boils down to a misconstrued sense of manners (at best) or political preferences (at worst). Want to say that ethanol is a stupid idea? THE DEBATE IS CLOSED - NO SERIOUS SCIENTIST BELIEVES THAT GLOBAL WARMING IS ANYTHING OTHER THAN A THREAT OF EXTINCTION. Want to say that women and men have (a) different average heights; (b) different standard deviations in intelligence; (c) different hormone levels; (d) massively different thicknesses in their corpus collosums, and therefore one or the other might on average, make better programmers/accountants/engineers? THE DEBATE IS CLOSED. IT IS UNACCEPTABLE TO SPECULATE ON THIS TOPIC. I call bullshit on that. Intellectually honest people respond to facts and arguments with OTHER facts and arguments. Intellectually dishonest people try to shut down debates using social control. WilliamLP writes "People need to stop writing shit like this" That's the phrase of a bully, and/or a censor.
- jfarmer 17y agoThat's a lot of text to say not much more than "Standard deviation can be as important as the mean; be careful about confounding variables; and if you're an engineer, spend more time learning statistics." Oh, also, "You're all assholes and I rock." Truly the Carl Sagan of software.
- kl4m 17y agoWhy is he mixing computer science with programming? Computer science is just a branch of mathematics. It has not much to do with what he talks about afterwards.
- billswift 17y agoComputer science is to programming as physics is to engineering. The way SOME programmers constantly put down computer science, especially academic computer science, is a sign of the anti-professionalism in the field. (I decided I better emphasize some programmers since they actually seem to be a very vocal minority.)
- randome 17y agopat on the back for you -- you understand intro college statistics... maybe you just work with shitty programmers that don't understand statistics????? i dunno but most of my colleagues know stats pretty damn well. but maybe thats just because i'm in school
- enki 17y agoi think we'll have a huge shortage of programmers shortly
- voberoi 17y agoI think this article is great once you get past the rant-iness of it. He makes a bunch of valid points and it's true: a lot people actually don't take these pretty crucial details into account when they're dealing with statistics. It's kind of a shame that his message is diluted by its delivery. Case in point: about half of the comments here aren't even about statistics!
- ThomPete 17y ago99% of all statistics are made up
- nathanwdavis 17y agoI love it when Zed Shaw gets pissed. It's fun!!