9 ms·

Fact and folklore in software engineering

135 points by Morendil 16y ago

pg 16y ago
When I found myself reading sentences like this Citation itself often presents as a modality, with an associated degree of confidence. I started to wonder if this article was a practical joke. Why was someone writing about programming using the pompous language of literary theory? Was someone trying to pull a Sokal? Then I saw the endnote. (I realize this is only DH2, but it's such an extreme case that I can't help thinking how lucky we are not to have programming ordinarily discussed in this way.)
- Morendil 16y ago
  Which part of the above did you have trouble with? "Citation" is the topic being discussed. It's a technical term that refers to how scientific papers refer to other scientific papers. "Modality" is the concept I've been introducing for three paragraphs. You're supposed to know what it means by then, or I've not been clear enough. "Present as" is an intransitive verb phrase. I could say the same thing, perhaps more simply, by saying "you can think of a citation as a modality". Perhaps it's this phrase that has thrown you? [EDIT: I've changed the article to try and make the sentence clearer. Thanks for your feedback.] It seems to me that "degree of confidence" is clear enough, and that it's clear enough that a modality can have a degree of confidence associated with it. So the article introduces two technical terms you're perhaps not familiar with, but I'd suppose someone who can read a CS paper or write a program can cope with that much.
  - j_baker 16y ago
    > You're supposed to know what it means by then, or I've not been clear enough. I hate to break it to you, but people (especially on a site like HN) have a tendency to skim over text. I hate to repeat myself as much as the next guy, but when you're communicating with human beings it's quite often that you have to beat a dead horse. The other thing is that the overall sentence isn't necessarily very difficult to read, but the fact that there are so many $10 words gives the impression that it's more difficult to read than it is. So most people (like me) don't try. With the above noted, I don't know if there's any way it can be improved. Perhaps that's the best way to say it. But those are the reasons others might find it difficult to read.
  - JesseAldridge 16y ago
    It's the word "modality" that throws me. The word by itself is pretty vague -- "somehow related to a mode". The linked Wikipedia article is dense and confusing. You explain it as a way of modifying a statement. Then you say a publication is an "extended modality". That requires another mental stretch. How does the abstract idea of "a publication" connect to modifying a statement? One expects you to explain that in the following sentences, but you don't really. Instead you follow with the seemingly unrelated statement, "A researcher may wish to state a conclusion: “water boils at 100 degrees celsius”. Convention dictates that he should initially add various hedges to his conclusion: ..." So when writing a publication a researcher will customarily add hedges. What does that have to do with a publication being an "extended modality"? Is it really necessary to use such an obscure term? It seems like it shouldn't be this hard to understand what you're saying.
    - meestaplu 16y ago
      I definitely agree about the linked Wikipedia article on modality. Literary theory is probably not a good way to talk about modality with programmers, but the idea of modality isn't just about linguistics and literature, and isn't as obscure as you think. A better starting point for programmers and computer scientists would have been modal logic, which uses the modal operators of necessity and possibility. For example, classical logic uses propositions. I can say "P" in classical logic. In modal logic, I can say "P", "Necessarily P", and "Possibly P", where logical necessity and possibility are modalities. See http://en.wikipedia.org/wiki/Modal_logic http://en.wikipedia.org/wiki/Modal_logic -- it's a pretty good overview and it links the logical and epistemological senses of modality.
- mycroftiv 16y ago
  I find this article to be very clear and readable overall, despite the presence of a few "cultural markers" such as the above language. In comparison with the truly opaque and mostly content-free works of the most notorious poststructuralists, the claim of the article ("assertions about programmer productivity are presented as scientific facts but are not founded on a large body of experimental evidence we expect from such claims") is clearly stated. That said, my own personal experience and observation is that the 10x claim probably underestimates the difference in productivity between the best and worst coders.
  - j_baker 16y ago
    Ahem... I really hate to make ad hominem arguments, but when you say things like "In comparison with the truly opaque and mostly content-free works of the most notorious poststructuralists, the claim of the article ... is clearly stated", I have to question if your definition of "clearly stated" is the same as mine. :-)
    - mycroftiv 16y ago
      I'm sure our definitions differ. For comparison, here is a quote from a translation of Guattari's "Chaosmosis" (1992) which I would say is unclear: "The ontological relativity advocated here is inseparable from an enunciative relativity. Knowledge of a Universe (in an astrophysical or axiological sense) is only possible through the mediation of autopoietic machines. A zone of self-belonging needs to exist somewhere for the coming into cognitive existence of any being or any modality of being." That is the kind of language which made many European intellectuals in the later half of the 20th century notorious. The problem is not just an unusual vocabulary and elaborate sentence structure, but fundamental incoherence and haphazard use of undefined terms. I don't believe the linked article under discussion is really in the same category.
- clyfe 16y ago
  Indeed, French is a fluid language if I may label it as such. (note of knowledge: the text is a translation from French)
- j_baker 16y ago
  (It should be noted that DH2 is "disagreement hierarchy 2" - responding to tone: http://www.paulgraham.com/disagree.html http://www.paulgraham.com/disagree.html)
- nonce756 16y ago
  The author has bad style, but he's not writing nonsense. He edited the statement because he was bullied into it, not because it was nonsense. I would try to avoid words such as "modality"---"qualification" might be better---but the point he makes is clear. He spends several paragraphs defining what he's talking about and then makes a case. It actually follows the lemma, lemma, theorem formula closely. He goes on to offer evidence for his point. Summarized, he says: over time citations can lend credibility to an otherwise unproved statement. He then says programmer productivity disparities haven't been conclusively demonstrated. That might have been a better starting point for the article. He's guilty of disorganized writing, and being influenced by literary theory, but his point is otherwise legitimate and capable of being proved or disproved. (I don't know how much he's edited the article since.)
- robinhouston 16y ago
  Like others, I'm a little puzzled by this objection to what seemed to me to be a coherent and interesting piece. I love the self-awareness implied by referencing your own disagreement hierarchy, and humbly wonder whether the following insightful remark may be pertinent here: “Someone who has a chip on their shoulder about some topic might be offended by a tone that to other readers seemed neutral.”
- dctoedt 16y ago
  If I understand the author correctly, "modality" might usefully be replaced by something like confidence indicator or confidence signal (in which case the last part of the sentence quoted by pg could be omitted).
deleted 16y ago
[deleted]
huherto 16y ago
My personal opinion is that the big gains in programming productivity in large projects have have to do with how code is organized and structured. The right abstractions are also key to keep the inherent complexity under control. It is not about implementing x feature in t time. It is more about implementing the feature in a coherent way, and the gain is long term. To measure and compare that, you would need to track 2 projects implementing the same features over a period of time. That experiment is hard to do.
- yxhuvud 16y ago
  Don't forget how the people is organized. Both programmers and the people around them.
kgo 16y ago
Article talks a lot about science, and then provides no actual studies that refute the original claim. That's not how science works. Devise a hypothesis. Develop a test to confirm or deny. Run test. And then he tears apart the study with strange assumptions. "Well they were debugging, not programming." Because no programmers in the real world spend a significant amount of their time debugging. I'd consider that to be a significant amount of time for a programmers day. Wish I had code complete in front of me, because I know it did do a break-down of developers time. The one that said that most developers spend more time walking than reading software books. If you're going to refute McConnell or even Peopleware, you've got to do a study. You can't just say "well cubes seem more productive than an office."
- Morendil 16y ago
  You're missing the point: I didn't set out to refute the claim of 10x. I set out to refute that the claim had been proven. > If you're going to refute McConnell or even Peopleware, you've got to do a study. What McConnell has published isn't empirical, or scientific; argument suffices to refute it. That goes for DeMarco and Lister, too - Peopleware was a popular book, not an empirical study aiming at establishing scientific fact.
- salvadors 16y ago
  He's not refuting "X", he's refuting the claim that "studies have shown X". To do that you don't need to run your own studies and show different results from X — in fact "X" might even be true. But if the existing studies don't exist, or can be shown to be flawed, then it's perfectly valid to simply point out the holes. Your debugging criticism is somewhat valid, but the context here in that the original Sackman paper was mostly testing the difference between "online" and "offline" programming (i.e. programming whilst sitting at a computer or not — nothing to do with the internet!): see, for example, http://dustyvolumes.com/archives/497 http://dustyvolumes.com/archives/497 and http://dustyvolumes.com/archives/500 http://dustyvolumes.com/archives/500 The research was on whether debugging whilst sitting at a computer would make you more lazy, which is an interesting experiment for its time, but isn't really that connected to the modern concepts of debugging vs programming.
j_baker 16y ago
It's important to note that (as far as I can tell) the author doesn't actually debunk the claim that good programmers are 10x more productive, only the claim that studies show that programmers are 10x as productive. Personally, I think the author is unintentionally also pointing out the problem of relying on studies of programmer productivity: it's next to impossible to measure. I seriously doubt that there is no valid research on this because no one has gotten around to it. Too many peoples' businesses rely on understanding programmer productivity inside and out for that to be the case. Personally, I agree with whoever it was that argued that we need to view this as a soft science like psychology or sociology. We need to focus on working in spite of our imperfect means of measuring productivity rather than dismissing all of our research based on it. Such is the nature of studying the human mind.
- Morendil 16y ago
  I've just been pointed to a recent article by Tom Gilb on measuring productivity, I'll post it when I've had a chance to read it. http://www.gilb.com/tiki-download_file.php?fileId=144 http://www.gilb.com/tiki-download_file.php?fileId=144
- Retric 16y ago
  The real issue is studding programmers is expensive. In class assignments or contests you can find a huge range of time to completion. But, paying a statistically significant number of programmers to solve the same task is prohibitively expensive let alone a wide range of identical tasks. PS: I have completed the same task as other programmers in less than 10% of the time. However, assessing overall productivity requires that someone is consistently faster on a wide range of tasks not just that they happen to solve something in 10% or even 1% of the time.
  - PaulHoule 16y ago
    I dunno, introductory level programming classes create an opportunity to observe anywhere from 5 to 200 people solving the same problem. I've never taught a CS course, but back in my college days I helped a friend with his CS homework by taking advantage of weak file permissions to copy other people's homework. The shocking thing was that, looking at a small sample, most of the programs didn't work correctly. To complete the assignment our way, we had to fix bugs in programs that other people wrote, which is fantastic preparation for a professional programmer ;-)
JimboOmega 16y ago
As the article points out (and I have before), even if it wasn't a myth, there's no interviewing technique to determine who is at what level. Also, it should be kept in mind that there is a chance that the original study was contaminated by bad programmers. The truly bad - those who don't really know how to write code at all - are going to vastly underperform those of average competence. If you had a test that screens out basic competency (an ability to handle CS 101 tasks like assignment, for instance) - how wide would the range be? Certainly based on 1 study from the 60's... we can't make a good judgment. It's not even a coding study! Even if it was a coding study, it doesn't measure how well they work in teams, or how well they adapt to new languages, or the myriad other things that matter in a programmer's career. . Here's a question that deserves study - what interview techniques can make productive teams? What productivity difference is there between those good at brain teasers (programming ones or otherwise), and those that are bad at them? What about those given syntax questions? Those who feel like a good cultural fit? General IQ?
equark 16y ago
I don't see how it's really debatable that some programmers can be 10x more productive in certain domains. This is especially true in technical domains where it can take years to gain the knowledge required to solve the problem at all. I can think of real-world problems that would require the average programmer several years to solve, yet I know people that could solve them in a day. In fact, almost any specialized task works like that.
- mattmanser 16y ago
  It would take others years to solve what someone else can solve in a day? I would love to hear an example of that as that seems a pretty extreme claim.
  - kaib 16y ago
    I don't claim as extreme productivity differences as the parent but I've seen people do pretty impressive engineering feats and have heard first hand stories of others. In the cases I've seen what makes people a lot more productive is experience. At Google, Ken and Russ ran circles around me in terms of writing compilers, but I had only written half a dozen toy compilers while someone like Ken has written many more production ones. My current co-worker Mikko just spent a few weeks re-writing the code to extract a mesh from our level set data. It's a distributed algorithm and our literature search didn't indicate it had ever been published. Starting with someone without domain experience it would be several weeks until the algorithm made sense. It probably would take years to replicate the feat. Programming is a very specialized endeavor. Some parts of more than others. My own experience indicates that individuals can have magnitudes of difference in their performance when applied to some of these specialized areas.
  - varjag 16y ago
    Remember the Sudoku debacle? Or should I say Sudokugate..
  - pmiller2 16y ago
    It's not specifically programming, but I've done homework exercises in a night or a weekend as a grad student in mathematics that were main results of PhD dissertations years ago. It's all about point of view. If you have the right way of looking at the problem, things become possible that would look like miracles to someone looking at the problem a different way.
PaulHoule 16y ago
I think the "10x" myth persists because we all imagine that we're the programmers who are "10x" better than average. It's nominally true because many programmers have zero or lower productivity, sometimes due to no fault of their own. If you work on a project for three years, and then it gets canceled, you could say your work had no economic value because it never got in front of customers. If some PHP hack manages to pound something out in a month that brings in $100,000 of business, he looks like a hero, no matter how bad the code is. In my view, superproductivity is about alignment with your environment. If you work for 12 months a year, but spend 3 of them on side projects that go nowhere, spend 3 months being a sysadmin, and then waste another 5 months on reworking a GUI, there goes about 90% of your productivity. The "Mythical Man Month" says that about 20% of the time on a software project is spend coding, the rest is requirements work, testing and that kind of stuff. A superprogrammer with supertools might be able to do that in time that approaches zero asymptotically, but unless you can get rid of the other 80%, project cost and time don't get much worse, even in comparison with a coder who is half as productive as average. The moral? If you want to look like a genius, find some place where (i) requirement work takes little time (and wastes little time downstream with changes) and (ii) testing, deployment and all that is minimal.
- lispm 16y ago
  I have seen programmers that are more than ten times faster than some slow programmers. There are debugging tasks where some people struggle for days, while others would find it in one hour.
  - PaulHoule 16y ago
    Sure, which leads one to thinking about it like auto racing. You don't win a race by "driving fast", you win a race by never driving slowly. It doesn't matter how fast your car is if you spend 30% of your time stacked up at intersections. Similarly, programmers spent a lot of time dealing with various barriers to productivity. (For instance, bugs you can't fix) If you want to increase your productivity, you need to attack these barriers by every means necessary.
  - 16y ago
gacba 16y ago
I would love to know how a study could be conducted in such a way to prove or disprove this notion. Any ideas? Any existing data sets that could reliably mined for info?
- Morendil 16y ago
  I wouldn't start with a data set. I would start by stating clearly enough the question we are looking for an answer to. Doing that might involve getting clear on the terms. "Productivity" is too abstract and ambiguous to be used as-is, so we first need to say "here is what we are going to measure".
  - kenjackson 16y ago
    Bingo. Software engineering is filled with throwing about this term "productivity" yet its unclear as to what it means. And even what the impact is. With that said, this is not trivial even for activities where the metrics seem a lot clearer, e.g., basketball or basetball.
  - narag 16y ago
    I believe that what people try to determine is not a very precise measure of productivity, but the fact that productivity measurement doesn't matter so much. The central message that I remember from Peopleware is that software teams are more about people than about numbers. Performance is going to vary, often wildly (often for the same people at different times). I loved the anecdote of the woman that wasn't very productive herself, but was a "catalyst", maybe simply a fancy way to say she was fun and nice to work with. Unless you are Google and have the right process to detect the best and resources to attract them, your best bet is to treat people well and apply common sense... as opposed to apply hard metrics and put pressure, demoralizing your people.
- daviding 16y ago
  It would be hard - http://www.martinfowler.com/bliki/CannotMeasureProductivity.html http://www.martinfowler.com/bliki/CannotMeasureProductivity....
- 16y ago
wazoox 16y ago
Heck, even one given programmer can see his own productivity varying wildly. There are times I could hack away thousands of lines of working code in a single week, and some other times I'd tinker for months getting no actual job done. Overall I managed to ship from time to time, though I blankly admit I failed the most ambitious projects because I was probably not up to the task. Even programming at my very basic ability level (actually I never considered myself to be a professional programmer * ) I've seen a number of people who simply never get anything done, and are 10x less productive than I am. The sort of people who are the heroes of thedailywtf. They never do any better than tinkering around; with some luck they may actually implement some simple code if given very precise requirements because they miss the basic impetus, or the necessary comprehension to start by themselves. Maybe in some huge java drones team, they may add some value, I don't know for sure, I've never worked in a team of more than five programmers anyway. OTOH I've worked with some guys, thumping the keyboard 16 hours a day for months straight, getting working production code out of the frigging door every single day. So I dare saying I've seen it all : the 10x programmer, the 1x programmer (myself and most others), and the 0.1x programmer (I don't know how many). All of this sure isn't scientific fact, but simply what I witnessed myself in the past 25 years. It's enough information to guide me through :) * I'm a professional dilettante :)
deleted 16y ago
[deleted]
prewett 16y ago
I suspect the 10X observation is due in large part to the fact that some programmers are more experienced. I'm much more productive than I used to be, because I've been doing it for 10 years. I've already had to think about a lot of problems, which means I already know solutions for a lot of things. It doesn't mean I'm inherently better than I was, just more experienced. I expect that this 10X figure is also applicable to things besides software engineering. I'd guess that auto mechanics with 30 yrs of experience running their own auto shop are 10X more productive at solving problems than new guy at the dealership. (Note: solving problems, not just replacing the part. Knowing what part to replace) The Ph.D. who's been researching for 30 years probably is a lot faster at finding solutions than the guy who just passed his qualifiers.
abecedarius 16y ago
One set of studies: http://norvig.com/java-lisp.html http://norvig.com/java-lisp.html The focus was more on the language factor, and some programmers were students while others were pros, and in some they were self-selected; but still, >10x variation in development time among the Java programmers. 30x if you allow all the above differences. The 10x thing gets quoted way more confidently than the amount of study justifies, but it's not just made up.
- ewjordan 16y ago
  I've already made this comment elsewhere on this thread, but just for completeness sake... The problem with reading off a 30x variation (or even >10x) from Norvig's results is that it discounts the possibility that the same person might take different amounts of time for tasks (relative to the others in the group). For instance, you might have people in the group that have already worked some variation of that problem. Or you might have some that had colds, were distracted, or just do badly on that sort of enumeration problem, but are killer at other problems. Every one of these possible discrepancies cuts away at the expected magnitude of the actual overall productivity difference between programmers, and should be accounted for. Of course, since that difference is not what was being studied, they didn't set up the experiments to gather the data we'd need to best estimate the real productivity differences, so it's hard to say what we should conclude...I'm sure there are people that are 1/30th as productive as the best programmers, but what we'd really want to know is how typical they are, and what the overall distribution looks like. That's far less clear.
sdepablos 16y ago
Anyone who says that "net negative productivity programmers" don't exist is a person who has been really lucky in his professional career.
Chris_Newton 16y ago
The 10x productivity claim is one of the unfortunate ones, because anyone with a few years of programming experience can relate to the idea, but it's very difficult to define an objective hypothesis that can be tested quantitatively in any meaningful way. First you have to define what things like "productivity" are for programmers, and to my knowledge no-one has really done a robust job of even that yet. Then you have to establish who the "best" and "worst" cases are. After all, I suspect many of us would agree that some programmers make a net negative contribution to their projects, taking more time away from more efficient programmers to fix the resulting problems than it would have taken the stronger programmers to do the work themselves in the first place. This may be inevitable, given the learning curve involved in a field as complicated and diverse as programming. That all said, I will just mention that Glass also covers this subject, with various other citations, in "Facts and Fallacies of Software Engineering". Given that his final fact is "Many researchers advocate rather than investigate", a criticism very much along the lines of this article, it would be interesting to know whether his sources stack up to more robust criticism if anyone has access to them.
dimitar 16y ago
Is there "software engineering" engineering at all? It is more like "social engineering" or "financial engineering". More to do with ingenuity, than engineering as a practice. Engineering is an application of scientific - mathematical and physical knowledge to create something. If you don't do the Math, you probably are not an engineer. There is little room for subjectivity. So, I guess most programmers (if not all) are software technicians. They have practical knowledge, but not the proven theory behind it. So you will have "Fact and folklore in software engineering", until you can synthesise "something-driven development" using knowledge mathematics and science. Currently you have some ideologies and studies on them. In control engineering for contrast to solve a problem, for example a controller regulating the flow of fuel in an engine you have the following procedure: 1. You make a model of the open and closed loop system. You know how do the processes look like in derivatives and you can make transfer functions and state-space representations. 2. You correct the system according to specification, adding to the models. 3. Knowing the physical properties of the different parts of the system you are left with some choices of implementation, but they have different trade-off (physical and economic). You talk with the client and make the appropriate choice. :-) So, you have the heuristics to solve the problems all build on both practice and science and you don't have the engineers split in schools or ideologies :-) I guess it should be the same with programmers.
- elai 16y ago
  Software engineer is a term that can help with visa applications. Programming feels a lot like engineering, architecture and shop work combined. A guy with a machine shop and an idea can create a machine that becomes a very useful mass market product even if he didn't do any mathematics other than measurement and arithmetic. What an engineer does with lots of math can create a similarly identical product. Some problems you need an engineer's sophistication with mathematics and physics to create (like a bridge), and similarly in programming, some problems in graphics and what not require sophistication in mathematics and physics to even create it or to advance the field.
mkramlich 16y ago
This is one of those things that's yet another example of something being treated academicaly which does not need to be treated that way. And of making something much more complex than needed. In real life, in actual practice, any working software engineer or heck even open source hobbyist programmer can tell you that talent and ability and productivity vary significantly between individuals. And this isn't just true among programmers, it's true in other areas/fields as well. This is not controversial, or at least it shouldn't be. And yes, some individuals are clearly 10x or 20x or whatever better than others -- for whatever reason, and the reason(s) doesn't matter too much honestly. It just is.
pschlump 16y ago
I have looked back in the notes from my last startup. I was CTO and I had to spend time doing fund raising, making sales calls, editing documentation and designs, going to customer meetings. I was never full time in development. I wrote 69 times as much code as the least productive developer and 12 times as much as our "Lead Programmer". Based on my experience 10x is a low estimate. I would say that some people are 100x as productive as others in developing software.
joblessjunkie 16y ago
The example chosen for the first half of this blog post was rather annoying: water boils at 100C by definition, and thus there has never been any scientific debate about this. And yet, his point is made. :-)

135 points by Morendil 16y ago

pg 16y ago

When I found myself reading sentences like this Citation itself often presents as a modality, with an associated degree of confidence. I started to wonder if this article was a practical joke. Why was someone writing about programming using the pompous language of literary theory? Was someone trying to pull a Sokal? Then I saw the endnote. (I realize this is only DH2, but it's such an extreme case that I can't help thinking how lucky we are not to have programming ordinarily discussed in this way.)

Morendil 16y ago
Which part of the above did you have trouble with? "Citation" is the topic being discussed. It's a technical term that refers to how scientific papers refer to other scientific papers. "Modality" is the concept I've been introducing for three paragraphs. You're supposed to know what it means by then, or I've not been clear enough. "Present as" is an intransitive verb phrase. I could say the same thing, perhaps more simply, by saying "you can think of a citation as a modality". Perhaps it's this phrase that has thrown you? [EDIT: I've changed the article to try and make the sentence clearer. Thanks for your feedback.] It seems to me that "degree of confidence" is clear enough, and that it's clear enough that a modality can have a degree of confidence associated with it. So the article introduces two technical terms you're perhaps not familiar with, but I'd suppose someone who can read a CS paper or write a program can cope with that much.
- j_baker 16y ago
  > You're supposed to know what it means by then, or I've not been clear enough. I hate to break it to you, but people (especially on a site like HN) have a tendency to skim over text. I hate to repeat myself as much as the next guy, but when you're communicating with human beings it's quite often that you have to beat a dead horse. The other thing is that the overall sentence isn't necessarily very difficult to read, but the fact that there are so many $10 words gives the impression that it's more difficult to read than it is. So most people (like me) don't try. With the above noted, I don't know if there's any way it can be improved. Perhaps that's the best way to say it. But those are the reasons others might find it difficult to read.
- JesseAldridge 16y ago
  It's the word "modality" that throws me. The word by itself is pretty vague -- "somehow related to a mode". The linked Wikipedia article is dense and confusing. You explain it as a way of modifying a statement. Then you say a publication is an "extended modality". That requires another mental stretch. How does the abstract idea of "a publication" connect to modifying a statement? One expects you to explain that in the following sentences, but you don't really. Instead you follow with the seemingly unrelated statement, "A researcher may wish to state a conclusion: “water boils at 100 degrees celsius”. Convention dictates that he should initially add various hedges to his conclusion: ..." So when writing a publication a researcher will customarily add hedges. What does that have to do with a publication being an "extended modality"? Is it really necessary to use such an obscure term? It seems like it shouldn't be this hard to understand what you're saying.
  - meestaplu 16y ago
    I definitely agree about the linked Wikipedia article on modality. Literary theory is probably not a good way to talk about modality with programmers, but the idea of modality isn't just about linguistics and literature, and isn't as obscure as you think. A better starting point for programmers and computer scientists would have been modal logic, which uses the modal operators of necessity and possibility. For example, classical logic uses propositions. I can say "P" in classical logic. In modal logic, I can say "P", "Necessarily P", and "Possibly P", where logical necessity and possibility are modalities. See http://en.wikipedia.org/wiki/Modal_logic http://en.wikipedia.org/wiki/Modal_logic -- it's a pretty good overview and it links the logical and epistemological senses of modality.
mycroftiv 16y ago
I find this article to be very clear and readable overall, despite the presence of a few "cultural markers" such as the above language. In comparison with the truly opaque and mostly content-free works of the most notorious poststructuralists, the claim of the article ("assertions about programmer productivity are presented as scientific facts but are not founded on a large body of experimental evidence we expect from such claims") is clearly stated. That said, my own personal experience and observation is that the 10x claim probably underestimates the difference in productivity between the best and worst coders.
- j_baker 16y ago
  Ahem... I really hate to make ad hominem arguments, but when you say things like "In comparison with the truly opaque and mostly content-free works of the most notorious poststructuralists, the claim of the article ... is clearly stated", I have to question if your definition of "clearly stated" is the same as mine. :-)
  - mycroftiv 16y ago
    I'm sure our definitions differ. For comparison, here is a quote from a translation of Guattari's "Chaosmosis" (1992) which I would say is unclear: "The ontological relativity advocated here is inseparable from an enunciative relativity. Knowledge of a Universe (in an astrophysical or axiological sense) is only possible through the mediation of autopoietic machines. A zone of self-belonging needs to exist somewhere for the coming into cognitive existence of any being or any modality of being." That is the kind of language which made many European intellectuals in the later half of the 20th century notorious. The problem is not just an unusual vocabulary and elaborate sentence structure, but fundamental incoherence and haphazard use of undefined terms. I don't believe the linked article under discussion is really in the same category.
clyfe 16y ago
Indeed, French is a fluid language if I may label it as such. (note of knowledge: the text is a translation from French)
j_baker 16y ago
(It should be noted that DH2 is "disagreement hierarchy 2" - responding to tone: http://www.paulgraham.com/disagree.html http://www.paulgraham.com/disagree.html)
nonce756 16y ago
The author has bad style, but he's not writing nonsense. He edited the statement because he was bullied into it, not because it was nonsense. I would try to avoid words such as "modality"---"qualification" might be better---but the point he makes is clear. He spends several paragraphs defining what he's talking about and then makes a case. It actually follows the lemma, lemma, theorem formula closely. He goes on to offer evidence for his point. Summarized, he says: over time citations can lend credibility to an otherwise unproved statement. He then says programmer productivity disparities haven't been conclusively demonstrated. That might have been a better starting point for the article. He's guilty of disorganized writing, and being influenced by literary theory, but his point is otherwise legitimate and capable of being proved or disproved. (I don't know how much he's edited the article since.)
robinhouston 16y ago
Like others, I'm a little puzzled by this objection to what seemed to me to be a coherent and interesting piece. I love the self-awareness implied by referencing your own disagreement hierarchy, and humbly wonder whether the following insightful remark may be pertinent here: “Someone who has a chip on their shoulder about some topic might be offended by a tone that to other readers seemed neutral.”
dctoedt 16y ago
If I understand the author correctly, "modality" might usefully be replaced by something like confidence indicator or confidence signal (in which case the last part of the sentence quoted by pg could be omitted).

deleted 16y ago

[deleted]

huherto 16y ago

My personal opinion is that the big gains in programming productivity in large projects have have to do with how code is organized and structured. The right abstractions are also key to keep the inherent complexity under control. It is not about implementing x feature in t time. It is more about implementing the feature in a coherent way, and the gain is long term. To measure and compare that, you would need to track 2 projects implementing the same features over a period of time. That experiment is hard to do.

yxhuvud 16y ago
Don't forget how the people is organized. Both programmers and the people around them.

kgo 16y ago

Article talks a lot about science, and then provides no actual studies that refute the original claim. That's not how science works. Devise a hypothesis. Develop a test to confirm or deny. Run test. And then he tears apart the study with strange assumptions. "Well they were debugging, not programming." Because no programmers in the real world spend a significant amount of their time debugging. I'd consider that to be a significant amount of time for a programmers day. Wish I had code complete in front of me, because I know it did do a break-down of developers time. The one that said that most developers spend more time walking than reading software books. If you're going to refute McConnell or even Peopleware, you've got to do a study. You can't just say "well cubes seem more productive than an office."

Morendil 16y ago
You're missing the point: I didn't set out to refute the claim of 10x. I set out to refute that the claim had been proven. > If you're going to refute McConnell or even Peopleware, you've got to do a study. What McConnell has published isn't empirical, or scientific; argument suffices to refute it. That goes for DeMarco and Lister, too - Peopleware was a popular book, not an empirical study aiming at establishing scientific fact.
salvadors 16y ago
He's not refuting "X", he's refuting the claim that "studies have shown X". To do that you don't need to run your own studies and show different results from X — in fact "X" might even be true. But if the existing studies don't exist, or can be shown to be flawed, then it's perfectly valid to simply point out the holes. Your debugging criticism is somewhat valid, but the context here in that the original Sackman paper was mostly testing the difference between "online" and "offline" programming (i.e. programming whilst sitting at a computer or not — nothing to do with the internet!): see, for example, http://dustyvolumes.com/archives/497 http://dustyvolumes.com/archives/497 and http://dustyvolumes.com/archives/500 http://dustyvolumes.com/archives/500 The research was on whether debugging whilst sitting at a computer would make you more lazy, which is an interesting experiment for its time, but isn't really that connected to the modern concepts of debugging vs programming.

j_baker 16y ago

It's important to note that (as far as I can tell) the author doesn't actually debunk the claim that good programmers are 10x more productive, only the claim that studies show that programmers are 10x as productive. Personally, I think the author is unintentionally also pointing out the problem of relying on studies of programmer productivity: it's next to impossible to measure. I seriously doubt that there is no valid research on this because no one has gotten around to it. Too many peoples' businesses rely on understanding programmer productivity inside and out for that to be the case. Personally, I agree with whoever it was that argued that we need to view this as a soft science like psychology or sociology. We need to focus on working in spite of our imperfect means of measuring productivity rather than dismissing all of our research based on it. Such is the nature of studying the human mind.

Morendil 16y ago
I've just been pointed to a recent article by Tom Gilb on measuring productivity, I'll post it when I've had a chance to read it. http://www.gilb.com/tiki-download_file.php?fileId=144 http://www.gilb.com/tiki-download_file.php?fileId=144
Retric 16y ago
The real issue is studding programmers is expensive. In class assignments or contests you can find a huge range of time to completion. But, paying a statistically significant number of programmers to solve the same task is prohibitively expensive let alone a wide range of identical tasks. PS: I have completed the same task as other programmers in less than 10% of the time. However, assessing overall productivity requires that someone is consistently faster on a wide range of tasks not just that they happen to solve something in 10% or even 1% of the time.
- PaulHoule 16y ago
  I dunno, introductory level programming classes create an opportunity to observe anywhere from 5 to 200 people solving the same problem. I've never taught a CS course, but back in my college days I helped a friend with his CS homework by taking advantage of weak file permissions to copy other people's homework. The shocking thing was that, looking at a small sample, most of the programs didn't work correctly. To complete the assignment our way, we had to fix bugs in programs that other people wrote, which is fantastic preparation for a professional programmer ;-)

JimboOmega 16y ago

As the article points out (and I have before), even if it wasn't a myth, there's no interviewing technique to determine who is at what level. Also, it should be kept in mind that there is a chance that the original study was contaminated by bad programmers. The truly bad - those who don't really know how to write code at all - are going to vastly underperform those of average competence. If you had a test that screens out basic competency (an ability to handle CS 101 tasks like assignment, for instance) - how wide would the range be? Certainly based on 1 study from the 60's... we can't make a good judgment. It's not even a coding study! Even if it was a coding study, it doesn't measure how well they work in teams, or how well they adapt to new languages, or the myriad other things that matter in a programmer's career. . Here's a question that deserves study - what interview techniques can make productive teams? What productivity difference is there between those good at brain teasers (programming ones or otherwise), and those that are bad at them? What about those given syntax questions? Those who feel like a good cultural fit? General IQ?

equark 16y ago

I don't see how it's really debatable that some programmers can be 10x more productive in certain domains. This is especially true in technical domains where it can take years to gain the knowledge required to solve the problem at all. I can think of real-world problems that would require the average programmer several years to solve, yet I know people that could solve them in a day. In fact, almost any specialized task works like that.

mattmanser 16y ago
It would take others years to solve what someone else can solve in a day? I would love to hear an example of that as that seems a pretty extreme claim.
- kaib 16y ago
  I don't claim as extreme productivity differences as the parent but I've seen people do pretty impressive engineering feats and have heard first hand stories of others. In the cases I've seen what makes people a lot more productive is experience. At Google, Ken and Russ ran circles around me in terms of writing compilers, but I had only written half a dozen toy compilers while someone like Ken has written many more production ones. My current co-worker Mikko just spent a few weeks re-writing the code to extract a mesh from our level set data. It's a distributed algorithm and our literature search didn't indicate it had ever been published. Starting with someone without domain experience it would be several weeks until the algorithm made sense. It probably would take years to replicate the feat. Programming is a very specialized endeavor. Some parts of more than others. My own experience indicates that individuals can have magnitudes of difference in their performance when applied to some of these specialized areas.
- varjag 16y ago
  Remember the Sudoku debacle? Or should I say Sudokugate..
- pmiller2 16y ago
  It's not specifically programming, but I've done homework exercises in a night or a weekend as a grad student in mathematics that were main results of PhD dissertations years ago. It's all about point of view. If you have the right way of looking at the problem, things become possible that would look like miracles to someone looking at the problem a different way.

PaulHoule 16y ago

I think the "10x" myth persists because we all imagine that we're the programmers who are "10x" better than average. It's nominally true because many programmers have zero or lower productivity, sometimes due to no fault of their own. If you work on a project for three years, and then it gets canceled, you could say your work had no economic value because it never got in front of customers. If some PHP hack manages to pound something out in a month that brings in $100,000 of business, he looks like a hero, no matter how bad the code is. In my view, superproductivity is about alignment with your environment. If you work for 12 months a year, but spend 3 of them on side projects that go nowhere, spend 3 months being a sysadmin, and then waste another 5 months on reworking a GUI, there goes about 90% of your productivity. The "Mythical Man Month" says that about 20% of the time on a software project is spend coding, the rest is requirements work, testing and that kind of stuff. A superprogrammer with supertools might be able to do that in time that approaches zero asymptotically, but unless you can get rid of the other 80%, project cost and time don't get much worse, even in comparison with a coder who is half as productive as average. The moral? If you want to look like a genius, find some place where (i) requirement work takes little time (and wastes little time downstream with changes) and (ii) testing, deployment and all that is minimal.

lispm 16y ago
I have seen programmers that are more than ten times faster than some slow programmers. There are debugging tasks where some people struggle for days, while others would find it in one hour.
- PaulHoule 16y ago
  Sure, which leads one to thinking about it like auto racing. You don't win a race by "driving fast", you win a race by never driving slowly. It doesn't matter how fast your car is if you spend 30% of your time stacked up at intersections. Similarly, programmers spent a lot of time dealing with various barriers to productivity. (For instance, bugs you can't fix) If you want to increase your productivity, you need to attack these barriers by every means necessary.
- 16y ago

gacba 16y ago

I would love to know how a study could be conducted in such a way to prove or disprove this notion. Any ideas? Any existing data sets that could reliably mined for info?

Morendil 16y ago
I wouldn't start with a data set. I would start by stating clearly enough the question we are looking for an answer to. Doing that might involve getting clear on the terms. "Productivity" is too abstract and ambiguous to be used as-is, so we first need to say "here is what we are going to measure".
- kenjackson 16y ago
  Bingo. Software engineering is filled with throwing about this term "productivity" yet its unclear as to what it means. And even what the impact is. With that said, this is not trivial even for activities where the metrics seem a lot clearer, e.g., basketball or basetball.
- narag 16y ago
  I believe that what people try to determine is not a very precise measure of productivity, but the fact that productivity measurement doesn't matter so much. The central message that I remember from Peopleware is that software teams are more about people than about numbers. Performance is going to vary, often wildly (often for the same people at different times). I loved the anecdote of the woman that wasn't very productive herself, but was a "catalyst", maybe simply a fancy way to say she was fun and nice to work with. Unless you are Google and have the right process to detect the best and resources to attract them, your best bet is to treat people well and apply common sense... as opposed to apply hard metrics and put pressure, demoralizing your people.
daviding 16y ago
It would be hard - http://www.martinfowler.com/bliki/CannotMeasureProductivity.html http://www.martinfowler.com/bliki/CannotMeasureProductivity....
16y ago

wazoox 16y ago

Heck, even one given programmer can see his own productivity varying wildly. There are times I could hack away thousands of lines of working code in a single week, and some other times I'd tinker for months getting no actual job done. Overall I managed to ship from time to time, though I blankly admit I failed the most ambitious projects because I was probably not up to the task. Even programming at my very basic ability level (actually I never considered myself to be a professional programmer * ) I've seen a number of people who simply never get anything done, and are 10x less productive than I am. The sort of people who are the heroes of thedailywtf. They never do any better than tinkering around; with some luck they may actually implement some simple code if given very precise requirements because they miss the basic impetus, or the necessary comprehension to start by themselves. Maybe in some huge java drones team, they may add some value, I don't know for sure, I've never worked in a team of more than five programmers anyway. OTOH I've worked with some guys, thumping the keyboard 16 hours a day for months straight, getting working production code out of the frigging door every single day. So I dare saying I've seen it all : the 10x programmer, the 1x programmer (myself and most others), and the 0.1x programmer (I don't know how many). All of this sure isn't scientific fact, but simply what I witnessed myself in the past 25 years. It's enough information to guide me through :) * I'm a professional dilettante :)

deleted 16y ago

[deleted]

prewett 16y ago

I suspect the 10X observation is due in large part to the fact that some programmers are more experienced. I'm much more productive than I used to be, because I've been doing it for 10 years. I've already had to think about a lot of problems, which means I already know solutions for a lot of things. It doesn't mean I'm inherently better than I was, just more experienced. I expect that this 10X figure is also applicable to things besides software engineering. I'd guess that auto mechanics with 30 yrs of experience running their own auto shop are 10X more productive at solving problems than new guy at the dealership. (Note: solving problems, not just replacing the part. Knowing what part to replace) The Ph.D. who's been researching for 30 years probably is a lot faster at finding solutions than the guy who just passed his qualifiers.

abecedarius 16y ago

One set of studies: http://norvig.com/java-lisp.html http://norvig.com/java-lisp.html The focus was more on the language factor, and some programmers were students while others were pros, and in some they were self-selected; but still, >10x variation in development time among the Java programmers. 30x if you allow all the above differences. The 10x thing gets quoted way more confidently than the amount of study justifies, but it's not just made up.

ewjordan 16y ago
I've already made this comment elsewhere on this thread, but just for completeness sake... The problem with reading off a 30x variation (or even >10x) from Norvig's results is that it discounts the possibility that the same person might take different amounts of time for tasks (relative to the others in the group). For instance, you might have people in the group that have already worked some variation of that problem. Or you might have some that had colds, were distracted, or just do badly on that sort of enumeration problem, but are killer at other problems. Every one of these possible discrepancies cuts away at the expected magnitude of the actual overall productivity difference between programmers, and should be accounted for. Of course, since that difference is not what was being studied, they didn't set up the experiments to gather the data we'd need to best estimate the real productivity differences, so it's hard to say what we should conclude...I'm sure there are people that are 1/30th as productive as the best programmers, but what we'd really want to know is how typical they are, and what the overall distribution looks like. That's far less clear.

sdepablos 16y ago

Anyone who says that "net negative productivity programmers" don't exist is a person who has been really lucky in his professional career.

Chris_Newton 16y ago

The 10x productivity claim is one of the unfortunate ones, because anyone with a few years of programming experience can relate to the idea, but it's very difficult to define an objective hypothesis that can be tested quantitatively in any meaningful way. First you have to define what things like "productivity" are for programmers, and to my knowledge no-one has really done a robust job of even that yet. Then you have to establish who the "best" and "worst" cases are. After all, I suspect many of us would agree that some programmers make a net negative contribution to their projects, taking more time away from more efficient programmers to fix the resulting problems than it would have taken the stronger programmers to do the work themselves in the first place. This may be inevitable, given the learning curve involved in a field as complicated and diverse as programming. That all said, I will just mention that Glass also covers this subject, with various other citations, in "Facts and Fallacies of Software Engineering". Given that his final fact is "Many researchers advocate rather than investigate", a criticism very much along the lines of this article, it would be interesting to know whether his sources stack up to more robust criticism if anyone has access to them.

dimitar 16y ago

Is there "software engineering" engineering at all? It is more like "social engineering" or "financial engineering". More to do with ingenuity, than engineering as a practice. Engineering is an application of scientific - mathematical and physical knowledge to create something. If you don't do the Math, you probably are not an engineer. There is little room for subjectivity. So, I guess most programmers (if not all) are software technicians. They have practical knowledge, but not the proven theory behind it. So you will have "Fact and folklore in software engineering", until you can synthesise "something-driven development" using knowledge mathematics and science. Currently you have some ideologies and studies on them. In control engineering for contrast to solve a problem, for example a controller regulating the flow of fuel in an engine you have the following procedure: 1. You make a model of the open and closed loop system. You know how do the processes look like in derivatives and you can make transfer functions and state-space representations. 2. You correct the system according to specification, adding to the models. 3. Knowing the physical properties of the different parts of the system you are left with some choices of implementation, but they have different trade-off (physical and economic). You talk with the client and make the appropriate choice. :-) So, you have the heuristics to solve the problems all build on both practice and science and you don't have the engineers split in schools or ideologies :-) I guess it should be the same with programmers.

elai 16y ago
Software engineer is a term that can help with visa applications. Programming feels a lot like engineering, architecture and shop work combined. A guy with a machine shop and an idea can create a machine that becomes a very useful mass market product even if he didn't do any mathematics other than measurement and arithmetic. What an engineer does with lots of math can create a similarly identical product. Some problems you need an engineer's sophistication with mathematics and physics to create (like a bridge), and similarly in programming, some problems in graphics and what not require sophistication in mathematics and physics to even create it or to advance the field.

mkramlich 16y ago

This is one of those things that's yet another example of something being treated academicaly which does not need to be treated that way. And of making something much more complex than needed. In real life, in actual practice, any working software engineer or heck even open source hobbyist programmer can tell you that talent and ability and productivity vary significantly between individuals. And this isn't just true among programmers, it's true in other areas/fields as well. This is not controversial, or at least it shouldn't be. And yes, some individuals are clearly 10x or 20x or whatever better than others -- for whatever reason, and the reason(s) doesn't matter too much honestly. It just is.

pschlump 16y ago

I have looked back in the notes from my last startup. I was CTO and I had to spend time doing fund raising, making sales calls, editing documentation and designs, going to customer meetings. I was never full time in development. I wrote 69 times as much code as the least productive developer and 12 times as much as our "Lead Programmer". Based on my experience 10x is a low estimate. I would say that some people are 100x as productive as others in developing software.

joblessjunkie 16y ago

The example chosen for the first half of this blog post was rather annoying: water boils at 100C by definition, and thus there has never been any scientific debate about this. And yet, his point is made. :-)