4 ms·

> I have heard Dr Lilach Mollick, dir of Pedagogy at Wharton, say this has been shown numerous times. People who follow her husband, Ethan, are probably aware a

> I have heard Dr Lilach Mollick, dir of Pedagogy at Wharton, say this has been shown numerous times. People who follow her husband, Ethan, are probably aware already. I'd be curious to see the sources. Basically every study I have ever read making some claim about programming (efficacy of IDEs, TDD, static typing, pair programming, formal CS education, ai assistants, etc...) has been a house of cards that falls apart with even modest prodding. They are usually premised on one or more inherently flawed metrics like number of github issues, LoC or whatever. That would be somewhat forgivable since there are not really any good metrics to go on, but then the studies invariably make only a perfunctory effort to disentangle even the most obvious of confounding variables, making all the results kind of pointless. Would be happy if anyone here knew of good papers that would change my mind on this point.

nyarlathotep_ 1y ago
> Basically every study I have ever read making some claim about programming (efficacy of IDEs, TDD, static typing, pair programming, formal CS education, ai assistants, etc...) has been a house of cards that falls apart with even modest prodding. Isn't this true about most things in software? I mean is there anything quantifiable about "microservices" vs "monolith"? Test-driven development, containers, whatever? I mean all of these things are in some way good, in some contexts, but it seems impossible to quantify benefits of any of them. I'm a believer that most decisions made in software are somewhat arbitrary, driven by trends and popularity and it seems like little effort is expended to come to overarching, data-backed results. When they are, they're rare and, like you said, fall apart under investigation or scrutiny. I've always found this strange about software in general. Even every time there's a "we rewrote $THING in $NEW_LANG" and it improved memory use/speed/latency whatever, there's a chorus of (totally legitimate) criticism and inquiry about how things were measured, what attempts were made to optimize the original solutions, if changes were made along the way outside of the language choice that impacted performance etc etc.
- jaredklewis 1y ago
  I think we agree. To be clear I am not arguing that tools and practices like TDD, microservices, ai assistants, and so on have no effect. They almost certainly have an effect (good or bad). It’s just the unfortunate reality that quantitatively measuring these effects in a meaningful way seems to basically be impossible (or at least I’ve never see it done). With enough resources I can’t think of any reason it shouldn’t be possible, but apparently those resources are not available because there are no good studies on these topics. Thus my skepticism of the “studies” referenced earlier in the thread.

nyarlathotep_ 1y ago

> Basically every study I have ever read making some claim about programming (efficacy of IDEs, TDD, static typing, pair programming, formal CS education, ai assistants, etc...) has been a house of cards that falls apart with even modest prodding. Isn't this true about most things in software? I mean is there anything quantifiable about "microservices" vs "monolith"? Test-driven development, containers, whatever? I mean all of these things are in some way good, in some contexts, but it seems impossible to quantify benefits of any of them. I'm a believer that most decisions made in software are somewhat arbitrary, driven by trends and popularity and it seems like little effort is expended to come to overarching, data-backed results. When they are, they're rare and, like you said, fall apart under investigation or scrutiny. I've always found this strange about software in general. Even every time there's a "we rewrote $THING in $NEW_LANG" and it improved memory use/speed/latency whatever, there's a chorus of (totally legitimate) criticism and inquiry about how things were measured, what attempts were made to optimize the original solutions, if changes were made along the way outside of the language choice that impacted performance etc etc.

jaredklewis 1y ago
I think we agree. To be clear I am not arguing that tools and practices like TDD, microservices, ai assistants, and so on have no effect. They almost certainly have an effect (good or bad). It’s just the unfortunate reality that quantitatively measuring these effects in a meaningful way seems to basically be impossible (or at least I’ve never see it done). With enough resources I can’t think of any reason it shouldn’t be possible, but apparently those resources are not available because there are no good studies on these topics. Thus my skepticism of the “studies” referenced earlier in the thread.