5 ms·

Experience report: It will never work in theory

64 points by mpweiher 3y ago

hoc 3y ago
First time I heard about the site... (... was back then, and I was amazed, of course.)
Nevermark 3y ago
> the real challenge in software engineering research is not what to do about ChatGPT or whatever else Silicon Valley is gushing about at the moment. I know this is really just a turn of phrase, but at the rate language models are becoming programming models, it is hard to think of any problem with the lack of hard engineering, science, math, ..., in what we call "Computer Engineering", "Computer Science", "Computer Math", ..., that won't be heavily impacted by model/co-model driven development. For instance, a major programming language of the near future might develop adaptively as multiple coder models communicate between each other. Reviewing, testing and improving upon each other's designs and co-optimizing the way they communicate code as they do so. Resulting in representations and solutions to existing development problems completely opaque to the humans currently working on those problems. The rising effect of massive data + compute outdoing slow careful research in yet another area. Today's models are already being trained across the board on practical/actual and theoretical/research artifacts. No persistent pernicious borders there. Not to take away from anything being said.
- xanderlewis 3y ago
  > any problem with the lack of hard engineering, science, math, … To be fair, that does exclude quite a lot of problems.
  - JadeNB 3y ago
    > > any problem with the lack of hard engineering, science, math, … > To be fair, that does exclude quite a lot of problems. I had some trouble parsing the comment, but I eventually settled on the reading that it means what it literally says, a problem caused by the lack of engineering in computer engineering, of science in computer science, and of math in "computer math" (a phrase I have never heard), rather than a problem other than hard engineering, science, and math, which I agree with you excludes an awful lot of problems.
    - Nevermark 3y ago
      Yes, I meant the phrase respectively! Apologies for lack of clarity. My first computing programming class was called Computer Math. With not even a soft math emphasis.
- wizzwizz4 3y ago
  > Resulting in representations and solutions to existing development problems completely opaque to the humans currently working on those problems. But why would anyone want that? I agree that this would result, but my takeaway is "and therefore we shouldn't do that".
  - Nevermark 3y ago
    > But why would anyone want that? I agree that this would result, but my takeaway is "and therefore we shouldn't do that". My guess is that a lot of CEO's already consider the implementation details of their companies' software systems as opaque. So that is essentially a pre-existing market for performative but opaque systems. CEO's. CFO's. Shareholders, etc. Even in engineering organizations. Look at how Boeing and NASA both demoted critical engineering concerns when external concerns became overwhelmingly more important to their leaders. Now factor in the models actually doing a good job, checking each other, and generally getting better year to year.
sethammons 3y ago
How do we see a list of the referenced papers that they feel applicable to industry that are not gaining purchase in mindshare?
- sfink 3y ago
  https://neverworkintheory.org/ https://neverworkintheory.org/ is the web site associated with this (from the 1st reference in the paper)
  - dj_mc_merlin 3y ago
    These are pretty interesting talks but they very much feel like just random talks from any conference? Interesting but I have no way to apply them if I wasn't writing a tool for developers. edit: On another level I am put off by their tendency to describe things such as "teaching undergraduates" or "debugging problems" as algorithms. I understand that you can't really analyze unclear, hazy things without putting them down in clear terms, but something in me doesn't really accept that you _can_ do this without losing so much detail and information as to make your conclusion useless. They do point out that SWE often think of studies this way: > Most software developers in industry have never heard of any findings more recent than The Mythical Man-Month: Essays on Software Engineering (which few of them ever actually read) and routinely dismiss studies as “not statistically significant,” even when those studies are carefully done and directly relevant to their work.
    - cratermoon 3y ago
      Ah yes, the front page featuring the recorded talks kind of hides the real substance. Here's the full collection of papers, by category https://neverworkintheory.org/category/ https://neverworkintheory.org/category/
  - sethammons 3y ago
    Awesome, thanks. I jumped into the first one that grabbed my attention: CDD - cognitive driven development (link below, 15min video). I love the premise but find the recommended results silly. TL;DW: cognitive research shows humans have working memory limited to 7 +/- 2 things. This means that classes should be limited in what they do. So far, I agree. And I think this can be extended, in practice. However, they back up this idea with a single case study and use a decorator to help count complexity to keep the classes small and state that methods not extending beyond 24 lines of code are ways to do this. This will just lead to cargo culture like "don't comment code." The decorator is not needed because code should (authoring or in review) be obvious if handling more than 5 to 9 things. The number of lines might be correlated to the number of things a function does, but that doesn't mean cognitive overhead. Like, a 100 line function that maps state abbreviations to state names is fine. Based on the presented data and mental performative drop off, it is not 7 +/- 2, it is really five + 0 to 2. I hereby dub this the Rule of Five. A practical example better than what they present in the talk: early returns. Every nested block adds to cognitive load, early returns allow you to discard that load as you continue reading through the function. The cyclomatic complexity reaching over 5 levels means many people will start to fail to keep all the blocks in working human memory. Therefore, early returns are a good implementation of Cognitive Driven Development. The case study they used seems meh. With a couple decades of experience with high growth start ups becoming "real" companies, I just don't buy their results on decorators and function line length. But they are on to something with limiting code to keep minimal cognitive overload down. https://m.youtube.com/watch?v=g_ht_LDm5Hw&feature=youtu.be https://m.youtube.com/watch?v=g_ht_LDm5Hw&feature=youtu.be
cushychicken 3y ago
Here’s my problem: I don’t want to work with researchers. Culturally, I find we’re simply not motivated by the same things. Researchers want to publish research. I want to ship working software. The more researchers I’ve met, the more convinced I become that those two goals are at odds with one another.
- spacecadet 3y ago
  Seems like a pov, I like working with researchers and shipping interesting working software. Why is it a problem they are at odds? Its not a researchers problem to create scalable production/consumer software or hell even git repos or libraries for you. Its yours! Thats your interest.
  - trws 3y ago
    I’ve run into both. I sit between a research organization and a production organization and have to produce artifacts on both sides, this suits me quite well because I’m very much an applied research person. If it’s not feasible or tangible then it tends not to interest me much. Doing applied research and working with people who want to do that tends to work reasonably well with production, because the point is to research techniques and algorithms and so forth that help improve production. Theoretical or more future-focused research tends to need to focus on building minimal scaffolding to be able to test ideas, when that kind of scaffolding gets integrated into production software, that’s where the problems arise, because by necessity it’s a kind of MVP. I am right now working on rearchitecting a component that was built this way and labeled as production without going through hardening first (though it did go through a great deal of testing it didn’t get the code review it needed). In the end I suppose my point is that research and production aren’t inherently at odds, or even orthogonal. Applied research can be valuable to improving production software by studying current stable code and processes and researching improvements, but trying to do blue-sky research by incorporating code into production can be orthogonal or in some cases truly at odds with stability. It matters a great deal who is doing it, and how it’s managed.
    - spacecadet 3y ago
smitty1e 3y ago
My favorite Herb Sutter quote: "The difference between theory and practice is greater in practice than in theory."
- sethammons 3y ago
  "In theory, it will work in practice; in practice it is the other way around"
- xanderlewis 3y ago
  A slightly stronger form: "In theory there is no difference between theory and practice — in practice there is." - Yogi Berra.
  - smitty1e 3y ago
    I like the way Sutter's ends on a prepositional phrase. Almost like the infinite loopiness of Celtic art, in theory.
glyphy177 3y ago
[dead]
pyb 3y ago
First time I hear of this, but one issue is that they've reviewed a pretty overwhelming number of papers, which dilutes their impact. In general, practitioners would be interested in hearing from advances in academia, but usually don't want to read papers. Building fully working systems - programming languages or libraries - is more impactful, but much higher-effort.
- cratermoon 3y ago
  > practitioners would be interested in hearing from advances in academia, but usually don't want to read papers. Building fully working systems Isn't that kind of the whole problem? Practitioners ignoring research results because all they want to look at is what other practitioners do? Or are you really suggesting that researchers also become practitioners who develop working systems based on the ideas they discover? Perhaps I'm misreading, but your statement seems to encapsulate the entire problem the article is highlighting.
  - pyb 3y ago
    Researchers should become practictioners to some extent, if they want their ideas to spread. Example : Donald Knuth and TeX. Racket programming language. etc... Of course in this publish-or-perish era, this is probably no longer possible.
freeone3000 3y ago
Bridging the gap between research and practical programs is so great there’s an entire position, the RSDE, around it. Which you only encounter in places you have researchers and want to make their work a reality. Unfortunately, most researchers are publishing work about improving process, creating new and interesting things, and developing new ways of thinking about the science of computation, instead of what actually makes money, throwing out half-baked apps faster. There’s really no need for a formal understanding of what you’re actually doing and why if you can throw out a few trial-and-error projects and one gains enough traction.
tgv 3y ago
> students design a small study or experiment, collect data, analyze them, and figure out what (if anything) they’ve proven Based on ample experience, both first and second hand, that's going to fail. It is really hard to instill proper methodology, domain modelling, and statistics in students, and since the premise already is that it "would not disrupt other curricula" (PM speak for: little time and attention), students are going to walk away with the wrong ideas and think that they now understand experimental science.
- moron4hire 3y ago
  I did both a group-project software engineering course and a research course in my undergraduate Computer Science program. It's definitely possible to do.
llm_trw 3y ago
>In the wake of recent political events, our community’s energy and attention should be focused on more important things. Living like this sounds exhausting. The only thing worse I can imagine is having to listen to people who actually think like this.
- brian_spiering 3y ago
  I was working with a non-profit that taught people around the world how to teach tech skills in their local communities. There was a shift to focusing on recent political events which was a major reason for my leaving. My take was if you personally want to do that, go for it. However, reacting to recent political events was not part of the group's mission. I wanted to spend my limited time on the group's primary mission of education.
  - llm_trw 3y ago
    The non-profit implosion over the last 8 years has been something to behold. It's like someone took the CIA handbook on how to sabotage institutions and used it on every organization more left wing than the John Birch Society.
- travisjungroth 3y ago
  There’s also a weird dynamic where this applies to production but not consumption. I can’t remember which, but there’s a social media site that is donation based. Every year they do a fundraising drive. Every year the users flip out they would try to raise money with Everything Going On Right Now. The founder posted on their blog about how every year has things going on, and they still need to raise money. And the users completely backlash against all forms of monetization. What I didn’t see anyone mention is the users are still using the site. Apparently using the site is really important. Just not making it, or earning a living while making it.

64 points by mpweiher 3y ago

hoc 3y ago

First time I heard about the site... (... was back then, and I was amazed, of course.)

> the real challenge in software engineering research is not what to do about ChatGPT or whatever else Silicon Valley is gushing about at the moment. I know this is really just a turn of phrase, but at the rate language models are becoming programming models, it is hard to think of any problem with the lack of hard engineering, science, math, ..., in what we call "Computer Engineering", "Computer Science", "Computer Math", ..., that won't be heavily impacted by model/co-model driven development. For instance, a major programming language of the near future might develop adaptively as multiple coder models communicate between each other. Reviewing, testing and improving upon each other's designs and co-optimizing the way they communicate code as they do so. Resulting in representations and solutions to existing development problems completely opaque to the humans currently working on those problems. The rising effect of massive data + compute outdoing slow careful research in yet another area. Today's models are already being trained across the board on practical/actual and theoretical/research artifacts. No persistent pernicious borders there. Not to take away from anything being said.

xanderlewis 3y ago
> any problem with the lack of hard engineering, science, math, … To be fair, that does exclude quite a lot of problems.
- JadeNB 3y ago
  > > any problem with the lack of hard engineering, science, math, … > To be fair, that does exclude quite a lot of problems. I had some trouble parsing the comment, but I eventually settled on the reading that it means what it literally says, a problem caused by the lack of engineering in computer engineering, of science in computer science, and of math in "computer math" (a phrase I have never heard), rather than a problem other than hard engineering, science, and math, which I agree with you excludes an awful lot of problems.
  - Nevermark 3y ago
    Yes, I meant the phrase respectively! Apologies for lack of clarity. My first computing programming class was called Computer Math. With not even a soft math emphasis.
wizzwizz4 3y ago
> Resulting in representations and solutions to existing development problems completely opaque to the humans currently working on those problems. But why would anyone want that? I agree that this would result, but my takeaway is "and therefore we shouldn't do that".
- Nevermark 3y ago
  > But why would anyone want that? I agree that this would result, but my takeaway is "and therefore we shouldn't do that". My guess is that a lot of CEO's already consider the implementation details of their companies' software systems as opaque. So that is essentially a pre-existing market for performative but opaque systems. CEO's. CFO's. Shareholders, etc. Even in engineering organizations. Look at how Boeing and NASA both demoted critical engineering concerns when external concerns became overwhelmingly more important to their leaders. Now factor in the models actually doing a good job, checking each other, and generally getting better year to year.

sethammons 3y ago

How do we see a list of the referenced papers that they feel applicable to industry that are not gaining purchase in mindshare?

sfink 3y ago
https://neverworkintheory.org/ https://neverworkintheory.org/ is the web site associated with this (from the 1st reference in the paper)
- dj_mc_merlin 3y ago
  These are pretty interesting talks but they very much feel like just random talks from any conference? Interesting but I have no way to apply them if I wasn't writing a tool for developers. edit: On another level I am put off by their tendency to describe things such as "teaching undergraduates" or "debugging problems" as algorithms. I understand that you can't really analyze unclear, hazy things without putting them down in clear terms, but something in me doesn't really accept that you _can_ do this without losing so much detail and information as to make your conclusion useless. They do point out that SWE often think of studies this way: > Most software developers in industry have never heard of any findings more recent than The Mythical Man-Month: Essays on Software Engineering (which few of them ever actually read) and routinely dismiss studies as “not statistically significant,” even when those studies are carefully done and directly relevant to their work.
  - cratermoon 3y ago
    Ah yes, the front page featuring the recorded talks kind of hides the real substance. Here's the full collection of papers, by category https://neverworkintheory.org/category/ https://neverworkintheory.org/category/
- sethammons 3y ago
  Awesome, thanks. I jumped into the first one that grabbed my attention: CDD - cognitive driven development (link below, 15min video). I love the premise but find the recommended results silly. TL;DW: cognitive research shows humans have working memory limited to 7 +/- 2 things. This means that classes should be limited in what they do. So far, I agree. And I think this can be extended, in practice. However, they back up this idea with a single case study and use a decorator to help count complexity to keep the classes small and state that methods not extending beyond 24 lines of code are ways to do this. This will just lead to cargo culture like "don't comment code." The decorator is not needed because code should (authoring or in review) be obvious if handling more than 5 to 9 things. The number of lines might be correlated to the number of things a function does, but that doesn't mean cognitive overhead. Like, a 100 line function that maps state abbreviations to state names is fine. Based on the presented data and mental performative drop off, it is not 7 +/- 2, it is really five + 0 to 2. I hereby dub this the Rule of Five. A practical example better than what they present in the talk: early returns. Every nested block adds to cognitive load, early returns allow you to discard that load as you continue reading through the function. The cyclomatic complexity reaching over 5 levels means many people will start to fail to keep all the blocks in working human memory. Therefore, early returns are a good implementation of Cognitive Driven Development. The case study they used seems meh. With a couple decades of experience with high growth start ups becoming "real" companies, I just don't buy their results on decorators and function line length. But they are on to something with limiting code to keep minimal cognitive overload down. https://m.youtube.com/watch?v=g_ht_LDm5Hw&feature=youtu.be https://m.youtube.com/watch?v=g_ht_LDm5Hw&feature=youtu.be

cushychicken 3y ago

Here’s my problem: I don’t want to work with researchers. Culturally, I find we’re simply not motivated by the same things. Researchers want to publish research. I want to ship working software. The more researchers I’ve met, the more convinced I become that those two goals are at odds with one another.

spacecadet 3y ago
Seems like a pov, I like working with researchers and shipping interesting working software. Why is it a problem they are at odds? Its not a researchers problem to create scalable production/consumer software or hell even git repos or libraries for you. Its yours! Thats your interest.
- trws 3y ago
  I’ve run into both. I sit between a research organization and a production organization and have to produce artifacts on both sides, this suits me quite well because I’m very much an applied research person. If it’s not feasible or tangible then it tends not to interest me much. Doing applied research and working with people who want to do that tends to work reasonably well with production, because the point is to research techniques and algorithms and so forth that help improve production. Theoretical or more future-focused research tends to need to focus on building minimal scaffolding to be able to test ideas, when that kind of scaffolding gets integrated into production software, that’s where the problems arise, because by necessity it’s a kind of MVP. I am right now working on rearchitecting a component that was built this way and labeled as production without going through hardening first (though it did go through a great deal of testing it didn’t get the code review it needed). In the end I suppose my point is that research and production aren’t inherently at odds, or even orthogonal. Applied research can be valuable to improving production software by studying current stable code and processes and researching improvements, but trying to do blue-sky research by incorporating code into production can be orthogonal or in some cases truly at odds with stability. It matters a great deal who is doing it, and how it’s managed.
  - spacecadet 3y ago

smitty1e 3y ago

My favorite Herb Sutter quote: "The difference between theory and practice is greater in practice than in theory."

sethammons 3y ago
"In theory, it will work in practice; in practice it is the other way around"
xanderlewis 3y ago
A slightly stronger form: "In theory there is no difference between theory and practice — in practice there is." - Yogi Berra.
- smitty1e 3y ago
  I like the way Sutter's ends on a prepositional phrase. Almost like the infinite loopiness of Celtic art, in theory.

glyphy177 3y ago

[dead]

pyb 3y ago

First time I hear of this, but one issue is that they've reviewed a pretty overwhelming number of papers, which dilutes their impact. In general, practitioners would be interested in hearing from advances in academia, but usually don't want to read papers. Building fully working systems - programming languages or libraries - is more impactful, but much higher-effort.

cratermoon 3y ago
> practitioners would be interested in hearing from advances in academia, but usually don't want to read papers. Building fully working systems Isn't that kind of the whole problem? Practitioners ignoring research results because all they want to look at is what other practitioners do? Or are you really suggesting that researchers also become practitioners who develop working systems based on the ideas they discover? Perhaps I'm misreading, but your statement seems to encapsulate the entire problem the article is highlighting.
- pyb 3y ago
  Researchers should become practictioners to some extent, if they want their ideas to spread. Example : Donald Knuth and TeX. Racket programming language. etc... Of course in this publish-or-perish era, this is probably no longer possible.

freeone3000 3y ago

Bridging the gap between research and practical programs is so great there’s an entire position, the RSDE, around it. Which you only encounter in places you have researchers and want to make their work a reality. Unfortunately, most researchers are publishing work about improving process, creating new and interesting things, and developing new ways of thinking about the science of computation, instead of what actually makes money, throwing out half-baked apps faster. There’s really no need for a formal understanding of what you’re actually doing and why if you can throw out a few trial-and-error projects and one gains enough traction.

tgv 3y ago

> students design a small study or experiment, collect data, analyze them, and figure out what (if anything) they’ve proven Based on ample experience, both first and second hand, that's going to fail. It is really hard to instill proper methodology, domain modelling, and statistics in students, and since the premise already is that it "would not disrupt other curricula" (PM speak for: little time and attention), students are going to walk away with the wrong ideas and think that they now understand experimental science.

moron4hire 3y ago
I did both a group-project software engineering course and a research course in my undergraduate Computer Science program. It's definitely possible to do.

llm_trw 3y ago

>In the wake of recent political events, our community’s energy and attention should be focused on more important things. Living like this sounds exhausting. The only thing worse I can imagine is having to listen to people who actually think like this.

brian_spiering 3y ago
I was working with a non-profit that taught people around the world how to teach tech skills in their local communities. There was a shift to focusing on recent political events which was a major reason for my leaving. My take was if you personally want to do that, go for it. However, reacting to recent political events was not part of the group's mission. I wanted to spend my limited time on the group's primary mission of education.
- llm_trw 3y ago
  The non-profit implosion over the last 8 years has been something to behold. It's like someone took the CIA handbook on how to sabotage institutions and used it on every organization more left wing than the John Birch Society.
travisjungroth 3y ago
There’s also a weird dynamic where this applies to production but not consumption. I can’t remember which, but there’s a social media site that is donation based. Every year they do a fundraising drive. Every year the users flip out they would try to raise money with Everything Going On Right Now. The founder posted on their blog about how every year has things going on, and they still need to raise money. And the users completely backlash against all forms of monetization. What I didn’t see anyone mention is the users are still using the site. Apparently using the site is really important. Just not making it, or earning a living while making it.