7 ms·

The underwhelming impact of software engineering research

139 points by signa11 5y ago

deleted 5y ago
[deleted]
TheAceOfHearts 5y ago
This is a point of frustration to me. It seems like there's a significant disconnect between research in academia and research in the real world. Beyond that, a lot of software engineering research doesn't seem to be easily reachable or discovered unless you're in academia. Real world software engineering has to ship a real working product, while academic software engineering only has to produce a paper. Maybe that's an unfair representation though.
- sheikheddy 5y ago
  Try reading some of the older, classic papers. I think those have taught me a lot about the fundamentals.
  - 0des 5y ago
    There could be the next Knuth among us. Excited to see what they bring.
    - BoiledCabbage 5y ago
      Either optimizing ad-clicks on the web. Or a new NFT crypto tie in sham.
  - debbiedowner 5y ago
    What's your list of classic papers for language models for code, static analysis, or code summarization?
- corpMaverick 5y ago
  I feel that in the past there were more products shipped by Academics. Unix BSD from Berkley, Minix from Tanembaum, Kermit from Columbia, Pascal from USCD; Pascal, Modula and Oberon from N. Wirth, etc. Building these products would be a great teaching tool and contribution to society. Also there could be a lot of research done on how to build complex software where the product is the software itself and it can be applied to other sciences that badly need software. I for one would love to get a graduate degree while building some product where I get to try different software design approaches.
- hinkley 5y ago
  I told a coworker once that I subscribed to the ACM (which I don’t anymore), and he looked at me like I’d just parted the Red Sea. He was amazed I could understand them. I told him I could only understand about half (closer to 3/4) which mollified him a bit. There are any number of reasons I have stayed engineering track instead of going into management but the biggest is that talking about things and doing them are vastly different, and we’ve structured management and research around talking. Some days I worry I may know more about project management than I know about cloud or React, but it would kill me to be telling people how to do the job that I’m not also doing, and if I don’t like how they did it, have only words available to get it fixed. In one of my hobbies, time and the pandemic have slowed me down quite a bit and I’m trying to manage more and do less, but unlike work there are plenty of opportunities to lead by doing. Doing 5-10% of the work and then guiding people to do the rest is socially acceptable. In software we sometimes call that a lead developer, but that definition changes from situation to situation. But the more you do the less influence you tend to have on what you’re doing and why. Is that human nature or leftover from feudalism?
- ModernMech 5y ago
  > It seems like there's a significant disconnect between research in academia and research in the real world. It's funny, back in my undergrad days I would walk past the Google offices that opened near my campus. I'd look in and see a giant wall-sized candy machine, primary colors everywhere, bean bags, pool tables, an air hockey table. I honestly thought it was a daycare of some kind until I asked someone why there's never any kids in there. This is what they call the "real world" apparently, as a contrast to what goes on in academia. I can definitely say that in all my years in academia, I've never been treated to a candy wall.
adam_arthur 5y ago
I often read SWE research that is just kind of common sense to somebody in industry. Of course, it's worth trying to take an empirical approach to quantify merits of certain techniques, but the vast majority of the research that's done does not seem that novel.
- sheikheddy 5y ago
  I'm fairly certain that you could probably write and publish a decent quality paper yourself. But why would you, when industry is so much more lucrative? There are many competent people in CS Academia, but their motivations tend to be quite different from the average engineer, and they don't have access to the same resources or expertise that tech giants or startups do (outside of internships and fellowships that are commonplace for visiting researchers who spend a season or two working at a lab)
debbiedowner 5y ago
What about deep learning approaches like this one: https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html?m=1 https://ai.googleblog.com/2022/04/pathways-language-model-pa... ? Or GitHub copilot?
tinco 5y ago
I think it's not so much the lack of good research being done, but the lack of real world application of that research. And furthermore the real applicable research wasn't really theoretical research but more lessons learned by engineers in the industry. It's crazy to me that researchers prefer to be published over having world renowned software to their name. You could figure out the fanciest database research, but there was a period where every fancy new NoSQL database was based on BerkeleyDB despite all that, bolstering the reputation of the University of Berkeley. Despite all the research into garbage collectors nearly every new programming language I've seen uses either BoehmGC (from 1994) or simply gives up on the idea and uses reference counting. Imagine if university groups would band together to produce really state of the art production quality reference software. They could use them as base for their research, bolster their reputation by having them be great, and even draw funding from big companies looking to hire the PhD's produced by the efforts. Instead we have got all these half finished demonstration software that no one industry would touch with a ten foot pole. It took just a few people to develop the first versions of rust, a research based programming language that in my opinion has been the biggest shock to the industry in decades (if you disregard machine learning).
- amelius 5y ago
  People should code in Haskell more, and we would have massive amounts of reusable code. All the problems stem from premature optimization, and rush to market.
  - unmole 5y ago
    Yeah. Won't have any software problems if there isn't a software industry to begin with.
  - coldtea 5y ago
    >and we would have massive amounts of reusable code Haskell even has problems coming up with a single string / text interface. It's not some magic bullet for code reuse, any more "everybody would write Java and we would have had massive amounts of reusable jar libs".
HstryrsrBttn 5y ago
Barrin92 5y ago
I don't think the problem is located in academia or research, it's in the market and user obsession of the software industry. For a very long time we didn't produce software primarily for consumers, we produced software for engineers and researchers. Bell Labs and the corporate research lab as an institution with its significant freedoms is an example of this. A lot of innovation was the result, including practical innovation. Hardware still works this way today basically. The performance optimizations, practices and goals are engineering centric, not consumer centric. Consumer benefits happen downstream as a side effect. The hardware sector is still run by scientists and engineers, one only needs to look at the people running the companies. There's no shortage of great software research that could be put into practice, plenty of languages with great features are underused. It's the result of the mentality in software that puts costs, users and marketshare over an engineering culture.
- randcraw 5y ago
  I agree. I think a central reason that software, and the process for creating it, have not improved is that no standards for software quality have been demanded of the industry. The creation of no electrical or mechanical device on which people comparably rely is as underregulated. Few software developers are required to acquire professional certification, or perform in compliance to standard practices intended to assure due diligence in design, implementation, testing or documentation. Very few software systems must undergo compliance certification. It's unsurprising then that software engineering continues to founder and fail to advance as a discipline.
  - closeparen 5y ago
    I am glad we did not install standards to freeze the way software was developed in e.g. the 1990s, and I expect that in 25 years I’ll be equally pleased that we did not set standards today.
  - native_samples 5y ago
    It hasn't failed to advance as a discipline. I'm nowhere near retirement and even so I've seen massive advances in my lifetime: 1. The widespread adoption of unit testing and associated techniques like continuous integration. A massive boost to software engineering reliability and productivity. 2. Introduction of crash reporters and associated analytics, this enabled huge improvements in software stability. On the server side, good logging and monitoring systems provide an equivalent. 3. Sandboxing going from unheard of to relatively widely used. 4. Design, development and then open sourcing of extremely robust and well engineered runtimes like the JVM, .NET etc. This has huge benefits throughout the entire industry. 5. Open source and the huge code reuse it made possible. etc etc. All these things have been drastically more impactful than bureaucratic central planning a la certifications.
jillesvangurp 5y ago
I did a PhD in the field. And only then became an actual software engineer. As it turns out, doing software engineering research does not involve a whole lot of engineering software. So, I had a lot to learn. So, this criticism resonates a bit with me. The problem with software engineering as a scientific discipline is that many engineers probably would not consider it a proper engineering discipline to begin with. Also, many computer scientists look down on it as not really related to that either because it is a bit of a soft science. So, there's a problem with researchers in this field not really having the right background to do good research: - they are typically not experienced software engineers. Quite many, like me, start working on a PhD without actually having worked as a software engineer. - they typically have studied computer science which teaches you a lot of math, algorithms, formal methods, and other typical beta skills but is perhaps a bit light on the type of stuff a sociologist or anthropologist would learn. That's a problem because a lot of software engineering involves the behavior of people in groups and coordinating that behavior to produce software. Studying that behavior to verify theories you might have about how software engineering works or ought to work requires an empirical approach. That's not something a lot of computer scientists learn how to do. So there's a lot of software engineering research that gets published that you could accuse of being a combination of naive, wrong/misguided, and poorly executed in terms of its methodology. Mathematicians or computer scientists trying to be empirical is not great typically. Sometimes you get some decent papers but there is a lot of pseudo scientific nonsense that gets published. Doing empirical research properly is hard and many computer scientists have no clue how to do it properly nor any appreciation for doing so. Just looking at ICSE's program last year you can see from the titles a weird mix of empirical studies of various topics and other papers clearly written by more computer science leaning types working on algorithms, machine learning or tools. Looks like there were some interesting papers in there but also a lot eye brow raising titles like "How Java Programmers Test Exceptional Behavior". I actually skimmed through the pdf of that one and it fits the pattern beautifully. It's an actual empirical study that tries to answer the question in the title. That's it. There's no new theories being formulated. No remarkable findings that were reported. It's just a very dry survey that yielded exactly what you'd expect. The answers to the research question are almost beside the point and should be completely unsurprising to any Java programmer. Great example of a well executed emprical research paper that answers a question any Java developer in need of actually testing some code that throws exceptions would answer with a simple Google query in a few minutes. So, yes underwhelming impact indeed.
ta988 5y ago
I wish there would be a publication habit for industry SWE, we could all benefit from the reports from the trenches. I know we have blogs and the occasional books but I would really enjoy an in-between.
ford 5y ago
IMO most of the practical "research" I've seen comes in the form of best practices from Big Co. who a) hire very intelligent people and b) have extreme financial incentives to write the best code & have the best processes possible. In a world where the dependent variable isn't something that can easily be measured in a lab, the next best we can do is either 1\ Ask experts or 2\ experiment in an environment where it's financially advisable to experiment & measure (AKA an environment where running an experiment with 100 engineers is <1% of your engineering headcount). I.e. I don't work for Google, but I do share things like [0, 1] with new engineers who join my team. [0] https://google.github.io/eng-practices/ https://google.github.io/eng-practices/ [1] https://arxiv.org/ftp/arxiv/papers/1702/1702.01715.pdf https://arxiv.org/ftp/arxiv/papers/1702/1702.01715.pdf
wrnr 5y ago
One might just write an article about the underwhelming impact of commenting on the internet.
matthewdgreen 5y ago
This article would be a lot more convincing if it illustrated its points with examples. I have no doubt that there is truth here, but the objections here are fairly general and unsubstantial. For example, take the scaling objection: give some examples of useful techniques that won’t scale and then talk about what you might do differently.
nonrandomstring 5y ago
Sommerville was my graduate software engineering textbook in the late 1980s. Along with Jackson, Yourdon and a few others it was the bedrock of my thinking about how to manage software projects over the past 30 years. It's not as if I got stuck in that mind-set, having been a keen, early adopter (and early critic) of many faddish methodologies over the years - but I've been astounded by the almost total lack of deep traction these philosophies have had in cultivating a mature, long term, responsible engineering attitude in our business. I am really pleased to hear a heavyweight like Ian Sommerville pitching in on what I've been seeing as a cultural crisis in software spanning almost a couple of decades now. It's disappointing to hear the usual dismissal of research and lionising of "industry" as if a cowboy "move fast and break things" mind-set constituted a single coherent commercial ideology. Can you imagine if the pharmaceutical or aviation industry took a cavalier attitude towards "eggheads with their cell biology theories" or "crazy academics with their fancy aerodynamic equations". Is software the only engineering discipline where the product gets less efficient each year? Partly I think it's also the public's fault for accepting such shoddy produce, and of governments for not being stricter because of a Faustian bargain to enjoy mass surveillance and cybernetic social control as vicarious side effects of a hands-off approach. It must be frustrating to have put a lifetime into carefully considered formalisms and see so little uptake. Anyway, what we've ended up with a bunch of non-interoperable, insecure devices and services built by well paid but mostly unhappy workers who have almost no grasp of the complexity, history or societal import of what they create. Sooner or later the chickens are coming home to roost, as cybersecurity, resilience or interoperability issues. Says Sommerville: "As a community, senior members (and in this I include myself) have failed to create an inclusive environment where new ideas from outside of the traditional elite are supported. We do not act together when dealing with policy makers and funding agencies and, with rare exceptions, have not spoken out against the futile short-termism that has hampered the development of our field." I absolutely salute you Ian. I wrote similarly a year ago in my last book that I feel a great sense of regret as a computer scientist who did not speak out sooner and louder. Perhaps it's grandiose/omnipotent to think we ever could have made a difference, but if not our generation then I really hope a new, radical generation will emerge who have the courage to "take back tech".
Animats 5y ago
Is that all he has to say? That reads like a cover sheet. There's been considerable progress in programming languages and tools. It's now possible to write very large programs with extensive concurrency without having them crash frequently. There's been progress in project organization tools. There's Github, and Slack, and Stack Overflow. It's now possible for a large number of people to collaborate on something and still get it to work. That's pretty good, actually.
- ogogmad 5y ago
  > There's been progress in project organization tools. There's Github, and Slack, and Stack Overflow. It's now possible for a large number of people to collaborate on something and still get it to work. Does this stuff come from academia? That's what's being debated.
hyperpape 5y ago
I think it’s important to understand what the author means by “software engineering research”. If I understand it correctly, he means research about the discipline of software engineering, not the output of engineering. So research on analyzing agile and waterfall approaches would be “software engineering research”, a programming languages paper would not be. A paper on whether static typing reduces defects is a grey area to me—-I think it’s SE research, but I’m not sure. It’s clear that there’s been lots of progress in technology: in networking, in databases, in compilers, etc. It’s just that none of that was substantially influenced by research on the process of software engineering.
- jaimebuelta 5y ago
  The problem is obtaining solid data and not falling on cargo cult. If there’s a paper showing that static typing is safer, but is assuming A, B and C, it can be very easy to end up with a lot of people saying that only static typing should be used in every imaginable situation. Applying the results of research on these kind of practices can be incredibly difficult. Engineering processes are not a hard science…
lostdog 5y ago
As a software engineer in the industry, what problems am I hoping academic research could investigate. Compilers that work like devs do. Almost every time I compile, the code is almost exactly the same, but the compiler forgets everything and throws away details from the last compile. Probably all the optimizer's choices are going to be the same this time as last time, so maybe we could cache these? Better code merging. When my code diverges from the main branch, I typically have to do a bunch of merging work. Would a merging algorithm that understands code semantics be able to do a better job of understanding what I mean during the merge? Language safety and ergonomics is huge. I very much appreciate Rust's progress, and it's a shining example of academia brought to industry, but can we do more? Jai's idea of making constraints codeable and incremental is pretty cool, and I bet there's a lot of room to improve on it. Measures of code complexity and modularity. There's a bunch of silly measurements of the "goodness" of code, but maybe it's time for another! Maybe we could describe how "replaceable" components are automatically, and gives a solid idea of the level of real tech debt in the codebase. How about some project management focused studies? Have a programmer give an estimate for a task, try to force it to be a deadline, and watch as quality of the results drops. Try applying something like scaled agile framework to a real problem and watch as the meeting load snuffs out meaningful development progress. Or some systems work. Apache airflow is a pretty cool job running agent, but I have yet to see a good error handling workflow for distributed job agents. Can you come up with the "Result" type of job scheduling that's nicely composable? There's really so much to do.
- UncleMeat 5y ago
  Cached builds are a thing but actual incremental analysis is a very hard problem. Seemingly unrelated changes a pretty common sources of interesting analyses bailing so you get more spooky action at a distance than you might expect. This gets worse with whole program optimization.
  - jaimebuelta 5y ago
    Perhaps a good dependency process would be an interesting research. Knowing what parts change when changing a small bit it’s actually very difficult to know, and it seems like a common problem that could improve things in many areas (for example, compilation, but also knowing what tests to run if some piece of the code changes, for example)

139 points by signa11 5y ago

deleted 5y ago

[deleted]

This is a point of frustration to me. It seems like there's a significant disconnect between research in academia and research in the real world. Beyond that, a lot of software engineering research doesn't seem to be easily reachable or discovered unless you're in academia. Real world software engineering has to ship a real working product, while academic software engineering only has to produce a paper. Maybe that's an unfair representation though.

sheikheddy 5y ago
Try reading some of the older, classic papers. I think those have taught me a lot about the fundamentals.
- 0des 5y ago
  There could be the next Knuth among us. Excited to see what they bring.
  - BoiledCabbage 5y ago
    Either optimizing ad-clicks on the web. Or a new NFT crypto tie in sham.
- debbiedowner 5y ago
  What's your list of classic papers for language models for code, static analysis, or code summarization?
corpMaverick 5y ago
I feel that in the past there were more products shipped by Academics. Unix BSD from Berkley, Minix from Tanembaum, Kermit from Columbia, Pascal from USCD; Pascal, Modula and Oberon from N. Wirth, etc. Building these products would be a great teaching tool and contribution to society. Also there could be a lot of research done on how to build complex software where the product is the software itself and it can be applied to other sciences that badly need software. I for one would love to get a graduate degree while building some product where I get to try different software design approaches.
hinkley 5y ago
I told a coworker once that I subscribed to the ACM (which I don’t anymore), and he looked at me like I’d just parted the Red Sea. He was amazed I could understand them. I told him I could only understand about half (closer to 3/4) which mollified him a bit. There are any number of reasons I have stayed engineering track instead of going into management but the biggest is that talking about things and doing them are vastly different, and we’ve structured management and research around talking. Some days I worry I may know more about project management than I know about cloud or React, but it would kill me to be telling people how to do the job that I’m not also doing, and if I don’t like how they did it, have only words available to get it fixed. In one of my hobbies, time and the pandemic have slowed me down quite a bit and I’m trying to manage more and do less, but unlike work there are plenty of opportunities to lead by doing. Doing 5-10% of the work and then guiding people to do the rest is socially acceptable. In software we sometimes call that a lead developer, but that definition changes from situation to situation. But the more you do the less influence you tend to have on what you’re doing and why. Is that human nature or leftover from feudalism?
ModernMech 5y ago
> It seems like there's a significant disconnect between research in academia and research in the real world. It's funny, back in my undergrad days I would walk past the Google offices that opened near my campus. I'd look in and see a giant wall-sized candy machine, primary colors everywhere, bean bags, pool tables, an air hockey table. I honestly thought it was a daycare of some kind until I asked someone why there's never any kids in there. This is what they call the "real world" apparently, as a contrast to what goes on in academia. I can definitely say that in all my years in academia, I've never been treated to a candy wall.

adam_arthur 5y ago

I often read SWE research that is just kind of common sense to somebody in industry. Of course, it's worth trying to take an empirical approach to quantify merits of certain techniques, but the vast majority of the research that's done does not seem that novel.

sheikheddy 5y ago
I'm fairly certain that you could probably write and publish a decent quality paper yourself. But why would you, when industry is so much more lucrative? There are many competent people in CS Academia, but their motivations tend to be quite different from the average engineer, and they don't have access to the same resources or expertise that tech giants or startups do (outside of internships and fellowships that are commonplace for visiting researchers who spend a season or two working at a lab)

debbiedowner 5y ago

What about deep learning approaches like this one: https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html?m=1 https://ai.googleblog.com/2022/04/pathways-language-model-pa... ? Or GitHub copilot?

tinco 5y ago

I think it's not so much the lack of good research being done, but the lack of real world application of that research. And furthermore the real applicable research wasn't really theoretical research but more lessons learned by engineers in the industry. It's crazy to me that researchers prefer to be published over having world renowned software to their name. You could figure out the fanciest database research, but there was a period where every fancy new NoSQL database was based on BerkeleyDB despite all that, bolstering the reputation of the University of Berkeley. Despite all the research into garbage collectors nearly every new programming language I've seen uses either BoehmGC (from 1994) or simply gives up on the idea and uses reference counting. Imagine if university groups would band together to produce really state of the art production quality reference software. They could use them as base for their research, bolster their reputation by having them be great, and even draw funding from big companies looking to hire the PhD's produced by the efforts. Instead we have got all these half finished demonstration software that no one industry would touch with a ten foot pole. It took just a few people to develop the first versions of rust, a research based programming language that in my opinion has been the biggest shock to the industry in decades (if you disregard machine learning).

amelius 5y ago
People should code in Haskell more, and we would have massive amounts of reusable code. All the problems stem from premature optimization, and rush to market.
- unmole 5y ago
  Yeah. Won't have any software problems if there isn't a software industry to begin with.
- coldtea 5y ago
  >and we would have massive amounts of reusable code Haskell even has problems coming up with a single string / text interface. It's not some magic bullet for code reuse, any more "everybody would write Java and we would have had massive amounts of reusable jar libs".

HstryrsrBttn 5y ago

Barrin92 5y ago

I don't think the problem is located in academia or research, it's in the market and user obsession of the software industry. For a very long time we didn't produce software primarily for consumers, we produced software for engineers and researchers. Bell Labs and the corporate research lab as an institution with its significant freedoms is an example of this. A lot of innovation was the result, including practical innovation. Hardware still works this way today basically. The performance optimizations, practices and goals are engineering centric, not consumer centric. Consumer benefits happen downstream as a side effect. The hardware sector is still run by scientists and engineers, one only needs to look at the people running the companies. There's no shortage of great software research that could be put into practice, plenty of languages with great features are underused. It's the result of the mentality in software that puts costs, users and marketshare over an engineering culture.

randcraw 5y ago
I agree. I think a central reason that software, and the process for creating it, have not improved is that no standards for software quality have been demanded of the industry. The creation of no electrical or mechanical device on which people comparably rely is as underregulated. Few software developers are required to acquire professional certification, or perform in compliance to standard practices intended to assure due diligence in design, implementation, testing or documentation. Very few software systems must undergo compliance certification. It's unsurprising then that software engineering continues to founder and fail to advance as a discipline.
- closeparen 5y ago
  I am glad we did not install standards to freeze the way software was developed in e.g. the 1990s, and I expect that in 25 years I’ll be equally pleased that we did not set standards today.
- native_samples 5y ago
  It hasn't failed to advance as a discipline. I'm nowhere near retirement and even so I've seen massive advances in my lifetime: 1. The widespread adoption of unit testing and associated techniques like continuous integration. A massive boost to software engineering reliability and productivity. 2. Introduction of crash reporters and associated analytics, this enabled huge improvements in software stability. On the server side, good logging and monitoring systems provide an equivalent. 3. Sandboxing going from unheard of to relatively widely used. 4. Design, development and then open sourcing of extremely robust and well engineered runtimes like the JVM, .NET etc. This has huge benefits throughout the entire industry. 5. Open source and the huge code reuse it made possible. etc etc. All these things have been drastically more impactful than bureaucratic central planning a la certifications.

jillesvangurp 5y ago

I did a PhD in the field. And only then became an actual software engineer. As it turns out, doing software engineering research does not involve a whole lot of engineering software. So, I had a lot to learn. So, this criticism resonates a bit with me. The problem with software engineering as a scientific discipline is that many engineers probably would not consider it a proper engineering discipline to begin with. Also, many computer scientists look down on it as not really related to that either because it is a bit of a soft science. So, there's a problem with researchers in this field not really having the right background to do good research: - they are typically not experienced software engineers. Quite many, like me, start working on a PhD without actually having worked as a software engineer. - they typically have studied computer science which teaches you a lot of math, algorithms, formal methods, and other typical beta skills but is perhaps a bit light on the type of stuff a sociologist or anthropologist would learn. That's a problem because a lot of software engineering involves the behavior of people in groups and coordinating that behavior to produce software. Studying that behavior to verify theories you might have about how software engineering works or ought to work requires an empirical approach. That's not something a lot of computer scientists learn how to do. So there's a lot of software engineering research that gets published that you could accuse of being a combination of naive, wrong/misguided, and poorly executed in terms of its methodology. Mathematicians or computer scientists trying to be empirical is not great typically. Sometimes you get some decent papers but there is a lot of pseudo scientific nonsense that gets published. Doing empirical research properly is hard and many computer scientists have no clue how to do it properly nor any appreciation for doing so. Just looking at ICSE's program last year you can see from the titles a weird mix of empirical studies of various topics and other papers clearly written by more computer science leaning types working on algorithms, machine learning or tools. Looks like there were some interesting papers in there but also a lot eye brow raising titles like "How Java Programmers Test Exceptional Behavior". I actually skimmed through the pdf of that one and it fits the pattern beautifully. It's an actual empirical study that tries to answer the question in the title. That's it. There's no new theories being formulated. No remarkable findings that were reported. It's just a very dry survey that yielded exactly what you'd expect. The answers to the research question are almost beside the point and should be completely unsurprising to any Java programmer. Great example of a well executed emprical research paper that answers a question any Java developer in need of actually testing some code that throws exceptions would answer with a simple Google query in a few minutes. So, yes underwhelming impact indeed.

ta988 5y ago

I wish there would be a publication habit for industry SWE, we could all benefit from the reports from the trenches. I know we have blogs and the occasional books but I would really enjoy an in-between.

ford 5y ago

IMO most of the practical "research" I've seen comes in the form of best practices from Big Co. who a) hire very intelligent people and b) have extreme financial incentives to write the best code & have the best processes possible. In a world where the dependent variable isn't something that can easily be measured in a lab, the next best we can do is either 1\ Ask experts or 2\ experiment in an environment where it's financially advisable to experiment & measure (AKA an environment where running an experiment with 100 engineers is <1% of your engineering headcount). I.e. I don't work for Google, but I do share things like [0, 1] with new engineers who join my team. [0] https://google.github.io/eng-practices/ https://google.github.io/eng-practices/ [1] https://arxiv.org/ftp/arxiv/papers/1702/1702.01715.pdf https://arxiv.org/ftp/arxiv/papers/1702/1702.01715.pdf

wrnr 5y ago

One might just write an article about the underwhelming impact of commenting on the internet.

matthewdgreen 5y ago

This article would be a lot more convincing if it illustrated its points with examples. I have no doubt that there is truth here, but the objections here are fairly general and unsubstantial. For example, take the scaling objection: give some examples of useful techniques that won’t scale and then talk about what you might do differently.

nonrandomstring 5y ago

Sommerville was my graduate software engineering textbook in the late 1980s. Along with Jackson, Yourdon and a few others it was the bedrock of my thinking about how to manage software projects over the past 30 years. It's not as if I got stuck in that mind-set, having been a keen, early adopter (and early critic) of many faddish methodologies over the years - but I've been astounded by the almost total lack of deep traction these philosophies have had in cultivating a mature, long term, responsible engineering attitude in our business. I am really pleased to hear a heavyweight like Ian Sommerville pitching in on what I've been seeing as a cultural crisis in software spanning almost a couple of decades now. It's disappointing to hear the usual dismissal of research and lionising of "industry" as if a cowboy "move fast and break things" mind-set constituted a single coherent commercial ideology. Can you imagine if the pharmaceutical or aviation industry took a cavalier attitude towards "eggheads with their cell biology theories" or "crazy academics with their fancy aerodynamic equations". Is software the only engineering discipline where the product gets less efficient each year? Partly I think it's also the public's fault for accepting such shoddy produce, and of governments for not being stricter because of a Faustian bargain to enjoy mass surveillance and cybernetic social control as vicarious side effects of a hands-off approach. It must be frustrating to have put a lifetime into carefully considered formalisms and see so little uptake. Anyway, what we've ended up with a bunch of non-interoperable, insecure devices and services built by well paid but mostly unhappy workers who have almost no grasp of the complexity, history or societal import of what they create. Sooner or later the chickens are coming home to roost, as cybersecurity, resilience or interoperability issues. Says Sommerville: "As a community, senior members (and in this I include myself) have failed to create an inclusive environment where new ideas from outside of the traditional elite are supported. We do not act together when dealing with policy makers and funding agencies and, with rare exceptions, have not spoken out against the futile short-termism that has hampered the development of our field." I absolutely salute you Ian. I wrote similarly a year ago in my last book that I feel a great sense of regret as a computer scientist who did not speak out sooner and louder. Perhaps it's grandiose/omnipotent to think we ever could have made a difference, but if not our generation then I really hope a new, radical generation will emerge who have the courage to "take back tech".

Animats 5y ago

Is that all he has to say? That reads like a cover sheet. There's been considerable progress in programming languages and tools. It's now possible to write very large programs with extensive concurrency without having them crash frequently. There's been progress in project organization tools. There's Github, and Slack, and Stack Overflow. It's now possible for a large number of people to collaborate on something and still get it to work. That's pretty good, actually.

ogogmad 5y ago
> There's been progress in project organization tools. There's Github, and Slack, and Stack Overflow. It's now possible for a large number of people to collaborate on something and still get it to work. Does this stuff come from academia? That's what's being debated.

hyperpape 5y ago

I think it’s important to understand what the author means by “software engineering research”. If I understand it correctly, he means research about the discipline of software engineering, not the output of engineering. So research on analyzing agile and waterfall approaches would be “software engineering research”, a programming languages paper would not be. A paper on whether static typing reduces defects is a grey area to me—-I think it’s SE research, but I’m not sure. It’s clear that there’s been lots of progress in technology: in networking, in databases, in compilers, etc. It’s just that none of that was substantially influenced by research on the process of software engineering.

jaimebuelta 5y ago
The problem is obtaining solid data and not falling on cargo cult. If there’s a paper showing that static typing is safer, but is assuming A, B and C, it can be very easy to end up with a lot of people saying that only static typing should be used in every imaginable situation. Applying the results of research on these kind of practices can be incredibly difficult. Engineering processes are not a hard science…

lostdog 5y ago

As a software engineer in the industry, what problems am I hoping academic research could investigate. Compilers that work like devs do. Almost every time I compile, the code is almost exactly the same, but the compiler forgets everything and throws away details from the last compile. Probably all the optimizer's choices are going to be the same this time as last time, so maybe we could cache these? Better code merging. When my code diverges from the main branch, I typically have to do a bunch of merging work. Would a merging algorithm that understands code semantics be able to do a better job of understanding what I mean during the merge? Language safety and ergonomics is huge. I very much appreciate Rust's progress, and it's a shining example of academia brought to industry, but can we do more? Jai's idea of making constraints codeable and incremental is pretty cool, and I bet there's a lot of room to improve on it. Measures of code complexity and modularity. There's a bunch of silly measurements of the "goodness" of code, but maybe it's time for another! Maybe we could describe how "replaceable" components are automatically, and gives a solid idea of the level of real tech debt in the codebase. How about some project management focused studies? Have a programmer give an estimate for a task, try to force it to be a deadline, and watch as quality of the results drops. Try applying something like scaled agile framework to a real problem and watch as the meeting load snuffs out meaningful development progress. Or some systems work. Apache airflow is a pretty cool job running agent, but I have yet to see a good error handling workflow for distributed job agents. Can you come up with the "Result" type of job scheduling that's nicely composable? There's really so much to do.

UncleMeat 5y ago
Cached builds are a thing but actual incremental analysis is a very hard problem. Seemingly unrelated changes a pretty common sources of interesting analyses bailing so you get more spooky action at a distance than you might expect. This gets worse with whole program optimization.
- jaimebuelta 5y ago
  Perhaps a good dependency process would be an interesting research. Knowing what parts change when changing a small bit it’s actually very difficult to know, and it seems like a common problem that could improve things in many areas (for example, compilation, but also knowing what tests to run if some piece of the code changes, for example)