4 ms·

The black hole of software engineering research (2015)

62 points by mindcrime 5y ago

wyldfire 5y ago
> I’ve referred to empirical studies that strongly suggest the adoption of particular practices, but experience, anecdote, and context have usually won out over evidence like...what? which practices? > I’ve demoed exciting tools from research, but my team has found them mostly useless, since they aren’t production ready. just getting teams on board with "classic" tools like static analysis and sanitizers is major. And those can usually be described as 'production ready'
- secondaryacct 5y ago
  The guy clearly does it the wrong way, he has a solution and everyone is looking at him "but what problem do we have this solves". So yeah, I mean even if these tools were production ready, they wouldnt be used. If they solved a problem people had, you d bet we d stay at night to make them production ready. He seems to think his devs are like mindless zombies needing constant intellectual feeding from him, the researching CTO, but it s probably more like they raise their eyes "oh god what is the guy producing no value day to day going to show us again while we stress on the deadline pretending to listen to him"
- AtlasBarfed 5y ago
  For an article bemoaning the failed communication of academic software engineering research to actual practice... It fails to provide documentation of academic software engineering research that they have tried to apply to actual practice.
jrumbut 5y ago
I see a couple of other issues, which share a root cause. 1. A lot of research is based on mining open source projects which bear little resemblance to how software engineering is done behind closed doors. 2. Very few prospective trials (much less randomized controlled trials), which are considered the gold standard of evidence for a good reason. These retrospective studies are very useful, but to really have faith in a conclusion that X is better than Y you need to assign several engineering teams to condition X and several more to condition Y and then compare. Nobody has money for that and it would be hard to find even one team (let alone several) that is going to let a local professor flip a coin and decide whether they are going to do Scrum or Kanban. You'd also need quite a few teams because there would be so many uncontrolled aspects. Researchers in this area are in the very difficult situation where the interesting things are difficult to measure and the things that are easy to measure aren't that interesting. Somehow they still do good work and that's very impressive but I don't envy their task.
- kqr 5y ago
  I think the whole "you need too many subjects and too much money" objection to doing science is very weak. There are other fields of research where they find inventive ways of experimenting on a shoestring budget (I think some social sciences belong in this category.) You can even do proper causal analysis when all you have is an observational study, given that you can get an expert to produce a decent guess at the causal network. (And this guess can to some extent be verified by plain correlations.)[1] Sure, you might not get the extremely high external validity of e.g. particle physics, but in many cases something is better than nothing.[2] Heck, screw external validity. I'd be ecstatic to see more experiments just within an organisation, focusing on internal validity. Every engineering organisation can afford to experiment with different tools and practises. There are huge productivity gains to be unlocked (is my hypothesis, anyway), and the cost is small. If we get together enough small studies lacking external validity, but which point in the same direction, we might even be able to infer some external validity between them. [1]: For more on this approach, see Judea Pearl. [2]: To use an example close to my heart: pregnancy is a condition where you rarely see randomised trials, and any research is sorely underfunded. Yet when people actually take the time to sit down and answer questions with data, we get something meaningful out of it.
  - regularfry 5y ago
    It's worth pointing out that this blog post was published three years before Accelerate, by Forsgren, Humble, and Kim. That book demonstrates what you can do just with surveys. It's not about software engineering per se - it's got a devops focus - but the overlap is significant. As far as I'm concerned that book significantly changed the game, not only because of the findings, but because it showed that hard results are possible in this arena. The downside is that you need enough data to make the maths work, and that precludes applying the techniques to internal studies in most organisations. There's just too much noise.
  - jrumbut 5y ago
    I wouldn't be surprised if a few organizations had run major experiments over the years, with results never seeing the light of day. It would also be great if there was more awareness of the kind of data the research community needs so that if someone saw a natural experiment shaping up in their company they knew who to call. Regarding budgets, these researchers are a lot smarter than me and I bet they are doing their best with what they have. What might be good though is having several labs join forces and collaborate on one very good trial of a core issue. Then you have a hope of settling a question and moving the field forward while creating some infrastructure that will help answer other big questions. Maybe that's happening, I haven't seen a lot of it though.
devwastaken 5y ago
The reason there's little research is because there is little money and structure in STEM education in general. I have been to multiple name brand uni's that simply do not have basic lab equipment. Some high schools had more equipment and expertise in software than uni's did. Why? Because education as a business has fundamentally broken incentives. Most educators have never worked in the private industry, or did so for very little time. This means they don't actually know what they're talking about, they just follow courses accreditation allows them to. You will never make as much as a professor in software than you will privately, unless you are in one of the top schools that are inaccessible to the overwhelming majority of the population. The workplace politics are egregious, because universities are not a meritocracy, universities have created a system where your skin color matters a lot more than your qualifications to conduct your job. For example, someone becoming head of department because they're from India, not because the product of their research or overall conduct in their job was the superior choice. This trickles down, students become the professor, and over time we are now here. Tech companies are the wealthiest in the world out of any major, yet contribute very little funding. If anything they make a killing off of schools and students buying hardware and software. Look at a company like Apple, that goes out of its way to make exclusivity deals with university chains, yet still charge their artificial $1200 price tag for the mandatory macbook. These students are having the life chocked out of them, why would we expect them to want to keep doing that and do "research"?
ngcc_hk 5y ago
If you compare say diabetes abd perception of time (another post), you can see the issue of research in many topics. Study need to be relevant and on the ground. But if someone study emacs, vim vs vscode or ispf can this be even master thesis material.
lincpa 5y ago
Computer science is essentially a management science, and vice versa. ---- Lin Pengcheng Creator of Computer Science Management School Creator of Management Science Computer School The Math-based Grand Unified Programming Theory: The Pure Function Pipeline Data Flow with Principle-based Warehouse/Workshop Model https://github.com/linpengcheng/PurefunctionPipelineDataflow https://github.com/linpengcheng/PurefunctionPipelineDataflow My theory is based on mathematical model, operational research, and management science, Its mathematical prototype is the simple, classic, vivid, and widely used in social production practice, elementary school mathematics "water input/output of the pool". My theory rebuilt the theoretical foundation of the IT industry, It makes the computer theory system fully & perfectly related to mathematics in a simple and unified way: from hardware integrated circuits and computer architecture, to software programming methodology, architecture, programming language and so on. It solve the most fundamental and core major problems in the IT industry: The foundation and core of the IT theory lack mathematical support.
thow-58d4e8b 5y ago
The fundamental problem is the same as with business and management research - any useful signal is completely drowned in an ocean of noise. Success of a project depends on the characters of the people involved, connections of the project lead, market timing, sources of funding, luck, current phase of the business cycle, past project experience, marketing and hype, sales teams, relatedness of icons and animations, etc. - more so than on any technical or organizational factor. I've yet to work on a project where technical reasons were at the root cause of the lack of success
- jeffreygoesto 5y ago
  Couldn't it be that it is much more signal than noise? It might be too many and too densely interconnected to disentangle (yet) though. Being technical people and used to clear cause-relation analyses we might just give up and call it noise?
- regularfry 5y ago
  You're looking in the wrong place. It's a sort of survivorship bias. You might not have worked on an individual project where technical reasons were the root cause of failure, but I've seen many projects not happen at all because changing a badly engineered platform was deemed too risky or too expensive, or just couldn't get scheduled because another project - made slower by poor past engineering decisions, but important enough to go ahead anyway - was occupying the team's time. Focusing on projects and not on platforms and products creates these blinkers.
dgan 5y ago
How does one follows research on some specific topic? Say I am interested in verification. I am constantly hitting paywalls for research papers that I have absolutely no idea what the value is. Let's be honest, even those papers are amazing, as a self taught developer, no way i could extract every possible valuable information from it. This also means that I have no access to some university account where I could download papers "just to see what's going on" Research looks like a hermetic world and little osmosis happens with surroundings
- mpfundstein 5y ago
  try scihub
DeathArrow 5y ago
Many "engineers" don't know what a hash table is under the hood, what binary search is or what pointers are, but they know Framework X, or Tool Y. They lack curiosity, desire to learn how things work, desire to discover. They don't trust concepts unless those concepts have a relatively large following and are deemed as "fresh", "cool", "new" and "inspiring" by trend setters. Also having some popular articles on Medium helps with trust.
- Thiez 5y ago
  Presumably they are never going to have to implement or modify a hash table (which kind? Cuckoo, Hopscotch, Robin Hood, ...?), but always use one provided by a library (very likely the standard library of their programming language). Of course knowledge about the performance characteristics of a hash table (especially in comparison to lists) is useful. Binary search isn't all that great due to cache misses. It's likely you won't regret using a hash table instead. As for pointers, depending on the programming language you use they may not be exposed in a way that the knowledge is all that relevant. In a language such as C you can't reasonably avoid them, but for Java, or Python, SQL, or JavaScript you just have references, which tend to be much simpler. Knowledge about how to use Framework X will probably turn out to be a lot more relevant and helpful for the day-to-day work of most developers.
  - davidgrenier 5y ago
    There are cases however for binary search where a hash table won't help. Pulling a sorted slice of data, for example, is useful in many domain.
- secondaryacct 5y ago
  It's funny, when it s not the devs, what you describe sounds strangely exactly like what many ivory tower CTO are like, "researching" useless shiny craps by reading some headline like "google does monorepos" and coming back saying "but you devs are just mindless morons, look at google they do monorepo, you should have the curiosity to merge all our github stuff into one giant repo". Yes yes boss, we're on it.
- 5y ago
jillesvangurp 5y ago
I was a software engineering researcher (Ph. D. and everything) before I actually became a software engineer about 20 years ago. I became one because I realized it was becoming quite awkward that I wasn't one. There was more than a bit of impostor syndrome. Software Engineering as a science has always been a bit controversial; mainly because a lot of computer scientists are involved with it and it is a bit of a soft science that most hard beta types would be uncomfortable with. Additionally, the whole notion of software engineering actually being an engineering discipline is not something everyone agrees on either. Software Engineering is fundamentally a process that involves people interacting. That means a lot of the challenges and problems are related to that and a lot of the solutions are related to that as well. Simply put, a tool won't fix your process and people issues, generally. The software engineering challenges in the industry are numerous. But I'd say the number one problem is that the demographics are completely skewed to propelling very inexperienced individuals into leadership roles because there simply aren't enough more experienced people around. Basically, the number of programmers doubles every five years or so. I'm 47; I routinely work with engineers half my age. Having anyone else in a team even close to 35-40 is very rare for me. The reason is not that my generation chickened out and went off and did something else; it's because we are outnumbered by about 2^5= 32x by people aged 22 or so. And about 16x by people aged 27. And about 8x by people aged 32. The median age in this industry is is 25-28. If you are lucky those people actually finished a degree and had one or two meaningful jobs before they joined you on your current project. Looking back at myself at that age, I definitely wasn't very senior. But it's great fun to work with young people and it keeps me sharp. The reality is that young people don't have a lot of older peers to learn from. So, a lot of issues in this industry are repetive. Young person gets propelled into a senior role and gets to rediscover all the same issues that people were dealing with 30 years ago as well. And they get to make all the same mistakes before they figure out some solution. The smart ones, pick up a few software engineering books and find out that people were dealing with the same shit 40 years ago and found some pragmatic ways of dealing with stuff. However, I'd caution anyone to apply anything they read blindly and dogmatically. Very little of it has been empirically validated in a proper software engineering research study. Doing so is hard and requires a lot of work. Most of the software engineering books out there don't bother with doing that work. That doesn't mean it's wrong but just that you need to separate facts from opinions. If it has the word "Agile" on the cover, you might want to keep that in mind. Most of that stuff is written with the same level of enthusiasm and assertiveness as the stuff that came before that.
thenoblesunfish 5y ago
I think part of the issue here had to be that the it’s very difficult to produce robust, actionable results about complex systems of humans. Research (by small teams at universities with limited resources, looking for minimal publishable units, typically) isn’t as effective as it might be for topics where isolated experiments are more possible.

62 points by mindcrime 5y ago

> I’ve referred to empirical studies that strongly suggest the adoption of particular practices, but experience, anecdote, and context have usually won out over evidence like...what? which practices? > I’ve demoed exciting tools from research, but my team has found them mostly useless, since they aren’t production ready. just getting teams on board with "classic" tools like static analysis and sanitizers is major. And those can usually be described as 'production ready'

secondaryacct 5y ago
The guy clearly does it the wrong way, he has a solution and everyone is looking at him "but what problem do we have this solves". So yeah, I mean even if these tools were production ready, they wouldnt be used. If they solved a problem people had, you d bet we d stay at night to make them production ready. He seems to think his devs are like mindless zombies needing constant intellectual feeding from him, the researching CTO, but it s probably more like they raise their eyes "oh god what is the guy producing no value day to day going to show us again while we stress on the deadline pretending to listen to him"
AtlasBarfed 5y ago
For an article bemoaning the failed communication of academic software engineering research to actual practice... It fails to provide documentation of academic software engineering research that they have tried to apply to actual practice.

jrumbut 5y ago

I see a couple of other issues, which share a root cause. 1. A lot of research is based on mining open source projects which bear little resemblance to how software engineering is done behind closed doors. 2. Very few prospective trials (much less randomized controlled trials), which are considered the gold standard of evidence for a good reason. These retrospective studies are very useful, but to really have faith in a conclusion that X is better than Y you need to assign several engineering teams to condition X and several more to condition Y and then compare. Nobody has money for that and it would be hard to find even one team (let alone several) that is going to let a local professor flip a coin and decide whether they are going to do Scrum or Kanban. You'd also need quite a few teams because there would be so many uncontrolled aspects. Researchers in this area are in the very difficult situation where the interesting things are difficult to measure and the things that are easy to measure aren't that interesting. Somehow they still do good work and that's very impressive but I don't envy their task.

kqr 5y ago
I think the whole "you need too many subjects and too much money" objection to doing science is very weak. There are other fields of research where they find inventive ways of experimenting on a shoestring budget (I think some social sciences belong in this category.) You can even do proper causal analysis when all you have is an observational study, given that you can get an expert to produce a decent guess at the causal network. (And this guess can to some extent be verified by plain correlations.)[1] Sure, you might not get the extremely high external validity of e.g. particle physics, but in many cases something is better than nothing.[2] Heck, screw external validity. I'd be ecstatic to see more experiments just within an organisation, focusing on internal validity. Every engineering organisation can afford to experiment with different tools and practises. There are huge productivity gains to be unlocked (is my hypothesis, anyway), and the cost is small. If we get together enough small studies lacking external validity, but which point in the same direction, we might even be able to infer some external validity between them. [1]: For more on this approach, see Judea Pearl. [2]: To use an example close to my heart: pregnancy is a condition where you rarely see randomised trials, and any research is sorely underfunded. Yet when people actually take the time to sit down and answer questions with data, we get something meaningful out of it.
- regularfry 5y ago
  It's worth pointing out that this blog post was published three years before Accelerate, by Forsgren, Humble, and Kim. That book demonstrates what you can do just with surveys. It's not about software engineering per se - it's got a devops focus - but the overlap is significant. As far as I'm concerned that book significantly changed the game, not only because of the findings, but because it showed that hard results are possible in this arena. The downside is that you need enough data to make the maths work, and that precludes applying the techniques to internal studies in most organisations. There's just too much noise.
- jrumbut 5y ago
  I wouldn't be surprised if a few organizations had run major experiments over the years, with results never seeing the light of day. It would also be great if there was more awareness of the kind of data the research community needs so that if someone saw a natural experiment shaping up in their company they knew who to call. Regarding budgets, these researchers are a lot smarter than me and I bet they are doing their best with what they have. What might be good though is having several labs join forces and collaborate on one very good trial of a core issue. Then you have a hope of settling a question and moving the field forward while creating some infrastructure that will help answer other big questions. Maybe that's happening, I haven't seen a lot of it though.

devwastaken 5y ago

The reason there's little research is because there is little money and structure in STEM education in general. I have been to multiple name brand uni's that simply do not have basic lab equipment. Some high schools had more equipment and expertise in software than uni's did. Why? Because education as a business has fundamentally broken incentives. Most educators have never worked in the private industry, or did so for very little time. This means they don't actually know what they're talking about, they just follow courses accreditation allows them to. You will never make as much as a professor in software than you will privately, unless you are in one of the top schools that are inaccessible to the overwhelming majority of the population. The workplace politics are egregious, because universities are not a meritocracy, universities have created a system where your skin color matters a lot more than your qualifications to conduct your job. For example, someone becoming head of department because they're from India, not because the product of their research or overall conduct in their job was the superior choice. This trickles down, students become the professor, and over time we are now here. Tech companies are the wealthiest in the world out of any major, yet contribute very little funding. If anything they make a killing off of schools and students buying hardware and software. Look at a company like Apple, that goes out of its way to make exclusivity deals with university chains, yet still charge their artificial $1200 price tag for the mandatory macbook. These students are having the life chocked out of them, why would we expect them to want to keep doing that and do "research"?

ngcc_hk 5y ago

If you compare say diabetes abd perception of time (another post), you can see the issue of research in many topics. Study need to be relevant and on the ground. But if someone study emacs, vim vs vscode or ispf can this be even master thesis material.

lincpa 5y ago

Computer science is essentially a management science, and vice versa. ---- Lin Pengcheng Creator of Computer Science Management School Creator of Management Science Computer School The Math-based Grand Unified Programming Theory: The Pure Function Pipeline Data Flow with Principle-based Warehouse/Workshop Model https://github.com/linpengcheng/PurefunctionPipelineDataflow https://github.com/linpengcheng/PurefunctionPipelineDataflow My theory is based on mathematical model, operational research, and management science, Its mathematical prototype is the simple, classic, vivid, and widely used in social production practice, elementary school mathematics "water input/output of the pool". My theory rebuilt the theoretical foundation of the IT industry, It makes the computer theory system fully & perfectly related to mathematics in a simple and unified way: from hardware integrated circuits and computer architecture, to software programming methodology, architecture, programming language and so on. It solve the most fundamental and core major problems in the IT industry: The foundation and core of the IT theory lack mathematical support.

thow-58d4e8b 5y ago

The fundamental problem is the same as with business and management research - any useful signal is completely drowned in an ocean of noise. Success of a project depends on the characters of the people involved, connections of the project lead, market timing, sources of funding, luck, current phase of the business cycle, past project experience, marketing and hype, sales teams, relatedness of icons and animations, etc. - more so than on any technical or organizational factor. I've yet to work on a project where technical reasons were at the root cause of the lack of success

jeffreygoesto 5y ago
Couldn't it be that it is much more signal than noise? It might be too many and too densely interconnected to disentangle (yet) though. Being technical people and used to clear cause-relation analyses we might just give up and call it noise?
regularfry 5y ago
You're looking in the wrong place. It's a sort of survivorship bias. You might not have worked on an individual project where technical reasons were the root cause of failure, but I've seen many projects not happen at all because changing a badly engineered platform was deemed too risky or too expensive, or just couldn't get scheduled because another project - made slower by poor past engineering decisions, but important enough to go ahead anyway - was occupying the team's time. Focusing on projects and not on platforms and products creates these blinkers.

dgan 5y ago

How does one follows research on some specific topic? Say I am interested in verification. I am constantly hitting paywalls for research papers that I have absolutely no idea what the value is. Let's be honest, even those papers are amazing, as a self taught developer, no way i could extract every possible valuable information from it. This also means that I have no access to some university account where I could download papers "just to see what's going on" Research looks like a hermetic world and little osmosis happens with surroundings

mpfundstein 5y ago
try scihub

DeathArrow 5y ago

Many "engineers" don't know what a hash table is under the hood, what binary search is or what pointers are, but they know Framework X, or Tool Y. They lack curiosity, desire to learn how things work, desire to discover. They don't trust concepts unless those concepts have a relatively large following and are deemed as "fresh", "cool", "new" and "inspiring" by trend setters. Also having some popular articles on Medium helps with trust.

Thiez 5y ago
Presumably they are never going to have to implement or modify a hash table (which kind? Cuckoo, Hopscotch, Robin Hood, ...?), but always use one provided by a library (very likely the standard library of their programming language). Of course knowledge about the performance characteristics of a hash table (especially in comparison to lists) is useful. Binary search isn't all that great due to cache misses. It's likely you won't regret using a hash table instead. As for pointers, depending on the programming language you use they may not be exposed in a way that the knowledge is all that relevant. In a language such as C you can't reasonably avoid them, but for Java, or Python, SQL, or JavaScript you just have references, which tend to be much simpler. Knowledge about how to use Framework X will probably turn out to be a lot more relevant and helpful for the day-to-day work of most developers.
- davidgrenier 5y ago
  There are cases however for binary search where a hash table won't help. Pulling a sorted slice of data, for example, is useful in many domain.
secondaryacct 5y ago
It's funny, when it s not the devs, what you describe sounds strangely exactly like what many ivory tower CTO are like, "researching" useless shiny craps by reading some headline like "google does monorepos" and coming back saying "but you devs are just mindless morons, look at google they do monorepo, you should have the curiosity to merge all our github stuff into one giant repo". Yes yes boss, we're on it.
5y ago

jillesvangurp 5y ago

I was a software engineering researcher (Ph. D. and everything) before I actually became a software engineer about 20 years ago. I became one because I realized it was becoming quite awkward that I wasn't one. There was more than a bit of impostor syndrome. Software Engineering as a science has always been a bit controversial; mainly because a lot of computer scientists are involved with it and it is a bit of a soft science that most hard beta types would be uncomfortable with. Additionally, the whole notion of software engineering actually being an engineering discipline is not something everyone agrees on either. Software Engineering is fundamentally a process that involves people interacting. That means a lot of the challenges and problems are related to that and a lot of the solutions are related to that as well. Simply put, a tool won't fix your process and people issues, generally. The software engineering challenges in the industry are numerous. But I'd say the number one problem is that the demographics are completely skewed to propelling very inexperienced individuals into leadership roles because there simply aren't enough more experienced people around. Basically, the number of programmers doubles every five years or so. I'm 47; I routinely work with engineers half my age. Having anyone else in a team even close to 35-40 is very rare for me. The reason is not that my generation chickened out and went off and did something else; it's because we are outnumbered by about 2^5= 32x by people aged 22 or so. And about 16x by people aged 27. And about 8x by people aged 32. The median age in this industry is is 25-28. If you are lucky those people actually finished a degree and had one or two meaningful jobs before they joined you on your current project. Looking back at myself at that age, I definitely wasn't very senior. But it's great fun to work with young people and it keeps me sharp. The reality is that young people don't have a lot of older peers to learn from. So, a lot of issues in this industry are repetive. Young person gets propelled into a senior role and gets to rediscover all the same issues that people were dealing with 30 years ago as well. And they get to make all the same mistakes before they figure out some solution. The smart ones, pick up a few software engineering books and find out that people were dealing with the same shit 40 years ago and found some pragmatic ways of dealing with stuff. However, I'd caution anyone to apply anything they read blindly and dogmatically. Very little of it has been empirically validated in a proper software engineering research study. Doing so is hard and requires a lot of work. Most of the software engineering books out there don't bother with doing that work. That doesn't mean it's wrong but just that you need to separate facts from opinions. If it has the word "Agile" on the cover, you might want to keep that in mind. Most of that stuff is written with the same level of enthusiasm and assertiveness as the stuff that came before that.

thenoblesunfish 5y ago

I think part of the issue here had to be that the it’s very difficult to produce robust, actionable results about complex systems of humans. Research (by small teams at universities with limited resources, looking for minimal publishable units, typically) isn’t as effective as it might be for topics where isolated experiments are more possible.