I did a PhD in the field. And only then became an actual software engineer. As it turns out, doing software engineering research does not involve a whole lot of

I did a PhD in the field. And only then became an actual software engineer. As it turns out, doing software engineering research does not involve a whole lot of engineering software. So, I had a lot to learn. So, this criticism resonates a bit with me. The problem with software engineering as a scientific discipline is that many engineers probably would not consider it a proper engineering discipline to begin with. Also, many computer scientists look down on it as not really related to that either because it is a bit of a soft science. So, there's a problem with researchers in this field not really having the right background to do good research: - they are typically not experienced software engineers. Quite many, like me, start working on a PhD without actually having worked as a software engineer. - they typically have studied computer science which teaches you a lot of math, algorithms, formal methods, and other typical beta skills but is perhaps a bit light on the type of stuff a sociologist or anthropologist would learn. That's a problem because a lot of software engineering involves the behavior of people in groups and coordinating that behavior to produce software. Studying that behavior to verify theories you might have about how software engineering works or ought to work requires an empirical approach. That's not something a lot of computer scientists learn how to do. So there's a lot of software engineering research that gets published that you could accuse of being a combination of naive, wrong/misguided, and poorly executed in terms of its methodology. Mathematicians or computer scientists trying to be empirical is not great typically. Sometimes you get some decent papers but there is a lot of pseudo scientific nonsense that gets published. Doing empirical research properly is hard and many computer scientists have no clue how to do it properly nor any appreciation for doing so. Just looking at ICSE's program last year you can see from the titles a weird mix of empirical studies of various topics and other papers clearly written by more computer science leaning types working on algorithms, machine learning or tools. Looks like there were some interesting papers in there but also a lot eye brow raising titles like "How Java Programmers Test Exceptional Behavior". I actually skimmed through the pdf of that one and it fits the pattern beautifully. It's an actual empirical study that tries to answer the question in the title. That's it. There's no new theories being formulated. No remarkable findings that were reported. It's just a very dry survey that yielded exactly what you'd expect. The answers to the research question are almost beside the point and should be completely unsurprising to any Java programmer. Great example of a well executed emprical research paper that answers a question any Java developer in need of actually testing some code that throws exceptions would answer with a simple Google query in a few minutes. So, yes underwhelming impact indeed.