I really want this kind of research to flourish, but the first click I made on an interesting looking paper led me to: "To measure tool usage, we randomly sampl

I really want this kind of research to flourish, but the first click I made on an interesting looking paper led me to: "To measure tool usage, we randomly sampled code changes from four Eclipse and eight Mylyn developers and ascertained, for each refactoring, if it was performed manually or with tool support. We found that refactoring tools are seldom used: 11 percent by Eclipse developers and 9 percent by Mylyn developers." To be fair, I haven't read the paper and the rest of the abstract looks reasonable: "To understand refactoring practice at large, we drew from a variety of data sets spanning more than 39,000 developers, 240,000 tool-assisted refactorings, 2,500 developer hours, and 12,000 version control commits." But what the heck is the first bit for? Of 12 people I know, the vast majority don't use refactoring tools. That's not the basis on which to launch a study. I've skimmed the abstracts of the other papers as well and I'm not all that impressed. Everything seems to be doing comparisons against Eclipse. While Eclipse has a variety of different usability features, I'm actually not convinced any of them help at all (which is why I don't use it ;-) ). So at the very least, I'd like to see a baseline against a traditional text editor like Emacs or Vim. The "improvements" in usability that they are measuring may simply be the avoidance of problems in Eclipse. I'm not trying to start an editor war here, I'm just saying you can't assume that any single editor is a good baseline. I'd argue strenuously that feature rich editors like Eclipse, IntelliJ, VS Code, etc, etc are particularly bad candidates because nobody has really measured the effectiveness of their features. Without being too negative, I hope there are better papers on these kinds of topics because I think it's incredibly important.