3 ms·

> numbers of defects per project How do you measure that? You could look at the project's bug tracker, but the number of bugs in it will reflect a project's

> numbers of defects per project How do you measure that? You could look at the project's bug tracker, but the number of bugs in it will reflect a project's popularity more than anything else. You could look at commit messages, as the (original) paper does (see 2.2.1 / 2.2.2 in the reproduction paper), but the heuristic they use doesn't look very reliable to me; I've seen projects where every feature implementation had to have an issue in the bug tracker and every commit implementing that feature had to reference the issue, so almost every commit would be counted as a defect by their metric. And of course, all of that is data about known defects, and fixed (presumably) defects in the case of commits - how do you compare the number of unknown defects? There's an underlying assumption [edit: not implying it's yogthos' assumption, of course] that you can simply compare unrelated projects with vastly different policies and processes, and I believe that a lot more manual work is required to get reliable and comparable data.

yogthos 7y ago
>You could look at the project's bug tracker, but the number of bugs in it will reflect a project's popularity more than anything else. It's a numbers game, if we look at a lot of projects written in a lot of different languages, we'll see a sample of projects with different kinds of popularity. If there is a trend associated with a particular language that's an outlier that would be an indication that the particular language may play a role. >And of course, all of that is data about known defects, and fixed (presumably) defects in the case of commits - how do you compare the number of unknown defects? Again, it's about trends. If you look at thousands or millions of projects these things average out across them. You can even group projects into different categories based on their domain, popularity, and so on. The underlying idea is that you should be able to see measurable trends when looking at large numbers of projects across languages. If such trends are established, then we can dig deeper and make a hypothesis as to what might be the cause.

yogthos 7y ago

>You could look at the project's bug tracker, but the number of bugs in it will reflect a project's popularity more than anything else. It's a numbers game, if we look at a lot of projects written in a lot of different languages, we'll see a sample of projects with different kinds of popularity. If there is a trend associated with a particular language that's an outlier that would be an indication that the particular language may play a role. >And of course, all of that is data about known defects, and fixed (presumably) defects in the case of commits - how do you compare the number of unknown defects? Again, it's about trends. If you look at thousands or millions of projects these things average out across them. You can even group projects into different categories based on their domain, popularity, and so on. The underlying idea is that you should be able to see measurable trends when looking at large numbers of projects across languages. If such trends are established, then we can dig deeper and make a hypothesis as to what might be the cause.