3 ms·

> There are languages that were under- or over-represented by using public GitHub repos; there are communities that were under- or over-represented by looking a

> There are languages that were under- or over-represented by using public GitHub repos; there are communities that were under- or over-represented by looking at discussions happening in Stack Overflow. I'm not sure I understand. Granted, they're never going to get perfect data about language rankings. But, why would AI assistants affect the rankings of language in a way that makes their methodology more unsound? They mention a drop-off in SO tag usage, and in Github pull requests: that's an indication of something, but is there any reason to believe it distorts the results so as to give undue prominence to one language more than another? Intuitively, I can't think of how. "Undue" is the keyword: even if the prominence changed as a result of AI language assistants, wouldn't that be an indication that people are just using those languages more, with the help of AI assistants? I'm really just wondering why, given the unavoidable squishiness of the methodology, the decrease in magnitude would bother them so much. They probably don't see their work as fundamentally hand-wavey, and maybe I just do, and that's the problem. I guess they're just investigating it, but the results seem to be more or less what you'd expect to conclude ab initio. I could be wrong, often am.

bad_user 3y ago
I think this trend favours unpopular languages. ChatGPT can only provide answers for programming languages and libraries that were included in its training. Granted, unpopular programming languages have online activity, too, but sample size matters.
rezonant 3y ago
I imagine they're just trying not to tell the wrong story just because of a tangential change in how developers use these tools, so taking some time to look closely and account for things and provide the necessary context for the report is sensible. I guess it's just taking longer than they expected due to the unexpected GitHub changes.
stonemetal12 3y ago
Lets say you are tracking lion populations. You watch a couple of watering holes and count the number of lions. From there you use stats to make estimates of total number of lions in the world. Suddenly your data shows a drop of 30% and is continuing to decline. Are lions going extinct, or did they find a new watering hole? If they found a new watering hole then you need to change your measuring locations to account for the shift. I don't think software development has started slowing down. So the data is indicating that the way they were measuring is becoming less useful than it was, and that they will need to find new data sources if they are going to continue. If that really is AI, maybe Copilot and friends will become an interesting source of data on programming language use.
janalsncm 3y ago
A smaller sample size means there is higher variance in the results.

bad_user 3y ago

I think this trend favours unpopular languages. ChatGPT can only provide answers for programming languages and libraries that were included in its training. Granted, unpopular programming languages have online activity, too, but sample size matters.

rezonant 3y ago

I imagine they're just trying not to tell the wrong story just because of a tangential change in how developers use these tools, so taking some time to look closely and account for things and provide the necessary context for the report is sensible. I guess it's just taking longer than they expected due to the unexpected GitHub changes.

stonemetal12 3y ago

Lets say you are tracking lion populations. You watch a couple of watering holes and count the number of lions. From there you use stats to make estimates of total number of lions in the world. Suddenly your data shows a drop of 30% and is continuing to decline. Are lions going extinct, or did they find a new watering hole? If they found a new watering hole then you need to change your measuring locations to account for the shift. I don't think software development has started slowing down. So the data is indicating that the way they were measuring is becoming less useful than it was, and that they will need to find new data sources if they are going to continue. If that really is AI, maybe Copilot and friends will become an interesting source of data on programming language use.

janalsncm 3y ago

A smaller sample size means there is higher variance in the results.