7 ms·

The research methodology in this blog post is fundamentally flawed. The author only counts how many people move from X to Y, but he doesn't count how many of th

The research methodology in this blog post is fundamentally flawed. The author only counts how many people move from X to Y, but he doesn't count how many of them do not move at all. The whole diagonal of his (sample) transition matrix are actually missing values, but he treats them as zeroes. This greatly distorts the equilibrium distribution. As a result, he misinterprets each equilibrium probability as the "future popularity" of a language as well, when it at best only represents the future popularity of a language among those who constantly switch their languages.

wjn0 10y ago
I wonder if the data could be normalized against usage statistics somehow. Maybe not perfectly effectively, but at least better than as-is, precisely because the number of people "switching" is so much smaller than the number using a given language.
- cormacrelf 10y ago
  Presumably also on a time series. People used to switch from Ruby to Node.js, now they switch to ???
  - The_suffocated 10y ago
    Agree. Depending on what one wants to measure, giving every sample equal weight can be potentially wrong.
- The_suffocated 10y ago
  If you mean to account for the larger variances of rarer events, that would be difficult. Perhaps some bootstrap sampling methods may help, but we need a true statistician here.
- LeifCarrotson 10y ago
  Normalization would be extremely helpful indeed. C, C#, C++, Java, and Python are all very popular languages and have large values in both dimensions, which isn't particularly useful.
- brightball 10y ago
  It would be interesting if the results could be controlled for time. Age of the language, year to year switch rates, etc.
frei 10y ago
At the bottom he writes, "the stationary distribution is actually independent of adding a constant diagonal (identity) matrix," but I'm not sure how that could be true (intuitively it doesn't make sense, but I don't know the math) edit: A identity matrix wouldn't affect the stationary distribution, but if you had the actual "stay" probabilities they wouldn't all be the same, and thus not an identity matrix at all.
- The_suffocated 10y ago
  Adding a constant diagonal matrix indeed would not affect the equilibrium distribution. Mathematically, if the original transition matrix is A and you add a multiple of the identity matrix to it, then after normalisation by row sums, the new transition matrix becomes tA+(1-t)I for some 0<=t<=1. Since the Perron vector x of A corresponds to the eigenvalue 1, it must also be an eigenvector of tA+(1-t)I, because (tA+(1-t)I)x=tAx+(1-t)x=tx+(1-t)x=x. Hence the stationary distribution remains unchanged. But as you said, the stationary distribution does change if the missing diagonal is not constant. And there is no reason to believe it is constant in the first place. In the end, what the author measures is still different from what he thinks he measures.
  - tgb 10y ago
    I agree with you, but your last line is wrong: the footnotes make it clear the author knows the omission.
aurelianito 10y ago
The real issue is that he is not counting the projects that start from zero ('why did I do this project in php') and the ones that end ('why did I finish this project in Haskell'). If he adds this data to it's matrix somehow then it would be much more interesting.
aurelianito 10y ago
The real issue is that he is not counting the projects that start from zero ('why did I do this project in php') and the ones that end ('why did I finish this project in Haskell'). If he adds this data to it's matrix somehow then it would be much more interesting.
aurelianito 10y ago
The real issue is that he is not counting the projects that start from zero ('why did I do this project in php') and the ones that end ('why did I finish this project in Haskell'). If he adds this data to it's matrix somehow then it would be much more interesting.
eeZah7Ux 10y ago
Also, the likelihood of people blogging about a language change is not the same across languages. Some crowds (Go and nodejs enthusiasts) are notoriously vocal due to the hype. Finally, it's common practice for companies to have their marketing department to pay "media marketing specialist" to advertise for products (their language) by posting on forums.
- patmcguire 10y ago
  Yeah, many languages will be almost entirely silent in transition. COBOL, FORTRAN, any proprietary language. And I would guess those transitions are usually to the "boring" languages: Java, C#, etc.
- pmontra 10y ago
  Some languages and frameworks are simply too new to have a significant number of people moving away from them. Go and Vue fit that case. MySQL looks genuine, even if surprising.
mavdi 10y ago
Great. We'll wait for you to come up with a better research then.
techterrier 10y ago
I assumed it was a joke.
hasenj 10y ago
This is similar to people measuring popularity of a library or language by stackoverflow question counts. Sure, you can't get a lot of questions for an obscure library that on one uses, but the mere fact that tons of people post questions about it doesn't mean it's good or popular. It might mean lots of beginners want to use it because it's been marketed and hyped as easy to use. (Angular comes to mind).
- sirbranedamuj 10y ago
  It might also mean that it's confusing and poorly designed, leading many, many of its users to ask questions about it.
  - hasenj 10y ago
    Exactly! (That's what I intended to say but I just realized that I didn't emphasize the point).
  - mhermher 10y ago
    Or that the official documentation is poor.
anjc 10y ago
> when it at best only represents the future popularity of a language among those who constantly switch their languages. This is literally in the title >The eigenvector of "Why we moved from language X to language Y"
- JackFr 10y ago
  The parent's point is that it is a very poor proxy for future language popularity.
  - danielweber 10y ago
    I took this whole post as very tongue-in-cheek. No one should really be using this to predict the future. Instead, it was a fun data-analysis exercise.
omginternets 10y ago
I wouldn't call that a "flaw" so much as an "entirely different question".
- Certhas 10y ago
  No it's definitely a flaw. The analysis doesn't model what he thinks it models. He claims that it doesn't matter by (correctly) pointing out that adding a diagonal doesn't change anything, and that hides the modelling assumption he makes: That there is the same absolute number of people choosing to stay within every language studied. His analysis would only be correct if the number of people who choose to stay with Java at any moment is equal to the number of people who choose to stay with Rust. That is absurd and hence the eigenvector is meaningless. Doesn't exactly matter much given that this is just a bit of fun on a blog post though. :P
  - j2kun 10y ago
    Doesn't it make sense if you ask: Given that someone will change the language they're using (and writes a public blog post about it), which are they most likely to change to? It's not unreasonable to use that as a proxy for industry trends. I recall reading about manufacturing jobs which, sure, might have lots of factories not changing, but the ones that do change _definitely_ opt for more automation with fewer workers. That's still a trend worth thinking about.
galdosdi 10y ago
> but he doesn't count how many of them do not move at all. The whole diagonal of his (sample) transition matrix are actually missing values, but he treats them as zeroes Great point. It's interesting to think about how exactly this could bias the results -- would it bias it in favor of languages that developers tend to initially not choose for their project? I do see that as very consistent with C, C++, and Java being up there. For new projects, developers love to choose anything but those, but then they find themselves gravitating towards them when the project gets bigger and practical concerns intrude
erikbern 10y ago
Author here. You are absolutely right. As I mentioned in the notes, I think this matters a bit less than it might seem like (the stationary distribution does not change if you add a diagonal matrix) but clearly some languages will have a higher propensity for people to stay. I think this flaw is even smaller than the issue of using Google statistics to infer transition probabilities. It's just a shitty proxy, at best. At the end of the day, there's a lot of assumptions going into this analysis. I hope I didn't make it seem more serious than I meant it to be – it's really just a fun project and kind of a joke not to meant taken seriously. That being said, I think the conclusions are at least "directionally" correct. They might be off by a factor of 2x or 5x or even 10x, but the stationary distribution exhibits an even bigger spread (multiple orders of magnitude) so I suspect the final ranking is still "roughly" correct (with a very liberal definition of "rough")
- j2kun 10y ago
  The thing I like most about this post is that it's falsifiable. We will know in ten years whether C and Java are still popular, and whether Go succeeds in the sense this data suggests. So thank you for being concrete and clear, even if it's all in fun and other people don't like it :)
  - erikbern 10y ago
    This is a great point. It's a model that predicts something about the future. I could have backtested it on historical Google stats to figure out if it's a good model :)
    - j2kun 10y ago
      In an unrelated note: interested in doing a guest post over at Math ∩ Programming? I see you've got lots of cool stuff with high-dimensional NN :)
  - thaumasiotes 10y ago
    > whether Go succeeds in the sense this data suggests An interesting thing about this methodology is that it is extremely sensitive to the age of a language. It's possible to switch from an old language to a new language, but not the other way around -- so if you happen to do your measurements after a language has had some uptake but before it's been around for long enough that people have built significant projects on it and subsequently gotten sick of it, the future distribution by this method can only be 100% New Language. (Because sometimes people switch to New Language, but no one ever switches away.) Actually, to predict the future distribution of language use, you also need to know the rate of people moving from nothing ("I just had a brilliant idea!") to each language. If everyone eventually transitions to Go, but everyone starts in Ruby, then the division of market share between Go and Ruby depends in part on how frequently people start new projects.

wjn0 10y ago

I wonder if the data could be normalized against usage statistics somehow. Maybe not perfectly effectively, but at least better than as-is, precisely because the number of people "switching" is so much smaller than the number using a given language.

cormacrelf 10y ago
Presumably also on a time series. People used to switch from Ruby to Node.js, now they switch to ???
- The_suffocated 10y ago
  Agree. Depending on what one wants to measure, giving every sample equal weight can be potentially wrong.
The_suffocated 10y ago
If you mean to account for the larger variances of rarer events, that would be difficult. Perhaps some bootstrap sampling methods may help, but we need a true statistician here.
LeifCarrotson 10y ago
Normalization would be extremely helpful indeed. C, C#, C++, Java, and Python are all very popular languages and have large values in both dimensions, which isn't particularly useful.
brightball 10y ago
It would be interesting if the results could be controlled for time. Age of the language, year to year switch rates, etc.

frei 10y ago

At the bottom he writes, "the stationary distribution is actually independent of adding a constant diagonal (identity) matrix," but I'm not sure how that could be true (intuitively it doesn't make sense, but I don't know the math) edit: A identity matrix wouldn't affect the stationary distribution, but if you had the actual "stay" probabilities they wouldn't all be the same, and thus not an identity matrix at all.

The_suffocated 10y ago
Adding a constant diagonal matrix indeed would not affect the equilibrium distribution. Mathematically, if the original transition matrix is A and you add a multiple of the identity matrix to it, then after normalisation by row sums, the new transition matrix becomes tA+(1-t)I for some 0<=t<=1. Since the Perron vector x of A corresponds to the eigenvalue 1, it must also be an eigenvector of tA+(1-t)I, because (tA+(1-t)I)x=tAx+(1-t)x=tx+(1-t)x=x. Hence the stationary distribution remains unchanged. But as you said, the stationary distribution does change if the missing diagonal is not constant. And there is no reason to believe it is constant in the first place. In the end, what the author measures is still different from what he thinks he measures.
- tgb 10y ago
  I agree with you, but your last line is wrong: the footnotes make it clear the author knows the omission.

aurelianito 10y ago

The real issue is that he is not counting the projects that start from zero ('why did I do this project in php') and the ones that end ('why did I finish this project in Haskell'). If he adds this data to it's matrix somehow then it would be much more interesting.

aurelianito 10y ago

eeZah7Ux 10y ago

Also, the likelihood of people blogging about a language change is not the same across languages. Some crowds (Go and nodejs enthusiasts) are notoriously vocal due to the hype. Finally, it's common practice for companies to have their marketing department to pay "media marketing specialist" to advertise for products (their language) by posting on forums.

patmcguire 10y ago
Yeah, many languages will be almost entirely silent in transition. COBOL, FORTRAN, any proprietary language. And I would guess those transitions are usually to the "boring" languages: Java, C#, etc.
pmontra 10y ago
Some languages and frameworks are simply too new to have a significant number of people moving away from them. Go and Vue fit that case. MySQL looks genuine, even if surprising.

mavdi 10y ago

Great. We'll wait for you to come up with a better research then.

techterrier 10y ago

I assumed it was a joke.

hasenj 10y ago

This is similar to people measuring popularity of a library or language by stackoverflow question counts. Sure, you can't get a lot of questions for an obscure library that on one uses, but the mere fact that tons of people post questions about it doesn't mean it's good or popular. It might mean lots of beginners want to use it because it's been marketed and hyped as easy to use. (Angular comes to mind).

sirbranedamuj 10y ago
It might also mean that it's confusing and poorly designed, leading many, many of its users to ask questions about it.
- hasenj 10y ago
  Exactly! (That's what I intended to say but I just realized that I didn't emphasize the point).
- mhermher 10y ago
  Or that the official documentation is poor.

anjc 10y ago

> when it at best only represents the future popularity of a language among those who constantly switch their languages. This is literally in the title >The eigenvector of "Why we moved from language X to language Y"

JackFr 10y ago
The parent's point is that it is a very poor proxy for future language popularity.
- danielweber 10y ago
  I took this whole post as very tongue-in-cheek. No one should really be using this to predict the future. Instead, it was a fun data-analysis exercise.

omginternets 10y ago

I wouldn't call that a "flaw" so much as an "entirely different question".

Certhas 10y ago
No it's definitely a flaw. The analysis doesn't model what he thinks it models. He claims that it doesn't matter by (correctly) pointing out that adding a diagonal doesn't change anything, and that hides the modelling assumption he makes: That there is the same absolute number of people choosing to stay within every language studied. His analysis would only be correct if the number of people who choose to stay with Java at any moment is equal to the number of people who choose to stay with Rust. That is absurd and hence the eigenvector is meaningless. Doesn't exactly matter much given that this is just a bit of fun on a blog post though. :P
- j2kun 10y ago
  Doesn't it make sense if you ask: Given that someone will change the language they're using (and writes a public blog post about it), which are they most likely to change to? It's not unreasonable to use that as a proxy for industry trends. I recall reading about manufacturing jobs which, sure, might have lots of factories not changing, but the ones that do change _definitely_ opt for more automation with fewer workers. That's still a trend worth thinking about.

galdosdi 10y ago

> but he doesn't count how many of them do not move at all. The whole diagonal of his (sample) transition matrix are actually missing values, but he treats them as zeroes Great point. It's interesting to think about how exactly this could bias the results -- would it bias it in favor of languages that developers tend to initially not choose for their project? I do see that as very consistent with C, C++, and Java being up there. For new projects, developers love to choose anything but those, but then they find themselves gravitating towards them when the project gets bigger and practical concerns intrude

erikbern 10y ago

Author here. You are absolutely right. As I mentioned in the notes, I think this matters a bit less than it might seem like (the stationary distribution does not change if you add a diagonal matrix) but clearly some languages will have a higher propensity for people to stay. I think this flaw is even smaller than the issue of using Google statistics to infer transition probabilities. It's just a shitty proxy, at best. At the end of the day, there's a lot of assumptions going into this analysis. I hope I didn't make it seem more serious than I meant it to be – it's really just a fun project and kind of a joke not to meant taken seriously. That being said, I think the conclusions are at least "directionally" correct. They might be off by a factor of 2x or 5x or even 10x, but the stationary distribution exhibits an even bigger spread (multiple orders of magnitude) so I suspect the final ranking is still "roughly" correct (with a very liberal definition of "rough")

j2kun 10y ago
The thing I like most about this post is that it's falsifiable. We will know in ten years whether C and Java are still popular, and whether Go succeeds in the sense this data suggests. So thank you for being concrete and clear, even if it's all in fun and other people don't like it :)
- erikbern 10y ago
  This is a great point. It's a model that predicts something about the future. I could have backtested it on historical Google stats to figure out if it's a good model :)
  - j2kun 10y ago
    In an unrelated note: interested in doing a guest post over at Math ∩ Programming? I see you've got lots of cool stuff with high-dimensional NN :)
- thaumasiotes 10y ago
  > whether Go succeeds in the sense this data suggests An interesting thing about this methodology is that it is extremely sensitive to the age of a language. It's possible to switch from an old language to a new language, but not the other way around -- so if you happen to do your measurements after a language has had some uptake but before it's been around for long enough that people have built significant projects on it and subsequently gotten sick of it, the future distribution by this method can only be 100% New Language. (Because sometimes people switch to New Language, but no one ever switches away.) Actually, to predict the future distribution of language use, you also need to know the rate of people moving from nothing ("I just had a brilliant idea!") to each language. If everyone eventually transitions to Go, but everyone starts in Ruby, then the division of market share between Go and Ruby depends in part on how frequently people start new projects.