7 ms·

Author here Agreed, which is why I didn't say,"working remote causes developers to earn 22% more", only that developers who work remote earn 22% more, which is

Author here Agreed, which is why I didn't say,"working remote causes developers to earn 22% more", only that developers who work remote earn 22% more, which is an interesting fact in itself. Early in the article I adjust this for various controls to better get at causality. This includes observable factors like age, years of experience, hours worked, size of employer, programming languages used, etc. As I note in the article: >Much of the apparent premium earned by remote developers is in fact driven by seniority and tenure. These are older, more experienced developers who either prefer to work remote or whose organizations grant them that privilege. However, controlling for manually selected factors doesn't imply causality, so I use principled covariate selection to select the best set of controls and get closer to something that could be called causal. You can read more about this method, called Double Lasso Selection, in Urminsky, Hansen, and Chernozhukov [0]. This results in an adjusted pay premium of 9.4% for remote developers relative to those that never work remote. Hard to know for sure if this is causal either, but it's likely much closer to whatever the true causal impact is. Unsurprisingly, it's a lower number. Thanks for reading! [0] http://home.uchicago.edu/ourminsky/Variable_Selection.pdf http://home.uchicago.edu/ourminsky/Variable_Selection.pdf

wbc 6y ago
very cool! any thoughts on where that premium comes from, since we're controlling for the standard stuff?
- dx034 6y ago
  Pure speculation, but maybe you have to look around or negotiate to find remote work? Those who tend to do that probably also get better salaries than those that stick with one job for very long or don't compare much when looking for jobs.
  - deleted 6y ago
    [deleted]
- deleted 6y ago
  [deleted]
- thinkharderdev 6y ago
  One possibility is that remote developers are just more productive (or if you prefer, more productive developers are more able to negotiate a remote working allowance from their employer). In standard economic theory, the equilibrium wage is at the workers marginal revenue product (or the marginal revenue that their work produces for the organization). So you would expect remote workers to either earn more because they are more productive or, if remote work makes you more productive, to earn more as a result of working remotely and being more productive
whack 6y ago
> Early in the article I adjust this for various controls to better get at causality. This includes observable factors like age, years of experience, hours worked, size of employer, programming languages used, etc. There are likely many other factors which aren't "observable" by your methodology, which are causing the 9.4% pay-premium. Factors like competency, domain knowledge, and how essential they are to the organization. I find it much more believable that these unobservable factors are causing both the pay-premium, and working-remote flexibility. This is a common problem that comes up in such studies. Adjusting for various observable factors is great, but it still leaves behind the other unobservable factors. At which point you have to use your judgement to figure out whether those unobservable factors are more compelling than the hypothesis being tested. That said, this is still a very cool analysis that demonstrates an interesting correlation. Thanks for sharing!
- whoisnnamdi 6y ago
  Totally. The "etc" I used in my quote is doing a lot of work here - I control for a fairly extensive list of observed factors from the dataset. I just listed out a few because otherwise the list gets absurdly long. I control for domain knowledge via what programming languages, technologies, and tools the developers know. I use a proxy for importance to the organization by controlling for the control the developer has over buying decisions within their organization. I control for education levels and college major. I control for employment status (full-time, part-time, independent). Hours worked per week. The list goes on but those are a few. But your point remains that unobservables could be confounding the results, so one can never be sure. Thanks for reading!
- braythwayt 6y ago
  My go-to quip for making decisions based on observable variables when the actual causal relationship is between unobservable variables and outcomes is, "Looking for your watch under the street lamp." --- For those who aren't familiar with the aphorism, it's from a joke that goes: One night, I came upon a man staring at the ground under a street lamp. "Looking for something?" I asked, and the man nodded, "My watch." "You lost it here?" I asked, but the man shook his head. "I lost it further down the street." "Why look here, if the watch is over there?" The light's better under the street lamp."
novok 6y ago
I feel like for those Sr. Eng's who can negotiate a remote position, if they don't go remote their careers and pay would go even further. It's one reason why I don't go remote, it's hard to reach staff or sr management positions typically when you do that. It's hard to tease that out in a survey although.
- RHSeeger 6y ago
  > hard to reach staff or sr management positions There's a lot of us that don't want to move into positions where we don't work on software development at the code level. Unfortunately, that also tends to put a cap on how much we can make because a lot of people see management as an upward movement rather than a lateral one.
  - rootusrootus 6y ago
    I work at a company with dual paths (management vs individual contributor) and a requirement to work at the code level would cap your progress upward on either path. As a senior engineer you are expected to spend significantly more time doing architecture work, mentoring other developers, and other non-code activities.
    - eyelidlessness 6y ago
      Well, that's a shame. Doing architecture work and mentoring other developers absolutely can include working with actual code.
munificent 6y ago
One easy dumb trick I use to filter out "interesting" results that are likely just getting causality backwards is to flip the title around and see if it immediately sounds like an obvious truism. In the case of this article, it goes from: > Remote Software Developers Earn 22% More Than Non-Remote Developers To something like: > Higher-paid software developers more likely to be working remotely And at that point, yes, it's fairly obviously going to be true. Working remotely is a perk for those who prefer it so we should expect it to be positively correlated with other forms of compensation. I think all that your data really shows is a hidden variable: competence. Better software developers get paid more, work remotely more, and probably also get more paid time off, larger bonuses, and all sorts of other perks.
- deleted 6y ago
  [deleted]
- AmericanChopper 6y ago
  You could also speculate that employers who offer WFH are more likely to pay more. It’s seen as a perk by many people, and suggesting that employers who offer more perks might pay more doesn’t seem unlikely to me. The headline would seem silly if it said “engineers who get free hot meals at work earn more than those who don’t”.
ulucs 6y ago
You can't really remove the selection effect of higher paid developers choosing to be remote by adding external controls. You could try using an instrument for remote work or use a Simultaneous Equation Model to remove the selection effect bias. Counter to your analysis, I'd actually expect the pay premium for remote work would be negative simply because the opportunity to work remote is worth paying for.
6gvONxR4sf7o 6y ago
I tried to find the factors you controlled for. You wrote > (including age, experience, hours worked, size of employer, programming languages, and more) but the "and more" link didn't give me more insight into what controls you used. The interpretation here is super duper sensitive to what those were, so you have just a straight up list of what your controls were? (before & after feature selection) I am most curious about geographic controls, which are going to be tricky, especially since you can't just one-hot encode them and throw them into double-LASSO. The association might be as simple as "remote workers more likely to be paid in USD" or something analogous, and I'm trying to determine whether/how you controlled for that.
NovemberWhiskey 6y ago
> Agreed, which is why I didn't say,"working remote causes developers to earn 22% more", only that developers who work remote earn 22% more, which is an interesting fact in itself. ... but you have a heading right at the start of the article: "Remote work pays", what was that supposed to mean then?
NovemberWhiskey 6y ago
I see you published some code on GitHub that appears to relate either to this study or another one you published recently. Can you clarify whether you're using the methodology from that code for limiting your sample? https://github.com/whoisnnamdi/highest-paid-software-developer-2019/blob/master/data_prep.py#L24 https://github.com/whoisnnamdi/highest-paid-software-develop... e.g. only those reporting a salary between $10K and $250K?
- whoisnnamdi 6y ago
  Thanks for checking out the code. Confirming that, yes, I am using this methodology for the results reported in this post.

wbc 6y ago

very cool! any thoughts on where that premium comes from, since we're controlling for the standard stuff?

dx034 6y ago
Pure speculation, but maybe you have to look around or negotiate to find remote work? Those who tend to do that probably also get better salaries than those that stick with one job for very long or don't compare much when looking for jobs.
- deleted 6y ago
  [deleted]
deleted 6y ago
[deleted]
thinkharderdev 6y ago
One possibility is that remote developers are just more productive (or if you prefer, more productive developers are more able to negotiate a remote working allowance from their employer). In standard economic theory, the equilibrium wage is at the workers marginal revenue product (or the marginal revenue that their work produces for the organization). So you would expect remote workers to either earn more because they are more productive or, if remote work makes you more productive, to earn more as a result of working remotely and being more productive

whack 6y ago

> Early in the article I adjust this for various controls to better get at causality. This includes observable factors like age, years of experience, hours worked, size of employer, programming languages used, etc. There are likely many other factors which aren't "observable" by your methodology, which are causing the 9.4% pay-premium. Factors like competency, domain knowledge, and how essential they are to the organization. I find it much more believable that these unobservable factors are causing both the pay-premium, and working-remote flexibility. This is a common problem that comes up in such studies. Adjusting for various observable factors is great, but it still leaves behind the other unobservable factors. At which point you have to use your judgement to figure out whether those unobservable factors are more compelling than the hypothesis being tested. That said, this is still a very cool analysis that demonstrates an interesting correlation. Thanks for sharing!

whoisnnamdi 6y ago
Totally. The "etc" I used in my quote is doing a lot of work here - I control for a fairly extensive list of observed factors from the dataset. I just listed out a few because otherwise the list gets absurdly long. I control for domain knowledge via what programming languages, technologies, and tools the developers know. I use a proxy for importance to the organization by controlling for the control the developer has over buying decisions within their organization. I control for education levels and college major. I control for employment status (full-time, part-time, independent). Hours worked per week. The list goes on but those are a few. But your point remains that unobservables could be confounding the results, so one can never be sure. Thanks for reading!
braythwayt 6y ago
My go-to quip for making decisions based on observable variables when the actual causal relationship is between unobservable variables and outcomes is, "Looking for your watch under the street lamp." --- For those who aren't familiar with the aphorism, it's from a joke that goes: One night, I came upon a man staring at the ground under a street lamp. "Looking for something?" I asked, and the man nodded, "My watch." "You lost it here?" I asked, but the man shook his head. "I lost it further down the street." "Why look here, if the watch is over there?" The light's better under the street lamp."

novok 6y ago

I feel like for those Sr. Eng's who can negotiate a remote position, if they don't go remote their careers and pay would go even further. It's one reason why I don't go remote, it's hard to reach staff or sr management positions typically when you do that. It's hard to tease that out in a survey although.

RHSeeger 6y ago
> hard to reach staff or sr management positions There's a lot of us that don't want to move into positions where we don't work on software development at the code level. Unfortunately, that also tends to put a cap on how much we can make because a lot of people see management as an upward movement rather than a lateral one.
- rootusrootus 6y ago
  I work at a company with dual paths (management vs individual contributor) and a requirement to work at the code level would cap your progress upward on either path. As a senior engineer you are expected to spend significantly more time doing architecture work, mentoring other developers, and other non-code activities.
  - eyelidlessness 6y ago
    Well, that's a shame. Doing architecture work and mentoring other developers absolutely can include working with actual code.

munificent 6y ago

One easy dumb trick I use to filter out "interesting" results that are likely just getting causality backwards is to flip the title around and see if it immediately sounds like an obvious truism. In the case of this article, it goes from: > Remote Software Developers Earn 22% More Than Non-Remote Developers To something like: > Higher-paid software developers more likely to be working remotely And at that point, yes, it's fairly obviously going to be true. Working remotely is a perk for those who prefer it so we should expect it to be positively correlated with other forms of compensation. I think all that your data really shows is a hidden variable: competence. Better software developers get paid more, work remotely more, and probably also get more paid time off, larger bonuses, and all sorts of other perks.

deleted 6y ago
[deleted]
AmericanChopper 6y ago
You could also speculate that employers who offer WFH are more likely to pay more. It’s seen as a perk by many people, and suggesting that employers who offer more perks might pay more doesn’t seem unlikely to me. The headline would seem silly if it said “engineers who get free hot meals at work earn more than those who don’t”.

ulucs 6y ago

You can't really remove the selection effect of higher paid developers choosing to be remote by adding external controls. You could try using an instrument for remote work or use a Simultaneous Equation Model to remove the selection effect bias. Counter to your analysis, I'd actually expect the pay premium for remote work would be negative simply because the opportunity to work remote is worth paying for.

6gvONxR4sf7o 6y ago

I tried to find the factors you controlled for. You wrote > (including age, experience, hours worked, size of employer, programming languages, and more) but the "and more" link didn't give me more insight into what controls you used. The interpretation here is super duper sensitive to what those were, so you have just a straight up list of what your controls were? (before & after feature selection) I am most curious about geographic controls, which are going to be tricky, especially since you can't just one-hot encode them and throw them into double-LASSO. The association might be as simple as "remote workers more likely to be paid in USD" or something analogous, and I'm trying to determine whether/how you controlled for that.

NovemberWhiskey 6y ago

> Agreed, which is why I didn't say,"working remote causes developers to earn 22% more", only that developers who work remote earn 22% more, which is an interesting fact in itself. ... but you have a heading right at the start of the article: "Remote work pays", what was that supposed to mean then?

NovemberWhiskey 6y ago

I see you published some code on GitHub that appears to relate either to this study or another one you published recently. Can you clarify whether you're using the methodology from that code for limiting your sample? https://github.com/whoisnnamdi/highest-paid-software-developer-2019/blob/master/data_prep.py#L24 https://github.com/whoisnnamdi/highest-paid-software-develop... e.g. only those reporting a salary between $10K and $250K?

whoisnnamdi 6y ago
Thanks for checking out the code. Confirming that, yes, I am using this methodology for the results reported in this post.