5 ms·

I am 100% sold on developer experience being key for fast iteration and high quality software operations, but the actual research cited here seems fairly flimsy

I am 100% sold on developer experience being key for fast iteration and high quality software operations, but the actual research cited here seems fairly flimsy. It was based on a web survey by DX (love them by the way), that had 219 completed responses and asked likert scale (1-5) questions about how they felt. Good developer experience increases self-reported productivity. I do suspect that it correlates with actual productivity and business outcomes, especially around reducing the productivity loss that comes from error remediation.

geekjock 3y ago
I'm one of the co-authors of the study. Your critique is valid though by research standards, for this type of study, our sample is sufficient. We are planning to replicate this study on a larger scale in the future, though!
- Lienetic 3y ago
  For the uninitiated among us - can you share more context on the research standards and the reasoning behind it? I'm interested and would like this to influence some decisions I have but would like to understand the confidence here :).
- vlovich123 3y ago
  Are there any plans to figure out objective ways to measure productivity and what distinguishes “good devex” from “bad devex”? I’ve worked at a lot of big tech companies that do surveys about internal tooling and every year it’s rated as a weak spot, across years and companies this seemed like a consistent trend. And yet everyone had teams dedicated to improving various aspects of devex so it’s unclear if these teams are just improving the wrong things or if productivity really is improving and it’s something else (eg the amount of code debt grows faster than devex improvements or people are asked to go faster than the devex improvements can keep up or the devex is being improved but the size of the survey means not enough people feel it because you optimize smaller subsets of engineering orgs). That’s another thing to be mindful about large scale and small scale surveys - the latter might be sampling specific teams adopting the tool whereas the former might find there’s no way to make everyone happy and it all turns into a wash.
  - geekjock 3y ago
    "Are there any plans to figure out objective ways to measure productivity" You can't measure developer productivity objectively, assuming you're referring to metrics like lines of code, number of pull requests, or velocity points which are infamous. There's broad agreement on this both within the research community as well as practitioners at leading tech companies. Here are some examples: https://newsletter.pragmaticengineer.com/p/measuring-developer-productivity-bae https://newsletter.pragmaticengineer.com/p/measuring-develop... "... and what distinguishes 'good devex' from 'bad devex'" Yes to this. This is an ongoing effort - we have two previous journal papers that touch on this which may be of interest to you: - https://getdx.com/research/devex-what-actually-drives-productivity/ https://getdx.com/research/devex-what-actually-drives-produc... - https://getdx.com/research/conceptual-framework-for-developer-experience/ https://getdx.com/research/conceptual-framework-for-develope...
    - vlovich123 3y ago
      > You can't measure developer productivity objectively, assuming you're referring to metrics like lines of code, number of pull requests, or velocity points which are infamous. There's broad agreement on this both within the research community as well as practitioners at leading tech companies. No those are metrics you’re suggesting. There are better ones as someone else mentioned (time to get code from PR to production, some way of measuring the quality of work getting to production, etc). Yes, the obvious metrics are poor and better metrics are difficult to measure and quantify. And obviously no single metric is going to capture something as multidimensional as code development. Also, the link you reference doesn’t support your argument. > Across the board, all companies shared that they use both qualitative and quantitative measures Throughout it discusses that people do use quantitative metrics to help guide their analysis and none of them try to do the obviously naive ones as mentioned. This isn’t intended as a critique, but as an engineering profession anything that isn’t quantifiable means it’s open to interpretation and argument weakening forward progress to be restricted to what becomes adopted as industry standard which is in many ways more of a popularity contest of fads rather than concrete technical improvement.
      - 3y ago
- datadrivenangel 3y ago
  I do love to see more research here, thank you! Having done internal developer & analyst tooling work (and used DX), this type of survey is great for internal prioritization when you have dedicated capacity for improvements. I'd be curious to see more about organizational outcomes, as this is piece of DevOps/DevEx data that I feel is weakest and requires the most faith. DORA did some research here, but it's still not always enough to convince leadership.
  - geekjock 3y ago
    Agreed with this. We're working on it!
IshKebab 3y ago
I agree. I think this is just one of those things that we know is true but can't easily be proven via the scientific method. I expect some HN readers will balk at the idea that we can know things but can't prove them scientifically but only because they haven't really thought about it. E.g. does giving variables sensible names reduce bug counts? Obviously. But it's very difficult to prove even something as obvious as that.
- thaumaturgy 3y ago
  It's peculiar that "developer productivity" is even a term. It doesn't have a generally-agreed-upon definition. I don't think a definition is even possible. So, we keep trying to come up with objective measures of a term that is impossible to objectively measure. Isn't that weird? Why not "developer happiness", or, if that's too hippy-dippy for the suits, "developer satisfaction"? Surely we can reason that, all other things being equal, happier developers are more productive. Then, surveys like this one make perfect sense. "Hey, how satisfied are you with the tools / processes / whathaveyou at your employer?" That's a valid question, and collecting a pile of answers does start to yield some useful insights into what works and what doesn't. But no, we're trying to use "developer productivity" instead, as though we're all simple machines that provide value by converting coffee into code, and the winning business is the one that is able to extract the most SLOC out of the cheapest and most miniscule quantity of coffee.
  - lioeters 3y ago
    I think in general, that's the industrial-age worldview of the "suits", where a business is a money-making machine, with worker productivity measured as an objective quantity, a mechanical force that converts to profit. It's a kind of scientism, a worship of numbers and charts, like economics.
  - IshKebab 3y ago
    > It doesn't have a generally-agreed-upon definition. It just means how productive developers are. How much they get done. I don't see what's peculiar about that. Is it peculiar that "productivity" is a term? > provide value by converting coffee into code, and the winning business is the one that is able to extract the most SLOC out of the cheapest and most miniscule quantity of coffee That is pretty much the case, though you don't need to word it so awfully. We provide value by spending our time to producing useful software. The winning business model (all other factors being equal; which they never are) is the one that is able to help developers waste less time when producing said software. See now it doesn't sound evil.
herval 3y ago
> Good developer experience increases self-reported productivity. So do ineffective DX, most times. There's "proper" way to assess productivity gains that don't rely on subjective self-reporting (eg the DORA metrics). The "improving feedback loops" item on this report makes sense on that light - the other two are questionable as a direct proxy
- Paul-Craft 3y ago
  > So do ineffective DX, most times. I don't find this hard to believe, but do you know of any research showing this?
  - herval 3y ago
    I haven't looked for research tbh, just personal experience. It's kinda hard to prove a negative tbh (how do you organize a research that can measure if something subjectively "bad" made people subjectively less effective?). I've been to multiple organizations that invested heavily in DX, and as an external observer you could see they became actually slower (eg one went from test-in-prod and deploy-to-prod to a convoluted CI pipeline) - it made engineers feel like they were doing a lot less work (you just push code and forget), but in reality, an entire department was hired to deal with issues in prod, and the teams didn't really deliver anything faster at all. But 100% of the devs swore it was better/faster/more efficient. On another, the added free time converted directly into more complexity, so tons of overengineered microservices (sometimes to the tune of 2-3x the number of individuals on a team) ensued - it got so easy to do X, everyone overdid it.
- makeitdouble 3y ago
  > DORA metrics I generally agree with your point, with the caveat that metrics like DORA are the first to be gamed when applied to an organization. It's the usual cycle of applying the metrics first in an informal way, then using it for team/process evaluations as it seems to be somewhat reliable, then teams start to reajust how they make the tickets and the deploy cycle in ways that dramatically increase the metrics while reality hasn't changed much. Analyzing productivity with completely objective metrics will I think always bear these flaws (I'd assume even measuring things at the highest level as "wages vs revenue" would be gamed on how wages and revenue are calculated ?)

geekjock 3y ago

I'm one of the co-authors of the study. Your critique is valid though by research standards, for this type of study, our sample is sufficient. We are planning to replicate this study on a larger scale in the future, though!

Lienetic 3y ago
For the uninitiated among us - can you share more context on the research standards and the reasoning behind it? I'm interested and would like this to influence some decisions I have but would like to understand the confidence here :).
vlovich123 3y ago
Are there any plans to figure out objective ways to measure productivity and what distinguishes “good devex” from “bad devex”? I’ve worked at a lot of big tech companies that do surveys about internal tooling and every year it’s rated as a weak spot, across years and companies this seemed like a consistent trend. And yet everyone had teams dedicated to improving various aspects of devex so it’s unclear if these teams are just improving the wrong things or if productivity really is improving and it’s something else (eg the amount of code debt grows faster than devex improvements or people are asked to go faster than the devex improvements can keep up or the devex is being improved but the size of the survey means not enough people feel it because you optimize smaller subsets of engineering orgs). That’s another thing to be mindful about large scale and small scale surveys - the latter might be sampling specific teams adopting the tool whereas the former might find there’s no way to make everyone happy and it all turns into a wash.
- geekjock 3y ago
  "Are there any plans to figure out objective ways to measure productivity" You can't measure developer productivity objectively, assuming you're referring to metrics like lines of code, number of pull requests, or velocity points which are infamous. There's broad agreement on this both within the research community as well as practitioners at leading tech companies. Here are some examples: https://newsletter.pragmaticengineer.com/p/measuring-developer-productivity-bae https://newsletter.pragmaticengineer.com/p/measuring-develop... "... and what distinguishes 'good devex' from 'bad devex'" Yes to this. This is an ongoing effort - we have two previous journal papers that touch on this which may be of interest to you: - https://getdx.com/research/devex-what-actually-drives-productivity/ https://getdx.com/research/devex-what-actually-drives-produc... - https://getdx.com/research/conceptual-framework-for-developer-experience/ https://getdx.com/research/conceptual-framework-for-develope...
  - vlovich123 3y ago
    > You can't measure developer productivity objectively, assuming you're referring to metrics like lines of code, number of pull requests, or velocity points which are infamous. There's broad agreement on this both within the research community as well as practitioners at leading tech companies. No those are metrics you’re suggesting. There are better ones as someone else mentioned (time to get code from PR to production, some way of measuring the quality of work getting to production, etc). Yes, the obvious metrics are poor and better metrics are difficult to measure and quantify. And obviously no single metric is going to capture something as multidimensional as code development. Also, the link you reference doesn’t support your argument. > Across the board, all companies shared that they use both qualitative and quantitative measures Throughout it discusses that people do use quantitative metrics to help guide their analysis and none of them try to do the obviously naive ones as mentioned. This isn’t intended as a critique, but as an engineering profession anything that isn’t quantifiable means it’s open to interpretation and argument weakening forward progress to be restricted to what becomes adopted as industry standard which is in many ways more of a popularity contest of fads rather than concrete technical improvement.
    - 3y ago
datadrivenangel 3y ago
I do love to see more research here, thank you! Having done internal developer & analyst tooling work (and used DX), this type of survey is great for internal prioritization when you have dedicated capacity for improvements. I'd be curious to see more about organizational outcomes, as this is piece of DevOps/DevEx data that I feel is weakest and requires the most faith. DORA did some research here, but it's still not always enough to convince leadership.
- geekjock 3y ago
  Agreed with this. We're working on it!

IshKebab 3y ago

I agree. I think this is just one of those things that we know is true but can't easily be proven via the scientific method. I expect some HN readers will balk at the idea that we can know things but can't prove them scientifically but only because they haven't really thought about it. E.g. does giving variables sensible names reduce bug counts? Obviously. But it's very difficult to prove even something as obvious as that.

thaumaturgy 3y ago
It's peculiar that "developer productivity" is even a term. It doesn't have a generally-agreed-upon definition. I don't think a definition is even possible. So, we keep trying to come up with objective measures of a term that is impossible to objectively measure. Isn't that weird? Why not "developer happiness", or, if that's too hippy-dippy for the suits, "developer satisfaction"? Surely we can reason that, all other things being equal, happier developers are more productive. Then, surveys like this one make perfect sense. "Hey, how satisfied are you with the tools / processes / whathaveyou at your employer?" That's a valid question, and collecting a pile of answers does start to yield some useful insights into what works and what doesn't. But no, we're trying to use "developer productivity" instead, as though we're all simple machines that provide value by converting coffee into code, and the winning business is the one that is able to extract the most SLOC out of the cheapest and most miniscule quantity of coffee.
- lioeters 3y ago
  I think in general, that's the industrial-age worldview of the "suits", where a business is a money-making machine, with worker productivity measured as an objective quantity, a mechanical force that converts to profit. It's a kind of scientism, a worship of numbers and charts, like economics.
- IshKebab 3y ago
  > It doesn't have a generally-agreed-upon definition. It just means how productive developers are. How much they get done. I don't see what's peculiar about that. Is it peculiar that "productivity" is a term? > provide value by converting coffee into code, and the winning business is the one that is able to extract the most SLOC out of the cheapest and most miniscule quantity of coffee That is pretty much the case, though you don't need to word it so awfully. We provide value by spending our time to producing useful software. The winning business model (all other factors being equal; which they never are) is the one that is able to help developers waste less time when producing said software. See now it doesn't sound evil.

herval 3y ago

> Good developer experience increases self-reported productivity. So do ineffective DX, most times. There's "proper" way to assess productivity gains that don't rely on subjective self-reporting (eg the DORA metrics). The "improving feedback loops" item on this report makes sense on that light - the other two are questionable as a direct proxy

Paul-Craft 3y ago
> So do ineffective DX, most times. I don't find this hard to believe, but do you know of any research showing this?
- herval 3y ago
  I haven't looked for research tbh, just personal experience. It's kinda hard to prove a negative tbh (how do you organize a research that can measure if something subjectively "bad" made people subjectively less effective?). I've been to multiple organizations that invested heavily in DX, and as an external observer you could see they became actually slower (eg one went from test-in-prod and deploy-to-prod to a convoluted CI pipeline) - it made engineers feel like they were doing a lot less work (you just push code and forget), but in reality, an entire department was hired to deal with issues in prod, and the teams didn't really deliver anything faster at all. But 100% of the devs swore it was better/faster/more efficient. On another, the added free time converted directly into more complexity, so tons of overengineered microservices (sometimes to the tune of 2-3x the number of individuals on a team) ensued - it got so easy to do X, everyone overdid it.
makeitdouble 3y ago
> DORA metrics I generally agree with your point, with the caveat that metrics like DORA are the first to be gamed when applied to an organization. It's the usual cycle of applying the metrics first in an informal way, then using it for team/process evaluations as it seems to be somewhat reliable, then teams start to reajust how they make the tickets and the deploy cycle in ways that dramatically increase the metrics while reality hasn't changed much. Analyzing productivity with completely objective metrics will I think always bear these flaws (I'd assume even measuring things at the highest level as "wages vs revenue" would be gamed on how wages and revenue are calculated ?)